{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T18:29:03Z","timestamp":1767637743333,"version":"3.48.0"},"reference-count":33,"publisher":"Maximum Academic Press","license":[{"start":{"date-parts":[[2017,8,24]],"date-time":"2017-08-24T00:00:00Z","timestamp":1503532800000},"content-version":"unspecified","delay-in-days":235,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["The Knowledge Engineering Review"],"published-print":{"date-parts":[[2017]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Multi-agent systems (MASs) are a form of distributed intelligence, where multiple autonomous agents act in a common environment. Numerous complex, real world systems have been successfully optimized using multi-agent reinforcement learning (MARL) in conjunction with the MAS framework. In MARL agents learn by maximizing a scalar reward signal from the environment, and thus the design of the reward function directly affects the policies learned. In this work, we address the issue of appropriate multi-agent credit assignment in stochastic resource management games. We propose two new stochastic games to serve as testbeds for MARL research into resource management problems: the tragic commons domain and the shepherd problem domain. Our empirical work evaluates the performance of two commonly used reward shaping techniques: potential-based reward shaping and difference rewards. Experimental results demonstrate that systems using appropriate reward shaping techniques for multi-agent credit assignment can achieve near-optimal performance in stochastic resource management games, outperforming systems learning using unshaped local or global evaluations. We also present the first empirical investigations into the effect of expressing the same heuristic knowledge in state- or action-based formats, therefore developing insights into the design of multi-agent potential functions that will inform future work.<\/jats:p>","DOI":"10.1017\/s026988891700011x","type":"journal-article","created":{"date-parts":[[2017,8,29]],"date-time":"2017-08-29T01:08:31Z","timestamp":1503968911000},"source":"Crossref","is-referenced-by-count":5,"title":["Multi-agent credit assignment in stochastic resource management games"],"prefix":"10.48130","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7951-878X","authenticated-orcid":false,"given":"Patrick","family":"Mannion","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sam","family":"Devlin","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jim","family":"Duggan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Enda","family":"Howley","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"27968","published-online":{"date-parts":[[2017,8,24]]},"reference":[{"key":"S026988891700011X_ref28","unstructured":"Watkins C. J. C. H. 1989. Learning from Delayed Rewards. PhD thesis, King\u2019s College."},{"key":"S026988891700011X_ref5","unstructured":"de Jong S. & Tuyls K. 2009. Learning to cooperate in a continuous tragedy of the commons. In Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems \u2013 Volume 2, 1185\u20131186. International Foundation for Autonomous Agents and Multiagent Systems."},{"key":"S026988891700011X_ref9","unstructured":"Devlin S. , Grzes M. & Kudenko D. 2011b. Multi-agent, potential-based reward shaping for robocup keepaway (extended abstract). In Proceedings of the 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 1227\u20131228."},{"volume-title":"Playing for Real: A Text on Game Theory","year":"2012","author":"Binmore","key":"S026988891700011X_ref2"},{"key":"S026988891700011X_ref17","unstructured":"Mannion P. , Duggan J. & Howley E. 2017. Analysing the effects of reward shaping in multi-objective stochastic games. In Proceedings of the Adaptive and Learning Agents Workshop (at AAMAS 2017)."},{"volume-title":"Introduction to Multiagent Systems","year":"2001","author":"Wooldridge","key":"S026988891700011X_ref33"},{"key":"S026988891700011X_ref27","doi-asserted-by":"publisher","DOI":"10.1007\/BF00992698"},{"key":"S026988891700011X_ref20","unstructured":"Mason K. , Mannion P. , Duggan J. & Howley E. 2016. Applying multi-agent reinforcement learning to watershed management. In Proceedings of the Adaptive and Learning Agents Workshop (at AAMAS 2016)."},{"key":"S026988891700011X_ref14","unstructured":"Malialis K. , Devlin S. & Kudenko D. 2016. Resource abstraction for reinforcement learning in multiagent congestion problems. In Proceedings of the 15th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 503\u2013511."},{"key":"S026988891700011X_ref10","unstructured":"Devlin S. , Yliniemi L. , Kudenko D. & Tumer K. 2014. Potential-based difference rewards for multiagent reinforcement learning. In Proceedings of the 13th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 165\u2013172."},{"key":"S026988891700011X_ref31","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1613\/jair.995","article-title":"Collective intelligence, data routing and Braess\u2019 paradox","volume":"16","author":"Wolpert","year":"2002","journal-title":"Journal of Artificial Intelligence Research"},{"key":"S026988891700011X_ref1","first-page":"406","article-title":"Inductive reasoning and bounded rationality","volume":"84","author":"Arthur","year":"1994","journal-title":"The American Economic Review"},{"key":"S026988891700011X_ref18","unstructured":"Mannion P. , Mason K. , Devlin S. , Duggan J. & Howley E. 2016c. Dynamic economic emissions dispatch optimisation using multi-agent reinforcement learning. In Proceedings of the Adaptive and Learning Agents Workshop (at AAMAS 2016)."},{"key":"S026988891700011X_ref22","doi-asserted-by":"publisher","DOI":"10.1002\/9780470316887"},{"key":"S026988891700011X_ref29","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-27645-3"},{"key":"S026988891700011X_ref12","doi-asserted-by":"publisher","DOI":"10.1142\/S0219525911002986"},{"key":"S026988891700011X_ref32","doi-asserted-by":"publisher","DOI":"10.1209\/epl\/i2000-00208-x"},{"key":"S026988891700011X_ref19","unstructured":"Mannion P. , Mason K. , Devlin S. , Duggan J. & Howley E. 2016d. Multi-objective dynamic dispatch optimisation using multi-agent reinforcement learning. In Proceedings of the 15th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 1345\u20131346."},{"key":"S026988891700011X_ref24","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2006.02.006"},{"key":"S026988891700011X_ref8","doi-asserted-by":"publisher","DOI":"10.1142\/S0219525911002998"},{"key":"S026988891700011X_ref30","unstructured":"Wiewiora E. , Cottrell G. & Elkan C. 2003. Principled methods for advising reinforcement learning agents. In Proceedings of the Twentieth International Conference on Machine Learning, 792\u2013799."},{"key":"S026988891700011X_ref11","doi-asserted-by":"crossref","unstructured":"Grze\u015b M. & Kudenko D. 2009. Theoretical and empirical analysis of reward shaping in reinforcement learning. In International Conference on Machine Learning and Applications, 2009. ICMLA\u201909, 337\u2013344. IEEE.","DOI":"10.1109\/ICMLA.2009.33"},{"key":"S026988891700011X_ref26","doi-asserted-by":"publisher","DOI":"10.4018\/978-1-60566-226-8.ch012"},{"key":"S026988891700011X_ref4","unstructured":"Colby M. , Duchow-Pressley T. , Chung J. J. & Tumer K. 2016. Local approximation of difference evaluation functions. In Proceedings of the 15th International Conference on Autonomous Agents & Multiagent Systems (AAMAS), 521\u2013529."},{"key":"S026988891700011X_ref25","unstructured":"Tumer K. & Agogino A. 2007. Distributed agent-based air traffic flow management. In Proceedings of the 6th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 330\u2013337."},{"key":"S026988891700011X_ref3","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-14435-6_7"},{"key":"S026988891700011X_ref21","unstructured":"Ng A. Y. , Harada D. & Russell S. J. 1999. Policy invariance under reward transformations: theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, ICML \u201999, 278\u2013287. Morgan Kaufmann Publishers Inc."},{"key":"S026988891700011X_ref23","unstructured":"Randl\u00f8v J. & Alstr\u00f8m P. 1998. Learning to drive a bicycle using reinforcement learning and shaping. In Proceedings of the Fifteenth International Conference on Machine Learning, ICML \u201998, 463\u2013471. Morgan Kaufmann Publishers Inc."},{"key":"S026988891700011X_ref7","unstructured":"Devlin S. & Kudenko D. 2012. Dynamic potential-based reward shaping. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 433\u2013440."},{"key":"S026988891700011X_ref6","unstructured":"Devlin S. & Kudenko D. 2011. Theoretical considerations of potential-based reward shaping for multi-agent systems. In Proceedings of the 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 225\u2013232."},{"key":"S026988891700011X_ref15","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-25808-9_4"},{"key":"S026988891700011X_ref16","unstructured":"Mannion P. , Duggan J. & Howley E. 2016b. Generating multi-agent potential functions using counterfactual estimates. In Proceedings of Learning, Inference and Control of Multi-Agent Systems (at NIPS 2016)."},{"key":"S026988891700011X_ref13","doi-asserted-by":"publisher","DOI":"10.1080\/09540091.2015.1031082"}],"container-title":["The Knowledge Engineering Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S026988891700011X","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T14:42:07Z","timestamp":1767624127000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S026988891700011X\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017]]},"references-count":33,"alternative-id":["S026988891700011X"],"URL":"https:\/\/doi.org\/10.1017\/s026988891700011x","relation":{},"ISSN":["0269-8889","1469-8005"],"issn-type":[{"type":"print","value":"0269-8889"},{"type":"electronic","value":"1469-8005"}],"subject":[],"published":{"date-parts":[[2017]]},"article-number":"e16"}}