Abstract
In this paper, we propose a method for ameliorating the state-space explosion that can occur in the context of multiagent reinforcement learning. In our method, an agent considers other agents’ states only when they interfere with each other in attaining their goals. Our idea is that the initial state-space of each agent does not include information about other spaces. Agents then automatically expand their state-space if they detect interference states. We adopt the information theory measure of entropy to detect the interference states for which agents should consider the state information of other agents. We demonstrate the advantage of our method with respect to the efficiency of global convergence.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Busoniu, L., De Schutter, B., Babuška, R.: A comprehensive survey of multiagent reinforcement learning. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 38(2), 156–172 (2008)
Watkins, C., Dayan, P.: Q-learning. Mach. Learn. 8(3), 279–292 (1992)
Littman, M.: Markov games as a framework for multi-agent reinforcement learning. In: Proceedings of the Eleventh International Conference on Machine Learning, pp. 242–250 (1994)
Littman, M.: Value-function reinforcement learning in Markov games. Cogn. Syst. Res. 2(1), 55–66 (2001)
Wang, X., Sandholm, T.: Reinforcement learning to play an optimal Nash equilibrium in team Markov games. In: Advances in Neural Information Processing Systems vol. 15, pp. 1571–1578 (2002)
Hu, J., Wellman, M.: Nash Q-learning for general-sum stochastic games. J. Mach. Learn. Res. 4, 1039–1069 (2003)
Greenwald, A., Zinkevich, M., Kaelbling, P.: Correlated Q-learning. In: Proceedings of the Twentieth International Conference on Machine Learning, pp. 242–249 (2003)
Busoniu, L., De Schutter, B., Babuška, R.: Multiagent reinforcement learning with adaptive state focus. In: Proceedings of the Seventeenth Belgian-Dutch Conference on Artificial Intelligence, pp. 35–42 (2005)
Kok, J.R., Vlassis, N.: Sparse cooperative Q-learning. In: Proceedings of the Twenty-First International Conference on Machine Learning, pp. 61–68 (2004)
Kok, J.R., Hoen, P., Bakker, B., Vlassis, N.: Utile coordination: learning interdependencies among cooperative agents. In: Proceedings of the IEEE Symposium on Computational Intelligence and Games (CIG), pp. 29–36 (2005)
Spaan, M.T.J., Melo, F.S.: Interaction-driven Markov games for decentralized multiagent planning under uncertainty. In: Proceedings of the 7th International Conference on Autonomous Agents and Multiagent Systems, pp. 525–532 (2008)
Melo, F.S., Veloso, M.: Learning of coordination: exploiting sparse interactions in multiagent systems. In: Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems, pp. 773–780 (2009)
De Hauwere, Y., Vrancx, P., Nowé, A.: Learning what to observe in multi-agent systems. In: Proceedings of the Twentieth Belgian-Dutch Conference on Artificial Intelligence, pp. 83–90 (2009)
De Hauwere, Y., Vrancx, P., Nowé, A.: Learning multi-agent state space representations. In: Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems, pp. 715–722 (2010)
De Hauwere, Y., Vrancx, P., Nowé, A.: Adaptive state representations for multiagent reinforcement learning. In: Proceedings of the 3rd International Conference on Agents and Artificial Intelligence, pp. 181–189 (2011)
Arai, S., Ishigaki, Y.: Information theoretic approach for measuring interaction in multiagent domain. J. Adv. Comput. Intell. Intell. Inform. 13(6), 649–657 (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Arai, S., Xu, H. (2016). Faster Convergence to Cooperative Policy by Autonomous Detection of Interference States in Multiagent Reinforcement Learning. In: Booth, R., Zhang, ML. (eds) PRICAI 2016: Trends in Artificial Intelligence. PRICAI 2016. Lecture Notes in Computer Science(), vol 9810. Springer, Cham. https://doi.org/10.1007/978-3-319-42911-3_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-42911-3_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-42910-6
Online ISBN: 978-3-319-42911-3
eBook Packages: Computer ScienceComputer Science (R0)