Abstract
Incorporating skills in reinforcement learning methods results in accelerate agents learning performance. The key problem of automatic skill discovery is to find subgoal states and create skills to reach them. Among the proposed algorithms, those based on graph centrality measures have achieved precise results. In this paper we propose a new graph centrality measure for identifying subgoal states that is crucial to develop useful skills. The main advantage of the proposed centrality measure is that this measure considers both local and global information of the agent states to score them that result in identifying real subgoal states. We will show through simulations for three benchmark tasks, namely, “four-room grid world”, “taxi driver grid world” and “soccer simulation grid world” that a procedure based on the proposed centrality measure performs better than the procedure based on the other centrality measures.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: a survey. Journal of Artificial Intelligence Research 4, 237–285 (1996)
Barto, A.G., Mahadevan, S.: Recent Advances in Hierarchical Reinforcement Learning. Discrete Event Dynamic Systems 13, 341–379 (2003)
Sutton, R., Precup, D., Singh, S.: Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artif. Intell. 112, 181–211 (1999)
Dietterich, T.G.: Hierarchical reinforcement learning with the MAXQ value function decomposition. J. Artif. Int. Res. 13, 227–303 (2000)
Parr, R., Russell, S.: Reinforcement learning with hierarchies of machines. In: Conference Reinforcement Learning with Hierarchies of Machines, pp. 1043–1049. MIT Press, Cambridge (1998)
Digney, B.L.: Learning hierarchical control structures for multiple tasks and changing environments. In: Proceedings of the Fifth International Conference on Simulation of Adaptive Behavior on From Animals to Animats 5, pp. 321–330. MIT Press, Univ. of Zurich, Zurich, Switzerland (1998)
McGovern, A., Barto, A.G.: Automatic Discovery of Subgoals in Reinforcement Learning using Diverse Density. In: Conference Automatic Discovery of Subgoals in Reinforcement Learning using Diverse Density, pp. 361–368. Morgan Kaufmann, San Francisco (2001)
Şimşek, Ö., Barto, A.G.: Learning Skills in Reinforcement Learning Using Relative Novelty, pp. 367–374 (2005)
Shi, C., Huang, R., Shi, Z.: Automatic Discovery of Subgoals in Reinforcement Learning Using Unique-Dreiction Value. In: IEEE International Conference on Cognitive Informatics, pp. 480–486 (2007)
Goel, S., Huber, M.: Subgoal Discovery for Hierarchical Reinforcement Learning Using Learned Policies. In: Conference Subgoal Discovery for Hierarchical Reinforcement Learning Using Learned Policies, pp. 346–350. AAAI Press, Menlo Park (2003)
Asadi, M., Huber, M.: Autonomous subgoal discovery and hierarchical abstraction for reinforcement learning using Monte Carlo method. In: Proceedings of the 20th National Conference on Artificial Intelligence, vol. 4, pp. 1588–1589. AAAI Press, Pittsburgh (2005)
Kazemitabar, S., Beigy, H.: Automatic Discovery of Subgoals in Reinforcement Learning Using Strongly Connected Components. In: Proceedings of the 15th International Conference on Advances in Neuro-Information Processing, pp. 829–834 (2009)
Ajdari Rad, A., Moradi, P., Hasler, M.: Automatic Skill Acquisition in Reinforcement Learning using Connection Graph Stability Centrality. In: Conference The IEEE International Symposium on Circuits and Systems, ISCAS 2010 (2010)
Moradi, P., Ajdari Rad, A., Khadivi, K., Hasler, M.: Automatic Discovery of Subgoals in Reinforcement Learning using Betweeness Centrality Measures. In: Conference 18th IEEE Workshop on Nonlinear Dynamics of Electronic Systems, NDES 2010 (2010)
Moradi, P., Ajdari Rad, A., Khadivi, A., Hasler, M.: Automatic Skill Acquisition using Complex Network Measures. In: Conference International Conference on Artificial Intelligence and Pattern Recognition, AIPR 2010 (2010)
Kheradmandian, G., Rahmati, M.: Automatic abstraction in reinforcement learning using data mining techniques. Robotics and Autonomous Systems 57, 1119–1128 (2009)
Şimşek, Ö., Barto, A.G.: Skill Characterization Based on Betweenness. In: Koller, D., Schuurmans, D., Bengio, Y., Bottou, L. (eds.) Advances in Neural Information Processing Systems, vol. 21, pp. 1497–1504 (2009)
Şimşek, Ö., Wolfe, A.P., Barto, A.G.: Identifying useful subgoals in reinforcement learning by local graph partitioning. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 816–823. ACM, Bonn (2005)
Mannor, S., Menache, I., Hoze, A., Klein, U.: Dynamic abstraction in reinforcement learning via clustering. In: Proceedings of the Twenty-First International Conference on Machine Learning, p. 71. ACM, Banff (2004)
Menache, I., Mannor, S., Shimkin, N.: Q-Cut - Dynamic Discovery of Sub-goals in Reinforcement Learning. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) ECML 2002. LNCS (LNAI), vol. 2430, pp. 295–306. Springer, Heidelberg (2002)
Jing, S., Guochang, G., Haibo, L.: Automatic option generation in hierarchical reinforcement learning via immune clustering. In: Conference Automatic Option Generation in Hierarchical Reinforcement Learning Via Immune Clustering, p. 4, p. 500 (2007)
Kazemitabar, S., Beigy, H.: Using Strongly Connected Components as a Basis for Autonomous Skill Acquisition in Reinforcement Learning. In: Yu, W., He, H., Zhang, N. (eds.) ISNN 2009. LNCS, vol. 5551, pp. 794–803. Springer, Heidelberg (2009)
Brandes, U.: A faster algorithm for betweenness centrality. Journal of Mathematical Sociology 25, 163–177 (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Moradi, P., Shiri, M.E., Entezari, N. (2010). Automatic Skill Acquisition in Reinforcement Learning Agents Using Connection Bridge Centrality. In: Kim, Th., Vasilakos, T., Sakurai, K., Xiao, Y., Zhao, G., Ślęzak, D. (eds) Communication and Networking. FGCN 2010. Communications in Computer and Information Science, vol 120. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-17604-3_6
Download citation
DOI: https://doi.org/10.1007/978-3-642-17604-3_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-17603-6
Online ISBN: 978-3-642-17604-3
eBook Packages: Computer ScienceComputer Science (R0)