Abstract
In recent years, Deep Reinforcement Learning (DRL) has achieved great successes in many large scale applications, e.g., the Deep Q-Network (DQN) surpasses the level of professional human players in most of the challenging Atari 2600 games. As DQN transforms the whole input frames into some feature vectors by using convolutional neural networks (CNNs) at each decision step, all objects in the system are treated equally in the process of the feature extraction. However, in reality, for complex systems where many objects exist, the optimal action taken by the agent may only be affected by some important objects, which may lead to inefficiency or poor performance of DQN. In order to alleviate this problem, in this paper, we introduce two approaches that integrate global and local attention mechanisms respectively into the DQN model. For the approach with global attention, the agent is able to focus on all objects to varying degrees; for the approach with local attention, the agent is allowed to focus only on a few objects of great importance with the result that a better strategy can be learned by the agent. The performance of our proposed approaches are demonstrated on some benchmark domains. Source code is available at https://github.com/DMU-XMU/Attention-based-DQN.
This work was supported by the National Natural Science Foundation of China (No. 61772438 and No. 61375077).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ba, J., Mnih, V., Kavukcuoglu, K.: Multiple object recognition with visual attention. arXiv preprint arXiv:1412.7755 (2014)
Egeth, C.E., Connor, H.E.: Visual attention: bottom-up versus top-down. Curr. Biol. 14, R850–R852 (2004)
Gregor, K., Danihelka, I., Graves, A., Wierstra, D.: DRAW: A recurrent neural network for image generation. CoRR abs/1502.04623 (2015). http://arxiv.org/abs/1502.04623
Hausknecht, M.J., Stone, P.: Deep recurrent q-learning for partially observable mdps. CoRR abs/1507.06527 (2015). http://arxiv.org/abs/1507.06527
Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Liu, Y., Tang, Y., Zeng, Y.: Predictive state representations with state space partitioning. In: Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 1259–1266 (2015)
Liu, Y., Zheng, J.: Online learning and planning in partially observable domains without prior knowledge. In: Generative Modeling and Model-Based Reasoning for Robotics and AI at ICML 2019 (2019)
Liu, Y., Zhu, H., Zeng, Y., Dai, Z.: Learning predictive state representations via Monte-carlo tree search. In: Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI) (2016)
Luong, M., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. CoRR abs/1508.04025 (2015). http://arxiv.org/abs/1508.04025
Mnih, V., Heess, N., Graves, A., Kavukcuoglu, k.: Recurrent models of visual attention. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 27, pp. 2204–2212. Curran Associates, Inc. (2014). http://papers.nips.cc/paper/5542-recurrent-models-of-visual-attention.pdf
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529 (2015)
Sorokin, I., Seleznev, A., Pavlov, M., Fedorov, A., Ignateva, A.: Deep attention recurrent q-network. CoRR abs/1512.01693 (2015). http://arxiv.org/abs/1512.01693
Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: International Conference on Machine Learning, pp. 2048–2057 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Ni, K., Yu, D., Liu, Y. (2019). Attention-Based Deep Q-Network in Complex Systems. In: Gedeon, T., Wong, K., Lee, M. (eds) Neural Information Processing. ICONIP 2019. Communications in Computer and Information Science, vol 1142. Springer, Cham. https://doi.org/10.1007/978-3-030-36808-1_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-36808-1_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-36807-4
Online ISBN: 978-3-030-36808-1
eBook Packages: Computer ScienceComputer Science (R0)