Abstract
As a famous deep reinforcement learning approach, deep deterministic policy gradient (DDPG) is able to deal with the problems in the domain of continuous control. To remove temporal correlations among the observed transitions, DDPG uses a sampling mechanism called experience reply which replays transitions at random from the replay buffer. Experience reply removes correlations among different transitions. However, random sampling does not consider the importance of transitions in replay buffer which leads to the longer training time and poor performance. In this paper, we propose a novel efficient sampling mechanism which we call deep deterministic policy gradient with clustered prioritized sampling (CPS-DDPG). CPS-DDPG clusters the transitions by K-means in order to reduce the complexity of the algorithm. In addition, CPS-DDPG samples transitions from different categories according to priorities so as to train targeted transitions. The key idea of CPS-DDPG is to set high priorities to the valuable categories and increase the priorities of the categories that have not been selected for long time appropriately in order to increase the diversity of the transitions. The experimental results show that the proposed model achieves better performance than the traditional deep reinforcement learning model in the continuous domain.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Playing Atari with deep reinforcement learning. In: Proceedings of Workshops at the 26th Neural Information Processing Systems, Lake Tahoe, USA (2013)
Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Silver, D., Huang, A., Maddison, C.J., et al.: Mastering the game of Go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)
Schulman, J., Moritz, P., Levine, S., et al.: High-dimensional continuous control using generalized advantage estimation. In: Proceedings of the International Conference on Learning Representations (2015)
Watter, M., Springenberg, J.T., Riedmiller, M., et al.: Embed to control: a locally linear latent dynamics model for control from raw images. In: Advances in Neural Information Processing System, pp. 2728–2736 (2015)
Lillicrap, T.P., Hunt, J.J., Pritzel, A., et al.: Continuous control with deep reinforcement learning. In: Proceedings of the International Conference on Learning Representations (2016)
Silver, D., Lever, G., Heess, N., et al.: Deterministic policy gradient algorithms. In: Proceedings of the International Conference on Machine Learning, pp. 387–395 (2014)
Heess, N., Wayne, G., Silver, D., et al.: Learning continuous control policies by stochastic value gradients. In: Advances in Neural Information Processing System, pp. 2926–2934 (2015)
Levine, S., Levine, S., Levine, S., et al.: Continuous deep Q-learning with model-based acceleration. In: Proceedings of the 33th International Conference on Machine Learning, pp. 2829–2838 (2016)
Levine, S., Abbeel, P.: Learning neural network policies with guided policy search under unknown dynamics. In: Advances in Neural Information Processing Systems, pp. 1071–1079 (2014)
O’Neill, J., Pleydell-Bouverie, B., Dupret, D., et al.: Play it again: reactivation of waking experience and memory. Trends Neurosci. 33(5), 220–229 (2010)
Wawrzyński, P., Tanwani, A.K.: Autonomous reinforcement learning with experience replay. Neural Netw. 41(5), 156 (2013)
Kang, S.H., Sandberg, B., Yip, A.M.: A regularized k-means and multiphase scale segmentation. Inverse Probl. Imaging 5(2), 407–429 (2017)
Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis of the multiarmed bandit problem. Mach. Learn. 47(2–3), 235–256 (2002)
Mnih, V., Badia, A.P., Mirza, M., et al.: Asynchronous methods for deep reinforcement learning. In: Proceedings of the International Conference on Machine Learning, New York, USA, pp. 1928–1937 (2016)
Acknowledgements
This work was supported by the National Natural Science Foundation of China (61303108, 61373094, 61772355); Jiangsu College Natural Science Research Key Program (17KJA520004); Program of the Provincial Key Laboratory for Computer Information Processing Technology (Soochow University) (KJS1524); China Scholarship Council Project (201606920013).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Wu, W., Zhu, F., Fu, Y., Liu, Q. (2018). Deep Deterministic Policy Gradient with Clustered Prioritized Sampling. In: Cheng, L., Leung, A., Ozawa, S. (eds) Neural Information Processing. ICONIP 2018. Lecture Notes in Computer Science(), vol 11302. Springer, Cham. https://doi.org/10.1007/978-3-030-04179-3_57
Download citation
DOI: https://doi.org/10.1007/978-3-030-04179-3_57
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-04178-6
Online ISBN: 978-3-030-04179-3
eBook Packages: Computer ScienceComputer Science (R0)