Abstract
As a decision-making problem with interaction between vehicles, it is difficult to describe intelligent vehicle lane change state space using a rule-based decision system. The deep deterministic policy gradient (DDPG) algorithm offers good performance for autonomous driving decision, but still has slow convergence and high collision probability in learning process when applied to lane change. Therefore, we propose an improved deep deterministic policy gradient algorithm with barrier function (DDPG-BF) algorithm to address these problems. The barrier function is constructed depending on the safety distance required for lane changes, and DDPG algorithm optimization is improved by guiding the vehicle to choose actions within safety constraints. Simulation results on TORCS confirmed that the proposed method converged in hundreds of training episodes, and reduced the unsafe behavior ratio to less than 0.05. Compared with DDPG and FEC-DDPG algorithm, the proposed method has the contribution to improve the convergence speed of learning and maintain the safe distance between vehicles in lane change.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Zhang, X., et al.: Overview of deep learning intelligent driving methods. J. Tsinghua Univ. (Sci. Technol.) 58(4), 438–444 (2018)
Chae, H., Kang, C M., Kim, B D., et al.: Autonomous braking system via deep reinforcement learning. In: ITSC: 2017 IEEE 20th International Conference on Intelligent Transportation Systems, pp. 1–6. IEEE (2017). https://doi.org/10.1109/ITSC.2017.8317839
Lillicrap, T P., et al.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)
Kendall, A., et al.: Learning to drive in a day. In: 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada, 2019, pp. 8248–8254. https://doi.org/10.1109/ICRA.2019.8793742
Fulton, N., Platzer, A.: Safe reinforcement learning via formal methods: toward safe control through proof and learning. In: Thirty-Second AAAI Conference on Artificial Intelligence, vol. 32, no. 1 (2018)
Yang, Y., et al.: Safe reinforcement learning for dynamical games. Int. J. Robust Nonlinear Control 30(9), 3706–3726 (2020)
Alshiekh, M., Bloem, R., Ehlers, R., et al.: Safe reinforcement learning via shielding. In: Thirty-Second AAAI Conference on Artificial Intelligence, vol. 32, no. 1 (2018)
Sibai, H., et al.: Safe Reinforcement Learning for Control Systems: A Hybrid Systems Perspective and Case Study (2019). http://publish.illinois.edu/husseinsibai/files/2019/10/Safe_RL_with_Continuous_Dynamics___HSCC2019-4.pdf
Zhang, B., et al.: Self-driving via improved DDPG algorithm. Comput. Eng. Appl. 55(10), 264–270 (2019)
Cheng, R., et al.: End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33(01), pp. 3387–3395 (2019). https://doi.org/10.1609/aaai.v33i01.33013387
Wang, P., Li, H., Chan, C.: Continuous control for automated lane change behavior based on deep deterministic policy gradient algorithm. In: IEEE Intelligent Vehicles Symposium (IV), Paris, France, 2019, pp. 1454–1460 (2019). https://doi.org/10.1109/IVS.2019.8813903
Zhang, L., Liu, J.: Calculation of safe speed of car in curve. Phys. Teacher 25(7),62–63 (2004)
Zang, L., et al.: Modeling and simulation of lateral safety distance of automobile in the bend. J. Chongqing Jiaotong Univ. 219(04), 15–20 (2020)
Bertolazzi, E., Biral, F., et al.: Supporting drivers in keeping safe speed and safe distance: the SASPENCE subproject within the European framework programme 6 integrating project PReVENT. IEEE Trans. Intell. Transp. Syst. 11(3), 525–538 (2010). https://doi.org/10.1109/TITS.2009.2035925
Luo, Q., Xun, L., et al.: Simulation analysis and study on car-following safety distance model based on braking process of leading vehicle. In 2011 9th World Congress on Intelligent Control and Automation, pp. 740–743. https://doi.org/10.1109/WCICA.2011.5970612
Silver, D., Heess, N., et al.: Deterministic policy gradient algorithms. In: International Conference on Machine Learning, PMLR, pp. 387–395 (2014)
Barto, A.G., et al.: Neuron like elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybern. 13(5), 834–846 (1970)
Heess, N., Silver, D., et al.: Actor- critic reinforcement learning with energy-based policies, pp. 45–58. PMLR, In European Workshop on Reinforcement Learning (2013)
Nocedal, J., Wright, S.: Numerical optimization. Springer Science & Business Media (2006)
Guo, C., Li, D., Zhang, G., et al.: Dynamic interior point method for vehicular traffic optimization. IEEE Trans. Vehicular Technol. 69(5), 4855–4868 (2020)
Boyd, S., Boyd, S.P., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)
Acknowledgment
This work was supported in part by the National Key Research and Development Program of China (Project No. 2018YFB1305105) and National Natural Science Foundation of China under Grant 62003361.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Feng, T., Xu, X., Zhang, X., Zhang, X. (2021). An Improved DDPG Algorithm with Barrier Function for Lane-Change Decision-Making of Intelligent Vehicles. In: Fang, L., Chen, Y., Zhai, G., Wang, J., Wang, R., Dong, W. (eds) Artificial Intelligence. CICAI 2021. Lecture Notes in Computer Science(), vol 13070. Springer, Cham. https://doi.org/10.1007/978-3-030-93049-3_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-93049-3_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-93048-6
Online ISBN: 978-3-030-93049-3
eBook Packages: Computer ScienceComputer Science (R0)