Abstract
This study presents a Deep-Sarsa based path planning and obstacle avoidance method for unmanned aerial vehicles (UAVs). Deep-Sarsa is an on-policy reinforcement learning approach, which gains information and rewards from the environment and helps UAV to avoid moving obstacles as well as finds a path to a target based on a deep neural network. It has a significant advantage over dynamic environment compared to other algorithms. In this paper, a Deep-Sarsa model is trained in a grid environment and then deployed in an environment in ROS-Gazebo for UAVs. The experimental results show that the trained Deep-Sarsa model can guide the UAVs to the target without any collisions. This is the first time that Deep-Sarsa has been developed to achieve autonomous path planning and obstacle avoidance of UAVs in a dynamic environment.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Gan, S.K., Sukkarieh, S.: Multi-UAV target search using explicit decentralized gradient-based negotiation. In: IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, pp. 751–756 (2011)
Fu, C., Carrio, A., Campoy, P.: Efficient visual odometry and mapping for unmanned aerial vehicle using ARM-based stereo vision pre-processing system. In: International Conference on Unmanned Aircraft Systems (ICUAS), Colorado, USA, pp. 957–962 (2015)
Maza, I., Kondak, K., Bernard, M., Ollero, A.: Multi-UAV cooperation and control for load transportation and deployment. J. Intell. Robot. Syst. 57(1), 417–449 (2009)
Fu, C., Carrio, A., Olivares-Mendez, M.A., Suarez-Fernandez, R., Campoy, P.: Robust real-time vision-based aircraft tracking from unmanned aerial vehicles. In: IEEE International Conference on Robotics and Automation (ICRA) (2014)
Hayat, S., Yanmaz, E., Brown, T.X., Bettstetter, C.: Multi-objective UAV path planning for search and rescue. In: IEEE International Conference on Robotics and Automation (ICRA), Singapore, pp. 5569–5574 (2017)
Sathyaraj, B.M., Jain, L.C., Finn, A., Drake, S.: Multiple UAVs path planning algorithms: a comparative study. Fuzzy Optim. Decis. Mak. 7(3), 257–267 (2008)
Hrabar, S.: 3D path planning and stereo-based obstacle avoidance for rotorcraft UAVs. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Nice, France, pp. 807–814 (2008)
Bounini, F., Gingras, D., Pollart, H., Gruyer, D.: Modified artificial potential field method for online path planning applications. In: IEEE Intelligent Vehicles Symposium (IV), Los Angeles, USA, pp. 180–185 (2017)
Galceran, E., Carreras, M.: A survey on coverage path planning for robotics. Robot. Auton. Syst. 61(12), 1258–1276 (2013)
Zhao, Y., Zheng, Z., Zhang, X., Liu, Y.: Q learning algorithm based UAV path learning and obstacle avoidence approach. In: 36th Chinese Control Conference (CCC), Dalian, China, pp. 3397–3402 (2017)
Imanberdiyev, N., Fu, C., Kayacan, E., Chen, I.-M.: Autonomous navigation of UAV by using real-time model-based reinforcement learning. In: 14th International Conference on Control, Automation, Robotics and Vision, Phuket, Thailand, pp. 1–6 (2016)
Kubat, M.: Reinforcement learning. In: An Introduction to Machine Learning, pp. 331–339 (2017)
Zhao, D., Wang, H., Shao, K., Zhu, Y.: Deep reinforcement learning with experience replay based on SARSA. In: IEEE Symposium Series on Computational Intelligence (SSCI) (2016)
Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y.: ROS: an open-source robot operating system. In: ICRA Workshop on Open Source Software, Kobe, Japan, pp. 1–6 (2009)
Koenig, N., Howard, A.: Design and use paradigms for Gazebo, an open-source multi-robot simulator. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Sendai, Japan, vol. 3, pp. 2149–2154 (2004)
Kober, J., Bagnell, J.A., Peters, J.: Reinforcement learning in robotics: a survey. Int. J. Robot. Res. 32(11), 1238–1274 (2013)
Singh, S., Jaakkola, T., Littman, M.L., Szepesvári, C.: Convergence results for single-step on-policy reinforcement-learning algorithms. Mach. Learn. 38(3), 287–308 (2000). https://doi.org/10.1007/978-981-10-7515-5_11
Sutton, R.S.: Generalization in reinforcement learning: successful examples using sparse coarse coding. In: Touretzky, D.S., Mozer, M.C., Hasselmo, M.E. (eds.) Advances in Neural Information Processing Systems, pp. 1038–1044. MIT Press (1996)
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Ketkar, N.: Introduction to keras. In: Deep Learning with Python, pp. 97–111 (2017)
Huang, A.S., Olson, E., Moore, D.C.: LCM: lightweight communications and marshalling. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Taipei, Taiwan, pp. 4057–4062 (2010)
Acknowledgements
This work is supported by the project of National Natural Science Foundation of China (No. 61603277), the 13th-Five-Year-Plan on Common Technology, key project (No. 41412050101), and the Shanghai Aerospace Science and Technology Innovation Fund (SAST 2016017). Meanwhile, this work is also partially supported by the Youth 1000 program project (No. 1000231901), as well as by the Key Basic Research Project of Shanghai Science and Technology Innovation Plan (No. 15JC1403300). All these supports are highly appreciated.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Luo, W., Tang, Q., Fu, C., Eberhard, P. (2018). Deep-Sarsa Based Multi-UAV Path Planning and Obstacle Avoidance in a Dynamic Environment. In: Tan, Y., Shi, Y., Tang, Q. (eds) Advances in Swarm Intelligence. ICSI 2018. Lecture Notes in Computer Science(), vol 10942. Springer, Cham. https://doi.org/10.1007/978-3-319-93818-9_10
Download citation
DOI: https://doi.org/10.1007/978-3-319-93818-9_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-93817-2
Online ISBN: 978-3-319-93818-9
eBook Packages: Computer ScienceComputer Science (R0)