Skip to main content
SpringerLink
Log in

Performing the Kick During Walking for RoboCup 3D Soccer Simulation League Using Reinforcement Learning Algorithm

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

Nowadays, humanoid soccer serves as a benchmark for artificial intelligence and robotic problems. The factors such as the kicking speed and the number of kicks by robot soccer players are the most significant aims that the participating teams are pursued in the RoboCup 3D Soccer Simulation League. The proposed method presents a kicking strategy during walking for humanoid soccer robots. Achieving an accurate and powerful kicking while robots are moving requires a dynamic optimization of the speed and motion parameters of the robot. In this paper, a curved motion path has been designed based on the robot position relative to the ball and the goal. Ultimately, the robot will be able to kick at the goal by walking along this curve path. The speed and angle of the walking robot are set towards the ball with regard to the robots curved motion path. After the final step of the robot, the accurate and effective adjustment of these two parameters ensures that the robot is located in the ideal position to perform the perfect kick. Due to the noise and walking condition of the robot, it is essential that the speed and angle of motion to be measured more accurately. For this purpose, we use a reinforcement learning model to adjust the robots step size and so does achieve the optimal value of two abovementioned parameters. Using reinforcement learning, robot would learn to pursue an optimal policy to correctly kick towards designated points. Therefore, the proposed method is a model-free and based on dynamic programming. The experiments reveal that the proposed method has significantly improved the team overall performance and robots ability to kick. Our proposed method has been 9.32% successful on average and outperformed the UTAustinVilla agent in terms of goal-scoring time in a non-opponent simulator.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  1. Kitano H, Okuno HG, Nakadai K, Sabisch T, Matsui T (2000) Design and architecture of SIG the humanoid: an experimental platform for integrated perception in RoboCup humanoid challenge. In: IEEE/RSJ international conference on intelligent robots and systems (IROS 2000), pp 181–190. https://doi.org/10.1109/IROS.2000.894602

  2. Haddadin S, Laue T, Frese U, Wolf S, Albu-Schäffer A, Hirzinger G (2009) Kick it with elasticity: safety and performance in human–robot soccer. Robot Auton Syst 57:761–775. https://doi.org/10.1016/j.robot.2009.03.004

    Article  Google Scholar 

  3. Carlson Z, Lemmon L, Higgins M, Frank D, Shahrezaie RS, Feil-Seifer D (2019) Perceived mistreatment and emotional capability following aggressive treatment of robots and computers. Int J Social Robot 11:1–13. https://doi.org/10.1007/s12369-019-00599-8

    Article  Google Scholar 

  4. RoboCup, http://www.robocup.org/. Accessed 29 April 2020

  5. Shi H, Xu L, Zhang L, Pan W, Xu G (2015) Research on self-adaptive decision-making mechanism for competition strategies in robot soccer. Front Comput Sci 9:485–494. https://doi.org/10.1007/s11704-015-4210-7

    Article  Google Scholar 

  6. Colbaugh R, Glass K, Seraji H (1991) An adaptive inverse kinematics algorithm for robot manipulators. Int J Model Simul 11:33–38. https://doi.org/10.1080/02286203.1991.11760118

    Article  Google Scholar 

  7. Jouandeau N, Hugel V (2014) Optimization of parametrised kicking motion for humanoid soccer player. In: IEEE international conference on autonomous robot systems and competitions, pp 241–246. https://doi.org/10.1109/ICARSC.2014.6849793

  8. Hereid A, Cousineau EA, Hubicki CM, Ames AD (2016). 3D dynamic walking with underactuated humanoid robots: A direct collocation framework for optimizing hybrid zero dynamics. In: IEEE international conference on robotics and automation, pp 1447–1454. https://doi.org/10.1109/ICRA.2016.7487279

  9. SimSpark, http://simspark.sourceforge.net/. Accessed 29 April 2020

  10. Böckmann A, Laue T (2016) Kick motions for the nao robot using dynamic movement primitives. In: Robot world cup, Springer, Cham, pp 33–44. https://doi.org/10.1007/978-3-319-68792-6_3

  11. Claret JA, Venture G, Basañez L (2017) Exploiting the robot kinematic redundancy for emotion conveyance to humans as a lower priority task. Int J Soc Robot 9:277–292. https://doi.org/10.1007/s12369-016-0387-2

    Article  Google Scholar 

  12. Watkins CJ, Dayan P (1992) Q-learning. Mach Learn 8:279–292. https://doi.org/10.1007/BF00992698

    Article  MATH  Google Scholar 

  13. Akgun B, Cakmak M, Jiang K, Thomaz AL (2012) Keyframe-based learning from demonstration. Int J Soc Robot 4:343–355. https://doi.org/10.1007/s12369-012-0160-0

    Article  Google Scholar 

  14. MacAlpine P, Stone P (2016) UT Austin Villa robocup 3D simulation base code release. In: Robot World Cup, Springer, Cham, pp 135–143. https://doi.org/10.1007/978-3-319-68792-6_11

  15. Müller J, Laue T, Röfer T (2010) Kicking a ball–modeling complex dynamic motions for humanoid robots. In: Robot soccer world cup, Springer, Berlin, Heidelberg, pp 109–120. https://doi.org/10.1007/978-3-642-20217-9_10

  16. Fu B, Zhang PX, Wang CF (2012) A cooperation strategy for shooting in robot soccer competition based on the multi-suppose tree. Procedia Engineering 29:1462–1466. https://doi.org/10.1016/j.proeng.2012.01.155

    Article  Google Scholar 

  17. Nasrollahi P, Jafari S, Ebrahimi M (2012) Action classification of humanoid soccer robots using machine learning. In: Artificial intelligence and signal processing, CSI, IEEE, pp 598–603. https://doi.org/10.1109/AISP.2012.6313816

  18. Röfer T, Laue T, Hasselbring A, Richter-Klug J, Röhrig E (2017) B-Human 2017–team tactics and robot skills in the standard platform league. In: Robot world cup, Springer, Cham, pp. 461–472. https://doi.org/10.1007/978-3-030-00308-1_38

  19. Röfer T, Laue T (2013) On B-human’s code releases in the standard platform league–software architecture and impact. In: Robot soccer world cup, Springer, Berlin, Heidelberg, pp 648–655. https://doi.org/10.1007/978-3-662-44468-9_61

  20. Genter K, MacAlpine P, Menashe J, Hannah J, Liebman E, Narvekar S, Stone P (2016) UT Austin Villa: Project-driven research in ai and robotics. IEEE Intell Syst 31:94–101. https://doi.org/10.1109/MIS.2016.35

    Article  Google Scholar 

  21. Liang Z, Shen P, Li X (2015) Walking motion design of humanoid robots in RoboCup3D simulation platform. Int J Model Simul 35:35–42. https://doi.org/10.1080/02286203.2015.1080453

    Article  Google Scholar 

  22. Depinet M, MacAlpine P, Stone P (2014) Keyframe sampling, optimization, and behavior integration: towards long-distance kicking in the robocup 3d simulation league. In: Robot soccer world cup, Springer, Cham, pp 571–582. https://doi.org/10.1007/978-3-319-18615-3_47

  23. Ficht G, Allgeuer P, Farazi H, Behnke S (2017) NimbRo-OP2: grown-up 3D printed open humanoid platform for research. In: IEEE-RAS 17th international conference on humanoid robotics (humanoids), pp 669–675. https://doi.org/10.1109/HUMANOIDS.2017.8246944

  24. Brindza J, Lee A, Majumdar A, Scharfman B, Schneider A, Shor R, Lee D (2009) Upennalizers robocup standard platform league team report 2009. University of Pennsylvania, Technical Report http://www.ais.uni-bonn.de/humanoidsoccer/ws10/papers/HSR10_02.pdf. Accessed 20 August 2019

  25. Li X, Liang Z, Feng H (2015) Kicking motion planning of Nao robots based on CMA-ES. In: IEEE 27th Chinese control and decision conference (CCDC), pp 6158–6161. https://doi.org/10.1109/CCDC.2015.7161918

  26. Beyer HG, Sendhoff B (2017) Simplify your covariance matrix adaptation evolution strategy. IEEE Trans Evol Comput 21:746–759. https://doi.org/10.1109/TEVC.2017.2680320

    Article  Google Scholar 

  27. Stone P, Sutton RS, Kuhlmann G (2005) Reinforcement learning for robocup soccer keepaway. Adapt Behav 13:165–188. https://doi.org/10.1177/105971230501300301

    Article  Google Scholar 

  28. Leottau DL, Ruiz-del-Solar J, MacAlpine P, Stone P (2015) A study of layered learning strategies applied to individual behaviors in robot soccer. In: Robot soccer world cup, Springer, Cham, pp 290–302. https://doi.org/10.1007/978-3-319-29339-4_24

  29. Hester T, Quinlan M, Stone P (2010) Generalized model learning for reinforcement learning on a humanoid robot. In: IEEE international conference on robotics and automation, pp 2369–2374. https://doi.org/10.1109/ROBOT.2010.5509181

  30. Fahami MA, Roshanzamir M, Izadi NH (2017) A reinforcement learning approach to score goals in RoboCup 3D soccer simulation for nao humanoid robot. In: 7th international conference on computer and knowledge engineering (ICCKE), IEEE, pp 450–454. https://doi.org/10.1109/ICCKE.2017.8167920

  31. Teixeira H, Silva T, Abreu M, Reis LP (2020) Humanoid robot kick in motion ability for playing robotic soccer. In: International conference on autonomous robot systems and competitions (ICARSC), IEEE, pp 34–39. https://doi.org/10.1109/ICARSC49921.2020.9096073

  32. Lu W, Zhang J, Zhao X, Wang J, Dang J (2017) Multimodal sensory fusion for soccer robot self-localization based on long short-term memory recurrent neural network. J Ambient Intell Humaniz Comput 8:885–893. https://doi.org/10.1007/s12652-017-0483-7

    Article  Google Scholar 

  33. Torabian S, HoseinAlipour S, Mirzargar A, Tavakkolian M (2013) Improving the localization of humanoid soccer robots in specified fields: a neural network approach. In: First RSI/ISM international conference on robotics and mechatronics (ICRoM), IEEE, pp 443–448. https://doi.org/10.1109/ICRoM.2013.6510148

  34. Yazdi E, Haghighat AT (2010) Evolution of biped walking using neural oscillators controller and harmony search algorithm optimizer. J Comput 2:111–118. https://arxiv.org/abs/1006.4553v1

  35. Ferreira R, Reis LP, Moreira AP, Lau N (2012) Development of an omnidirectional kick for a nao humanoid robot. In: Ibero-American conference on artificial intelligence, Springer, Berlin, Heidelberg, pp 571–580. https://doi.org/10.1007/978-3-642-34654-5_58

  36. Uchitane T, Hatanaka T (2011) Applying evolution strategies for biped locomotion learning in RoboCup 3D Soccer Simulation. In: IEEE congress of evolutionary computation (CEC), pp 179–185. https://doi.org/10.1109/CEC.2011.5949616

  37. Huang Q, Yokoi K, Kajita S, Kaneko K, Arai H, Koyachi N, Tanie K (2001) Planning walking patterns for a biped robot. IEEE Trans Robot Autom 17:280–289. https://doi.org/10.1109/70.938385

    Article  Google Scholar 

  38. Shamsuddin S, Ismail LI, Yussof H, Zahari NI, Bahari S, Hashim H, Jaffar A (2011) Humanoid robot NAO: review of control and motion exploration. In: IEEE international conference on control system, computing and engineering (ICCSCE), pp 511–516. https://doi.org/10.1109/ICCSCE.2011.6190579

  39. MacAlpine P, Barrett S, Urieli D, Vu V, Stone P (2012) Design and optimization of an omnidirectional humanoid walk: a winning approach at the RoboCup 2011 3D simulation competition. In: 26th AAAI conference on artificial intelligence, pp 1047–1053. https://www.cs.utexas.edu/~pstone/Papers/bib2html-links/AAAI12-MacAlpine.pdf. Accessed 29 Apr 2020

  40. Han SC, Li HX (2004) Invertible incline matrices and Cramer’s rule over inclines. Linear Algebra Appl 389:121–138. https://doi.org/10.1016/j.laa.2004.03.025

    Article  MathSciNet  MATH  Google Scholar 

  41. Gouaillier D, Collette C, Kilner C (2010) Omni-directional closed-loop walk for NAO. In: 10th IEEE-RAS international conference on humanoid robots (humanoids), pp 448–454. https://doi.org/10.1109/ICHR.2010.5686291

  42. Muniz F, Maximo MR, Ribeiro CH (2016) Keyframe movement optimization for simulated humanoid robot using a parallel optimization framework. In: Robotics symposium and IV Brazilian robotics symposium (LARS/SBR), XIII Latin American, IEEE, pp 79–84. https://doi.org/10.1109/LARS-SBR.2016.20

  43. Zhang PY, Lü TS, Song LB (2004) Soccer robot path planning based on the artificial potential field approach with simulated annealing. Robotica 22:563–566. https://doi.org/10.1017/S0263574703005666

    Article  Google Scholar 

  44. Sun K, Liu L, Qiu J, Feng G (2020) Fuzzy adaptive finite-time fault-tolerant control for strict-feedback nonlinear systems. IEEE Trans Fuzzy Syst. https://doi.org/10.1109/TFUZZ.2020.2965890

    Article  Google Scholar 

  45. Sun K, Jianbin Q, Karimi HR, Fu Y (2020) Event-triggered robust fuzzy adaptive finite-time control of nonlinear systems with prescribed performance. IEEE Trans Fuzzy Syst. https://doi.org/10.1109/TFUZZ.2020.2979129

    Article  Google Scholar 

  46. Ghalehgolabi M, Rezaeipanah A (2017) Intrusion detection system using genetic algorithm and data mining techniques based on the reduction. Int J Comput Appl Technol Res 6:461–466. https://doi.org/10.7753/IJCATR0611.1003

    Article  Google Scholar 

  47. Andersson SA, Madigan D, Perlman MD (2001) Alternative Markov properties for chain graphs. Scand J Stat 28:33–85. https://doi.org/10.1111/1467-9469.00224

    Article  MathSciNet  MATH  Google Scholar 

  48. Sunehag P, Evans R, Dulac-Arnold G, Zwols Y, Visentin D, Coppin B (2015) Deep reinforcement learning with attention for slate Markov decision processes with high-dimensional states and actions. arXiv preprint arXiv:1512.01124

  49. Szepesvári C (2010) Algorithms for reinforcement learning. Synth Lect Artif Intell Mach Learn 4:1–103. https://doi.org/10.2200/S00268ED1V01Y201005AIM009

    Article  MATH  Google Scholar 

  50. Tokic M (2010) Adaptive ε-greedy exploration in reinforcement learning based on value differences. In: Annual conference on artificial intelligence, Springer, Berlin, Heidelberg, pp 203–210. https://doi.org/10.1007/978-3-642-16111-7_23

  51. Park JH (2001) Impedance control for biped robot locomotion. IEEE Trans Robot Autom 17:870–882. https://doi.org/10.1109/70.976014

    Article  Google Scholar 

  52. Jafarian S, Abshirini A, Jafari S (2013) Online stability and direction fall detection for robotic soccer players using feedback from server. In: The 5th conference on information and knowledge technology, IEEE, pp 323–327. https://doi.org/10.1109/IKT.2013.6620087

  53. Goldenberg A, Benhabib B, Fenton R (1985) A complete generalized solution to the inverse kinematics of robots. IEEE J Robot Autom 1:14–20. https://doi.org/10.1109/JRA.1985.1086995

    Article  Google Scholar 

  54. Kofinas N, Orfanoudakis E, Lagoudakis MG (2015) Complete analytical forward and inverse kinematics for the NAO humanoid robot. J Intell Robot Syst 77:251–264. https://doi.org/10.1007/s10846-013-0015-4

    Article  Google Scholar 

  55. Binaries File, https://archive.robocup.info/Soccer/Simulation/3D/binaries/RoboCup/. Accessed 29 Apr 2020

  56. Xu Y, Vatankhah H (2013) Simspark: an open source robot simulator developed by the robocup community. In: Robot soccer world cup, Springer, Berlin, Heidelberg, pp 632–639. https://doi.org/10.1007/978-3-662-44468-9_59

  57. Allan K (2013) Introduction. In: Disability in science fiction. Palgrave Macmillan, New York, pp 1–15. https://doi.org/10.1057/9781137343437_1

  58. Riedmiller M, Gabel T, Hafner R, Lange S (2009) Reinforcement learning for robot soccer. Autonomous Robots 27:55–73. https://doi.org/10.1007/s10514-009-9120-4

    Article  Google Scholar 

  59. Sutton RS (1988) Learning to predict by the methods of temporal differences. Mach Learn 3:9–44. https://doi.org/10.1007/BF00115009

    Article  Google Scholar 

  60. Fang NC, Tsai TN, Wu LF, Cheng CH, Huang CY, Liu CY, Li THS (2017) Multi-robot coordination strategy for 3 vs. 3 teen-sized humanoid robot soccer game. In: automatic control conference (CACS), IEEE, pp 1–6. https://doi.org/10.1109/CACS.2017.8284265

  61. Nasrollahi P, Jafari S, Jamaseb M, Nikooee A (2013). Decision making of humanoid soccer robots using rule based expert systems. In: The 5th conference on information and knowledge technology, IEEE, pp 258–261. https://doi.org/10.1109/IKT.2013.6620075

  62. Gao L, Wang C, Zhang Y, Liu J. AUA3D Team Description Paper For RoboCup (2012) Robocup archive. http://archive.robocup.info/Soccer/Simulation/3D/TDPs/RoboCup/2012/AUA_SS3D_RC2012_TDP.pdf. Accessed 29 Apr 2020

  63. Li C, Liu R, Feng X, Chen Y, Lu S, CIT3D Soccer Simulation Team Description for RoboCup (2012) Robocup archive. https://hedayat.fedorapeople.org/misc/rc2011tdps/cit3d_tdp.pdf. Accessed 29 Apr 2020

  64. Scheunemann MM, van Dijk SG, Miko R, Barry D, Evans GM, Rossi A, Polani D (2019) Bold hearts team description for RoboCup 2019 (humanoid kid size league). arXiv preprint arXiv:1904.10066

Download references

Author information

Authors and Affiliations

  1. Department of Computer, Faculty of Computer, University of Rahjuyan Danesh Borazjan, Bushehr, Iran

    Amin Rezaeipanah

  2. Department of Computer Engineering, Ahram Branch, Islamic Azad University, Bushehr, Iran

    Parvin Amiri

  3. School of Electrical and Computer, Engineering, Shiraz University, Shiraz, Iran

    Shahram Jafari

Authors
  1. Amin Rezaeipanah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Parvin Amiri
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shahram Jafari
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amin Rezaeipanah.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rezaeipanah, A., Amiri, P. & Jafari, S. Performing the Kick During Walking for RoboCup 3D Soccer Simulation League Using Reinforcement Learning Algorithm. Int J of Soc Robotics 13, 1235–1252 (2021). https://doi.org/10.1007/s12369-020-00712-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-020-00712-2

Keywords

Search

Navigation