Skip to main content
SpringerLink
Log in

Intuitive Bare-Hand Teleoperation of a Robotic Manipulator Using Virtual Reality and Leap Motion

  • Conference paper
  • First Online:
Towards Autonomous Robotic Systems (TAROS 2019)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11650))

Included in the following conference series:

Abstract

Despite various existing works on intuitive human-robot interaction (HRI) for teleoperation of robotic manipulators, to the best of our knowledge, the following research question has not been investigated yet: Can we have a teleoperated robotic manipulator that simply copies a human operator’s bare hand posture and gesture in a real-time manner without having any hand-held devices? This paper presents a novel teleoperation system that attempts to address this question. Firstly, we detail how to set up the system practically by using a Universal Robots UR5, a Robotiq 3-finger gripper, and a Leap Motion based on Unity and ROS, and describe specifically what information is communicated between each other. Furthermore, we provide the details of the ROS nodes developed for controlling the robotic arm and gripper, given the information of a human’s bare hands sensed by the Leap Motion. Then, we demonstrate our system executing a simple pick-and-place task, and discuss possible benefits and costs of this HRI concept.

This project has been supported by the RAIN Hub, which is funded by the Industrial Strategy Challenge Fund, part of the government’s modern Industrial Strategy. The fund is delivered by UK Research and Innovation and managed by EPSRC [EP/R026084/1].

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/siemens/ros-sharp/.

  2. 2.

    See comparison in https://youtu.be/7HnfG0a6Gfg.

  3. 3.

    https://leapmotion.github.io/UnityModules/interaction-engine.html.

  4. 4.

    https://gallery.leapmotion.com/detection-example/.

  5. 5.

    https://github.com/ros-industrial/universal_robot.

  6. 6.

    https://github.com/ros-industrial/robotiq.

  7. 7.

    https://github.com/ros-industrial/ur_modern_driver.

  8. 8.

    The URScript Programming Language, ver 3.5.4, April 12, 2018.

  9. 9.

    http://wiki.ros.org/ur_kin_py.

  10. 10.

    Watch also the video: https://youtu.be/lu-0yrl9J5g.

  11. 11.

    https://www.ultrahaptics.com/products-programs/.

References

  1. Allspaw, J., Roche, J., Lemiesz, N., Yannuzzi, M., Yanco, H.A.: Remotely teleoperating a humanoid robot to perform fine motor tasks with virtual reality. In: Waste Management Symposium (WM 2018), Phoenix, AZ (2018)

    Google Scholar 

  2. Cancedda, L., Cannavò, A., Garofalo, G., Lamberti, F., Montuschi, P., Paravati, G.: Mixed reality-based user interaction feedback for a hand-controlled interface targeted to robot teleoperation. In: De Paolis, L.T., Bourdot, P., Mongelli, A. (eds.) AVR 2017. LNCS, vol. 10325, pp. 447–463. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-60928-7_38

    Chapter  Google Scholar 

  3. Chae, J., Jin, Y., Sung, Y., Cho, K.: Genetic algorithm-based motion estimation method using orientations and EMGs for robot controls. Sensors 18(2), 183 (2018). https://doi.org/10.3390/s18010183

    Article  Google Scholar 

  4. Chen, S., Ma, H., Yang, C., Fu, M.: Hand gesture based robot control system using leap motion. In: Liu, H., Kubota, N., Zhu, X., Dillmann, R., Zhou, D. (eds.) ICIRA 2015. LNCS (LNAI), vol. 9244, pp. 581–591. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-22879-2_53

    Chapter  Google Scholar 

  5. Crick, C., Jay, G., Osentoski, S., Pitzer, B., Jenkins, O.C.: Rosbridge: ROS for non-ROS users. Springer Tracts Adv. Robot. 100, 493–504 (2017)

    Article  Google Scholar 

  6. Hawkins, K.P.: Analytic inverse kinematics for the universal robots UR-5/UR-10 arms. Technical report, Georgia Institute of Technology (2013). https://smartech.gatech.edu/bitstream/handle/1853/50782/ur_kin_tech_report_1.pdf

  7. Hokayem, P.F., Spong, M.W.: Bilateral teleoperation: an historical survey. Automatica 42(12), 2035–2057 (2006). https://doi.org/10.1016/j.automatica.2006.06.027

    Article  MathSciNet  MATH  Google Scholar 

  8. Jin, H., Chen, Q., Chen, Z., Hu, Y., Zhang, J.: Multi-LeapMotion sensor based demonstration for robotic refine tabletop object manipulation task. CAAI Trans. Intell. Technol. 1(1), 104–113 (2016). https://doi.org/10.1016/j.trit.2016.03.010. https://linkinghub.elsevier.com/retrieve/pii/S2468232216000111

    Article  Google Scholar 

  9. Krupke, D., Einig, L., Langbehn, E., Zhang, J., Steinicke, F.: Immersive remote grasping: realtime gripper control by a heterogenous robot control system. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST 02–04-Nove, pp. 337–338 (2016). https://doi.org/10.1145/2993369.2996345

  10. Kruusamae, K., Pryor, M.: High-precision telerobot with human-centered variable perspective and scalable gestural interface. In: Proceedings - 2016 9th International Conference on Human System Interactions, HSI 2016, pp. 190–196 (2016). https://doi.org/10.1109/HSI.2016.7529630

  11. Li, C., Yang, C., Wan, J., Annamalai, A.S.S., Cangelosi, A.: Teleoperation control of Baxter robot using Kalman filter-based sensor fusion. Syst. Sci. Control Eng. 5(1), 156–167 (2017). https://doi.org/10.1080/21642583.2017.1300109

    Article  Google Scholar 

  12. Lipton, J.I., Fay, A.J., Rus, D.: Baxter’s homunculus: virtual reality spaces for teleoperation in manufacturing. IEEE Robot. Autom. Lett. 3(1), 179–186 (2018). https://doi.org/10.1109/LRA.2017.2737046

    Article  Google Scholar 

  13. Makris, S., et al.: Dual arm robot in cooperation with humans for flexible assembly. CIRP Ann. 66(1), 13–16 (2017). https://doi.org/10.1016/j.cirp.2017.04.097

    Article  Google Scholar 

  14. Nuño, E., Basañez, L., Ortega, R.: Passivity-based control for bilateral teleoperation: A tutorial. Automatica 47(3), 485–495 (2011). https://doi.org/10.1016/j.automatica.2011.01.004

    Article  MathSciNet  MATH  Google Scholar 

  15. Pancake, D., et al.: A novel and cost effective approach to the decommissioning and decontamination of legacy glove boxes - minimizing TRU waste and maximizing LLW waste - 13634. In: Waste Management Symposium (WM 2013), Phoenix, AZ (2013)

    Google Scholar 

  16. Peppoloni, L., Brizzi, F., Avizzano, C.A., Ruffaldi, E.: Immersive ROS-integrated framework for robot teleoperation. In: 2015 IEEE Symposium on 3D User Interfaces (3DUI), pp. 177–178. IEEE, March 2015. https://doi.org/10.1109/3DUI.2015.7131758

  17. Roldán, J.J., et al.: Multi-robot systems, virtual reality and ros: developing a new generation of operator interfaces. In: Koubaa, A. (ed.) Robot Operating System (ROS). SCI, vol. 778, pp. 29–64. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-91590-6_2

    Chapter  Google Scholar 

  18. Tang, G., Webb, P.: The design and evaluation of an ergonomic contactless gesture control system for industrial robots. J. Robot. (2018). https://doi.org/10.1155/2018/9791286

    Article  Google Scholar 

  19. Whitney, D., Rosen, E., Phillips, E., Konidaris, G., Tellex, S.: Comparing robot grasping teleoperation across desktop and virtual reality with ROS reality. In: International Symposium on Robotics Research, pp. 1–16 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The University of Manchester, Manchester, M13 9PL, UK

    Inmo Jang, Joaquin Carrasco, Andrew Weightman & Barry Lennox

Authors
  1. Inmo Jang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joaquin Carrasco
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrew Weightman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Barry Lennox
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Inmo Jang .

Editor information

Editors and Affiliations

  1. Queen Mary University of London, London, UK

    Kaspar Althoefer

  2. Queen Mary University of London, London, UK

    Jelizaveta Konstantinova

  3. Queen Mary University of London, London, UK

    Ketao Zhang

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jang, I., Carrasco, J., Weightman, A., Lennox, B. (2019). Intuitive Bare-Hand Teleoperation of a Robotic Manipulator Using Virtual Reality and Leap Motion. In: Althoefer, K., Konstantinova, J., Zhang, K. (eds) Towards Autonomous Robotic Systems. TAROS 2019. Lecture Notes in Computer Science(), vol 11650. Springer, Cham. https://doi.org/10.1007/978-3-030-25332-5_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-25332-5_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-25331-8

  • Online ISBN: 978-3-030-25332-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation