Skip to main content
SpringerLink
Log in

Using open surgery simulation kinematic data for tool and gesture recognition

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

The use of motion sensors is emerging as a means for measuring surgical performance. Motion sensors are typically used for calculating performance metrics and assessing skill. The aim of this study was to identify surgical gestures and tools used during an open surgery suturing simulation based on motion sensor data.

Methods

Twenty-five participants performed a suturing task on a variable tissue simulator. Electromagnetic motion sensors were used to measure their performance. The current study compares GRU and LSTM networks, which are known to perform well on other kinematic datasets, as well as MS-TCN++, which was developed for video data and was adapted in this work for motion sensors data. Finally, we extended all architectures for multi-tasking.

Results

In the gesture recognition task the MS-TCN++ has the highest performance with accuracy of \(82.4 \pm 6.97\) and F1-Macro of \(78.92 \pm 8.5\), edit distance of \(86.30 \pm 8.42\) and F1@10 of \(89.30 \pm 7.01\) In the tool usage recognition task for the right hand, MS-TCN++ performs the best in most metrics with an accuracy score of \(94.69 \pm 3.57\), F1-Macro of \(86.06 \pm 7.06\), F1@10 of \(84.34 \pm 10.90\), and F1@25 of \(80.58 \pm 12.03\). The multi-task GRU performs best in all metrics in the left-hand case, with an accuracy of \(95.04 \pm 4.18\), edit distance of \(85.01 \pm 16.94\), F1-Macro of \(89.81 \pm 11.65\), F1@10 of \(89.17 \pm 13.28\), and F1@25 of \(88.64 \pm 13.6\).

Conclusion

In this study, using motion sensor data, we automatically identified the surgical gestures and the tools used during an open surgery suturing simulation. Our methods may be used for computing more detailed performance metrics and assisting in automatic workflow analysis. MS-TCN++ performed better in gesture recognition as well as right-hand tool recognition, while the multi-task GRU provided better results in the left-hand case. It should be noted that our multi-task GRU network is significantly smaller and has achieved competitive results in the rest of the tasks as well.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Vozenilek J, Huff JS, Reznek M, Gordon JA (2004) See one, do one, teach one: advanced technology in medical education. Acad Emerg Med 11(11):1149–1154

    Article  Google Scholar 

  2. Ziv A, Wolpe PR, Small SD, Glick S (2003) Simulation-based medical education: an ethical imperative. Acad Med 78(8):783–788

    Article  Google Scholar 

  3. Aggarwal R, Mytton OT, Derbrew M, Hananel D, Heydenburg M, Issenberg B, MacAulay C, Mancini ME, Morimoto T, Soper N, Ziv A, Reznick R (2010) Training and simulation for patient safety. BMJ Qual Saf 19(Suppl 2):34–43

    Article  Google Scholar 

  4. Reznick RK, MacRae H (2006) Teaching surgical skills-changes in the wind. N Engl J Med 355(25):2664–2669

    Article  CAS  Google Scholar 

  5. Fonseca AL, Evans LV, Gusberg RJ (2013) Open surgical simulation in residency training: a review of its status and a case for its incorporation. J Surg Educ 70(1):129–137

    Article  Google Scholar 

  6. Epstein RM (2007) Assessment in medical education. N Engl J Med 356(4):387–396

    Article  CAS  Google Scholar 

  7. Moorthy K, Munz Y, Sarker SK, Darzi A (2003) Objective assessment of technical skills in surgery. Bmj 327(7422):1032–1037

    Article  Google Scholar 

  8. Martin J, Regehr G, Reznick R, Macrae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84(2):273–278

    CAS  PubMed  Google Scholar 

  9. D’Angelo A-LD, Rutherford DN, Ray RD, Laufer S, Mason A, Pugh CM (2016) Working volume: validity evidence for a motion-based metric of surgical efficiency. Am J Surg 211(2):445–450

    Article  Google Scholar 

  10. Howells NR, Brinsden MD, Gill RS, Carr AJ, Rees JL (2008) Motion analysis: a validated method for showing skill levels in arthroscopy. Arthrosc J Arthrosc Relat Surg 24(3):335–342

    Article  Google Scholar 

  11. Yamaguchi S, Yoshida D, Kenmotsu H, Yasunaga T, Konishi K, Ieiri S, Nakashima H, Tanoue K, Hashizume M (2011) Objective assessment of laparoscopic suturing skills using a motion-tracking system. Surg Endosc 25(3):771–775

    Article  Google Scholar 

  12. Kowalewski K-F, Hendrie JD, Schmidt MW, Garrow CR, Bruckner T, Proctor T, Paul S, Adigüzel D, Bodenstedt S, Erben A, Kenngott H, Erben Y, Speidel S, Müller-Stich BP, Nickel F (2017) Development and validation of a sensor-and expert model-based training system for laparoscopic surgery: the iSurgeon. Surg Endosc 31(5):2155–2165

    Article  Google Scholar 

  13. D’Angelo A-LD, Rutherford DN, Ray RD, Laufer S, Kwan C, Cohen ER, Mason A, Pugh CM (2015) Idle time: an underdeveloped performance metric for assessing surgical skill. Am J Surg 209(4):645–651

    Article  Google Scholar 

  14. Buescher JF, Mehdorn A-S, Neumann P-A, Becker F, Eichelmann A-K, Pankratius U, Bahde R, Foell D, Senninger N, Rijcken E (2018) Effect of continuous motion parameter feedback on laparoscopic simulation training: a prospective randomized controlled trial on skill acquisition and retention. J Surg Educ 75(2):516–526

    Article  Google Scholar 

  15. Jantscher, W.H., Pandey, S., Agarwal, P., Richardson, S.H., Lin, B.R., Byrne, M.D., O’Malley, M.K.: Toward improved surgical training: Delivering smoothness feedback using haptic cues. In: 2018 IEEE Haptics Symposium (HAPTICS), pp. 241–246 (2018). IEEE

  16. Gao, Y., Vedula, S.S., Reiley, C.E., Ahmidi, N., Varadarajan, B., Lin, H.C., Tao, L., Zappella, L., Béjar, B., Yuh, D.D., Chen, C.C.G., Vida, R., Khudanpur, S., Hager, G.G.: Jhu-isi gesture and skill assessment working set (jigsaws): A surgical activity dataset for human motion modeling. In: MICCAI Workshop: M2cai, vol. 3, p. 3 (2014)

  17. Fawaz HI, Forestier G, Weber J, Idoumghar L, Muller P-A (2019) Accurate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networks. Int J Comput Assist Radiol Surg 14(9):1611–1617

    Article  Google Scholar 

  18. DiPietro, R., Lea, C., Malpani, A., Ahmidi, N., Vedula, S.S., Lee, G.I., Lee, M.R., Hager, G.D.: Recognizing surgical activities with recurrent neural networks. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp. 551–558 (2016). Springer

  19. van Amsterdam, B., Clarkson, M.J., Stoyanov, D.: Multi-task recurrent neural network for surgical gesture recognition and progress prediction. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 1380–1386 (2020). IEEE

  20. Lea, C., Vidal, R., Reiter, A., Hager, G.D.: Temporal convolutional networks: a unified approach to action segmentation. In: European Conference on Computer Vision, pp. 47–54 (2016). Springer

  21. Funke, I., Bodenstedt, S., Oehme, F., von Bechtolsheim, F., Weitz, J., Speidel, S.: Using 3d convolutional neural networks to learn spatiotemporal features for automatic surgical gesture recognition in video. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 467–475 (2019). Springer

  22. Liu, D., Jiang, T.: Deep reinforcement learning for surgical gesture segmentation and classification. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp. 247–255 (2018). Springer

  23. Li, S.-J., AbuFarha, Y., Liu, Y., Cheng, M.-M., Gall, J.: Ms-tcn++: multi-stage temporal convolutional network for action segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020)

  24. Richard, A., Kuehne, H., Gall, J.: Weakly supervised action learning with rnn based fine-to-coarse modeling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 754–763 (2017)

  25. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  CAS  Google Scholar 

  26. DiPietro R, Ahmidi N, Malpani A, Waldram M, Lee GI, Lee MR, Vedula SS, Hager GD (2019) Segmenting and classifying activities in robot-assisted surgery with recurrent neural networks. Int J Comput Assist Radiol Surg 14(11):2005–2020

    Article  Google Scholar 

  27. Cho, K., Van Merriënboer, B., Bahdanau, D., Bengio, Y.: On the properties of neural machine translation: encoder–decoder approaches. arXiv preprint arXiv:1409.1259 (2014)

  28. Goldbraikh A, D’Angelo A-L, Pugh CM, Laufer S (2022) Video-based fully automatic assessment of open surgery suturing skills. Int J Comput Assist Radiol Surg 17:1–12

    Article  Google Scholar 

  29. McClellan JH, Parks TW (2005) A personal history of the Parks–McClellan algorithm. IEEE Signal Process Mag 22(2):82–86

    Article  Google Scholar 

  30. Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299–6308 (2017)

  31. Kuehne, H., Arslan, A., Serre, T.: The language of actions: recovering the syntax and semantics of goal-directed human activities. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 780–787 (2014)

  32. Huang C, Huang X, Fang Y, Xu J, Qu Y, Zhai P, Fan L, Yin H, Xu Y, Li J (2020) Sample imbalance disease classification model based on association rule feature selection. Pattern Recognit Lett 133:280–286

    Article  Google Scholar 

  33. Lea, C., Vidal, R., Hager, G.D.: Learning convolutional action primitives for fine-grained action recognition. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 1642–1649 (2016). IEEE

  34. Lea, C., Flynn, M.D., Vidal, R., Reiter, A., Hager, G.D.: Temporal convolutional networks for action segmentation and detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 156–165 (2017)

  35. Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M.: Optuna: a next-generation hyperparameter optimization framework. In: Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2019)

  36. Hutter, F., Hoos, H., Leyton-Brown, K.: An efficient approach for assessing hyperparameter importance. In: International Conference on Machine Learning, pp. 754–762 (2014). PMLR

  37. Goldbraikh, A., Shubi, O., Shamir, N., Carla, Laufer, S.: Efficient assessment of surgical maneuvers: Human versus machine. In: 2022 Surgeons and Engineers: A Dialogue on Surgical Simulation (2022). ACS

  38. Twinanda AP, Shehata S, Mutter D, Marescaux J, De Mathelin M, Padoy N (2016) Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging 36(1):86–97

    Article  Google Scholar 

  39. Jin, A., Yeung, S., Jopling, J., Krause, J., Azagury, D., Milstein, A., Fei-Fei, L.: Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 691–699 (2018). IEEE

  40. Sarikaya D, Corso JJ, Guru KA (2017) Detection and localization of robotic tools in robot-assisted surgery videos using deep neural networks for region proposal and detection. IEEE Trans Med Imaging 36(7):1542–1549

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Applied Mathematics Department, Technion – Israel Institute of Technology, 3200003, Haifa, Israel

    Adam Goldbraikh

  2. Faculty of Industrial Engineering and Management, Technion – Israel Institute of Technology, 3200003, Haifa, Israel

    Tomer Volk & Shlomi Laufer

  3. Department of Surgery, Stanford University School of Medicine, Stanford, CA, 610101, USA

    Carla M. Pugh

Authors
  1. Adam Goldbraikh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tomer Volk
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Carla M. Pugh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shlomi Laufer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adam Goldbraikh.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study. *Figures 4 and 5 were created with BioRender.com.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Code: https://github.com/AdamGoldbraikh/Using-Open-Surgery-Simulation-Kinematic-Data-for-Tool-and-Gesture-Recognition.git

Appendix

Appendix

See Figs. 13 and 14.

Fig. 14
figure 14

Gesture recognition and tool recognition networks results, confusion matrix for each task. Each test of each cross-validation fold is summed 8 times for the entire dataset

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Goldbraikh, A., Volk, T., Pugh, C.M. et al. Using open surgery simulation kinematic data for tool and gesture recognition. Int J CARS 17, 965–979 (2022). https://doi.org/10.1007/s11548-022-02615-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-022-02615-1

Keywords

Search

Navigation