Skip to main content
SpringerLink
Log in

Deep learning to detect anterior cruciate ligament tear on knee MRI: multi-continental external validation

  • Musculoskeletal
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objectives

To develop a deep-learning algorithm for anterior cruciate ligament (ACL) tear detection and to compare its accuracy using two external datasets.

Methods

A database of 19,765 knee MRI scans (17,738 patients) issued from different manufacturers and magnetic fields was used to build a deep learning–based ACL tear detector. Fifteen percent showed partial or complete ACL rupture. Coronal and sagittal fat-suppressed proton density or T2-weighted sequences were used. A Natural Language Processing algorithm was used to automatically label reports associated with each MRI exam. We compared the accuracy of our model on two publicly available external datasets: MRNet, Bien et al, USA (PLoS Med 15:e1002699, 2018); and KneeMRI, Stajduhar et al, Croatia (Comput Methods Prog Biomed 140:151-164, 2017). Receptor operating characteristics (ROC) curves, area under the curve (AUC), sensitivity, specificity, and accuracy were used to evaluate our model.

Results

Our neural networks achieved an AUC value of 0.939 for detection of ACL tears, with a sensitivity of 87% (0.875) and a specificity of 91% (0.908). After retraining our model on Bien dataset and Stajduhar dataset, our algorithm achieved AUC of 0.962 (95% CI 0.930–0.988) and 0.922 (95% CI 0.875, 0.962) respectively. Sensitivity, specificity, and accuracy were respectively 85% (95% CI 75–94%, 0.852), 89% (95% CI 82–97%, 0.894), 0.875 (95% CI 0.817–0.933) for Bien dataset, and 68% (95% CI 54–81%, 0.681), 93% (95% CI 89–97%, 0.934), and 0.870 (95% CI 0.821–0.913) for Stajduhar dataset.

Conclusion

Our algorithm showed high performance in the detection of ACL tears with AUC on two external datasets, demonstrating its generalizability on different manufacturers and populations.

Summary

This study shows the performance of an algorithm for detecting anterior cruciate ligament tears with an external validation on populations from countries and continents different from the study population.

Key Points

• An algorithm for detecting anterior cruciate ligament ruptures was built from a large dataset of nearly 20,000 MRI with AUC values of 0.939, sensitivity of 87%, and specificity of 91%.

• This algorithm was tested on two external populations from different other countries: a dataset from an American population and a dataset from a Croatian population. Performance remains high on these two external validation populations (AUC of 0.962 and 0.922 respectively).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3.
Fig. 4

Similar content being viewed by others

Abbreviations

ACL:

Anterior cruciate ligament

CNN:

Convolutional neural networks

IoU:

Intersection over Union

NLP:

Natural Language Processing

PD:

Proton density

ReLU:

Rectified linear unit

SD:

Standard deviation

References

  1. Musahl V, Karlsson J (2019) Anterior cruciate ligament tear. N Engl J Med 380:2341–2348

    Article  PubMed  Google Scholar 

  2. Shea K, Carey J (2015) Management of anterior cruciate ligament injuries. J Am Acad Orthop Surg 23:e1–e5

    Article  PubMed  Google Scholar 

  3. Li K, Du J, Huang L, Ni L, Liu T, Yang H (2017) The diagnostic accuracy of magnetic resonance imaging for anterior cruciate ligament injury in comparison to arthroscopy: a meta-analysis. Sci Rep 7:7583

    Article  PubMed  PubMed Central  Google Scholar 

  4. Challen J, Tang Y, Hazratwala K, Stuckey S (2007) Accuracy of MRI diagnosis of internal derangement of the knee in a non-specialized tertiary level referral teaching hospital. Australas Radiol 51:426–431

    Article  CAS  PubMed  Google Scholar 

  5. Singh R, Kalra M, Nitiwarangkul C et al (2018) Deep learning in chest radiography: detection of findings and presence of change. PLoS One 13:e0204155

    Article  PubMed  PubMed Central  Google Scholar 

  6. Chilamkurthy S, Ghosh R, Tanamala S et al (2018) Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. Lancet 392:2388–2396

  7. Gyftopoulos S, Lin D, Knoll F, Doshi A, Rodrigues T, Recht M (2019) Artificial intelligence in musculoskeletal imaging: current status and future directions. AJR Am J Roentgenol 213:506–513

    Article  PubMed  PubMed Central  Google Scholar 

  8. Bien N, Rajpurkar P, Ball R et al (2018) Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of MRNet. PLoS Med 15:e1002699

    Article  PubMed  PubMed Central  Google Scholar 

  9. Štajduhar I, Mamula M, Miletić D, Ünal G (2017) Semi-automated detection of anterior cruciate ligament injury from MRI. Comput Methods Prog Biomed 140:151–164

    Article  Google Scholar 

  10. Rizk B, Brat H, Zille P, Guillin R, Pouchy C (2021) Adam C, et al Meniscal lesion detection and characterization in adult knee MRI: A deep learning model approach with external validation. Phys Med 83:64–71

    Article  CAS  PubMed  Google Scholar 

  11. Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. arXiv:1301.3781. Available from: http://arxiv.org/abs/1301.3781

  12. Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:14123555. Available from: http://arxiv.org/abs/1412.3555

  13. Beaufils P, Hulet C, Dhénain M, Nizard R, Nourissat G, Pujol N (2009) Clinical practice guidelines for the management of meniscal lesions and isolated lesions of the anterior cruciate ligament of the knee in adults. Orthop Traumatol Surg Res 95(6):437–442

    Article  CAS  PubMed  Google Scholar 

  14. DiCiccio T, Efron B (1996) Bootstrap confidence intervals. Stat Sci 11:189–228

    Article  Google Scholar 

  15. Liu F, Guan B, Zhou Z et al (2019) Fully automated diagnosis of anterior cruciate ligament tears on knee MR images by using deep learning. Radiol Artif Intell 1:180091

    Article  PubMed  PubMed Central  Google Scholar 

  16. Germann C, Marbach G, Civardi F et al (2020) Deep convolutional neural network-based diagnosis of anterior cruciate ligament tears: performance comparison of homogenous versus heterogeneous knee MRI cohorts with different pulse sequence protocols and 1.5-T and 3-T magnetic field strengths. Invest Radiol 55:499–506

    CAS  PubMed  PubMed Central  Google Scholar 

  17. Zhang L, Li M, Zhou Y, Lu G, Zhou Q (2020) Deep learning approach for anterior cruciate ligament lesion detection: evaluation of diagnostic performance using arthroscopy as the reference standard. J Magn Reson Imaging 52:1745–1752

    Article  PubMed  Google Scholar 

  18. Pinto dos Santos D, Brodehl S, Baeßler B et al (2019) Structured report data can be used to develop deep learning algorithms: a proof of concept in ankle radiographs. Insights Imaging 10:93

    Article  PubMed  PubMed Central  Google Scholar 

  19. Chang P, Wong T, Rasiej M (2019) Deep learning for detection of complete anterior cruciate ligament tear. J Digit Imaging 32:980–986

    Article  PubMed  PubMed Central  Google Scholar 

  20. Van Dyck P, Vanhoenacker F, Gielen J et al (2010) Three tesla magnetic resonance imaging of the anterior cruciate ligament of the knee: can we differentiate complete from partial tears? Skelet Radiol 40:701–707

  21. Phelan N, Rowland P, Galvin R, O’Byrne J (2015) A systematic review and meta-analysis of the diagnostic accuracy of MRI for suspected ACL and meniscal tears of the knee. Knee Surg Sports Traumatol Arthrosc 24:1525–1539

    Article  PubMed  Google Scholar 

Download references

Funding

The authors state that this work has not received any funding.

Author information

Authors and Affiliations

  1. Department of Radiology, Hôpital Européen Georges Pompidou, Assistance Publique-Hôpitaux de Paris; Université de Paris, 20 Rue Leblanc, 75015, Paris, France

    Alexia Tran

  2. Réseau d’imagerie Sud Francilien, Evry, France

    Louis Lassalle

  3. Incepto Medical, Paris, France

    Pascal Zille, Chloé Adam, Martin Charachon, Maxence Wallaert & Gaspard d’Assignies

  4. Department of Radiology, Centre Hospitalier Universitaire de Rennes, Rennes, France

    Raphaël Guillin

  5. Department of Radiology, Radiologie B, Hôpital Cochin, Assistance Publique-Hôpitaux de Paris; Université de Paris, Paris, France

    Etienne Pluot

  6. Institut de Radiologie de Sion, Groupe 3R, Sion, Switzerland

    Hugues Brat & Benoît Rizk

  7. Department of Radiology, Centre Hospitalier Départemental Vendée, La Roche-sur-Yon, France

    Gaspard d’Assignies

Authors
  1. Alexia Tran
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Louis Lassalle
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pascal Zille
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Raphaël Guillin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Etienne Pluot
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Chloé Adam
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Martin Charachon
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Hugues Brat
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Maxence Wallaert
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Gaspard d’Assignies
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Benoît Rizk
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexia Tran.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Pascal Zille.

Conflict of interest

The authors of this manuscript declare relationships with the following companies:

• A.T. was an intern at Incepto Medical, but at the time of submission of the article disclosed no relevant relationships.

• P.Z., C.A., M.C., and M.W. are employed by Incepto Medical.

• G.A. is the founder and Chief Medical Officer of Incepto Medical.

• R.G. disclosed no conflict of interest.

• H.B. disclosed no conflict of interest.

• B.R. disclosed no conflict of interest.

• L.L. has a consulting activity for Incepto Medical.

Statistics and biometry

No complex statistical methods were necessary for this paper.

Informed consent

Written informed consent was obtained from all subjects (patients) in this study.

Ethical approval

Institutional Review Board approval was obtained.

Study subjects or cohorts overlap

Some study subjects have been previously reported in Rizk B, Brat H, Zille P, Guillin R, Pouchy C, Adam C, et al Meniscal lesion detection and characterization in adult knee MRI: a deep learning model approach with external validation. Physica Medica. 2021 Mar;83:64–71.

Methodology

• retrospective

• diagnostic study

• multicenter study

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

*Alexia Tran and Louis Lassalle are joint first authors.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tran, A., Lassalle, L., Zille, P. et al. Deep learning to detect anterior cruciate ligament tear on knee MRI: multi-continental external validation. Eur Radiol 32, 8394–8403 (2022). https://doi.org/10.1007/s00330-022-08923-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-022-08923-z

Keywords

Search

Navigation