Skip to main content
SpringerLink
Log in

Automated detection of pulmonary embolism in CT pulmonary angiograms using an AI-powered algorithm

  • Computed Tomography
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objectives

To evaluate the performance of an AI-powered algorithm for the automatic detection of pulmonary embolism (PE) on chest computed tomography pulmonary angiograms (CTPAs) on a large dataset.

Methods

We retrospectively identified all CTPAs conducted at our institution in 2017 (n = 1499). Exams with clinical questions other than PE were excluded from the analysis (n = 34). The remaining exams were classified into positive (n = 232) and negative (n = 1233) for PE based on the final written reports, which defined the reference standard. The fully anonymized 1-mm series in soft tissue reconstruction served as input for the PE detection prototype algorithm that was based on a deep convolutional neural network comprising a Resnet architecture. It was trained and validated on 28,000 CTPAs acquired at other institutions. The result series were reviewed using a web-based feedback platform. Measures of diagnostic performance were calculated on a per patient and a per finding level.

Results

The algorithm correctly identified 215 of 232 exams positive for pulmonary embolism (sensitivity 92.7%; 95% confidence interval [CI] 88.3–95.5%) and 1178 of 1233 exams negative for pulmonary embolism (specificity 95.5%; 95% CI 94.2–96.6%). On a per finding level, 1174 of 1352 findings marked as embolus by the algorithm were true emboli. Most of the false positive findings were due to contrast agent–related flow artifacts, pulmonary veins, and lymph nodes.

Conclusion

The AI prototype algorithm we tested has a high degree of diagnostic accuracy for the detection of PE on CTPAs. Sensitivity and specificity are balanced, which is a prerequisite for its clinical usefulness.

Key Points

• An AI-based prototype algorithm showed a high degree of diagnostic accuracy for the detection of pulmonary embolism on CTPAs.

• It can therefore help clinicians to automatically prioritize exams with a high suspection of pulmonary embolism and serve as secondary reading tool.

• By complementing traditional ways of worklist prioritization in radiology departments, this can speed up the diagnostic and therapeutic workup of patients with pulmonary embolism and help to avoid false negative calls.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Abbreviations

ADMIRE:

Advanced Modeled Iterative Reconstruction

CAD:

Computer-assisted detection

CTPA:

CT pulmonary angiogram

DCNN:

Deep convolutional neural network

DICOM:

Digital imaging and communications in medicine

FN:

False negative

FP:

False positive

FTE:

File transfer protocol

IR:

Iterative reconstruction

NPV:

Negative predictive value

PACS:

Picture Archiving and Communication System

PE:

Pulmonary embolism

PGY:

Postgraduate year

PPV:

Positive predictive value

SAFIRE:

Sinogram Affirmed Iterative Reconstruction

SWCCE:

Sample weighted categorical cross-entropy

TP:

True positive

References

  1. Brown J, Shesser R (2004) Computed tomography scan use is rising faster than other investigational modalities in the emergency department evaluation of patients. Ann Emerg Med 44:S33. https://doi.org/10.1016/j.annemergmed.2004.07.109

    Article  Google Scholar 

  2. Kocher KE, Meurer WJ, Fazel R, Scott PA, Krumholz HM, Nallamothu BK (2011) National trends in use of computed tomography in the emergency department. Ann Emerg Med 58:452–462.e3. https://doi.org/10.1016/J.ANNEMERGMED.2011.05.020

  3. Estrada-Y-Martin RM, Oldham SA (2011) CTPA as the gold standard for the diagnosis of pulmonary embolism. Int J Comput Assist Radiol Surg 6:557–563. https://doi.org/10.1007/s11548-010-0526-4

    Article  Google Scholar 

  4. Pernod G, Caterino J, Maignan M et al (2017) D-dimer use and pulmonary embolism diagnosis in emergency units: why is there such a difference in pulmonary embolism prevalence between the United States of America and countries outside USA? PLoS One 12:e0169268. https://doi.org/10.1371/journal.pone.0169268

    Article  CAS  Google Scholar 

  5. Smith SB, Geske JB, Maguire JM, Zane NA, Carter RE, Morgenthaler TI (2010) Early anticoagulation is associated with reduced mortality for acute pulmonary embolism. Chest 137:1382–1390

  6. Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86:2278–2324. https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  7. Prevedello LM, Erdal BS, Ryu JL et al (2017) Automated critical test findings identification and online notification system using artificial intelligence in imaging. Radiology 285:923–931. https://doi.org/10.1148/radiol.2017162664

    Article  Google Scholar 

  8. Nagel S, Sinha D, Day D et al (2017) e-ASPECTS software is non-inferior to neuroradiologists in applying the ASPECT score to computed tomography scans of acute ischemic stroke patients. Int J Stroke 12:615–622. https://doi.org/10.1177/1747493016681020

    Article  Google Scholar 

  9. Winkel DJ, Heye T, Weikert TJ, Boll DT, Stieltjes B (2018) Evaluation of an AI-based detection software for acute findings in abdominal computed tomography scans. Invest Radiol:1. https://doi.org/10.1097/RLI.0000000000000509

  10. Liu K, Li Q, Ma J et al (2019) Evaluating a fully automated pulmonary nodule detection approach and its impact on radiologist performance. Radiol Artif Intell 1:e180084. https://doi.org/10.1148/ryai.2019180084

    Article  Google Scholar 

  11. Vorontsov E, Cerny M, Régnier P et al (2019) Deep learning for automated segmentation of liver lesions at CT in patients with colorectal cancer liver metastases. Radiol Artif Intell 1:180014. https://doi.org/10.1148/ryai.2019180014

    Article  Google Scholar 

  12. Thian YL, Li Y, Jagmohan P, Sia D, Chan VE, Tan RT (2019) Convolutional neural networks for automated fracture detection and localization on wrist radiographs. Radiol Artif Intell 1:e180001. https://doi.org/10.1148/ryai.2019180001

  13. Li L, Liu Z, Huang H, Lin M, Luo D (2018) Evaluating the performance of a deep learning-based computer-aided diagnosis (DL-CAD) system for detecting and characterizing lung nodules: comparison with the performance of double reading by radiologists. Thorac Cancer 10:1759–7714.12931. https://doi.org/10.1111/1759-7714.12931

  14. Ekert T, Krois J, Meinhold L et al (2019) Deep learning for the radiographic detection of apical lesions. J Endod. https://doi.org/10.1016/j.joen.2019.03.016

  15. Cheng C-T, Ho T-Y, Lee T-Y et al (2019) Application of a deep learning algorithm for detection and visualization of hip fractures on plain pelvic radiographs. Eur Radiol. https://doi.org/10.1007/s00330-019-06167-y

  16. Ye W, Gu W, Guo X et al (2019) Detection of pulmonary ground-glass opacity based on deep learning computer artificial intelligence. Biomed Eng Online 18:6. https://doi.org/10.1186/s12938-019-0627-4

    Article  Google Scholar 

  17. Maizlin ZV, Vos PM, Godoy MB, Cooperberg PL (2007) Computer-aided detection of pulmonary embolism on CT angiography. J Thorac Imaging 22:324–329. https://doi.org/10.1097/RTI.0b013e31815b89ca

  18. Engelke C, Schmidt S, Bakai A, Auer F, Marten K (2008) Computer-assisted detection of pulmonary embolism: performance evaluation in consensus with experienced and inexperienced chest radiologists. Eur Radiol 18:298–307. https://doi.org/10.1007/s00330-007-0770-3

  19. Bouma H, Sonnemans JJ, Vilanova A, Gerritsen FA (2009) Automatic detection of pulmonary embolism in CTA images. IEEE Trans Med Imaging 28:1223–1230. https://doi.org/10.1109/TMI.2009.2013618

    Article  Google Scholar 

  20. Lee CW, Seo JB, Song J-W et al (2011) Evaluation of computer-aided detection and dual energy software in detection of peripheral pulmonary embolism on dual-energy pulmonary CT angiography. Eur Radiol 21:54–62. https://doi.org/10.1007/s00330-010-1903-7

    Article  Google Scholar 

  21. Kligerman SJ, Lahiji K, Galvin JR, Stokum C, White CS (2014) Missed pulmonary emboli on CT angiography: assessment with pulmonary embolism–computer-aided detection. Am J Roentgenol 202:65–73. https://doi.org/10.2214/AJR.13.11049

  22. Masutani Y, MacMahon H, Doi K (2002) Computerized detection of pulmonary embolism in spiral CT angiography based on volumetric image analysis. IEEE Trans Med Imaging 21:1517–1523. https://doi.org/10.1109/TMI.2002.806586

    Article  Google Scholar 

  23. Pichon E, Novak CL, Kiraly AP, Naidich DP (2004) A novel method for pulmonary emboli visualization from high-resolution CT images. Proceedings of the SPIE, Volume 5367, p 161-170 (2004). p 161

  24. Liang J, Bi J (2007) Computer aided detection of pulmonary embolism with tobogganing and mutiple instance classification in CT pulmonary angiography. Inf Process Med Imaging 20:630–641

    Google Scholar 

  25. Zhou C, Chan H-P, Sahiner B et al (2009) Computer-aided detection of pulmonary embolism in computed tomographic pulmonary angiography (CTPA): performance evaluation with independent data sets. Med Phys 36:3385–3396. https://doi.org/10.1118/1.3157102

    Article  Google Scholar 

  26. Özkan H, Osman O, Şahin S, Boz AF (2014) A novel method for pulmonary embolism detection in CTA images. Comput Methods Programs Biomed 113:757–766. https://doi.org/10.1016/j.cmpb.2013.12.014

    Article  Google Scholar 

  27. Digumarthy S, Kagay C, Legasto A, Muse V, Wittram C, Shepard J (2006) Computer-aided detection (CAD) of acute pulmonary emboli: evaluation in patients without significant pulmonary disease. Radiological Society of North America 2006 Scientific Assembly and Annual Meeting, November 26 - December 1, 2006, Chicago IL

  28. Schoepf UJ, Schneider AC, Das M, Wood SA, Cheema JI, Philip Costello P (2007) Pulmonary embolism: computer-aided detection at multidetector row spiral computed tomography. J Thorac Imaging 22:319–323. https://doi.org/10.1097/RTI.0b013e31815842a9

  29. Wittenberg R, Peters JF, Sonnemans JJ, Prokop M, Schaefer-Prokop CM (2010) Computer-assisted detection of pulmonary embolism: evaluation of pulmonary CT angiograms performed in an on-call setting. Eur Radiol 20:801–806. https://doi.org/10.1007/s00330-009-1628-7

  30. Blackmon KN, Florin C, Bogoni L et al (2011) Computer-aided detection of pulmonary embolism at CT pulmonary angiography: can it improve performance of inexperienced readers? Eur Radiol 21:1214–1223. https://doi.org/10.1007/s00330-010-2050-x

    Article  Google Scholar 

  31. Wittenberg R, Berger FH, Peters JF et al (2012) Acute pulmonary embolism: effect of a computer-assisted detection prototype on diagnosis—an observer study. Radiology 262:305–313. https://doi.org/10.1148/radiol.11110372

    Article  Google Scholar 

  32. Sang Cheol Park SC, Chapman BE, Bin Zheng B (2011) A multistage approach to improve performance of computer-aided detection of pulmonary embolisms depicted on CT images: preliminary investigation. IEEE Trans Biomed Eng 58:1519–1527. https://doi.org/10.1109/TBME.2010.2063702

    Article  Google Scholar 

  33. Das M, Mühlenbruch G, Helm A et al (2008) Computer-aided detection of pulmonary embolism: influence on radiologists’ detection performance with respect to vessel segments. Eur Radiol 18:1350–1355. https://doi.org/10.1007/s00330-008-0889-x

    Article  Google Scholar 

  34. Lee G, Lee HY, Park H et al (2017) Radiomics and its emerging role in lung cancer research, imaging biomarkers and clinical management: state of the art. Eur J Radiol 86:297–307. https://doi.org/10.1016/J.EJRAD.2016.09.005

    Article  Google Scholar 

  35. Lahiji K, Kligerman S, Jeudy J, White C (2014) Improved accuracy of pulmonary embolism computer-aided detection using iterative reconstruction compared with filtered back projection. AJR Am J Roentgenol 203:763–771. https://doi.org/10.2214/AJR.13.11838

    Article  Google Scholar 

  36. He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. Proceedings of the IEEE Conference on Cumputer Vision and Pattern Recognition: 770-778. https://doi.org/10.1109/CVPR.2016.90

  37. Wilson EB (1927) Probable inference, the law of succession, and statistical inference. J Am Stat Assoc 22:209. https://doi.org/10.2307/2276774

    Article  Google Scholar 

  38. Wittenberg R, Peters JF, van den Berk IAH et al (2013) Computed tomography pulmonary angiography in acute pulmonary embolism. J Thorac Imaging 28:315–321. https://doi.org/10.1097/RTI.0b013e3182870b97

    Article  Google Scholar 

  39. Bhargavan M, Kaye AH, Forman HP, Sunshine JH (2009) Workload of radiologists in United States in 2006-2007 and trends since 1991-1992. Radiology 252:458–467. https://doi.org/10.1148/radiol.2522081895

    Article  Google Scholar 

  40. Tajbakhsh N, Gotway MB, Liang J (2015) Computer-aided pulmonary embolism detection using a novel vessel-aligned multiplanar image representation and convolutional neural networks. In: Navab N, Hornegger J, Wells W, Frangi A (eds) Medical image computing and computer-assisted intervention --MICCAI 2015. MICCAI 2015. Lecture notes in computer science, vol 9350. Springer, Cham

  41. Tajbakhsh N, Shin JY, Gotway MB, Liang J (2019) Computer-aided detection and visualization of pulmonary embolism using a novel, compact, and discriminative image representation. Med Image Anal 58:101541. https://doi.org/10.1016/J.MEDIA.2019.101541

    Article  Google Scholar 

  42. Beydilli İ, Yılmaz F, Sönmez BM et al (2016) Thrombolytic therapy delay is independent predictor of mortality in acute pulmonary embolism at emergency service. Kaohsiung J Med Sci 32:572–578. https://doi.org/10.1016/j.kjms.2016.09.004

    Article  Google Scholar 

  43. Annarumma M, Withey SJ, Bakewell RJ, Pesce E, Goh V, Montana G (2019) Automated triaging of adult chest radiographs with deep artificial neural networks. Radiology:291

  44. Rayan JC, Reddy N, Kan JH, Zhang W, Annapragada A (2019) Binomial classification of pediatric elbow fractures using a deep learning multiview approach emulating radiologist decision making. Radiol Artif Intell 1:e180015. https://doi.org/10.1148/ryai.2019180015

  45. Penaloza A, Kline J, Verschuren F et al (2012) European and American suspected and confirmed pulmonary embolism populations: comparison and analysis. J Thromb Haemost 10:375–381. https://doi.org/10.1111/j.1538-7836.2012.04631.x

    Article  CAS  Google Scholar 

Download references

Funding

The authors state that this work has not received any funding.

Author information

Authors and Affiliations

  1. Clinic of Radiology & Nuclear Medicine, University Hospital Basel, University of Basel, Petersgraben 4, 4031, Basel, Switzerland

    Thomas Weikert, David J. Winkel, Jens Bremerich, Bram Stieltjes, Victor Parmar, Alexander W. Sauter & Gregor Sommer

Authors
  1. Thomas Weikert
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David J. Winkel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jens Bremerich
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bram Stieltjes
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Victor Parmar
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Alexander W. Sauter
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Gregor Sommer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thomas Weikert.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Gregor Sommer, MD, PhD (University Hospital Basel, Switzerland).

Conflict of interest

The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.

Statistics and biometry

No complex statistical methods were necessary for this paper.

Informed consent

Written informed consent was waived by the Institutional Review Board (project number 2019-01050). We added the document of the IRB decision to this submission.

Ethical approval

Institutional Review Board approval was obtained.

Methodology

• retrospective

• diagnostic study

• performed at one institution

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Weikert, T., Winkel, D.J., Bremerich, J. et al. Automated detection of pulmonary embolism in CT pulmonary angiograms using an AI-powered algorithm. Eur Radiol 30, 6545–6553 (2020). https://doi.org/10.1007/s00330-020-06998-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-020-06998-0

Keywords

Search

Navigation