Abstract
The explosion of medical imaging data along with the advent of big data analytics has launched an exciting era for clinical research. One factor affecting the ability to aggregate large medical image collections for research is the lack of infrastructure for automated data annotation. Among all imaging modalities, annotation of magnetic resonance (MR) images is particularly challenging due to the non-standard labeling of MR image types. In this work, we aimed to train a deep neural network to annotate MR image sequence type for scans of brain tumor patients. We focused on the four most common MR sequence types within neuroimaging: T1-weighted (T1W), T1-weighted post-gadolinium contrast (T1Gd), T2-weighted (T2W), and T2-weighted fluid-attenuated inversion recovery (FLAIR). Our repository contains images acquired using a variety of pulse sequences, sequence parameters, field strengths, and scanner manufacturers. Image selection was agnostic to patient demographics, diagnosis, and the presence of tumor in the imaging field of view. We used a total of 14,400 two-dimensional images, each visualizing a different part of the brain. Data was split into train, validation, and test sets (9600, 2400, and 2400 images, respectively) and sets consisted of equal-sized groups of image types. Overall, the model reached an accuracy of 99% on the test set. Our results showed excellent performance of deep learning techniques in predicting sequence types for brain tumor MR images. We conclude deep learning models can serve as tools to support clinical research and facilitate efficient database management.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Commission, M. P. A, Book AD, others: Healthcare spending and the Medicare program. Washington, DC: MedPAC, 2012
Smith-Bindman R, Miglioretti DL, Larson EB: Rising use of diagnostic medical imaging in a large integrated health system. Health Aff 27:1491–1502, 2008
Nitz WR: MR imaging: Acronyms and clinical applications. Eur. Radiol. 9:979–997, 1999
MRI sequences acronyms. IMAIOS Available at: https://www.imaios.com/en/e-Courses/e-MRI/MRI-Sequences/Sequences-acronyms. (Accessed: 30th April 2019)
Enlarge, C. T. T. O. & Ge, G. GRE Acronyms.
Swanson KR, Rockne RC, Claridge J, Chaplain MA, Alvord EC, Anderson ARA: Quantifying the role of angiogenesis in malignant progression of gliomas: in silico modeling integrates imaging and histology. Cancer Res. 71:7366–7375, 2011
Jackson PR, Juliano J, Hawkins-Daarud A, Rockne RC, Swanson KR: Patient-specific mathematical neuro-oncology: Using a simple proliferation and invasion tumor model to inform clinical practice. Bull. Math. Biol. 77:846–856, 2015
Hawkins-Daarud A, Rockne R, Corwin D, Anderson ARA, Kinahan P, Swanson KR: In silico analysis suggests differential response to bevacizumab and radiation combination therapy in newly diagnosed glioblastoma. J. R. Soc. Interface 12:20150388, 2015
Rockne RC. et al.: A patient-specific computational model of hypoxia-modulated radiation resistance in glioblastoma using 18F-FMISO-PET. J. R. Soc. Interface 12, 2015
Alfonso JCL, Talkenberger K, Seifert M, Klink B, Hawkins-Daarud A, Swanson KR, Hatzikirou H, Deutsch A: The biology and mathematical modelling of glioma invasion: a review. J. R. Soc. Interface 14:20170490, 2017
Rayfield CA, Grady F, de Leon G, Rockne R, Carrasco E, Jackson P, Vora M, Johnston SK, Hawkins-Daarud A, Clark-Swanson KR, Whitmire S, Gamez ME, Porter A, Hu L, Gonzalez-Cuyar L, Bendok B, Vora S, Swanson KR: Distinct phenotypic clusters of glioblastoma growth and response kinetics predict survival. JCO Clinical Cancer Informatics 2:1–14, 2018
Randall EC, Emdal KB, Laramy JK, Kim M, Roos A, Calligaris D, Regan MS, Gupta SK, Mladek AC, Carlson BL, Johnson AJ, Lu FK, Xie XS, Joughin BA, Reddy RJ, Peng S, Abdelmoula WM, Jackson PR, Kolluri A, Kellersberger KA, Agar JN, Lauffenburger DA, Swanson KR, Tran NL, Elmquist WF, White FM, Sarkaria JN, Agar NYR: Integrated mapping of pharmacokinetics and pharmacodynamics in a patient-derived xenograft model of glioblastoma. Nat. Commun. 9:4904, 2018
Swanson KR, Rostomily RC, Alvord, Jr EC: A mathematical modelling tool for predicting survival of individual patients following resection of glioblastoma: A proof of principle. Br. J. Cancer 98:113–119, 2008
Massey SC, Rockne RC, Hawkins-Daarud A, Gallaher J, Anderson ARA, Canoll P, Swanson KR: Simulating PDGF-Driven Glioma Growth and Invasion in an Anatomically Accurate Brain Domain. Bull. Math. Biol. 80:1292–1309, 2018
Johnston SK. et al.: ENvironmental Dynamics Underlying Responsive Extreme Survivors (ENDURES) of Glioblastoma: a Multi-disciplinary Team-based. Multifactorial Analytical Approach. bioRxiv 461236, 2018. doi:https://doi.org/10.1101/461236
Barnholtz-Sloan JS, Swanson KR: Sex differences in GBM revealed by analysis of patient imaging, transcriptome, and survival data. Sci. Transl. Med, 2019
Massey SC et al.: Image-based metric of invasiveness predicts response to adjuvant temozolomide for primary glioblastoma. bioRxiv 509281, 2019. doi:https://doi.org/10.1101/509281
Brown RW et al.: Magnetic Resonance Imaging: Physical Principles and Sequence Design. (John Wiley & Sons, 2014
Cha S: Neuroimaging in neuro-oncology. Neurotherapeutics 6:465–477, 2009
Armstrong TS, Cohen MZ, Weinberg J, Gilbert MR: Imaging techniques in neuro-oncology. Semin. Oncol. Nurs. 20:231–239, 2004
Krizhevsky A, Sutskever I, Hinton GE: ImageNet classification with deep convolutional neural networks. in Advances in Neural Information Processing Systems 25 (eds. Pereira, F., Burges, C. J. C., Bottou, L. & Weinberger, K. Q.) 1097–1105 (Curran Associates, Inc., 2012
Mohsen H, El-Dahshan E-SA, El-Horbaty E-SM, Salem A-BM: Classification using deep learning neural networks for brain tumors. Future Computing and Informatics Journal 3:68–71, 2018
Huynh BQ, Li H, Giger ML: Digital mammographic tumor classification using transfer learning from deep convolutional neural networks. Journal of Medical Imaging 3:034501, 2016
Chen Y-JY-J, Hua K-L, Hsu C-H, Cheng W-H, Hidayati SC: Computer-aided classification of lung nodules on computed tomography images via deep learning technique. OncoTargets and Therapy, 2015. https://doi.org/10.2147/ott.s80733
Simonyan K, Zisserman A: Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv [cs.CV], 2014
Chollet F, Others. Keras. 2015
Abadi, M. et al.: Tensorflow: A system for large-scale machine learning. in 12th ${USENIX} Symposium on Operating Systems Design and Implementation ({OSDI}$ 16) 265–283, 2016
Kingma DP, Adam BJ: A method for stochastic optimization. arXiv [cs.LG], 2014
Pedregosa F et al.: Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 12:2825–2830, 2011
Gong, Y., Jia, Y., Leung, T., Toshev, A. & Ioffe, S. Deep convolutional ranking for multilabel image annotation. arXiv [cs.CV], 2013
Wu, F., Wang Z., Zhang Z., Yang Y., Luo J., Zhu W., Zhuang Y. Weakly Semi-Supervised Deep Learning for Multi-Label Image Annotation. IEEE Transactions on Big Data 1, 109–122 (2015), 109, 122.
Wu J, Yu Y, Huang C, Yu K: Deep multiple instance learning for image classification and auto-annotation. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 3460–3469, 2015
Murthy VN, Maji S, Manmatha R: Automatic Image Annotation Using Deep Learning Representations. in Proceedings of the 5th ACM on International Conference on Multimedia Retrieval 603–606 (ACM), 2015.
Ojha U, Adhikari U, Singh DK: Image annotation using deep learning: A review. 2017 International Conference on Intelligent Computing and Control (I2C2), 2017. https://doi.org/10.1109/i2c2.2017.8321819
Hamm CA, Wang CJ, Savic LJ, Ferrante M, Schobert I, Schlachter T, Lin MD, Duncan JS, Weinreb JC, Chapiro J, Letzen B: Deep learning for liver tumor diagnosis part I: development of a convolutional neural network classifier for multi-phasic MRI. Eur. Radiol. 29:3338–3347, 2019. https://doi.org/10.1007/s00330-019-06205-9
Hermessi, H., Mourali, O. & Zagrouba, E. Deep feature learning for soft tissue sarcoma classification in MR images via transfer learning. Expert Systems with Applications 120, (2019), 116, 127.
Margeta J, Criminisi A, Cabrera Lozoya R, Lee DC, Ayache N: Fine-tuned convolutional neural nets for cardiac MRI acquisition plane recognition. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization 5:339–349, 2017
Funding
This study received support from the James S. McDonnell Foundation, the Ivy Foundation, the Mayo Clinic, the Zicarelli Foundation, and the NIH (R01 NS060752, R01 CA164371, U54 CA210180, U54 CA143970, U54 CA193489, U01 CA220378).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Statement of Impact
To the best of our knowledge, no previous work has focused on automatic annotation of MR image types. This form of classification can be highly useful for building large-scale imaging repositories that can receive submissions from heterogeneous data sources.
Rights and permissions
About this article
Cite this article
Ranjbar, S., Singleton, K.W., Jackson, P.R. et al. A Deep Convolutional Neural Network for Annotation of Magnetic Resonance Imaging Sequence Type. J Digit Imaging 33, 439–446 (2020). https://doi.org/10.1007/s10278-019-00282-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10278-019-00282-4