Abstract
Multivariate pattern analysis (MVPA) methods have become an important tool in neuroimaging, revealing complex associations and yielding powerful prediction models. Despite methodological developments and novel application domains, there has been little effort to compile benchmark results that researchers can reference and compare against. This study takes a significant step in this direction. We employed three classes of state-of-the-art MVPA algorithms and common types of structural measurements from brain Magnetic Resonance Imaging (MRI) scans to predict an array of clinically relevant variables (diagnosis of Alzheimer’s, schizophrenia, autism, and attention deficit and hyperactivity disorder; age, cerebrospinal fluid derived amyloid-β levels and mini-mental state exam score). We analyzed data from over 2,800 subjects, compiled from six publicly available datasets. The employed data and computational tools are freely distributed (https://www.nmr.mgh.harvard.edu/lab/mripredict), making this the largest, most comprehensive, reproducible benchmark image-based prediction experiment to date in structural neuroimaging. Finally, we make several observations regarding the factors that influence prediction performance and point to future research directions. Unsurprisingly, our results suggest that the biological footprint (effect size) has a dramatic influence on prediction performance. Though the choice of image measurement and MVPA algorithm can impact the result, there was no universally optimal selection. Intriguingly, the choice of algorithm seemed to be less critical than the choice of measurement type. Finally, our results showed that cross-validation estimates of performance, while generally optimistic, correlate well with generalization accuracy on a new dataset.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
http://fcon_1000.projects.nitrc.org/indi/adhd200/general/ADHD-200_PhenotypicKey.pdf.
References
Ashburner, J., & Friston, K. J. (2000). VVoxel-based morphometry: the methods. NeuroImage, 11, 805–821.
Batmanghelich, N., Taskar, B., Davatzikos, C. (2009). A general and unifying framework for feature construction, in image-based pattern classification. Information Processing in Medical Imaging. Springer, pp. 423–434.
Benjamini, Y., Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 289–300.
Brown, M.R., Sidhu, G.S., Greiner, R., Asgarian, N., Bastani, M., Silverstone, P.H., Greenshaw, A.J., Dursun, S.M. (2012). ADHD-200 Global Competition: diagnosing ADHD using personal characteristic data can outperform resting state fMRI measurements. Frontiers in systems neuroscience 6.
Chang, C.-C., & Lin, C.-J. (2011). LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2, 27.
Cho, Y., Seong, J.-K., Jeong, Y., & Shin, S. Y. (2012). Individual subject classification for Alzheimer’s disease based on incremental learning using a spatial frequency representation of cortical thickness data. NeuroImage, 59, 2217–2230.
Chu, C., Hsu, A.-L., Chou, K.-H., Bandettini, P., & Lin, C. (2012). Does feature selection improve classification accuracy? Impact of sample size and feature selection on classification using anatomical magnetic resonance images. NeuroImage, 60, 59–70.
Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20, 273–297.
Costafreda, S. G., Chu, C., Ashburner, J., & Fu, C. H. (2009). Prognostic and diagnostic potential of the structural neuroanatomy of depression. PloS One, 4, e6353.
Criminisi, A., Shotton, J., Konukoglu, E., (2011). Decision forests for classification, regression, density estimation, manifold learning and semi-supervised learning. Microsoft Research Cambridge, Tech. Rep. MSRTR-2011-114 5, 12.
Cuingnet, R., Gerardin, E., Tessieras, J., Auzias, G., Lehericy, S., Habert, M.-O., Chupin, M., Benali, H., & Colliot, O. (2011). Automatic classification of patients with Alzheimer’s disease from structural MRI: a comparison of ten methods using the ADNI database. NeuroImage, 56, 766–781.
Dale, A. M., Fischl, B., & Sereno, M. I. (1999). Cortical surface-based analysis: I. Segmentation and surface reconstruction. NeuroImage, 9, 179–194.
Davatzikos, C., Resnick, S. M., Wu, X., Parmpi, P., & Clark, C. (2008). Individual patient diagnosis of AD and FTD via high-dimensional pattern classification of MRI. NeuroImage, 41, 1220–1227.
Davatzikos, C., Xu, F., An, Y., Fan, Y., & Resnick, S. M. (2009). Longitudinal progression of Alzheimer’s-like patterns of atrophy in normal older adults: the SPARE-AD index. Brain, 132, 2026–2035.
DeLong, E.R., DeLong, D.M., Clarke-Pearson, D.L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 837–845.
Duchesnay, E., Cachia, A., Roche, A., Rivière, D., Cointepas, Y., Papadopoulos-Orfanos, D., Zilbovicius, M., Martinot, J.-L., & Mangin, J. F. (2007). Classification based on cortical folding patterns. Medical Imaging, IEEE Transactions, 26(4), 553–565.
Duchesne, S., Rolland, Y., & Verin, M. (2009). Automated computer differential classification in Parkinsonian syndromes via pattern analysis on MRI. Academic Radiology, 16, 61–70.
Ecker, C., Rocha-Rego, V., Johnston, P., Mourao-Miranda, J., Marquand, A., Daly, E. M., Brammer, M. J., Murphy, C., & Murphy, D. G. (2010). Investigating the predictive value of whole-brain structural MR scans in autism: a pattern classification approach. NeuroImage, 49, 44–56.
Fan, Y., Shen, D., Gur, R. C., Gur, R. E., & Davatzikos, C. (2007). COMPARE: classification of morphological patterns using adaptive regional elements. Medical Imaging, IEEE Transactions, 26, 93–105.
Feinstein, A., Roy, P., Lobaugh, N., Feinstein, K., O’Connor, P., & Black, S. (2004). Structural brain abnormalities in multiple sclerosis patients with major depression. Neurology, 62(4), 586–590.
Fischl, B. (2012). Free surfer. NeuroImage, 62, 774–781.
Fischl, B., Sereno, M. I., & Dale, A. M. (1999a). Cortical surface-based analysis: II: Inflation, flattening, and a surface-based coordinate system. NeuroImage, 9, 195–207.
Fischl, B., Sereno, M. I., Tootell, R. B., & Dale, A. M. (1999b). High-resolution intersubject averaging and a coordinate system for the cortical surface. Human Brain Mapping, 8, 272–284.
Fischl, B., Salat, D. H., Busa, E., Albert, M., Dieterich, M., Haselgrove, C., van der Kouwe, A., Killiany, R., Kennedy, D., & Klaveness, S. (2002). Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron, 33, 341–355.
Fischl, B., van Der Kouwe, A., Destrieux, C., Halgren, E., Ségonne, F., Salat, D. H., Busa, E., Seidman, L. J., Goldstein, J., Kennedy, D. & Dale, A. M. (2004). Automatically parcellating the human cerebral cortex. Cerebral Cortex, 14(1), 11–22.
Frisoni, G. B., Fox, N. C., Jack, C. R., Scheltens, P., & Thompson, P. M. (2010). The clinical use of structural MRI in Alzheimer disease. Nature Reviews Neurology, 6, 67–77.
Friston, K. J., Holmes, A. P., Worsley, K. J., Poline, J. Ä., Frith, C. D., & Frackowiak, R. S. (1994). Statistical parametric maps in functional imaging: a general linear approach. Human Brain Mapping, 2, 189–210.
Friston, K., Chu, C., Mourao-Miranda, J., Hulme, O., Rees, G., Penny, W., & Ashburner, J. (2008). Bayesian decoding of brain images. NeuroImage, 39, 181–205.
Gaonkar, B., & Davatzikos, C. (2013). Analytic estimation of statistical significance maps for support vector machine based multi-variate image analysis and classification. NeuroImage, 78, 270–283.
Gollub, R.L., Shoemaker, J.M., King, M.D., White, T., Ehrlich, S., Sponheim, S.R., Clark, V.P., Turner, J.A., Mueller, B.A., Magnotta, V. (2013). The MCIC Collection: A Shared Repository of Multi-Modal, Multi-Site Brain Image Data from a Clinical Investigation of Schizophrenia. Neuroinformatics, 1–22.
Guyon, I., & Elisseeff, A. (2003). An introduction to variable and feature selection. The Journal of Machine Learning Research, 3, 1157–1182.
Han, X., Jovicich, J., Salat, D., van der Kouwe, A., Quinn, B., Czanner, S., Busa, E., Pacheco, J., Albert, M., & Killiany, R. (2006). Reliability of MRI-derived measurements of human cerebral cortical thickness: the effects of field strength, scanner upgrade and manufacturer. NeuroImage, 32, 180–194.
Ho, B.-C., Andreasen, N. C., Nopoulos, P., Arndt, S., Magnotta, V., & Flaum, M. (2003). Progressive structural brain abnormalities and their relationship to clinical outcome: a longitudinal magnetic resonance imaging study early in schizophrenia. Archives of General Psychiatry, 60, 585.
Jack, C. R., Bernstein, M. A., Fox, N. C., Thompson, P., Alexander, G., Harvey, D., Borowski, B., Britson, P. J., Whitwell, J. L., & Ward, C. (2008). The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods. Journal of Magnetic Resonance Imaging, 27, 685–691.
Jain, A., & Zongker, D. (1997). Feature selection: evaluation, application, and small sample performance. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 19, 153–158.
Japkowicz, N., Shah, M. (2011). Evaluating learning algorithms: a classification perspective. Cambridge University Press.
Kawasaki, Y., Suzuki, M., Kherif, F., Takahashi, T., Zhou, S.-Y., Nakamura, K., Matsui, M., Sumiyoshi, T., Seto, H., & Kurachi, M. (2007). Multivariate voxel-based morphometry successfully differentiates schizophrenia patients from healthy controls. NeuroImage, 34, 235–242.
Kloppel, S., Stonnington, C. M., Chu, C., Draganski, B., Scahill, R. I., Rohrer, J. D., Fox, N. C., Jack, C. R., Ashburner, J., & Frackowiak, R. S. (2008). Automatic classification of MR scans in Alzheimer’s disease. Brain, 131, 681–689.
Kloppel, S., Chu, C., Tan, G., Draganski, B., Johnson, H., Paulsen, J., Kienzle, W., Tabrizi, S., Ashburner, J., & Frackowiak, R. (2009). Automatic detection of preclinical neurodegeneration Presymptomatic Huntington disease. Neurology, 72, 426–431.
Kloppel, S., Abdulkadir, A., Jack, C. R., Jr., Koutsouleris, N., Mour√£o-Miranda, J., & Vemuri, P. (2012). Diagnostic neuroimaging across diseases. NeuroImage, 61, 457–463.
Konukoglu, E., Glocker, B., Zikic, D., & Criminisi, A., (2013). Neighbourhood Approximation using randomized forests. Medical Image Analysis 17(7), 790–804.
Koutsouleris, N., Meisenzahl, E. M., Davatzikos, C., Bottlender, R., Frodl, T., Scheuerecker, J., Schmitt, G., Zetzsche, T., Decker, P., & Reiser, M. (2009). Use of neuroanatomical pattern classification to identify subjects in at-risk mental states of psychosis and predict disease transition. Archives of General Psychiatry, 66, 700.
Kriegeskorte, N., Goebel, R., & Bandettini, P. (2006). Information-based functional brain mapping. Proceedings of the National Academy of Sciences of the United States of America, 103, 3863–3868.
Lao, Z., Shen, D., Xue, Z., Karacali, B., Resnick, S. M., & Davatzikos, C. (2004). Morphological classification of brains via high-dimensional shape transformations and machine learning methods. NeuroImage, 21, 46–57.
Lerch, J. P., Pruessner, J., Zijdenbos, A. P., Collins, D. L., Teipel, S. J., Hampel, H., & Evans, A. C. (2008). Automated cortical thickness measurements from MRI can accurately separate Alzheimer’s patients from normal elderly controls. Neurobiology of Aging, 29, 23–30.
Liu, F., Guo, W., Yu, D., Gao, Q., Gao, K., Xue, Z., Du, H., Zhang, J., Tan, C., & Liu, Z. (2012). Classification of different therapeutic responses of major depressive disorder with multivariate pattern analysis method based on structural MR scans. PloS One, 7, e40968.
Lockhart, R., Taylor, J., Tibshirani, R.J., Tibshirani, R., 2012. A significance test for the lasso.
MacKay, D. J. (1992). The evidence framework applied to classification networks. Neural Computation, 4, 720–736.
Marcus, D. S., Olsen, T. R., Ramaratnam, M., & Buckner, R. L. (2007a). The extensible neuroimaging archive toolkit. Neuroinformatics, 5, 11–33.
Marcus, D. S., Wang, T. H., Parker, J., Csernansky, J. G., Morris, J. C., & Buckner, R. L. (2007b). Open access series of imaging studies (OASIS): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults. Journal of Cognitive Neuroscience, 19, 1498–1507.
Meinshausen, N., & Buhlmann, P. (2010). Stability selection. Journal of the Royal Statistical Society, Series B: Statistical Methodology, 72, 417–473.
Milham, M. P., Fair, D., Mennes, M., & Mostofsky, S. H. (2012). The ADHD-200 consortium: a model to advance the translational potential of neuroimaging in clinical neuroscience. Frontiers in Systems Neuroscience, 6, 62.
Mitchell, T. M., Hutchinson, R., Niculescu, R. S., Pereira, F., Wang, X., Just, M., & Newman, S. (2004). Learning to decode cognitive states from brain images. Machine Learning, 57, 145–175.
Mourao-Miranda, J., Bokde, A. L., Born, C., Hampel, H., & Stetter, M. (2005). Classifying brain states and determining the discriminating activation patterns: support vector machine on functional MRI data. NeuroImage, 28, 980–995.
Mourao-Miranda, J., Reinders, A., Rocha-Rego, V., Lappin, J., Rondina, J., Morgan, C., Morgan, K., Fearon, P., Jones, P., & Doody, G. (2012). Individualized prediction of illness course at the first psychotic episode: a support vector machine MRI study. Psychological Medicine, 42, 1037.
Mwangi, B., Matthews, K., & Steele, J. D. (2012). Prediction of illness severity in patients with major depression using structural MR brain scans. Journal of Magnetic Resonance Imaging, 35, 64–71.
Nie, K., Chen, J.-H., Yu, H. J., Chu, Y., Nalcioglu, O., & Su, M.-Y. (2008). Quantitative analysis of lesion morphology and texture features for diagnostic prediction in breast MRI. Academic Radiology, 15, 1513–1525.
Nielsen, J.A., Zielinski, B.A., Fletcher, P.T., Alexander, A.L., Lange, N., Bigler, E.D., Lainhart, J.E., Anderson, J.S., 2013. Multisite functional connectivity MRI classification of autism: ABIDE results. Frontiers in human neuroscience 7.
Nieuwenhuis, M., van Haren, N. E., Hulshoff Pol, H. E., Cahn, W., Kahn, R. S., & Schnack, H. G. (2012). Classification of schizophrenia patients and healthy controls from structural MRI scans in two large independent samples. NeuroImage, 61, 606–612.
Nouretdinov, I., Costafreda, S. G., Gammerman, A., Chervonenkis, A., Vovk, V., Vapnik, V., & Fu, C. H. (2011). Machine learning classification with confidence: application of transductive conformal predictors to MRI-based diagnostic and prognostic markers in depression. NeuroImage, 56, 809–813.
Parker, B. J., Günter, S., & Bedo, J. (2007). Stratification bias in low signal microarray studies. BMC Bioinformatics, 8(1), 326.
Pereira, F., Botvinick, M., (2011). Classification of functional magnetic resonance imaging data using informative pattern features. Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pp. 940–946.
Petersen, R. C., Smith, G. E., Waring, S. C., Ivnik, R. J., Tangalos, E. G., & Kokmen, E. (1999). Mild cognitive impairment: clinical characterization and outcome. Archives of Neurology, 56, 303.
Plant, C., Teipel, S. J., Oswald, A., Böhm, C., Meindl, T., Mourao-Miranda, J., Bokde, A. W., Hampel, H., & Ewers, M. (2010). Automated detection of brain atrophy patterns based on MRI for the prediction of Alzheimer’s disease. NeuroImage, 50(1), 162–174.
Rondina, J., Hahn, T., de Oliveira, L., Marquand, A., Dresler, T., Leitner, T., Fallgatter, A., Shawe-Taylor, J., Mourao-Miranda, J. (2013). SCoRS-a method based on stability for feature selection and mapping in neuroimaging.
Sabuncu, M. R., Van Leemput, K. (2012). The Relevance Voxel Machine (RVoxM): A self-tuning bayesian model for informative image-based prediction. Medical Imaging, IEEE Transactions on Medical Imaging, 31(12), 2290–2306.
Saeys, Y., Inza, I. a., & Larra√±aga, P. (2007). A review of feature selection techniques in bioinformatics. Bioinformatics, 23, 2507–2517.
Schnack, H. G., Nieuwenhuis, M., van Haren, N. E., Abramovic, L., Scheewe, T. W., Brouwer, R. M., Hulshoff Pol, H. E., & Kahn, R. S. (2014). Can structural MRI aid in clinical classification? A machine learning study in two independent samples of patients with schizophrenia, bipolar disorder and healthy subjects. NeuroImage, 84, 299–306.
Scholkopf, B., Smola, A.J. (2002). Learning with kernels: Support vector machines, regularization, optimization, and beyond. the MIT Press.
Scott, A., Courtney, W., Wood, D., De la Garza, R., Lane, S., King, M., Wang, R., Roberts, J., Turner, J.A., Calhoun, V.D., 2011. COINS: an innovative informatics and neuroimaging tool suite built for large heterogeneous datasets. Frontiers in neuroinformatics 5.
Seeley, W. W., Crawford, R. K., Zhou, J., Miller, B. L., & Greicius, M. D. (2009). Neurodegenerative diseases target large-scale human brain networks. Neuron, 62, 42–52.
Sonnenburg, S., Zien, A., Philips, P., & Rätsch, G. (2008). POIMs: positional oligomer importance matrices—understanding support vector machine-based signal detectors. Bioinformatics, 24(13), i6–i14.
Soriano-Mas, C., Pujol, J., Alonso, P., Cardoner, N., Menchon, J. M., Harrison, B. J., Deus, J., Vallejo, J., & Gaser, C. (2007). Identifying patients with obsessive-compulsive disorder using whole-brain anatomy. NeuroImage, 35, 1028–1037.
Stonnington, C. M., Chu, C., Kl√∂ppel, S., Jack, C. R., Jr., Ashburner, J., & Frackowiak, R. S. (2010). Predicting clinical scores from magnetic resonance scans in Alzheimer’s disease. NeuroImage, 51, 1405–1413.
Strobl, C., Boulesteix, A.-L., Kneib, T., Augustin, T., & Zeileis, A. (2008). Conditional variable importance for random forests. BMC Bioinformatics, 9, 307.
Teipel, S. J., Born, C., Ewers, M., Bokde, A. L., Reiser, M. F., M√∂ller, H.-J. R., & Hampel, H. (2007). Multivariate deformation-based analysis of brain atrophy to predict Alzheimer’s disease in mild cognitive impairment. NeuroImage, 38, 13–24.
Tipping, M. E. (2001). Sparse Bayesian learning and the relevance vector machine. The Journal of Machine Learning Research, 1, 211–244.
Vemuri, P., Whitwell, J. L., Kantarci, K., Josephs, K. A., Parisi, J. E., Shiung, M. S., Knopman, D. S., Boeve, B. F., Petersen, R. C., & Dickson, D. W. (2008). Antemortem MRI based Structural abnormality iNDex (STAND)-scores correlate with postmortem braak neurofibrillary tangle stage. NeuroImage, 42, 559–567.
Wang, X., Yang, J., Jensen, R., & Liu, X. (2006). Rough set feature selection and rule induction for prediction of malignancy degree in brain glioma. Computer Methods and Programs in Biomedicine, 83, 147–156.
Wang, Y., Fan, Y., Bhatt, P., & Davatzikos, C. (2010). High-dimensional pattern regression using machine learning: from medical images to continuous clinical variables. NeuroImage, 50, 1519–1535.
Wang, H., Nie, F., Huang, H., Risacher, S., Ding, C., Saykin, A.J., Shen, L. (2011). Sparse multi-task regression and feature selection to identify brain imaging predictors for memory performance. Computer Vision (ICCV), 2011 I.E. International Conference on. IEEE, pp. 557–562.
Westman, E., Simmons, A., Zhang, Y., Muehlboeck, J., Tunnard, C., Liu, Y., Collins, L., Evans, A., Mecocci, P., & Vellas, B. (2011). Multivariate analysis of MRI data for Alzheimer’s disease, mild cognitive impairment and healthy controls. NeuroImage, 54, 1178–1187.
Wilson, S. M., Ogar, J. M., Laluz, V., Growdon, M., Jang, J., Glenn, S., Miller, B. L., Weiner, M. W., & Gorno-Tempini, M. L. (2009). Automated MRI-based classification of primary progressive aphasia variants. NeuroImage, 47, 1558–1567.
Wolfe, D.A., Hollander, M., 1973. Nonparametric statistical methods. Nonparametric statistical methods.
Zien, A., Krämer, N., Sonnenburg, S.r., Rätsch, G. (2009). The feature importance ranking measure. In Machine Learning and Knowledge Discovery in Databases (pp. 694–709). Springer Berlin Heidelberg.
Acknowledgments
This research was carried out in whole or in part at the Athinoula A. Martinos Center for Biomedical Imaging at the Massachusetts General Hospital, using resources provided by the Center for Functional Neuroimaging Technologies, P41EB015896, a P41 Biotechnology Resource Grant supported by the National Institute of Biomedical Imaging and Bioengineering (NIBIB), National Institutes of Health. Dr. Sabuncu received support from an AHAF (BrightFocus) Alzheimer’s Disease pilot grant (AHAF A2012333), and an NIH K25 grant (NIBIB 1K25EB013649-01). Further support for this research was provided in part by the National Center for Research Resources (U24 RR021382), the National Institute for Biomedical Imaging and Bioengineering (R01EB006758), National Institute for Neurological Disorders and Stroke (R01 NS052585-01, 1R21NS072652-01, 1R01NS070963, R01NS083534), and was made possible by the resources provided by Shared Instrumentation Grants 1S10RR023401, 1S10RR019307, and 1S10RR023043. Additional support was provided by the NIH Blueprint for Neuroscience Research (5U01-MH093765), part of the multi-institutional Human Connectome Project.
Author Responsibilities
Both authors are responsible for manuscript preparation and review and data analysis.
Conflicts of Interest
No conflicts of interest exist for any of the named authors in this study.
Author information
Authors and Affiliations
Consortia
Corresponding author
Additional information
Mert R. Sabuncu and Ender Konukoglu contributed equally.
Data used in the preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database. As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. A complete listing of ADNI investigators is available at http://tinyurl.com/ADNI-main.
Electronic Supplementary Material
Below is the link to the electronic supplementary material.
ESM 1
(DOC 681 kb)
Rights and permissions
About this article
Cite this article
Sabuncu, M.R., Konukoglu, E. & for the Alzheimer’s Disease Neuroimaging Initiative. Clinical Prediction from Structural Brain MRI Scans: A Large-Scale Empirical Study. Neuroinform 13, 31–46 (2015). https://doi.org/10.1007/s12021-014-9238-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12021-014-9238-1