Abstract
Human brain is supposed to process information in multiple frequency bands. Therefore, we can extract diverse information from functional Magnetic Resonance Imaging (fMRI) data by processing it at multiple resolutions. We propose a framework, called Hierarchical Multi-resolution Mesh Networks (HMMNs), which establishes a set of brain networks at multiple resolutions of fMRI signal to represent the underlying cognitive process. Our framework, first, decomposes the fMRI signal into various frequency subbands using wavelet transform. Then, a brain network is formed at each subband by ensembling a set of local meshes. Arc weights of each local mesh are estimated by ridge regression. Finally, adjacency matrices of mesh networks obtained at different subbands are used to train classifiers in an ensemble learning architecture, called fuzzy stacked generalization (FSG). Our decoding performances on Human Connectome Project task-fMRI dataset reflect that HMMNs can successfully discriminate tasks with 99% accuracy, across 808 subjects. Diversity of information embedded in mesh networks of multiple subbands enables the ensemble of classifiers to collaborate with each other for brain decoding. The suggested HMMNs decode the cognitive tasks better than a single classifier applied to any subband. Also mesh networks have a better representation power compared to pairwise correlations or average voxel time series. Moreover, fusion of diverse information using FSG outperforms fusion with majority voting. We conclude that, fMRI data, recorded during a cognitive task, provide diverse information in multi-resolution mesh networks. Our framework fuses this complementary information and boosts the brain decoding performances obtained at individual subbands.
Similar content being viewed by others
References
Alkan, S., & Yarman-Vural, F.T. (2015). Ensembling brain regions for brain decoding. In 2015 37th annual international conference of the IEEE engineering in medicine and biology society (EMBC) (pp. 2948–2951).
Barch, D.M., Burgess, G.C., Harms, M.P., Petersen, S.E., Schlaggar, B.L., Corbetta, M., Glasser, M.F., Curtiss, S., Dixit, S., Feldt, C., Nolan, D., Bryant, E., Hartley, T., Footer, O., Bjork, J.M., Poldrack, R., Smith, S., Johansen-Berg, H., Snyder, A.Z., & Essen, D.C.V. (2013). Function in the human connectome: task-fmri and individual differences in behavior. NeuroImage, 80, 169–189.
Behroozi, M., & Daliri, M.R. (2014). Predicting brain states associated with object categories from fmri data. Journal of Integrative Neuroscience, 13(04), 645–667.
Behroozi, M., & Daliri, M.R. (2015). Rdlpfc area of the brain encodes sentence polarity: a study using fmri. Brain Imaging and Behavior, 9(2), 178–189.
Behroozi, M., Daliri, M.R., & Boyaci, H. (2011). Statistical analysis methods for the fmri data. Basic and Clinical Neuroscience, 2(4), 67–74.
Binder, J.R., Gross, W.L., Allendorfer, J.B., Bonilha, L., Chapin, J., Edwards, J.C., Grabowski, T.J., Langfitt, J.T., Loring, D.W., Lowe, M.J., Koenig, K., Morgan, P.S., Ojemann, J.G., Rorden, C., Szaflarski, J.P., Tivarus, M.E., & Weaver, K.E. (2011). Mapping anterior temporal lobe language areas with fmri: a multicenter normative study. NeuroImage, 54(2), 1465–1475.
Buckner, R.L., Krienen, F.M., Castellanos, A., Diaz, J.C., & Yeo, B.T.T. (2011). The organization of the human cerebellum estimated by intrinsic functional connectivity. Journal of Neurophysiology, 106(5), 2322–2345.
Bullmore, E., Fadili, J., Maxim, V., Şendur, L., Whitcher, B., Suckling, J., Brammer, M., & Breakspear, M. (2004). Wavelets and functional magnetic resonance imaging of the human brain. NeuroImage, 23(Supplement 1), S234–S249.
Cabral, C., Silveira, M., & Figueiredo, P. (2012). Decoding visual brain states from fmri using an ensemble of classifiers. Pattern Recognition, 45(6), 2064–2074.
Castelli, F., Happé, F., Frith, U., & Frith, C. (2000). Movement and mind: a functional imaging study of perception and interpretation of complex intentional movement patterns. NeuroImage, 12(3), 314–325.
Chen, M., Han, J., Hu, X., Jiang, X., Guo, L., & Liu, T. (2014). Survey of encoding and decoding of visual stimulus via fmri: an image analysis perspective. Brain Imaging and Behavior, 8(1), 7–23.
Cox, D.D., & Savoy, R.L. (2003). Functional magnetic resonance imaging (fmri) “brain reading”: detecting and classifying distributed patterns of fmri activity in human visual cortex. NeuroImage, 19(2), 261–270.
Daliri, M.R. (2012). Predicting the cognitive states of the subjects in functional magnetic resonance imaging signals using the combination of feature selection strategies. Brain Topography, 25(2), 129–135.
Delgado, M.R., Nystrom, L.E., Fissell, C., Noll, D.C., & Fiez, J.A. (2000). Tracking the hemodynamic responses to reward and punishment in the striatum. Journal of Neurophysiology, 84(6), 3072–3077.
Dinov, I.D., Boscardin, J.W., Mega, M.S., Sowell, E.L., & Toga, A.W. (2005). A wavelet-based statistical analysis of fmri data. Neuroinformatics, 3(4), 319–342.
Ekman, M., Derrfuss, J., Tittgemeyer, M., & Fiebach, C.J. (2012). Predicting errors from reconfiguration patterns in human brain networks. Proceedings of the National Academy of Sciences, 109(41), 16,714–16,719.
Fan, Y. (2003). On the approximate decorrelation property of the discrete wavelet transform for fractionally differenced processes. IEEE Transactions on Information Theory, 49(2), 516–521.
Fang, J., Hu, X., Han, J., Jiang, X., Zhu, D., Guo, L., & Liu, T. (2015). Data-driven analysis of functional brain interactions during free listening to music and speech. Brain Imaging and Behavior, 9(2), 162–177.
Fornito, A., Harrison, B.J., Zalesky, A., & Simons, J.S. (2012). Competitive and cooperative dynamics of large-scale brain functional networks supporting recollection. Proceedings of the National Academy of Sciences, 109 (31), 12,788–12,793.
Glasser, M.F., Sotiropoulos, S.N., Wilson, J.A., Coalson, T.S., Fischl, B., Andersson, J.L., Xu, J., Jbabdi, S., Webster, M., Polimeni, J.R., et al. (2013). The minimal preprocessing pipelines for the human connectome project. NeuroImage, 80, 105–124.
Hariri, A.R., Tessitore, A., Mattay, V.S., Fera, F., & Weinberger, D.R. (2002). The amygdala response to emotional stimuli: a comparison of faces and scenes. NeuroImage, 17(1), 317–323.
Kamitani, Y., & Tong, F. (2005). Decoding the visual and subjective contents of the human brain. Nature Neuroscience, 8(5), 679–685.
Kauppi, J.P., Jääskeläinen, I.P., Sams, M., & Tohka, J. (2010). Inter-subject correlation of brain hemodynamic responses during watching a movie: localization in space and frequency. Frontiers in Neuroinformatics, 4(5).
Kauppi, J.P., Pajula, J., & Tohka, J. (2014). A versatile software package for inter-subject correlation based analyses of fmri. Frontiers in Neuroinformatics, 8(2).
Kuncheva, L.I. (2004). Combining pattern classifiers: methods and algorithms. Wiley.
Kuncheva, L.I., Rodríguez, J J, Plumpton, C.O., Linden, D.E.J., & Johnston, S.J. (2010). Random subspace ensembles for fmri classification. IEEE Transactions on Medical Imaging, 29(2), 531–542.
Lindquist, M.A. (2008). The statistical analysis of fmri data. Statistical Science, 23(4), 439–464.
Mandelbrot, B.B. (1977). The fractal geometry of nature. New York: Springer.
Onal, I., Ozay, M., & Yarman Vural, F. (2015a). Modeling voxel connectivity for brain decoding. In PRNI, Stanford, CA, USA (pp. 5–8).
Onal, I., Ozay, M., & Yarman Vural, F.T. (2015b). Functional mesh model with temporal measurements for brain decoding. In Engineering in medicine and biology society (EMBC), 2015 37th annual international conference of the IEEE (pp. 2624–2628): IEEE.
Ozay, M., & Yarman-Vural, F.T. (2016). Hierarchical distance learning by stacking nearest neighbor classifiers. Information Fusion, 29, 14–31.
Richiardi, J., Eryilmaz, H., Schwartz, S., Vuilleumier, P., & Ville, D.V.D. (2011). Decoding brain states from fmri connectivity graphs. NeuroImage, 56(2), 616–626. Multivariate Decoding and Brain Reading.
Richiardi, J., Achard, S., Bunke, H., & De Ville, D.V. (2013). Machine learning with brain graphs: predictive modeling approaches for functional imaging in systems neuroscience. IEEE Signal Processing Magazine.
Shirer, W.R., Ryali, S., Rykhlevskaia, E., Menon, V., & Greicius, M.D. (2011). Decoding subject-driven cognitive states with whole-brain connectivity patterns. Cerebral Cortex, 22(1), 158–165.
Smith, R., Keramatian, K., & Christoff, K. (2007). Localizing the rostrolateral prefrontal cortex at the individual level. NeuroImage, 36(4), 1387–1396.
Thompson, W.H., & Fransson, P. (2015). The frequency dimension of fmri dynamic connectivity: network connectivity, functional hubs and integration in the resting brain. NeuroImage, 121, 227–242.
Van De Ville, D., Blu, T., & Unser, M. (2006). Surfing the brain—an overview of wavelet-based techniques for fMRI data analysis. IEEE Engineering in Medicine and Biology Magazine, 25(2), 65–78.
Wheatley, T., Milleville, S.C., & Martin, A. (2007). Understanding animate agents: distinct roles for the social network and mirror system. Psychological Science, 18(6), 469–474.
Xu, Z., & Chan, A.K. (2002). Encoding with frames in mri and analysis of the signal-to-noise ratio. IEEE Transactions on Medical Imaging, 21(4), 332–342.
Funding
Itir Onal Ertugrul is funded by TUBITAK 2211E. This work was supported by CREST, JST, and TUBITAK Project No 116E091.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interests
Itir Onal Ertugrul, Mete Ozay and Fatos Yarman Vural declare that they have no conflicts of interest.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors. Data used in this study were previously collected and archived in a data bank.
Additional information
This work was supported by CREST, JST, Grant Number JPMJCR14D1, the ImPACT Program of the Council for Science, Technology, and Innovation (Cabinet Office, Government of Japan) and TUBITAK Project No 116E091. Itir Onal Ertugrul was supported by TUBITAK 2211E.
Rights and permissions
About this article
Cite this article
Onal Ertugrul, I., Ozay, M. & Yarman Vural, F.T. Hierarchical multi-resolution mesh networks for brain decoding. Brain Imaging and Behavior 12, 1067–1083 (2018). https://doi.org/10.1007/s11682-017-9774-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11682-017-9774-z