Abstract
How to reconstruct the stimulus images from fMRI signals is an important problem in the field of neuroscience. Limited by the complexity and the acquisition accuracy of brain signals, it is still very difficult to completely reconstruct the realistic images from fMRI signals by the artificial intelligence related technology. In the experiments related to brain signals stimulated by real images, we found that: as subjects, when stimulated by continuous realistic images, they are more sensitive to the high-frequency information in the stimulus images, such as image contour, color upheaval area, etc., The human vision and cerebral cortex seem to respond more strongly to these. Based on this discovery, we propose a method that pays more attention to the image structure to reconstruct images from fMRI signals. In order to fully decode the voxels in fMRI signals and solve the problem of insufficient amount of fMRI data, we use a back-to-back model based on Variational Auto-Encoder, which can decode more voxels in fMRI recordings into meaningful image features and introduce more unlabeled data to improve the overall performance of the model. Experiments demonstrate that our method performers better than other mainstream methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Engel, S.A., Rumelhart, D.E., et al.: fmri of human visual cortex. Nature 369, 525 (1994)
Yoshida, T., et al.: Natural images are reliably represented by sparse and variable populations of neurons in visual cortex. Nat. Commun. 11(1), 1–19 (2020)
Miyawaki, Y., et al.: Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron 60(5), 915–929 (2008)
Zheng, N.n., Liu, Z.y, et al.: Hybrid-augmented intelligence: collaboration and cognition. Front. Inf. Technol. Elect. Eng. 18(2), 153–179 (2017)
Horikawa, T., Kamitani, Y.: Generic decoding of seen and imagined objects using hierarchical visual features. Nat. Commun. 8(1), 1–15 (2017)
Ren, Z., et al.: Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning. Neuroimage 228, 117602 (2021)
Naselaris, T., Prenger, R.J., et al.: Bayesian reconstruction of natural images from human brain activity. Neuron 63(6), 902–915 (2009)
VanRullen, R., Reddy, L.: Reconstructing faces from fmri patterns using deep generative neural networks. Commun. Bio. 2(1), 1–10 (2019)
Shen, G., Dwivedi, K., et al.: End-to-end deep image reconstruction from human brain activity. Front. Comput. Neurosci. 13, 21 (2019)
Shen, G., Horikawa, T., et al.: Deep image reconstruction from human brain activity. PLoS Comput. Biol. 15(1), e1006633 (2019)
Fang, T., Qi, Y., Pan, G.: Reconstructing perceptive images from brain activity by shape-semantic gan. Adv. Neural. Inf. Process. Syst. 33, 13038–13048 (2020)
St-Yves, G., Naselaris, T.: Generative adversarial networks conditioned on brain activity reconstruct seen images. In: 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1054–1061. IEEE(2018)
Seeliger, K., Güçlü, U., et al.: Generative adversarial networks for reconstructing natural images from brain activity. Neuroimage 181, 775–785 (2018)
Yu, S., Zheng, N., et al.: A novel brain decoding method: a correlation network framework for revealing brain connections. IEEE Trans. Cogn. Dev. Syst. 11(1), 95–106 (2018)
Beliy, R., Gaziv, G., et al.: From voxels to pixels and back: self-supervision in natural-image reconstruction from fmri. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Nonaka, S., Majima, K., et al.: Brain hierarchy score: which deep neural networks are hierarchically brain-like? Iscience 24(9), 103013 (2021)
Hammill, H.B., Holladay, T.M.: The effects of certain approximations in image quality evaluation from edge traces. Opt. Eng. 8(6), 223–228 (1970)
Ni, Z., Ma, L., et al.: Esim: Edge similarity for screen content image quality assessment. IEEE Trans. Image Process. 26(10), 4818–4831 (2017)
Birman, D., Gardner, J.L.: A flexible readout mechanism of human sensory representations. Nat. Commun. 10(1), 1–13 (2019)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
Chen, R.T., Li, X., et al.: Isolating sources of disentanglement in variational autoencoders. In: Advances in Neural Information Processing Systems, vol. 31 (2018)
Wang, Z., Bovik, A.C., et al.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Zhao, H., Gallo, O., et al.: Loss functions for image restoration with neural networks. IEEE Trans. Comput. Imaging 3(1), 47–57 (2016)
Deng, J., et al.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)
Bjontegaard, G.: Calculation of average psnr differences between rd-curves.VCEG-M33 (2001)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhao, Z., Jing, H., Wang, J., Wu, W., Ma, Y. (2022). Images Structure Reconstruction from fMRI by Unsupervised Learning Based on VAE. In: Pimenidis, E., Angelov, P., Jayne, C., Papaleonidas, A., Aydin, M. (eds) Artificial Neural Networks and Machine Learning – ICANN 2022. ICANN 2022. Lecture Notes in Computer Science, vol 13531. Springer, Cham. https://doi.org/10.1007/978-3-031-15934-3_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-15934-3_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-15933-6
Online ISBN: 978-3-031-15934-3
eBook Packages: Computer ScienceComputer Science (R0)