Abstract
Functional magnetic resonance imaging (fMRI) can be used to map patterns of brain activity and understand how information is expressed in the human brain. Using fMRI data to analyses the relationship between visual cortex and language semantic representation is of significance for building a new deep learning model. The cognition of visual semantic concept refers to the behavior that people can distinguish and classify the semantic concept of visual information they see. Many previous research literatures have revealed semantically active brain regions, but lack of modeling the relationship between visual information and language semantic concept in human brain, which is very important to understand the brain mechanism of concept learning. In this paper, we propose a Semantic Concept Cognitive Network (S-ConceptNet) model of brain cortical signals based on fMRI, The model organizes visual and linguistic semantic information into a unified representation framework, which can effectively analyses the generation process of semantic concept, and realize the function of semantic concept cognition. Based on S-ConceptNet, we also use the Dual-learning model to reconstruct the brain signal, judge the corresponding concept category through the S-ConceptNet, compare the reconstructed image with the image of this category. And finally output semantic information corresponding to the brain signal through similarity. We verify the effect of the model and conduct comparative experiments, and the experimental results are better than previous work, and prove the effectiveness of the model proposed in this paper.
This work was supported by the National Natural Science Foundation of China (NO. 62088102) and China National Postdoctoral Program for Innovative Talents from China Postdoctoral Science Foundation (NO. BX2021239).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Gazzaniga, M.S., Ivry, R.B., Mangun, G.: Cognitive Neuroscience; The Biology of the Mind, 3rd edn. Norton, New York (2014)
Robbins, S.: A misdirected principle with a catch: explicability for AI. Mind. Mach. 29(4), 495–514 (2019)
Zheng, N.-N., et al.: Hybrid-augmented intelligence: collaboration and cognition. Front. Inf. Technol. Electr. Eng. 18(2), 153–179 (2017). https://doi.org/10.1631/FITEE.1700053
Logothetis, N.K., Pauls, J., Augath, M., et al.: Neurophysiological investigation of the basis of the fMRI signal. Nature 412(6843), 150–157 (2001)
Fujiwara, Y., Miyawaki, Y., Kamitani, Y.: Modular encoding and decoding models derived from Bayesian canonical correlation analysis. Neural Comput. 25(4), 979–1005 (2013)
Miyawaki, Y., Uchida, H., Yamashita, O., et al.: Visual image reconstruction from human brain activity: a modular decoding approach. J. Phys. Conf. Ser. 197(1), 012021 (2009)
Shen, G., Dwivedi, K., Majima, K., et al.: End-to-end deep image reconstruction from human brain activity. Front. Comput. Neurosci. 13, 21 (2019)
Horikawa, T., Kamitani, Y.: Generic decoding of seen and imagined objects using hierarchical visual features. Nat. Commun. 8(1), 1–15 (2017)
Rakhimberdina, Z., Jodelet, Q., Liu, X., et al.: Natural image reconstruction from fMRI using deep learning: a survey. Front. Neurosci. 1730 (2021)
Mao, Z., Su, Y., Xu, G., et al.: Spatio-temporal deep learning method for ADHD fMRI classification. Inf. Sci. 499, 1–11 (2019)
Qureshi, M.N.I., Oh, J., Lee, B.: 3d-cnn based discrimination of schizophrenia using resting-state fMRI. Artif. Intell. Med. 98, 10–17 (2019)
Lin, Y., Li, J., Wang, H.: DCNN-GAN: reconstructing realistic image from fMRI. In: 16th International Conference on Machine Vision Applications (MVA), pp. 1–6. IEEE (2019)
Noonan, K A., Jefferies, E., Visser, M., et al.: Going beyond inferior prefrontal involvement in semantic control: evidence for the additional contribution of dorsal angular gyrus and posterior middle temporal cortex. J. Cogn. Neurosci. 25(11), 1824–1850 (2013)
Binder, J.R., Desai, R.H., Graves, W.W., et al.: Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. Cereb. Cortex 19(12), 2767–2796 (2009)
Jefferies, E.: The neural basis of semantic cognition: converging evidence from neuropsychology, neuroimaging and TMS. Cortex 49(3), 611–625 (2013)
Grodniewicz, J.P.: The process of linguistic understanding. Synthese 198(12), 11463–11481 (2020). https://doi.org/10.1007/s11229-020-02807-9
Zhang, Q., Wang, H., Luo, C., et al.: The neural basis of semantic cognition in Mandarin Chinese: a combined fMRI and TMS study. Hum. Brain Mapp. 40, 5412–5423 (2019)
Behrens, T.E.J., et al.: Non-invasive mapping of connections between human thalamus and cortex using diffusion imaging. Nat. Neurosci. 6(7), 750–757 (2003)
Du, C., Li, J., Huang, L., et al.: Brain encoding and decoding in fMRI with bidirectional deep generative models. Engineering 5(5), 948–953 (2019)
Wang, H., Huang, L., Du, C., et al.: Neural encoding for human visual cortex with deep neural networks learning “what” and “where”. In: IEEE Transactions on Cognitive and Developmental Systems, pp. 827–840 (2020)
Beliy, R., et al.: From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Yu, S., Zheng, N., Wu, H., et al.: Exploring brain effective connectivity in visual perception using a hierarchical correlation network. In: Proceedings of the IFIP International Conference on Artificial Intelligence Applications and Innovations 2019, pp. 223–235 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 IFIP International Federation for Information Processing
About this paper
Cite this paper
Jing, H., Du, M., Ma, Y., Zheng, N. (2022). Exploring the Relationship Between Visual Information and Language Semantic Concept in the Human Brain. In: Maglogiannis, I., Iliadis, L., Macintyre, J., Cortez, P. (eds) Artificial Intelligence Applications and Innovations. AIAI 2022. IFIP Advances in Information and Communication Technology, vol 646. Springer, Cham. https://doi.org/10.1007/978-3-031-08333-4_32
Download citation
DOI: https://doi.org/10.1007/978-3-031-08333-4_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-08332-7
Online ISBN: 978-3-031-08333-4
eBook Packages: Computer ScienceComputer Science (R0)
