Skip to main content
SpringerLink
Log in

Cross-Modal Integration of Identity and Gender Information Through Faces and Voices Involves a Similar Cortical Network

  • Chapter
  • First Online:
Integrating Face and Voice in Person Perception

  • 1901 Accesses

Abstract

We investigate the cerebral cross-modal interactions between human faces and voices involved during gender and identity categorization in two separate functional magnetic resonance imaging (fMRI) studies. In each of these experiments, participants were scanned in four runs that contained three conditions consisting in the presentation of faces, voices, or congruent face–voice pairs. The task consisted in categorizing each trial (visual, auditory, or associations) according to its gender or identity. The subtraction between the bimodal condition and the sum of the unimodal ones, as well as psychophysiological interaction analyses (PPI), were performed. Main results suggest that the cross-modal auditory–visual categorization of human gender and identity is sustained by a network of highly similar cerebral regions. This network included several regions such as the unimodal visual and auditory regions processing the perceived faces and voices and inter-connected via a subcortical relay located in the striatum, the left superior parietal gyrus, part of a larger parieto-motor network dispatching the attentional resources to the visual and auditory modalities, and the right inferior frontal gyrus sustaining the integration of the semantically congruent information into a coherent multimodal representation. Therefore, we suggest that cross-modal processing of human stimuli requires the activation of a network of cortical regions, including both unimodal visual and auditory regions and supramodal parietal and frontal regions involved in the integration of both faces and voices and in the cross-modal attentional processes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Beauchamp MS (2005) Statistical criteria in fMRI studies of multisensory integration. Neuroinformatics 3:93–113

    Article  PubMed  Google Scholar 

  • Beauchemin M et al (2006) Electrophysiological markers of voice familiarity. European Journal of Neuroscience 23:3081–3086

    Article  PubMed  Google Scholar 

  • Belin P, Zatorre RJ, Lafaille P, Ahad P, Pike B (2000) Voice-selective areas in human auditory cortex. Nature 403:309–312

    Article  PubMed  CAS  Google Scholar 

  • Bernstein LE, Auer ET Jr, Wagner M, Ponton CW (2008) Spatiotemporal dynamics of audiovisual speech processing. Neuroimage 39:423–435

    Article  PubMed  Google Scholar 

  • Bodamer J (1947) Die Prosop-Agnosia (Die Agnosie des Physionomeerkennens). Archives fur Psychiatrie und Nervenkrankenheiten 179:6–33

    Article  Google Scholar 

  • Bruce V, Young A (1986) Understanding face recognition. British Journal of Psychology 77(3):305–327

    Article  PubMed  Google Scholar 

  • Burton AM, Bruce V, Johnston RA (1990) Understanding face recognition with an interactive model. British Journal of Psychology 81:361–380

    Article  PubMed  Google Scholar 

  • Bushara KO, Hanakawa T, Immish I, Toma K, Kansaku K, Hallett M (2003) Neural correlates of cross-modal binding. Nature Neuroscience 6(2):190–195

    Article  PubMed  CAS  Google Scholar 

  • Bushara KO, Weeks RA, Ishii K, Catalan MJ, Tian B, Rauschecker JP et al (1999) Modality-specific frontal and parietal areas for auditory and visual spatial localization in humans. Nature Neuroscience 2:759–766

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA (2001) Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex 11:1110–1123

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in human heteromodal cortex. Current Biology 10:649–657

    Article  PubMed  CAS  Google Scholar 

  • Campanella S, Belin P (2007) Integrating face and voice in person perception. Trends in Cognitive Sciences 11(12):535–543

    Article  PubMed  Google Scholar 

  • Campanella S, Hanoteau C, Depy D, Rossion B, Bruyer R, Crommelinck M (2000) Right N170 modulation in a face discrimination task: An account for categorical perception of familiar faces. Psychophysiology 37:796–806

    Article  PubMed  CAS  Google Scholar 

  • Campanella S, Joassin F, Rossion B, De Volder AG, Bruyer R, Crommelinck M (2001) Associations of the distinct visual representations of faces and names: A PET activation study. Neuroimage 14:873–882

    Article  PubMed  CAS  Google Scholar 

  • Driver J, Spence C (2000) Multisensory perception: Beyond modularity and convergence. Current Biology 10:731–735

    Article  Google Scholar 

  • Ganel T, Goshen-Gottstein Y (2002) Perceptual integrity of sex and identity of faces: Further evidence for the single-route hypothesis. Journal of Experimental Psychology Human Perception and Performance 28:854–867

    Article  PubMed  Google Scholar 

  • Garrido L, Eisner F, McGettigan C, Stewart L, Sauter D, Hanley JR, Schweinberger SR, Warren JD, Duchaine B (2009) Developmental phonagnosia: A selective deficit of vocal identity recognition. Neuropsychologia 47(1):123–131

    Article  PubMed  Google Scholar 

  • Gauthier I, Skudlarski P, Gore JC, Anderson AW (2000) Expertise for cars and birds recruits brain areas involved in face recognition. Nature Neuroscience 3:191–197

    Article  PubMed  CAS  Google Scholar 

  • Gonzalo D, Shallice T, Dolan R (2000) Time-dependent changes in learning audiovisual associations: A single-trial fMRI study. Neuroimage 11:243–255

    Article  PubMed  CAS  Google Scholar 

  • Haruno M, Kawato M (2006) Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus–action–reward association learning. Neural Networks 19:1242–1254

    Article  PubMed  Google Scholar 

  • Haxby JV et al (2000) The distributed human neural system for face perception. Trends in Cognitive Sciences 4:223–233

    Article  PubMed  Google Scholar 

  • Hesling I, Clément S, Bordessoules M, Allard M (2005) Cerebral mechanisms of prosodic integration: Evidence from connected speech. Neuroimage 24:937–947

    Article  PubMed  Google Scholar 

  • Joassin F, Campanella S, Debatisse D, Guérit JM, Bruyer R, Crommelinck M (2004a) The electrophysiological correlates sustaining the retrieval of face–name associations: An ERP study. Psychophysiology 41:625–635

    Article  PubMed  CAS  Google Scholar 

  • Joassin F, Maurage P, Bruyer R, Crommelinck M, Campanella S (2004b) When audition alters vision: An event-related potential study of the cross-modal interactions between faces and voices. Neuroscience Letters 369:132–137

    Article  PubMed  CAS  Google Scholar 

  • Joassin F, Maurage P, Campanella S (2011a) The neural network sustaining the crossmodal processing of human gender from faces and voices: An fMRI study. Neuroimage 54(2):1654–1661

    Article  PubMed  Google Scholar 

  • Joassin F, Meert G, Campanella S, Bruyer R (2007) The associative processes involved in faces–proper names vs. objects–common names binding: A comparative ERP study. Biological Psychology 75(3):286–299

    Article  PubMed  Google Scholar 

  • Joassin F, Pesenti M, Maurage P, Verreckt E, Bruyer R, Campanella S (2011b) Cross-modal interactions between human faces and voices involved in person recognition. Cortex 47:367–376

    Article  PubMed  Google Scholar 

  • Kanwisher N, McDermott J, Chun MM (1997) The fusiform face area: A module in human extrastriate cortex specialized for face perception. The Journal of Neuroscience 9:462–475

    Google Scholar 

  • Kerlin JR, Shahin AJ, Miller LM (2010) Attentional grain control of ongoing cortical speech representations in a “cocktail party”. The Journal of Neuroscience 30(2):620–628

    Article  PubMed  CAS  Google Scholar 

  • Laurienti PJ, Perrault TJ, Stanford TR, Wallace MT, Stein BE (2005) On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Experimental Brain Research 166:289–297

    Article  Google Scholar 

  • Leube DT, Erb M, Grodd W, Bartels M, Kircher TTJ (2001) Differential activation in parahippocampal and prefrontal cortex during word and face encoding tasks. Neuroreport 12(12):2773–2777

    Article  PubMed  CAS  Google Scholar 

  • Magnée M, de Gelder B, van Engeland H, Kemner C (2008) Atypical processing of fearful face–voice pairs in Pervasive Developmental Disorder: An ERP study. Clinical Neurophysiology 119:2004–2010

    Article  Google Scholar 

  • Maurage P, Campanella S, Philippot P, Pham T, Joassin F (2007) The crossmodal facilitation effect is disrupted in alcoholism: A study with emotional stimuli. Alcohol and Alcoholism 42:552–559

    Article  PubMed  CAS  Google Scholar 

  • Maurage P, Philippot P, Joassin F, Alonso Prieto E, Palmero Soler E, Zanow F, Campanella S (2008) The auditory–visual integration of anger is disrupted in alcoholism: An ERP study. Journal of Psychiatry and Neuroscience 33(2):111–122

    PubMed  Google Scholar 

  • McNamara A, Buccino G, Menz MM, Gläsher J, Wolbers T, Baumgärtner A, Binkofski F (2008) Neural dynamics of learning sound–action associations. PLoS One 3(12):1–10

    Article  Google Scholar 

  • Melillo R, Leisman G (2009) Autistic spectrum disorders as functional disconnection syndrome. Reviews in the Neurosciences 20(2):111–131

    Article  PubMed  Google Scholar 

  • Monk C, Weng SJ, Wiggins J, Kurapati N, Louro H, Carrasco M, Maslowsky J, Risi S, Lord C (2010) Neural circuitry of emotional face processing in autism spectrum disorders. Journal of Psychiatry and Neuroscience 35(2):105–114

    Article  PubMed  Google Scholar 

  • Puce A, Allison T, Gore JC, McCarthy G (1995) Face-sensitive regions in human extrastriate cortex studied by functional MRI. Journal of Neurophysiology 74(3):1192–1199

    PubMed  CAS  Google Scholar 

  • Rama P, Courtney SM (2005) Functional topography of working memory for face or voice identity. NeuroImage 24:224–234

    Article  PubMed  Google Scholar 

  • Rolls ET (2000) The orbitofrontal cortex and reward. Cerebral Cortex 10:284–294

    Article  PubMed  CAS  Google Scholar 

  • Schweinberger SR, Robertson D, Kaufmann JM (2007) Hearing facial identities. The Quarterly Journal of Experimental Psychology 60(10):1446–1456

    Article  PubMed  Google Scholar 

  • Seiferth N, Pauly K, Kellermann T, Shah N, Ott G, Herpertz-Dahlmann B, Kircher T, Schneider F, Habel U (2009) Neuronal correlates of facial emotion discrimination in early onset schizophrenia. Neuropsychopharmacology 34:477–487

    Article  PubMed  Google Scholar 

  • Senkowski D, Schneider TR, Foxe JJ, Engel AK (2008) Crossmodal binding through neural coherence: Implications for multisensory processing. Trends in Cognitive Sciences 31(8):401–409

    CAS  Google Scholar 

  • Sheffert SM, Olson E (2004) Audiovisual speech facilitates voice learning. Perception & Psychophysics 66(2):352–362

    Article  Google Scholar 

  • Shomstein S, Yantis S (2004) Control of attention shifts between vision and audition in human cortex. The Journal of Neuroscience 24(47):10702–10706

    Article  PubMed  CAS  Google Scholar 

  • Smith EL, Grabowecky M, Suzuki S (2007) Auditory–visual crossmodal integration in perception of face gender. Current Biology 17:1680–1685

    Article  PubMed  CAS  Google Scholar 

  • Steeves J, Dricot L, Goltz HC, Sorger B, Peters J, Milner AD, Goodale MA, Goebel R, Rossion B (2009) Abnormal face identity coding in the middle fusiform gyrus of two brain-damaged prosopagnosic patients. Neuropsychologia 47(12):2584–2592

    Article  PubMed  Google Scholar 

  • Van Lancker DR, Canter GJ (1982) Impairment of voice and face recognition in patients with hemispheric damage. Brain and Cognition 1(2):185–195

    Article  PubMed  Google Scholar 

  • Von Kriegstein K, Giraud AL (2006) Implicit multisensory associations influence voice recognition. PLoS Biology 4:e326

    Article  Google Scholar 

  • Von Kriegstein K, Kleinschmidt A, Sterzer P, Giraud AL (2005) Interaction of face and voice areas during speaker recognition. Journal of Cognitive Neuroscience 17(3):367–376

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Laboratory of Psychological Medicine, Free University of Brussels, Brussels, Belgium

    Salvatore Campanella

  2. CHU Brugmann, Psychiatry Department (EEG), The Belgian Fund for Scientific Research (FNRS), Brussels, Belgium

    Salvatore Campanella

  3. Clinique de la mémoire, CHU Ambroise Paré, Mons, Belgium

    Frédéric Joassin

Authors
  1. Salvatore Campanella
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Frédéric Joassin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Salvatore Campanella .

Editor information

Editors and Affiliations

  1. Dept. Psychology, Voice Neurocognition Lab., University of Glasgow, Glasgow, Glasgow, G12 8QB, United Kingdom

    Pascal Belin

  2. Brugmann University Hospital, Brussels, B-1020, Belgium

    Salvatore Campanella

  3. University of Tubingen, Tubingen, 72074, Germany

    Thomas Ethofer

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer Science+Business Media New York

About this chapter

Cite this chapter

Campanella, S., Joassin, F. (2013). Cross-Modal Integration of Identity and Gender Information Through Faces and Voices Involves a Similar Cortical Network. In: Belin, P., Campanella, S., Ethofer, T. (eds) Integrating Face and Voice in Person Perception. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-3585-3_8

Download citation

Publish with us

Policies and ethics

Search

Navigation