Skip to main content
SpringerLink
Log in

The benefit of multisensory integration with biological motion signals

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

Assessing intentions, direction, and velocity of others is necessary for most daily tasks, and such information is often made available by both visual and auditory motion cues. Therefore, it is not surprising our great ability to perceive human motion. Here, we explore the multisensory integration of cues of biological motion walking speed. After testing for audiovisual asynchronies (visual signals led auditory ones by 30 ms in simultaneity temporal windows of 76.4 ms), in the main experiment, visual, auditory, and bimodal stimuli were compared to a standard audiovisual walker in a velocity discrimination task. Results in variance reduction conformed to optimal integration of congruent bimodal stimuli across all subjects. Interestingly, the perceptual judgements were still close to optimal for stimuli at the smallest level of incongruence. Comparison of slopes allows us to estimate an integration window of about 60 ms, which is smaller than that reported in audiovisual speech.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Alais D, Burr D (2004a) No direction-specific bimodal facilitation for audiovisual motion detection. Cog Brain Res 19:195–196

    Article  Google Scholar 

  • Alais D, Burr D (2004b) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14:257–262

    PubMed  CAS  Google Scholar 

  • Alais D, Carlile S (2005) Synchronizing to real events: subjective audiovisual alignment scales with perceived auditory depth and speed of sound. Proc Natl Acad Sci 102:2244–2247

    Article  PubMed  CAS  Google Scholar 

  • Andersen TS, Tiippana K, Sams M (2005) Maximum likelihood integration of rapid flashes and beeps. Neurosci Lett 380:155–160

    Article  PubMed  CAS  Google Scholar 

  • Arrighi R, Alais D, Burr D (2006) Perceptual synchrony of audiovisual streams for natural and artificial motion sequences. J Vision 6:260–268

    Article  Google Scholar 

  • Arrighi R, Marini F, Burr D (2009) Meaningful auditory information enhances perception of visual biological motion. J Vision 9:1–7

    Article  Google Scholar 

  • Barraclough NE, Xiao D, Baker CI, Oram MW, Perrett DI (2005) Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. J Cog Neurosci 17:377–391

    Article  Google Scholar 

  • Battaglia PW, Jacobs RA, Aslin RN (2003) Bayesian integration of visual and auditory signals for spatial localization. J Opt Soc Am 20:1391–1397

    Article  Google Scholar 

  • Bentvelzen A, Leung J, Alais D (2009) Discriminating audiovisual speed: Optimal integration defaults to probability summation when companent reliabilities diverge. Perception 28:966–987

    Article  Google Scholar 

  • Bidet-Caulet A, Voisin J, Bertrand O, Fonlupt P (2005) Listening to a walking human activates the temporal biological motion area. NeuroImage 28:132–139

    Article  PubMed  Google Scholar 

  • Brooks A, Van der Zwan R, Billard A, Petreska B, Clarke S, Blanke O (2007) Auditory motion affects visual biological motion processing. Neuropsychologia 45:523–530

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA (2001) Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb Cortex 11:1110–1123

    Article  PubMed  CAS  Google Scholar 

  • Clarke JJ, Yuille AL (1990) Data fusion for sensory information processing. Kluwer Academic, Boston

    Google Scholar 

  • Di Luca M, Machulla TK, Ernst MO (2009) Recalibration of multisensory simultaneity: Cross-modal transfer coincides with change in perceptual latency. J Vision 9:1–16

    Article  Google Scholar 

  • Efron B, Tibshirani RJ (1993) An introduction to the bootstrap. Chapman and Hall, New York

    Google Scholar 

  • Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415:429–433

    Article  PubMed  CAS  Google Scholar 

  • Ernst MO, Bülthoff HH (2004) Merging the senses into a robust percept. Tends Cog Sci 8:162–169

    Article  Google Scholar 

  • Garcia JO, Grossman ED (2007) Necessary but not sufficient: Motion perception is required for perceiving biological motion. Vision Res 48:1144–1149

    Article  Google Scholar 

  • Hiris E, Krebeck A, Edmonds J, Stout A (2005) What learning to see arbitrary motion tells us about biological motion perception. J Exp Psychol 31:1096–1106

    Google Scholar 

  • López-Moliner J, Soto-Faraco S (2007) Vision affects how fast we hear sounds move. J Vision 6:1–7

    Google Scholar 

  • Marcell MM, Borella D, Greene M, Kerr E, Rogers S (2000) Confrontation naming of environmental sounds. J Clin Exp Neuropsychol 22:830–864

    Article  PubMed  CAS  Google Scholar 

  • Massaro DW, Friedman D (1990) Models of integration given multiple sources of information given multiple sources of information. Psychol Rev 97:229–252

    Article  Google Scholar 

  • Meyer GF, Wuerger SM, Rohrbein F, Zetzsche C (2005) Low-level integration of auditory and visual motion signals requires spatial co-localization. Exp Brain Res 166:538–547

    Article  PubMed  Google Scholar 

  • Neri P, Morrone MC, Burr DC (1998) Seeing biological motion. Nature 395:894–896

    Article  PubMed  CAS  Google Scholar 

  • Saygin AP, Driver J, de Sa VR (2008) In the footsteps of biological motion and multisensory perception. Psychol Sci 19:469–475

    Article  PubMed  Google Scholar 

  • Shams L, Ma WJ, Beierholm M (2005) Sound-induced flash illusion as an optimal percept. NeuroReport 16:1923–1927

    Article  PubMed  Google Scholar 

  • Shiffrar M, Pinto J (2002) The visual analysis of bodily motion. In: Prinz W, Hommel B (eds) Common Mechanisms in Perception and Action–Attention and Performance XIX. Oxford University Press, Oxford, pp 381–399

    Google Scholar 

  • Shiffrar M, Lichtey L, Chatterjee SH (1997) The perception of biological motion across apertures. Percept Psychophys 59:51–59

    Article  PubMed  CAS  Google Scholar 

  • Sugita Y, Suzuki Y (2003) Implicit estimation of sound-arrival time. Nature 421:911

    Article  PubMed  CAS  Google Scholar 

  • Van der Zwan R, MacHatch C, Kozlowski D, Troje NF, Blanke O, Brooks A (2009) Gender bending: Auditory cues affect visual judgments of gender in biological motion displays. Exp Brain Res 198:373–382

    Article  PubMed  Google Scholar 

  • Van Eijk RLJ, Kohlrausch A, Juola JF, Van de Par S (2008) Audiovisual synchrony and temporal order judgments: Effects of experimental method and stimulus type. Percept Psychophys 70:955–968

    Article  PubMed  Google Scholar 

  • Van Wassenhove V, Grant KW, Poeppel D (2007) Temporal window of integration in audiovisual speech perception. Neurophychologia 45:598–607

    Article  Google Scholar 

  • Vatakis A, Spence C (2006) Audiovisual synchrony perception for music, speech, and object actions. Brain Res 111:134–142

    Article  Google Scholar 

  • Welsh RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88:638–667

    Article  Google Scholar 

  • Wuerger SM, Hofbauer M, Meyer GF (2003) The integration of auditory and visual motion signals at threshold. Percept Psychophys 65:1188–1196

    Article  PubMed  CAS  Google Scholar 

  • Wuerger S, Meyer G, Hofbauer M, Zetzsche C, Schill K (2010) Motion extrapolation of auditory-visual targets. Information Fusion 11:45–50

    Article  Google Scholar 

  • Yabe H, Tervaniemi M, Reinikainen K, Näätänen R (1997) Temporal window of integration revealed by MMN to sound omission. Neuroreport 8:1971–1974

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgments

This work was partly funded by the Portuguese Foundation for Science and Technology (SFRH/BD/36345/2007, PTDC/SAU-BEB/68455/2006, SFRH/BSAB/974/2009) and the Portugal-Spain Actions PT2009-0186 from the Spanish Government and E-134/10 from the Portuguese Conselho de Reitores das Universidades Portuguesas.

Author information

Authors and Affiliations

  1. School of Psychology, University of Minho, Braga, Portugal

    Catarina Mendonça & Jorge A. Santos

  2. Group of Vision and Control of Action, Institute for Brain, Cognition and Behavior, University of Barcelona, Catalonia, Spain

    Joan López-Moliner

  3. Laboratory of Visualization and Perception, Centro de Computação Gráfica, Campus de Azurém, 4800-058, Guimarães, Portugal

    Catarina Mendonça

Authors
  1. Catarina Mendonça
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jorge A. Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joan López-Moliner
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Catarina Mendonça.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Mendonça, C., Santos, J.A. & López-Moliner, J. The benefit of multisensory integration with biological motion signals. Exp Brain Res 213, 185–192 (2011). https://doi.org/10.1007/s00221-011-2620-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-011-2620-4

Keywords

Search

Navigation