Abstract
To overcome differences in physical transmission time and neural processing, the brain adaptively recalibrates the point of simultaneity between auditory and visual signals by adapting to audiovisual asynchronies. Here, we examine whether the prolonged recalibration process of passively sensed visual and auditory signals is affected by naturally occurring multisensory training known to enhance audiovisual perceptual accuracy. Hence, we asked a group of drummers, of non-drummer musicians and of non-musicians to judge the audiovisual simultaneity of musical and non-musical audiovisual events, before and after adaptation with two fixed audiovisual asynchronies. We found that the recalibration for the musicians and drummers was in the opposite direction (sound leading vision) to that of non-musicians (vision leading sound), and change together with both increased music training and increased perceptual accuracy (i.e. ability to detect asynchrony). Our findings demonstrate that long-term musical training reshapes the way humans adaptively recalibrate simultaneity between auditory and visual signals.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alais D, Carlile S (2005) Synchronizing to real events: subjective audiovisual alignment scales with perceived auditory depth and speed of sound. Proc Natl Acad Sci USA 102(6):2244–2247
Arnold DH, Johnston A, Nishida S (2005) Timing sight and sound. Vision Res 45(10):1275–1284
Aschersleben G, Prinz W (1995) Synchronizing actions with events: the role of sensory information. Percept Psychophys 57(3):305–317
Bigand E, Poulin-Charronnat B (2006) Are we “experienced listeners”? A review of the musical capacities that do not depend on formal musical training. Cognition 100(1):100–130
Bishop L, Goebl W (2014) Context-specific effects of musical expertise on audiovisual integration. Frontiers in Psychology 5:1123
Botella L (2008) Timekeeping is everything 1: rhythm and the construction of meaning. J Constructivist Psychol 21(4):309–320
Brainard DH (1997) The psychophysics toolbox. Spat Vis 10:433–436
Bruns P, Röder B (2015) Sensory recalibration integrates information from the immediate and the cumulative past. Sci Rep 5:12739
Calvo-Merino B, Glaser DE, Grezes J, Passingham RE, Haggard P (2005) Action observation and acquired motor skills: an FMRI study with expert dancers. Cereb Cortex 15(8):1243–1249
Calvo-Merino B, Grèzes J, Glaser DE, Passingham RE, Haggard P (2006) Seeing or doing? Influence of visual and motor familiarity in action observation. Curr Biol 16(19):1905–1910
De Niear MA, Noel JP, Wallace MT (2017) The impact of feedback on the different time courses of multisensory temporal recalibration. Neural Plasticity
Desantis A, Haggard P (2016) Action-outcome learning and prediction shape the window of simultaneity of audiovisual outcomes. Cognition 153:33–42
Di Luca M, Machulla TK, Ernst MO (2009) Recalibration of multisensory simultaneity: cross-modal transfer coincides with a change in perceptual latency. J Vision 9(12):7–7
Faul F, Erdfelder E, Lang AG, Buchner A (2007) G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods 39(2):175–191
Flatischler R (1992) The forgotten power of rhythm. Life Rhythm
Fontana F, Avanzini F, Rocchesso D (2004) Computation of nonlinear filter networks containing delay-free paths. In: Proceedings of the 7th International Conference on Digital Audio Effects (DAFX-04), Naples, Italy, pp 113–118
Foss-Feig JH, Kwakye LD, Cascio CJ, Burnette CP, Kadivar H, Stone WL, Wallace MT (2010) An extended multisensory temporal binding window in autism spectrum disorders. Exp Brain Res 203(2):381–389
Foxe JJ, Molholm S, Bene D, Frey VA, Russo HP, Blanco D, Ross LA (2013) Severe multisensory speech integration deficits in high-functioning school-aged children with autism spectrum disorder (ASD) and their resolution during early adolescence. Cerebral Cortex, bht213
Fujisaki W, Shimojo S, Kashino M, Nishida SY (2004) Recalibration of audiovisual simultaneity. Nat Neurosci 7(7):773–778
García-Pérez MA, Alcalá-Quintana R (2012) On the discrepant results in synchrony judgment and temporal-order judgment tasks: a quantitative model. Psychon Bull Rev 19(5):820–846
Harrar V, Harris LR (2008) The effect of exposure to asynchronous audio, visual, and tactile stimulus combinations on the perception of simultaneity. Exp Brain Res 186(4):517–524
Hodges DA, Hairston WD, Burdette JH (2005) Aspects of multisensory perception: the integration of visual and auditory information in musical experiences. Ann N Y Acad Sci 1060(1):175–185
Keetels M, Vroomen J (2007) No effect of auditory–visual spatial disparity on temporal recalibration. Exp Brain Res 182(4):559–565
King AJ (2005) Multisensory integration: strategies for synchronization. Curr Biol 15(9):R339–R341
Lee H, Noppeney U (2011) Long-term music training tunes how the brain temporally binds signals from multiple senses. Proc Natl Acad Sci 108(51):E1441–E1450
Love SA, Petrini K, Cheng A, Pollick FE (2013) A psychophysical investigation of differences between synchrony and temporal order judgments. PloS One, 8(1), e54798
Navarra J, García-Morera J, Spence C (2012) Temporal adaptation to audiovisual asynchrony generalizes across different sound frequencies. Front Psychol 3:152
Nichols J (2012) Music education in homeschooling: Jamie’s story. Narrative soundings: an anthology of narrative inquiry in music education. Springer Netherlands, pp 115–125
Noel JP, Łukowska M, Wallace M, Serino A (2016a) Multisensory simultaneity judgment and proximity to the body. J Vision 16(3):21–21
Noel JP, De Niear M, Van der Burg E, Wallace MT (2016b) Audiovisual simultaneity judgment and rapid recalibration throughout the lifespan. PloS One, 11(8):e0161698
Noel JP, De Niear MA, Stevenson R, Alais D, Wallace MT (2017) Atypical rapid audio-visual temporal recalibration in autism spectrum disorders. Autism Res 10(1):121–129
Oberman LM, Ramachandran VS (2008) Preliminary evidence for deficits in multisensory integration in autism spectrum disorders: the mirror neuron hypothesis. Soc Neurosci 3(3–4):348–355
Pelli DG (1997) The VideoToolbox software for visual psychophysics: Transforming numbers into movies. Spat Vis 10(4):437–442
Petrini K, Dahl S, Rocchesso D, Waadeland CH, Avanzini F, Puce A, Pollick FE (2009a) Multisensory integration of drumming actions: musical expertise affects perceived audiovisual asynchrony. Exp Brain Res 198(2–3):339–352
Petrini K, Russell M, Pollick F (2009b) When knowing can replace seeing in audiovisual integration of actions. Cognition 110(3):432–439
Petrini K, Holt SP, Pollick F (2010) Expertise with multisensory events eliminates the effect of biological motion rotation on audiovisual synchrony perception. J Vision 10(5):2–2
Petrini K, Pollick FE, Dahl S, McAleer P, McKay L, Rocchesso D, … Puce A (2011) Action expertise reduces brain activity for audiovisual matching actions: an fMRI study with expert drummers. Neuroimage 56(3):1480–1492
Proverbio AM, Attardo L, Cozzi M, Zani A (2015) The effect of musical practice on gesture/sound pairing. Front Psychol 6
Repp BH, Su YH (2013) Sensorimotor synchronization: a review of recent research (2006–2012) Psychon Bull Rev 20(3):403–452
Roach NW, Heron J, Whitaker D, McGraw PV (2010) Asynchrony adaptation reveals neural population code for audio-visual timing. Proc R Soc Lond B: Biol Sci rspb20101737
Rohde M, Ernst MO (2013) To lead and to lag–forward and backward recalibration of perceived visuo-motor simultaneity. Front Psychol 3:599
Rohde M, Ernst MO (2016) Time, agency, and sensory feedback delays during action. Curr Opin Behav Sci 8:193–199
Rohde M, Scheller M, Ernst MO (2014a) Effects can precede their cause in the sense of agency. Neuropsychologia 65:191–196
Rohde M, van Dam LC, Ernst MO (2014b) Predictability is necessary for closed-loop visual feedback delay adaptation. J Vision 14(3):4–4
Schroeder CE, Foxe JJ (2004) 18 multisensory convergence in early cortical processing. The handbook of multisensory processes, p 295
Shams L, Kamitani Y, Shimojo S (2000) Illusions: what you see is what you hear. Nature 408(6814):788
Simon DM, Noel JP, Wallace MT (2017) Event related potentials index rapid recalibration to audiovisual temporal asynchrony. Front Integr Neurosci, 11
Spence C, Squire S (2003) Multisensory integration: maintaining the perception of synchrony. Curr Biol 13(13):R519-R521
Stein BE, Meredith MA, Wallace MT (1993) The visually responsive neuron and beyond: multisensory integration in cat and monkey. Progress in Brain Research, vol. 95. Elsevier, pp 79–90
Stevenson RA, Segers M, Ferber S, Barense MD, Wallace MT (2015) The impact of multisensory integration deficits on speech perception in children with autism spectrum disorders. Multisensory Sensorimotor Interactions In Speech Perception, 249 Studia Musicologica Norvegica, 32:169–191
Turi M, Karaminis T, Pellicano E, Burr D (2016) No rapid audiovisual recalibration in adults on the autism spectrum. Sci Rep 6:21756
Van der Burg E, Goodbourn PT (2015) Rapid, generalized adaptation to asynchronous audiovisual speech. Proc R Soc Lond B: Biol Sci 282(1804):20143083
Van der Burg E, Alais D, Cass J (2013) Rapid recalibration to audiovisual asynchrony. J Neurosci 33(37):14633–14637
Van der Burg E, Alais D, Cass J (2015a) Audiovisual temporal recalibration occurs independently at two different time scales. Sci Rep, 5
Van der Burg E, Orchard-Mills E, Alais D (2015b) Rapid temporal recalibration is unique to audiovisual stimuli. Exp Brain Res 233(1):53–59
Vatakis A, Spence C (2006) Audiovisual synchrony perception for music, speech, and object actions. Brain Res 1111(1):134–142
Vatakis A, Navarra J, Soto-Faraco S, Spence C (2007) Temporal recalibration during asynchronous audiovisual speech perception. Exp Brain Res 181(1):173–181
Vines BW, Krumhansl CL, Wanderley MM, Levitin DJ (2006) Cross-modal interactions in the perception of musical performance. Cognition 101(1):80–113
Vroomen J, Keetels M, De Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cogn Brain Res 22(1):32–35
Acknowledgements
We would like to thank Dr Chris Bevan and Eliot Farmer for engaging and useful discussion on what it means to be a drummer.
Author information
Authors and Affiliations
Contributions
KP designed the study, KP and FEP piloted the study, CJ conducted the experiment, KP and MJP supervised CJ during the experiment conduction, KP and CJ analyzed the data, KP, CJ, FEP and MJP wrote the manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no competing financial interests.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Jicol, C., Proulx, M.J., Pollick, F.E. et al. Long-term music training modulates the recalibration of audiovisual simultaneity. Exp Brain Res 236, 1869–1880 (2018). https://doi.org/10.1007/s00221-018-5269-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00221-018-5269-4