Abstract
Electroencephalography (EEG) activity in the mu frequency band (8–13 Hz) is suppressed during both gesture performance and observation. However, it is not clear if or how particular characteristics within the kinematic execution of gestures map onto dynamic changes in mu activity. Mapping the time course of gesture kinematics onto that of mu activity could help understand which aspects of gestures capture attention and aid in the classification of communicative intent. In this work, we test whether the timing of inflection points within gesture kinematics predicts the occurrence of oscillatory mu activity during passive gesture observation. The timing for salient features of performed gestures in video stimuli was determined by isolating inflection points in the hands’ motion trajectories. Participants passively viewed the gesture videos while continuous EEG data was collected. We used wavelet analysis to extract mu oscillations at 11 Hz and at central electrodes and occipital electrodes. We used linear regression to test for associations between the timing of inflection points in motion trajectories and mu oscillations that generalized across gesture stimuli. Separately, we also tested whether inflection point occurrences evoked mu/alpha responses that generalized across participants. Across all gestures and inflection points, and pooled across participants, peaks in 11 Hz EEG waveforms were detected 465 and 535 ms after inflection points at occipital and central electrodes, respectively. A regression model showed that inflection points in the motion trajectories strongly predicted subsequent mu oscillations (\({R}^{2}=0.921, p\)<0.01); effects were weaker and non-significant for low (17 Hz) and high (21 Hz) beta activity. When segmented by inflection point occurrence rather than stimulus onset and testing participants as a random effect, inflection points evoked mu and beta activity from 308 to 364 ms at central electrodes, and broad activity from 226 to 800 ms at occipital electrodes. The results suggest that inflection points in gesture trajectories elicit coordinated activity in the visual and motor cortices, with prominent activity in the mu/alpha frequency band and extending into the beta frequency band. The time course of activity indicates that visual processing drives subsequent activity in the motor cortex during gesture processing, with a lag of approximately 80 ms.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00221-020-05724-y/MediaObjects/221_2020_5724_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00221-020-05724-y/MediaObjects/221_2020_5724_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00221-020-05724-y/MediaObjects/221_2020_5724_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00221-020-05724-y/MediaObjects/221_2020_5724_Fig4_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00221-020-05724-y/MediaObjects/221_2020_5724_Fig5_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00221-020-05724-y/MediaObjects/221_2020_5724_Fig6_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00221-020-05724-y/MediaObjects/221_2020_5724_Fig7_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00221-020-05724-y/MediaObjects/221_2020_5724_Fig8_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00221-020-05724-y/MediaObjects/221_2020_5724_Fig9_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00221-020-05724-y/MediaObjects/221_2020_5724_Fig10_HTML.png)
Similar content being viewed by others
References
Arbib MA (2005) Interweaving protosign and protospeech: Further developments beyond the mirror. Interact Stud 6:145–171
Avenanti A, Bueti D, Galati G, Aglioti SM (2005) Transcranial magnetic stimulation highlights the sensorimotor side of empathy for pain. Nat Neurosci 8(7):955
Bașar E (2010) Brain–body–mind in the nebulous Cartesian system: a holistic approach by oscillations. Springer, New York
Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Ser B 57(1):289–300
Bobick AF, Wilson AD (1997) A state-based approach to the representation and recognition of gesture. Pattern Anal Mach Intell IEEE Trans 19:1325–1337
Braadbaart L, Williams JH, Waiter GD (2013) Do mirror neuron areas mediate mu rhythm suppression during imitation and action observation? Int J Psychophysiol 89:99–105
Buchin K, Buchin M, Gudmundsson J, Löffler M, Luo J (2011) Detecting commuting patterns by clustering subtrajectories. Int J Comput Geom Appl 21:253–282
Cabrera ME, Novak K, Foti D, Voyles R, Wachs JP (2017) What makes a gesture a gesture? Neural signatures involved in gesture recognition. In: 2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017), pp 748–753
Chong TT-J, Cunnington R, Williams MA, Kanwisher N, Mattingley JB (2008) fMRI adaptation reveals mirror neurons in human inferior parietal cortex. Curr Biol 18:1576–1580
Cook R, Bird G, Catmur C, Press C, Heyes C (2014) Mirror neurons: from origin to function. Behav Brain Sci 37:177–192
Dahan A, Reiner M (2016) Recognition of the semantics and kinematics of gestures: neural responses to “what” and “how”? Int J Psychophysiol. https://doi.org/10.1016/j.ijpsycho.2016.10.009
Decety J, Grèzes J (1999) Neural mechanisms subserving the perception of human actions. Trends Cogn Sci 3:172–178
Despinoy F, Bouget D, Forestier G, Penet C, Zemiti N, Poignet P, Jannin P (2016) unsupervised trajectory segmentation for surgical gesture recognition in robotic training. IEEE Trans Biomed Eng 63:1280–1291. https://doi.org/10.1109/TBME.2015.2493100
Di Pellegrino G, Fadiga L, Fogassi L, Gallese V, Rizzolatti G (1992) Understanding motor events: a neurophysiological study. Exp Brain Res 91:176–180
Engel A, Burke M, Fiehler K, Bien S, Rösler F (2008) What activates the human mirror neuron system during observation of artificial movements: bottom-up visual features or top-down intentions? Neuropsychologia 46:2033–2042
Enticott PG, Johnston PJ, Herring SE, Hoy KE, Fitzgerald PB (2008) Mirror neuron activation is associated with facial emotion processing. Neuropsychologia 46:2851–2854
Escalera S, Gonzàlez J, Baró X, Reyes M, Lopes O, Guyon I, Athitsos V, Escalante H (2013) Multi-modal gesture recognition challenge 2013: Dataset and results. In: Proceedings of the 15th ACM on international conference on multimodal interaction. ACM, pp 445–452
Filimon F, Nelson JD, Hagler DJ, Sereno MI (2007) Human cortical representations for reaching: mirror neurons for execution, observation, and imagery. Neuroimage 37:1315–1328
Gallese V, Sinigaglia C (2012) Response to de Bruin and Gallagher: embodied simulation as reuse is a productive explanation of a basic form of mind-reading. Trends Cogn Sci 16:99–100
Gazzola V, Keysers C (2009) The observation and execution of actions share motor and somatosensory voxels in all tested subjects: single-subject analyses of unsmoothed fMRI data. Cereb Cortex 1991(19):1239–1255. https://doi.org/10.1093/cercor/bhn181
Gazzola V, Rizzolatti G, Wicker B, Keysers C (2007) The anthropomorphic brain: the mirror neuron system responds to human and robotic actions. Neuroimage 35:1674–1684
Glenberg AM, Sato M, Cattaneo L, Riggio L, Palumbo D, Buccino G (2008) Processing abstract language modulates motor system activity. Q J Exp Psychol 61:905–919
Herrmann CS, Grigutsch M, Busch NA (2005) 11 EEG oscillations and wavelet analysis. Event-related potentials: A methods handbook, p 229
Hobson HM, Bishop DV (2016) Mu suppression—a good measure of the human mirror neuron system? Cortex 82:290–310
Hogeveen J, Chartrand TL, Obhi SS (2014) Social mimicry enhances mu-suppression during action observation. Cereb Cortex 25:2076–2082
Iacoboni M, Woods RP, Brass M, Bekkering H, Mazziotta JC, Rizzolatti G (1999) Cortical mechanisms of human imitation. Science 286:2526–2528
Kita S, van Gijn I, van der Hulst H (1998) Movement phases in signs and co-speech gestures, and their transcription by human coders. In: Wachsmuth I, Fröhlich M (eds) Gesture and sign language in human-computer interaction. GW 1997, vol 1371. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, pp 23–35
Kühn S, Brass M (2008) Testing the connection of the mirror system and speech: how articulation affects imitation in a simple response task. Neuropsychologia 46:1513–1521
Loram ID, Gawthrop PJ, Lakie M (2006) The frequency of human, manual adjustments in balancing an inverted pendulum is constrained by intrinsic physiological factors. J Physiol 577:417–432
McNeill D (1992) Hand and mind: what gestures reveal about thought. University of Chicago Press, New York
Muthukumaraswamy SD, Johnson BW, McNair NA (2004) Mu rhythm modulation during observation of an object-directed grasp. Cogn Brain Res 19:195–201
Neuper C, Scherer R, Reiner M, Pfurtscheller G (2005) Imagery of motor actions: Differential effects of kinesthetic and visual–motor mode of imagery in single-trial EEG. Cogn Brain Res 25:668–677. https://doi.org/10.1016/j.cogbrainres.2005.08.014
Perry A, Stein L, Bentin S (2011) Motor and attentional mechanisms involved in social interaction—evidence from mu and alpha EEG suppression. Neuroimage 58:895–904
Pfurtscheller G, Neuper C, Mohl W (1994) Event-related desynchronization (ERD) during visual processing. Int J Psychophysiol 16:147–153
Quandt LC, Marshall PJ, Shipley TF, Beilock SL, Goldin-Meadow S (2012) Sensitivity of alpha and beta oscillations to sensorimotor characteristics of action: an EEG study of action production and gesture observation. Neuropsychologia 50:2745–2751
Raos V, Umiltá MA, Murata A, Fogassi L, Gallese V (2006) Functional properties of grasping-related neurons in the ventral premotor area F5 of the macaque monkey. J Neurophysiol 95(2):709–729
Rizzolatti G, Arbib MA (1998) Language within our grasp. Trends Neurosci 21:188–194
Rizzolatti G, Craighero L (2004) The mirror-neuron system. Annu Rev Neurosci 27:169–192
Rizzolatti G, Sinigaglia C (2010) The functional role of the parieto-frontal mirror circuit: interpretations and misinterpretations. Nat Rev Neurosci 11:264–274. https://doi.org/10.1038/nrn2805
Rizzolatti G, Fadiga L, Gallese V, Fogassi L (1996) Premotor cortex and the recognition of motor actions. Cogn Brain Res 3:131–141
Streltsova A, Berchio C, Gallese V, Umilta MA (2010) Time course and specificity of sensory-motor alpha modulation during the observation of hand motor acts and gestures: a high density EEG study. Exp Brain Res 205:363–373
Tadel F, Baillet S, Mosher JC, Pantazis D, Leahy RM (2011) Brainstorm: a user-friendly application for MEG/EEG analysis computational intelligence and neuroscience, vol 2011. https://doi.org/10.1155/2011/879716(Article ID 879716)
Tadel F, Bock EA, Niso G, Mosher JC, Cousineau M, Pantazis D, Leahy RM, Baillet S (2019) MEG/EEG group analysis with Brainstorm. Front Neurosci 13:76
Théoret H, Pascual-Leone A (2002) Language acquisition: do as you hear. Curr Biol 12:R736–R737
Urgen BA, Plank M, Ishiguro H, Poizner H, Saygin AP (2013) EEG theta and Mu oscillations during perception of human and robot actions. Front Neurorobotics 7:19. https://doi.org/10.3389/fnbot.2013.00019
Viviani P, Terzuolo C (1982) Trajectory determines movement dynamics. Neuroscience 7:431–437. https://doi.org/10.1016/0306-4522(82)90277-9
Wu YC, Coulson S (2005) Meaningful gestures: electrophysiological indices of iconic gesture comprehension. Psychophysiology 42:654–667. https://doi.org/10.1111/j.1469-8986.2005.00356.x
Acknowledgements
This work was supported by the Office of the Assistant Secretary of Defense for Health Affairs under Award No. W81XWH-14-1-0042 (JPW is Principal Investigator). Opinions, interpretations, conclusions and recommendations are those of the author and are not necessarily endorsed by the Department of Defense.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Gay R. Holstein .
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Cabrera, M.E., Novak, K., Foti, D. et al. Electrophysiological indicators of gesture perception. Exp Brain Res 238, 537–550 (2020). https://doi.org/10.1007/s00221-020-05724-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00221-020-05724-y