Abstract
Metaphoric (MP) co-speech gestures are commonly used during daily communication. They communicate about abstract information by referring to gestures that are clearly concrete (e.g., raising a hand for “the level of the football game is high”). To understand MP co-speech gestures, a multisensory integration at semantic level is necessary between abstract speech and concrete gestures. While semantic gesture-speech integration has been extensively investigated using functional magnetic resonance imaging, evidence from electroencephalography (EEG) is rare. In the current study, we set out an EEG experiment, investigating the processing of MP vs. iconic (IC) co-speech gestures in different contexts, to reveal the oscillatory signature of MP gesture integration. German participants (n = 20) viewed video clips with an actor performing both types of gestures, accompanied by either comprehensible German or incomprehensible Russian (R) speech, or speaking German sentences without any gestures. Time-frequency analysis of the EEG data showed that, when gestures were accompanied by comprehensible German speech, MP gestures elicited decreased gamma band power (50–70 Hz) between 500 and 700 ms in the parietal electrodes when compared to IC gestures, and the source of this effect was localized to the right middle temporal gyrus. This difference is likely to reflect integration processes, as it was reduced in the R language and no-gesture conditions. Our findings provide the first empirical evidence suggesting the functional relationship between gamma band oscillations and higher-level semantic processes in a multisensory setting.
Highlights
IntroductionIt is common to hear someone talking about the ups and downs of a day, and describing them as bitter or sweet
In the co-speech gesture condition, MP gestures yielded decreased gamma power when compared with IC gestures (MPG vs. iconic co-speech gestures (ICG)), and this effect was source localized to the right middle temporal gyrus
This paradigm has been applied to recent EEG studies that directly investigated the integration between gesture and speech (Biau and Soto-Faraco, 2013; Biau et al, 2015; He et al, 2015; Drijvers et al, 2018) and functional magnetic resonance imaging (fMRI) studies focusing on integration of MP gestures (Kircher et al, 2009; Straube et al, 2011)
Summary
It is common to hear someone talking about the ups and downs of a day, and describing them as bitter or sweet These constructions provide a means to communicate about abstract target representations by referring to source representations that are clearly more concrete (Lakoff and Johnson, 1980). Unlike linguistic metaphors, MP gesture is a multi-modal form in which the MP meaning is expressed or underpinned. In this sense, successful comprehension of MP co-speech gesture requires lower-level integration of audio–visual input, and higher-level semantic processes, so that the construction of a coherent, and MP semantic representation is possible. Recent evidence from neurophysiology has emerged, suggesting that both processes may be related to de/synchronized brain oscillations in the gamma band (>30 Hz)
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
