He Yifei, Nagels Arne, Schlesewsky Matthias, Straube Benjamin
Translational Neuroimaging Lab, Department of Psychiatry and Psychotherapy, Marburg Center for Mind, Brain and Behavior, Philipps-University Marburg, Marburg, Germany.
Department of General Linguistics, Johannes Gutenberg University Mainz, Mainz, Germany.
Front Psychol. 2018 Jul 30;9:1348. doi: 10.3389/fpsyg.2018.01348. eCollection 2018.
Metaphoric (MP) co-speech gestures are commonly used during daily communication. They communicate about abstract information by referring to gestures that are clearly concrete (e.g., raising a hand for "the level of the football game is high"). To understand MP co-speech gestures, a multisensory integration at semantic level is necessary between abstract speech and concrete gestures. While semantic gesture-speech integration has been extensively investigated using functional magnetic resonance imaging, evidence from electroencephalography (EEG) is rare. In the current study, we set out an EEG experiment, investigating the processing of MP vs. iconic (IC) co-speech gestures in different contexts, to reveal the oscillatory signature of MP gesture integration. German participants ( = 20) viewed video clips with an actor performing both types of gestures, accompanied by either comprehensible German or incomprehensible Russian (R) speech, or speaking German sentences without any gestures. Time-frequency analysis of the EEG data showed that, when gestures were accompanied by comprehensible German speech, MP gestures elicited decreased gamma band power (50-70 Hz) between 500 and 700 ms in the parietal electrodes when compared to IC gestures, and the source of this effect was localized to the right middle temporal gyrus. This difference is likely to reflect integration processes, as it was reduced in the R language and no-gesture conditions. Our findings provide the first empirical evidence suggesting the functional relationship between gamma band oscillations and higher-level semantic processes in a multisensory setting.
隐喻性（MP）伴随言语手势在日常交流中普遍使用。它们通过指代明显具体的手势来传达抽象信息（例如，举起手表示“足球比赛水平很高”）。为了理解MP伴随言语手势，在抽象言语和具体手势之间进行语义层面的多感官整合是必要的。虽然已经使用功能磁共振成像对语义手势 - 言语整合进行了广泛研究，但来自脑电图（EEG）的证据却很少。在当前的研究中，我们开展了一项EEG实验，研究在不同情境下MP与象似性（IC）伴随言语手势的处理过程，以揭示MP手势整合的振荡特征。德国参与者（ = 20）观看了演员做出这两种手势的视频片段，视频中伴有可理解的德语或不可理解的俄语（R）言语，或者演员只说德语句子而不做任何手势。对EEG数据的时频分析表明，当手势伴有可理解的德语言语时，与IC手势相比，MP手势在顶叶电极处500至700毫秒之间引发伽马波段功率（50 - 70赫兹）降低，并且这种效应的来源定位于右侧颞中回。这种差异可能反映了整合过程，因为在俄语和无手势条件下这种差异减小了。我们的研究结果提供了首个实证证据，表明在多感官环境中伽马波段振荡与高级语义过程之间的功能关系。
