Joint Doctoral Program Language and Communicative Disorders, San Diego State University and UC San Diego, United States.
Cognitive Science Department, UC San Diego, United States.
Brain Lang. 2021 May;216:104916. doi: 10.1016/j.bandl.2021.104916. Epub 2021 Feb 27.
Here we examine the role of visuospatial working memory (WM) during the comprehension of multimodal discourse with co-speech iconic gestures. EEG was recorded as healthy adults encoded either a sequence of one (low load) or four (high load) dot locations on a grid and rehearsed them until a free recall response was collected later in the trial. During the rehearsal period of the WM task, participants observed videos of a speaker describing objects in which half of the trials included semantically related co-speech gestures (congruent), and the other half included semantically unrelated gestures (incongruent). Discourse processing was indexed by oscillatory EEG activity in the alpha and beta bands during the videos. Across all participants, effects of speech and gesture incongruity were more evident in low load trials than in high load trials. Effects were also modulated by individual differences in visuospatial WM capacity. These data suggest visuospatial WM resources are recruited in the comprehension of multimodal discourse.
在这里，我们研究了在理解伴有同话标志性手势的多模态语篇时，视空间工作记忆 (WM) 的作用。记录了健康成年人在编码网格上的一个 (低负载) 或四个 (高负载) 点位置序列并进行排练，直到稍后在试验中收集自由回忆反应。在 WM 任务的排练期间，参与者观察了说话者描述物体的视频，其中一半的试验包括语义相关的同话手势 (一致)，另一半包括语义不相关的手势 (不一致)。在观看视频时，α 和 β 波段的振荡 EEG 活动对语篇处理进行了索引。在所有参与者中，与言语和手势不一致的影响在低负载试验中比在高负载试验中更为明显。个体差异对视空间 WM 能力的影响也有所调节。这些数据表明，在理解多模态语篇时，需要调用视空间 WM 资源。
