Language Science and Technology, Campus C7, Saarland University, 66123 Saarbrücken, Germany; Cluster of Excellence Multimodal Computing and Interaction (MMCI), Campus E1.7, Saarland University, 66123 Saarbrücken, Germany.
Language Science and Technology, Campus C7, Saarland University, 66123 Saarbrücken, Germany; Cluster of Excellence Multimodal Computing and Interaction (MMCI), Campus E1.7, Saarland University, 66123 Saarbrücken, Germany.
Brain Cogn. 2019 Oct;135:103571. doi: 10.1016/j.bandc.2019.05.009. Epub 2019 Jun 12.
Behavioral studies have shown that speaker gaze to objects in a co-present scene can influence listeners' sentence comprehension. To gain deeper insight into the mechanisms involved in gaze processing and integration, we conducted two ERP experiments (N = 30, Age: [18, 32] and [19, 33] respectively). Participants watched a centrally positioned face performing gaze actions aligned to utterances comparing two out of three displayed objects. They were asked to judge whether the sentence was true given the provided scene. We manipulated the second gaze cue to be either Congruent (baseline), Incongruent or Averted (Exp1)/Mutual (Exp2). When speaker gaze is used to form lexical expectations about upcoming referents, we found an attenuated N200 when phonological information confirms these expectations (Congruent). Similarly, we observed attenuated N400 amplitudes when gaze-cued expectations (Congruent) facilitate lexical retrieval. Crucially, only a violation of gaze-cued lexical expectations (Incongruent) leads to a P600 effect, suggesting the necessity to revise the mental representation of the situation. Our results support the hypothesis that gaze is utilized above and beyond simply enhancing a cued object's prominence. Rather, gaze to objects leads to their integration into the mental representation of the situation before they are mentioned.
行为研究表明，说话者在共同呈现场景中注视物体的行为可以影响听众对句子的理解。为了更深入地了解注视处理和整合所涉及的机制，我们进行了两项 ERP 实验（N=30，年龄分别为[18，32]和[19，33]）。参与者观看一个位于中央的面孔做出与话语对齐的注视动作，这些话语比较了三个显示对象中的两个。他们被要求根据提供的场景判断句子是否正确。我们操纵第二个注视线索使其与语音信息一致（基线）、不一致或偏离（Exp1）/相互（Exp2）。当说话者的注视用于形成对即将到来的指称的词汇期望时，当语音信息证实这些期望时，我们发现 N200 被削弱（一致）。同样，当注视提示的期望（一致）促进词汇检索时，我们观察到 N400 振幅的减弱。至关重要的是，只有注视提示的词汇期望被违反（不一致）才会导致 P600 效应，这表明有必要修改情况的心理表征。我们的研究结果支持这样一种假设，即注视不仅增强了被提示对象的突出性，而且还将其整合到情况的心理表征中，然后才提及这些对象。
