Maquate Katja, Knoeferle Pia
Psycholinguistics, Institute for German Language and Linguistics, Humboldt-Universität zu Berlin, Berlin, Germany.
Berlin School of Mind and Brain, Humboldt-Universität zu Berlin, Berlin, Germany.
Front Psychol. 2021 Aug 2;12:547360. doi: 10.3389/fpsyg.2021.547360. eCollection 2021.
Research findings on language comprehension suggest that many kinds of non-linguistic cues can rapidly affect language processing. Extant processing accounts of situated language comprehension model these rapid effects and are only beginning to accommodate the role of non-linguistic emotional, cues. To begin with a detailed characterization of distinct cues and their relative effects, three visual-world eye-tracking experiments assessed the relative importance of two cue types (action depictions vs. emotional facial expressions) as well as the effects of the degree of naturalness of social (facial) cues (smileys vs. natural faces). We predicted to replicate previously reported rapid effects of referentially mediated actions. In addition, we assessed distinct world-language relations. If how a cue is conveyed matters for its effect, then a verb referencing an action depiction should elicit a stronger immediate effect on visual attention and language comprehension than a speaker's emotional facial expression. The latter is mediated non-referentially the emotional connotations of an adverb. The results replicated a pronounced facilitatory effect of action depiction (relative to no action depiction). By contrast, the facilitatory effect of a preceding speaker's emotional face was less pronounced. How the facial emotion was rendered mattered in that the emotional face effect was present with natural faces (Experiment 2) but not with smileys (Experiment 1). Experiment 3 suggests that contrast, i.e., strongly opposing emotional valence information vs. non-opposing valence information, might matter for the directionality of this effect. These results are the first step toward a more principled account of how distinct visual (social) cues modulate language processing, whereby the visual cues that are referenced by language (the depicted action), copresent (the depicted action), and more natural (the natural emotional prime face) tend to exert more pronounced effects.
关于语言理解的研究结果表明，多种非语言线索能够迅速影响语言处理。现有的情境语言理解处理模型解释了这些快速影响，并且才刚刚开始考虑非语言情感线索的作用。为了详细描述不同线索及其相对影响，我们进行了三项视觉世界眼动追踪实验，评估了两种线索类型（动作描绘与情感面部表情）的相对重要性，以及社交（面部）线索自然程度（笑脸与自然面孔）的影响。我们预计会重现先前报道的参照介导动作的快速影响。此外，我们评估了不同的世界 - 语言关系。如果线索的传达方式对其效果有影响，那么一个指代动作描绘的动词应该比说话者的情感面部表情对视觉注意力和语言理解产生更强的即时影响。后者是通过副词的情感内涵进行非参照介导的。结果重现了动作描绘（相对于无动作描绘）的显著促进作用。相比之下，先前说话者情感面孔的促进作用不太明显。面部情感的呈现方式很重要，因为情感面孔效应在自然面孔（实验2）中存在，而在笑脸（实验1）中不存在。实验3表明，对比，即强烈对立的情感效价信息与非对立的效价信息，可能对这种效应的方向性很重要。这些结果是朝着更有原则地解释不同视觉（社交）线索如何调节语言处理迈出的第一步，由此语言所指代的视觉线索（所描绘的动作）、同时呈现的线索（所描绘的动作）以及更自然的线索（自然情感启动面孔）往往会产生更显著的影响。
