Cano Porras Desiderio, Louwerse Max M
Department of Cognitive Science and Artificial Intelligence, Tilburg University, the Netherlands.
Department of Cognitive Science and Artificial Intelligence, Tilburg University, the Netherlands.
Cognition. 2025 Mar;256:106047. doi: 10.1016/j.cognition.2024.106047. Epub 2024 Dec 25.
Making eye contact with our conversational partners is what is most common in multimodal communication. Yet, little is known about this behavior. Prior studies have reported different findings on what we look at in the narrator's face. Some studies show eye gaze is usually focused on our conversational partner's eyes, other studies have shown evidence for eye gaze primarily on the narrator's mouth, and yet others find evidence for fixations on the narrator's nose bridge perhaps as a transition for eye gaze between the eyes and mouth. The current study aimed to shed light on these different findings by investigating eye gaze on a narrator's face in a fixed cognitive task. Experiment 1 monitored participants' eye gaze when looking at videos of a male and female human narrator. Experiment 2 used a virtual human, allowing manipulation of different parts of the narrator's face to validate the findings in Experiment 1. Gaze behavior on the human faces (Experiment 1) and the virtual human face (Experiment 2) of the narrator was similar, with the narrator's eyes attracting most fixations seemingly serving as an anchor for communication, particularly at the start and the end of a conversation. The mouth, in turn, served as a communicative cue when eye contact has been established. When lip movements were impaired in the virtual human, the eyes immediately took over as the anchor again. These findings can be explained by the theoretical framework of action ladders in multimodal language use. They shed light on cognitive and social psychological aspects of human-human multimodal communication, both in human and embodied conversational agents.
与对话伙伴进行眼神交流是多模态交流中最常见的行为。然而，我们对这种行为却知之甚少。先前的研究报告了关于我们在叙述者脸上注视部位的不同发现。一些研究表明，目光通常聚焦在对话伙伴的眼睛上；另一些研究则表明，目光主要集中在叙述者的嘴上；还有一些研究发现，目光会固定在叙述者的鼻梁上，这可能是目光在眼睛和嘴巴之间转换的过渡。当前的研究旨在通过在一项固定的认知任务中调查对叙述者面部的目光注视，来阐明这些不同的发现。实验1监测了参与者观看男性和女性人类叙述者视频时的目光注视情况。实验2使用了虚拟人，通过操纵叙述者面部的不同部位来验证实验1的结果。叙述者的人脸（实验1）和虚拟人脸（实验2）上的注视行为相似，叙述者的眼睛吸引了大多数注视，似乎是交流的锚点，尤其是在对话开始和结束时。反过来，当眼神交流建立后，嘴巴则作为一种交流线索。当虚拟人的嘴唇动作受损时，眼睛会立即再次成为锚点。这些发现可以用多模态语言使用中的动作阶梯理论框架来解释。它们揭示了人与人多模态交流在人类和具身对话代理中的认知和社会心理方面。
