Department of Language and Linguistic Science, University of York, York YO10 5DD, UK.
Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands.
Philos Trans R Soc Lond B Biol Sci. 2023 Apr 24;378(1875):20210473. doi: 10.1098/rstb.2021.0473. Epub 2023 Mar 6.
Human communicative interaction is characterized by rapid and precise turn-taking. This is achieved by an intricate system that has been elucidated in the field of conversation analysis, based largely on the study of the auditory signal. This model suggests that transitions occur at points of possible completion identified in terms of linguistic units. Despite this, considerable evidence exists that visible bodily actions including gaze and gestures also play a role. To reconcile disparate models and observations in the literature, we combine qualitative and quantitative methods to analyse turn-taking in a corpus of multimodal interaction using eye-trackers and multiple cameras. We show that transitions seem to be inhibited when a speaker averts their gaze at a point of possible turn completion, or when a speaker produces gestures which are beginning or unfinished at such points. We further show that while the direction of a speaker's gaze does not affect the speed of transitions, the production of manual gestures does: turns with gestures have faster transitions. Our findings suggest that the coordination of transitions involves not only linguistic resources but also visual gestural ones and that the transition-relevance places in turns are multimodal in nature. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.
人类交际互动的特点是快速而准确的轮流发言。这是通过一个复杂的系统实现的，该系统在很大程度上是基于会话分析领域的研究，主要基于对听觉信号的研究。该模型表明，转换发生在根据语言单位确定的可能完成点处。尽管如此，仍有大量证据表明，包括目光和手势在内的可见身体动作也发挥了作用。为了调和文献中不同的模型和观察结果，我们结合定性和定量方法，使用眼动追踪器和多个摄像头分析了使用多模态交互的语料库中的轮流发言。我们表明，当说话者在可能的轮次完成点回避目光时，或者当说话者在这些点开始或未完成手势时，转换似乎会受到抑制。我们进一步表明，虽然说话者的目光方向不会影响转换的速度，但手动手势的产生会影响转换的速度：带有手势的轮次具有更快的转换速度。我们的研究结果表明，转换的协调不仅涉及语言资源，还涉及视觉手势资源，并且轮次中的转换相关位置在本质上是多模态的。本文是“面对面：推进社会互动科学”讨论会议的一部分。
