Macuch Silva Vinicius, Holler Judith, Ozyurek Asli, Roberts Seán G
Institute of Cognitive Science, Osnabrück University, Osnabrück, Germany.
Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands.
R Soc Open Sci. 2020 Jan 15;7(1):182056. doi: 10.1098/rsos.182056. eCollection 2020 Jan.
Face-to-face communication is multimodal at its core: it consists of a combination of vocal and visual signalling. However, current evidence suggests that, in the absence of an established communication system, visual signalling, especially in the form of visible gesture, is a more powerful form of communication than vocalization and therefore likely to have played a primary role in the emergence of human language. This argument is based on experimental evidence of how vocal and visual modalities (i.e. gesture) are employed to communicate about familiar concepts when participants cannot use their existing languages. To investigate this further, we introduce an experiment where pairs of participants performed a referential communication task in which they described unfamiliar stimuli in order to reduce reliance on conventional signals. Visual and auditory stimuli were described in three conditions: using visible gestures only, using non-linguistic vocalizations only and given the option to use both (multimodal communication). The results suggest that even in the absence of conventional signals, gesture is a more powerful mode of communication compared with vocalization, but that there are also advantages to multimodality compared to using gesture alone. Participants with an option to produce multimodal signals had comparable accuracy to those using only gesture, but gained an efficiency advantage. The analysis of the interactions between participants showed that interactants developed novel communication systems for unfamiliar stimuli by deploying different modalities flexibly to suit their needs and by taking advantage of multimodality when required.
它由声音信号和视觉信号组合而成。然而，目前的证据表明，在没有既定交流系统的情况下，视觉信号，尤其是以可见手势的形式，是比发声更强大的交流形式，因此很可能在人类语言的出现中发挥了主要作用。这一论点基于当参与者无法使用其现有语言时，声音和视觉模态（即手势）如何被用于交流熟悉概念的实验证据。为了进一步研究这一点，我们引入了一个实验，在该实验中，参与者成对执行一个指称性交流任务，在任务中他们描述不熟悉的刺激，以减少对传统信号的依赖。在三种条件下描述视觉和听觉刺激：仅使用可见手势、仅使用非语言发声以及可以选择同时使用两者（多模态交流）。结果表明，即使在没有传统信号的情况下，与发声相比，手势也是一种更强大的交流方式，但与仅使用手势相比，多模态也有优势。可以产生多模态信号的参与者与仅使用手势的参与者具有相当的准确性，但获得了效率优势。对参与者之间互动的分析表明，互动者通过灵活运用不同模态以满足自身需求，并在需要时利用多模态，为不熟悉的刺激开发了新颖的交流系统。
