Mazzi Giulia, Ferrari Ambra, Mencaroni Maria Laura, Valzolgher Chiara, Tommasini Mirko, Pavani Francesco, Benetti Stefania
Center for Mind/Brain Sciences (CIMeC), University of Trento, Rovereto, Italy.
Max Plank Institute for Psycholinguistics, Nijmegen, The Netherlands.
PLoS Comput Biol. 2025 Sep 12;21(9):e1013468. doi: 10.1371/journal.pcbi.1013468. eCollection 2025 Sep.
Face-to-face communication relies on the seamless integration of multisensory signals, including voice, gaze, and head movements, to convey meaning effectively. This poses a fundamental computational challenge: optimally binding signals sharing the same communicative intention (e.g., looking at the addressee while speaking) and segregating unrelated signals (e.g., looking away while coughing), all within the rapid turn-taking dynamics of conversation. Critically, the computational mechanisms underlying this extraordinary feat remain largely unknown. Here, we cast face-to-face communication as a Bayesian Causal Inference problem to formally test whether prior expectations arbitrate between the integration and segregation of vocal and bodily signals. Specifically, we asked whether there is a stronger prior tendency to integrate audiovisual signals that convey the same communicative intention, thus establishing a crossmodal pragmatic correspondence. Additionally, we evaluated whether observers solve causal inference by adopting optimal Bayesian decision strategies or non-optimal approximate heuristics. In a spatial localization task, participants watched audiovisual clips of a speaker where the audio (voice) and the video (bodily cues) were sampled either from congruent positions or at increasing spatial disparities. Crucially, we manipulated the pragmatic correspondence of the signals: in a communicative condition, the speaker addressed the participant with their head, gaze and speech; in a non-communicative condition, the speaker kept the head down and produced a meaningless vocalization. We measured audiovisual integration through the ventriloquist effect, which quantifies how much the perceived audio position is misplaced towards the video position. Combining psychophysics with computational modelling, we show that observers solved audiovisual causal inference using non-optimal heuristics that nevertheless approximate optimal Bayesian inference with high accuracy. Remarkably, participants showed a stronger tendency to integrate vocal and bodily information when signals conveyed congruent communicative intent, suggesting that pragmatic correspondences enhance multisensory integration. Collectively, our findings provide novel and compelling evidence that face-to-face communication is shaped by deeply ingrained expectations about how multisensory signals should be structured and interpreted.
面对面交流依赖于多感官信号的无缝整合，包括语音、目光和头部动作，以便有效地传达意义。这带来了一个基本的计算挑战：在对话快速的轮流交替动态中，最佳地绑定具有相同交流意图的信号（例如，说话时看着对方），并区分不相关的信号（例如，咳嗽时看向别处）。至关重要的是，这一非凡能力背后的计算机制在很大程度上仍然未知。在这里，我们将面对面交流视为一个贝叶斯因果推理问题，以正式测试先验期望是否在声音和身体信号的整合与区分之间起仲裁作用。具体而言，我们询问是否存在更强的先验倾向来整合传达相同交流意图的视听信号，从而建立跨模态语用对应关系。此外，我们评估了观察者是通过采用最优贝叶斯决策策略还是非最优近似启发式方法来解决因果推理问题。在一个空间定位任务中，参与者观看说话者的视听片段，其中音频（语音）和视频（身体线索）要么从一致的位置采样，要么在空间差异不断增大的情况下采样。关键的是，我们操纵了信号的语用对应关系：在交流条件下，说话者用头部、目光和言语与参与者交流；在非交流条件下，说话者低着头发出无意义的发声。我们通过口技效应来测量视听整合，口技效应量化了感知到的音频位置向视频位置偏移的程度。将心理物理学与计算建模相结合，我们表明观察者使用非最优启发式方法解决视听因果推理问题，然而这种方法能高精度地近似最优贝叶斯推理。值得注意的是，当信号传达一致的交流意图时，参与者表现出更强的整合声音和身体信息的倾向，这表明语用对应关系增强了多感官整合。总体而言，我们的研究结果提供了新颖且有说服力的证据，表明面对面交流是由对多感官信号应如何构建和解释的根深蒂固的期望所塑造的。
