Center for Brain and Cognition, Universitat Pompeu Fabra, Barcelona, Spain.
Humboldt-Universität zu Berlin, Berlin, Germany.
Hum Brain Mapp. 2024 Aug 1;45(11):e26797. doi: 10.1002/hbm.26797.
Speech comprehension is crucial for human social interaction, relying on the integration of auditory and visual cues across various levels of representation. While research has extensively studied multisensory integration (MSI) using idealised, well-controlled stimuli, there is a need to understand this process in response to complex, naturalistic stimuli encountered in everyday life. This study investigated behavioural and neural MSI in neurotypical adults experiencing audio-visual speech within a naturalistic, social context. Our novel paradigm incorporated a broader social situational context, complete words, and speech-supporting iconic gestures, allowing for context-based pragmatics and semantic priors. We investigated MSI in the presence of unimodal (auditory or visual) or complementary, bimodal speech signals. During audio-visual speech trials, compared to unimodal trials, participants more accurately recognised spoken words and showed a more pronounced suppression of alpha power-an indicator of heightened integration load. Importantly, on the neural level, these effects surpassed mere summation of unimodal responses, suggesting non-linear MSI mechanisms. Overall, our findings demonstrate that typically developing adults integrate audio-visual speech and gesture information to facilitate speech comprehension in noisy environments, highlighting the importance of studying MSI in ecologically valid contexts.
言语理解对于人类的社会互动至关重要，它依赖于在不同的表示层面上对听觉和视觉线索进行整合。虽然研究已经广泛地使用理想化的、受严格控制的刺激来研究多感觉整合（MSI），但我们需要理解在日常生活中遇到复杂的自然刺激时这一过程是如何进行的。本研究在自然的社会背景下，调查了神经正常的成年人在听到视觉言语时的行为和神经多感觉整合。我们的新范式结合了更广泛的社会情境背景、完整的单词和支持言语的图标手势，允许基于语境的语用学和语义先验。我们在存在单一模式（听觉或视觉）或互补的双重模式语音信号的情况下，调查了 MSI。与单一模式试验相比，在视听语音试验中，参与者更准确地识别出所说的单词，并且表现出更明显的α波功率抑制——这是整合负荷增加的指标。重要的是，在神经水平上，这些效应超过了单一模式反应的简单叠加，表明存在非线性的 MSI 机制。总的来说，我们的发现表明，正常发育的成年人整合听觉、视觉言语和手势信息，以促进嘈杂环境下的言语理解，突出了在生态有效背景下研究 MSI 的重要性。
