Brain and Cognitive Sciences, University of Rochester, Rochester, NY, 14627, USA.
Del Monte Institute for Neuroscience, University of Rochester, Rochester, NY, 14627, USA.
Atten Percept Psychophys. 2022 Aug;84(6):2016-2026. doi: 10.3758/s13414-022-02440-3. Epub 2022 Feb 24.
It is well established that in order to comprehend speech in noisy environments, listeners use the face of the talker in conjunction with the auditory speech. Yet how listeners use audiovisual speech correspondences along the multisensory speech processing pathway is not known. We engaged listeners in a pair of experiments using face rotation to partially dissociate linguistic and temporal information and two tasks to assess both overall integration and early integration specifically. In our first exploratory experiment, listeners performed a speech in noise task to determine which face rotation maximally disrupts speech comprehension and thus overall audiovisual integration. Our second experiment involved a dual pitch discrimination and visual catch task to test specifically for binding. The results showed that temporal coherence supports early integration, replicating the importance of temporal coherence seen for binding nonspeech stimuli. However, the benefit of temporal coherence was present in both upright and inverted positions, suggesting that binding is minimally affected by face rotation under these conditions. Together, our results suggest that different aspects of audio-visual speech are integrated at different stages of multisensory speech processing.
众所周知，为了在嘈杂的环境中理解言语，听众会将说话者的面部与听觉言语结合起来使用。然而，听众如何沿着多感官言语处理途径使用视听言语对应关系尚不清楚。我们通过使用面部旋转来部分分离语言和时间信息，让听众参与了两项实验，以评估整体整合和早期整合。在我们的第一个探索性实验中，听众执行了一项噪声中的言语任务，以确定哪种面部旋转最大程度地破坏言语理解，从而整体视听整合。我们的第二个实验涉及双重音高辨别和视觉捕获任务，以专门测试绑定。结果表明，时间连贯性支持早期整合，复制了对于绑定非言语刺激物的时间连贯性重要性。然而，时间连贯性的优势在直立和倒置位置都存在，这表明在这些条件下，面部旋转对绑定的影响最小。总之，我们的结果表明，视听言语的不同方面在多感官言语处理的不同阶段进行整合。
