Maguinness Corrina, Schall Sonja, Mathias Brian, Schoemann Martin, von Kriegstein Katharina
Chair of Cognitive and Clinical Neuroscience, Faculty of Psychology, Technische Universität Dresden, Dresden, Germany.
Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.
Q J Exp Psychol (Hove). 2024 Sep 20;78(7):17470218241278649. doi: 10.1177/17470218241278649.
Seeing the visual articulatory movements of a speaker, while hearing their voice, helps with understanding what is said. This multisensory enhancement is particularly evident in noisy listening conditions. Multisensory enhancement also occurs even in auditory-only conditions: auditory-only speech and voice-identity recognition are superior for speakers previously learned with their face, compared to control learning; an effect termed the "face-benefit." Whether the face-benefit can assist in maintaining robust perception in increasingly noisy listening conditions, similar to concurrent multisensory input, is unknown. Here, in two behavioural experiments, we examined this hypothesis. In each experiment, participants learned a series of speakers' voices together with their dynamic face or control image. Following learning, participants listened to auditory-only sentences spoken by the same speakers and recognised the content of the sentences (speech recognition, Experiment 1) or the voice-identity of the speaker (Experiment 2) in increasing levels of auditory noise. For speech recognition, we observed that 14 of 30 participants (47%) showed a face-benefit. 19 of 25 participants (76%) showed a face-benefit for voice-identity recognition. For those participants who demonstrated a face-benefit, the face-benefit increased with auditory noise levels. Taken together, the results support an audio-visual model of auditory communication and suggest that the brain can develop a flexible system in which learned facial characteristics are used to deal with varying auditory uncertainty.
在听说话者声音的同时看到其视觉发音动作，有助于理解所说内容。这种多感官增强在嘈杂的聆听环境中尤为明显。多感官增强甚至在仅听觉条件下也会发生：与对照学习相比，对于之前通过面部学习过的说话者，仅听觉的语音和语音识别能力更强；这种效应被称为“面部优势”。面部优势是否能像同时进行的多感官输入那样，在日益嘈杂的聆听环境中帮助维持强大的感知能力，目前尚不清楚。在此，我们通过两项行为实验对这一假设进行了检验。在每个实验中，参与者将一系列说话者的声音与其动态面部或对照图像一起学习。学习之后，参与者听取由相同说话者说出的仅听觉句子，并在不断增加的听觉噪声水平下识别句子内容（语音识别，实验1）或说话者的语音身份（实验2）。对于语音识别，我们观察到30名参与者中有14名（47%）表现出面部优势。25名参与者中有19名（76%）在语音身份识别方面表现出面部优势。对于那些表现出面部优势的参与者，面部优势随着听觉噪声水平的增加而增强。综上所述，这些结果支持了听觉交流的视听模型，并表明大脑可以发展出一个灵活的系统，其中学习到的面部特征被用于应对不同的听觉不确定性。
