Kislyuk Daniel S, Möttönen Riikka, Sams Mikko
Laboratory of Computational Engineering, Helsinki University of Technology, Finland.
J Cogn Neurosci. 2008 Dec;20(12):2175-84. doi: 10.1162/jocn.2008.20152.
The interaction between auditory and visual speech streams is a seamless and surprisingly effective process. An intriguing example is the "McGurk effect": The acoustic syllable /ba/ presented simultaneously with a mouth articulating /ga/ is typically heard as /da/ [McGurk, H., & MacDonald, J. Hearing lips and seeing voices. Nature, 264, 746-748, 1976]. Previous studies have demonstrated the interaction of auditory and visual streams at the auditory cortex level, but the importance of these interactions for the qualitative perception change remained unclear because the change could result from interactions at higher processing levels as well. In our electroencephalogram experiment, we combined the McGurk effect with mismatch negativity (MMN), a response that is elicited in the auditory cortex at a latency of 100-250 msec by any above-threshold change in a sequence of repetitive sounds. An "odd-ball" sequence of acoustic stimuli consisting of frequent /va/ syllables (standards) and infrequent /ba/ syllables (deviants) was presented to 11 participants. Deviant stimuli in the unisensory acoustic stimulus sequence elicited a typical MMN, reflecting discrimination of acoustic features in the auditory cortex. When the acoustic stimuli were dubbed onto a video of a mouth constantly articulating /va/, the deviant acoustic /ba/ was heard as /va/ due to the McGurk effect and was indistinguishable from the standards. Importantly, such deviants did not elicit MMN, indicating that the auditory cortex failed to discriminate between the acoustic stimuli. Our findings show that visual stream can qualitatively change the auditory percept at the auditory cortex level, profoundly influencing the auditory cortex mechanisms underlying early sound discrimination.
听觉和视觉语音流之间的相互作用是一个无缝且惊人有效的过程。一个有趣的例子是“麦格克效应”：与口型发出/ga/同时呈现的声学音节/ba/通常会被听成/da/ [麦格克，H.，& 麦克唐纳，J. 听唇音和看口型。《自然》，264，746 - 748，1976]。先前的研究已经证明了听觉和视觉流在听觉皮层水平的相互作用，但这些相互作用对于定性感知变化的重要性仍不明确，因为这种变化也可能是由更高处理水平的相互作用导致的。在我们的脑电图实验中，我们将麦格克效应与失匹配负波（MMN）相结合，MMN是一种在听觉皮层中由重复声音序列中任何高于阈值的变化在100 - 250毫秒潜伏期引发的反应。由频繁出现的/va/音节（标准音）和不频繁出现的/ba/音节（偏差音）组成的“异常球”声学刺激序列被呈现给11名参与者。单感官声学刺激序列中的偏差刺激引发了典型的MMN，反映了听觉皮层对声学特征的辨别。当声学刺激被配到一个口型持续发出/va/的视频上时，由于麦格克效应，偏差声学音/ba/被听成/va/，并且与标准音无法区分。重要的是，这样的偏差音没有引发MMN，表明听觉皮层未能区分声学刺激。我们的研究结果表明，视觉流可以在听觉皮层水平上定性地改变听觉感知，深刻影响早期声音辨别背后的听觉皮层机制。
