Division of Communication and Auditory Neuroscience, House Ear Institute, Los Angeles, California, USA.
J Exp Psychol Hum Percept Perform. 2011 Aug;37(4):1193-209. doi: 10.1037/a0023100.
When the auditory and visual components of spoken audiovisual nonsense syllables are mismatched, perceivers produce four different types of perceptual responses, auditory correct, visual correct, fusion (the so-called McGurk effect), and combination (i.e., two consonants are reported). Here, quantitative measures were developed to account for the distribution of the four types of perceptual responses to 384 different stimuli from four talkers. The measures included mutual information, correlations, and acoustic measures, all representing audiovisual stimulus relationships. In Experiment 1, open-set perceptual responses were obtained for acoustic /bɑ/ or /lɑ/ dubbed to video /bɑ, dɑ, gɑ, vɑ, zɑ, lɑ, wɑ, ðɑ/. The talker, the video syllable, and the acoustic syllable significantly influenced the type of response. In Experiment 2, the best predictors of response category proportions were a subset of the physical stimulus measures, with the variance accounted for in the perceptual response category proportions between 17% and 52%. That audiovisual stimulus relationships can account for perceptual response distributions supports the possibility that internal representations are based on modality-specific stimulus relationships.
当口语视听无意义音节的听觉和视觉成分不匹配时，感知者会产生四种不同类型的感知反应，即听觉正确、视觉正确、融合（所谓的麦格克效应）和组合（即报告两个辅音）。在这里，开发了定量措施来解释对来自四个说话者的 384 个不同刺激的四种感知反应的分布。这些措施包括互信息、相关性和声学措施，它们都代表视听刺激关系。在实验 1 中，为视频 /bɑ、dɑ、gɑ、vɑ、zɑ、lɑ、wɑ、ðɑ/ 配音的声学 /bɑ/ 或 /lɑ/ 获得了开放式感知反应。说话者、视频音节和声学音节对反应类型有显著影响。在实验 2 中，反应类别比例的最佳预测因子是物理刺激测量的一个子集，感知反应类别比例的方差在 17%到 52%之间。视听刺激关系可以解释感知反应分布，这支持了内部表示基于模态特定刺激关系的可能性。
