School of Computer Science and Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin 300350, PR China.
School of Computer Science and Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin 300350, PR China; State Key Laboratory of Intelligent Technology and Systems, National Laboratory for Information Science and Technology, Tsinghua University, Beijing 100084, PR China.
Neuroscience. 2018 May 1;377:12-25. doi: 10.1016/j.neuroscience.2018.01.051. Epub 2018 Feb 3.
Semantically congruent sounds can facilitate perception of visual objects in the human brain. However, the manner in which semantically congruent sounds affect cognitive processing for degraded visual stimuli remains unclear. We presented participants with naturalistic degraded images and semantically congruent sounds from different conceptual categories in three modalities: degraded visual only, auditory only, and auditory and degraded visual. Functional magnetic resonance imaging was performed to assess variations in brain-activation spatial patterns. In order to account for the facilitation of auditory modulation at different levels, four conceptual categories of stimuli were divided into coarse and fine groups. Conjunction analysis and multivariate pattern analysis were used to investigate integrative properties. Superadditive interactions were found in the visual association cortex and subadditive interactions were observed in the superior temporal sulcus/superior temporal gyrus (STS/STG). Our results demonstrate that the visual association cortex and STS/STG are involved in the integration of auditory and degraded visual information. In addition, the pattern classification results imply that semantically congruent sounds may facilitate identification of degraded images in both coarse and fine groups. Importantly, when naturalistic visual stimuli were further subdivided, facilitation through auditory modulation exhibited category selectivity.
语义一致的声音可以促进人类大脑对视觉对象的感知。然而，语义一致的声音如何影响对退化视觉刺激的认知加工尚不清楚。我们在三种模态下向参与者呈现了自然退化的图像和来自不同概念类别的语义一致的声音：仅退化的视觉、仅听觉和听觉与退化的视觉。功能性磁共振成像用于评估大脑激活空间模式的变化。为了说明不同水平上听觉调制的促进作用，将四个概念类别刺激分为粗分组和细分组。使用联合分析和多元模式分析来研究综合特性。在视觉联合皮层中发现了超相加相互作用，在颞上回/颞上回（STS/STG）中观察到了次相加相互作用。我们的结果表明，视觉联合皮层和 STS/STG 参与了听觉和退化视觉信息的整合。此外，模式分类结果表明，语义一致的声音可能有助于识别粗分组和细分组中的退化图像。重要的是，当进一步细分自然视觉刺激时，通过听觉调制的促进作用表现出了类别选择性。
