Research Center for Applied Mathematics and Machine Intelligence, Research Institute of Basic Theories, Zhejiang Lab, Hangzhou 311121, China.
Research Center for Applied Mathematics and Machine Intelligence, Research Institute of Basic Theories, Zhejiang Lab, Hangzhou 311121, China; Key Laboratory for Biomedical Engineering of Ministry of Education, College of Biomedical Engineering and Instrument Sciences, Zhejiang University, Hangzhou 310027, China; The MOE Frontier Science Center for Brain Science & Brain-machine Integration, Zhejiang University, Hangzhou 310012, China.
Neuroimage. 2022 Jul 15;255:119182. doi: 10.1016/j.neuroimage.2022.119182. Epub 2022 Apr 5.
Natural scenes contain multi-modal information, which is integrated to form a coherent perception. Previous studies have demonstrated that cross-modal information can modulate neural encoding of low-level sensory features. These studies, however, mostly focus on the processing of single sensory events or rhythmic sensory sequences. Here, we investigate how the neural encoding of basic auditory and visual features is modulated by cross-modal information when the participants watch movie clips primarily composed of non-rhythmic events. We presented audiovisual congruent and audiovisual incongruent movie clips, and since attention can modulate cross-modal interactions, we separately analyzed high- and low-arousal movie clips. We recorded neural responses using electroencephalography (EEG), and employed the temporal response function (TRF) to quantify the neural encoding of auditory and visual features. The neural encoding of sound envelope is enhanced in the audiovisual congruent condition than the incongruent condition, but this effect is only significant for high-arousal movie clips. In contrast, audiovisual congruency does not significantly modulate the neural encoding of visual features, e.g., luminance or visual motion. In summary, our findings demonstrate asymmetrical cross-modal interactions during the processing of natural scenes that lack rhythmicity: Congruent visual information enhances low-level auditory processing, while congruent auditory information does not significantly modulate low-level visual processing.
自然场景包含多模态信息，这些信息被整合在一起形成连贯的感知。先前的研究表明，跨模态信息可以调节低水平感觉特征的神经编码。然而，这些研究大多集中在单个感觉事件或有节奏的感觉序列的处理上。在这里，当参与者观看主要由非节奏事件组成的电影片段时，我们研究了跨模态信息如何调节基本听觉和视觉特征的神经编码。我们呈现了视听一致和视听不一致的电影片段，由于注意力可以调节跨模态相互作用，我们分别分析了高唤醒度和低唤醒度的电影片段。我们使用脑电图 (EEG) 记录神经反应，并采用时间响应函数 (TRF) 来量化听觉和视觉特征的神经编码。与不一致条件相比，在视听一致条件下，声音包络的神经编码得到增强，但这种效应仅在高唤醒度的电影片段中显著。相比之下，视听一致性并不显著调节视觉特征的神经编码，例如亮度或视觉运动。总之，我们的发现表明，在处理缺乏节奏的自然场景时，存在不对称的跨模态相互作用：一致的视觉信息增强了低水平的听觉处理，而一致的听觉信息并没有显著调节低水平的视觉处理。
