Center for Brain Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou, China.
PLoS One. 2011;6(6):e20801. doi: 10.1371/journal.pone.0020801. Epub 2011 Jun 29.
One of the central questions in cognitive neuroscience is the precise neural representation, or brain pattern, associated with a semantic category. In this study, we explored the influence of audiovisual stimuli on the brain patterns of concepts or semantic categories through a functional magnetic resonance imaging (fMRI) experiment. We used a pattern search method to extract brain patterns corresponding to two semantic categories: "old people" and "young people." These brain patterns were elicited by semantically congruent audiovisual, semantically incongruent audiovisual, unimodal visual, and unimodal auditory stimuli belonging to the two semantic categories. We calculated the reproducibility index, which measures the similarity of the patterns within the same category. We also decoded the semantic categories from these brain patterns. The decoding accuracy reflects the discriminability of the brain patterns between two categories. The results showed that both the reproducibility index of brain patterns and the decoding accuracy were significantly higher for semantically congruent audiovisual stimuli than for unimodal visual and unimodal auditory stimuli, while the semantically incongruent stimuli did not elicit brain patterns with significantly higher reproducibility index or decoding accuracy. Thus, the semantically congruent audiovisual stimuli enhanced the within-class reproducibility of brain patterns and the between-class discriminability of brain patterns, and facilitate neural representations of semantic categories or concepts. Furthermore, we analyzed the brain activity in superior temporal sulcus and middle temporal gyrus (STS/MTG). The strength of the fMRI signal and the reproducibility index were enhanced by the semantically congruent audiovisual stimuli. Our results support the use of the reproducibility index as a potential tool to supplement the fMRI signal amplitude for evaluating multimodal integration.
认知神经科学的核心问题之一是与语义类别相关的精确神经表示或大脑模式。在这项研究中，我们通过功能磁共振成像 (fMRI) 实验探讨了视听刺激对概念或语义类别的大脑模式的影响。我们使用模式搜索方法提取了两个语义类别“老年人”和“年轻人”的大脑模式。这些大脑模式是由属于这两个语义类别的语义一致的视听、语义不一致的视听、单模态视觉和单模态听觉刺激引起的。我们计算了可重复性指数，该指数衡量同一类别中模式的相似性。我们还从这些大脑模式中解码了语义类别。解码准确率反映了两个类别之间大脑模式的可区分性。结果表明，与单模态视觉和单模态听觉刺激相比，语义一致的视听刺激的大脑模式可重复性指数和解码准确率都显著更高，而语义不一致的刺激则不能引起具有更高可重复性指数或解码准确率的大脑模式。因此，语义一致的视听刺激增强了大脑模式的类内可重复性和大脑模式的类间可区分性，促进了语义类别或概念的神经表示。此外，我们分析了颞上回和中颞回 (STS/MTG) 的大脑活动。语义一致的视听刺激增强了 fMRI 信号的强度和可重复性指数。我们的结果支持将可重复性指数用作补充 fMRI 信号幅度的潜在工具，以评估多模态整合。
