Wang Yuwei, Zeng Yi
Research Center for Brain-inspired Intelligence, Institute of Automation, Chinese Academy of Sciences, Beijing, China.
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China.
Front Syst Neurosci. 2022 May 12;16:845177. doi: 10.3389/fnsys.2022.845177. eCollection 2022.
Concept learning highly depends on multisensory integration. In this study, we propose a multisensory concept learning framework based on brain-inspired spiking neural networks to create integrated vectors relying on the concept's perceptual strength of auditory, gustatory, haptic, olfactory, and visual. With different assumptions, two paradigms: Independent Merge (IM) and Associate Merge (AM) are designed in the framework. For testing, we employed eight distinct neural models and three multisensory representation datasets. The experiments show that integrated vectors are closer to human beings than the non-integrated ones. Furthermore, we systematically analyze the similarities and differences between IM and AM paradigms and validate the generality of our framework.
概念学习高度依赖多感官整合。在本研究中，我们提出了一种基于受大脑启发的脉冲神经网络的多感官概念学习框架，以根据概念在听觉、味觉、触觉、嗅觉和视觉方面的感知强度创建整合向量。基于不同假设，在该框架中设计了两种范式：独立合并（IM）和关联合并（AM）。为了进行测试，我们采用了八个不同的神经模型和三个多感官表征数据集。实验表明，整合向量比未整合的向量更接近人类。此外，我们系统地分析了IM和AM范式之间的异同，并验证了我们框架的通用性。
