He Zhipeng, Li Zina, Yang Fuzhou, Wang Lei, Li Jingcong, Zhou Chengju, Pan Jiahui
School of Software, South China Normal University, Foshan 528225, China.
School of Computer, South China Normal University, Guangzhou 510641, China.
Brain Sci. 2020 Sep 29;10(10):687. doi: 10.3390/brainsci10100687.
With the continuous development of portable noninvasive human sensor technologies such as brain-computer interfaces (BCI), multimodal emotion recognition has attracted increasing attention in the area of affective computing. This paper primarily discusses the progress of research into multimodal emotion recognition based on BCI and reviews three types of multimodal affective BCI (aBCI): aBCI based on a combination of behavior and brain signals, aBCI based on various hybrid neurophysiology modalities and aBCI based on heterogeneous sensory stimuli. For each type of aBCI, we further review several representative multimodal aBCI systems, including their design principles, paradigms, algorithms, experimental results and corresponding advantages. Finally, we identify several important issues and research directions for multimodal emotion recognition based on BCI.
随着脑机接口（BCI）等便携式无创人体传感技术的不断发展，多模态情感识别在情感计算领域受到了越来越多的关注。本文主要探讨基于BCI的多模态情感识别研究进展，并综述了三种类型的多模态情感BCI（aBCI）：基于行为和脑信号组合的aBCI、基于各种混合神经生理学模态的aBCI以及基于异构感觉刺激的aBCI。对于每种类型的aBCI，我们进一步综述了几个具有代表性的多模态aBCI系统，包括它们的设计原则、范式、算法、实验结果及相应优势。最后，我们确定了基于BCI的多模态情感识别的几个重要问题和研究方向。
