Academy of Medical Engineering and Translational Medicine, State Key Laboratory of Advanced Medical Materials and Devices, Haihe Laboratory of Brain-computer Interaction and Human-machine Integration, Tianjin Key Laboratory of Brain Science and Neural Engineering, Institute of Applied Psychology, Tianjin University, Tianjin 300072, China.
School of Biomedical Engineering, Medical School, Shenzhen University, Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen 518060, China.
Comput Biol Med. 2024 Oct;181:108973. doi: 10.1016/j.compbiomed.2024.108973. Epub 2024 Aug 30.
Emotion recognition is crucial for human-computer interaction, and electroencephalography (EEG) stands out as a valuable tool for capturing and reflecting human emotions. In this study, we propose a hierarchical hybrid model called Mixed Attention-based Convolution and Transformer Network (MACTN). This model is designed to collectively capture both local and global temporal information and is inspired by insights from neuroscientific research on the temporal dynamics of emotions. First, we introduce depth-wise temporal convolution and separable convolution to extract local temporal features. Then, a self-attention-based transformer is used to integrate the sparse global emotional features. Besides, channel attention mechanism is designed to identify the most task-relevant channels, facilitating the capture of relationships between different channels and emotional states. Extensive experiments are conducted on three public datasets under both offline and online evaluation modes. In the multi-class cross-subject online evaluation using the THU-EP dataset, MACTN demonstrates an approximate 8% enhancement in 9-class emotion recognition accuracy in comparison to state-of-the-art methods. In the multi-class cross-subject offline evaluation using the DEAP and SEED datasets, a comparable performance is achieved solely based on the raw EEG signals, without the need for prior knowledge or transfer learning during the feature extraction and learning process. Furthermore, ablation studies have shown that integrating self-attention and channel-attention mechanisms improves classification performance. This method won the Emotional BCI Competition's final championship in the World Robot Contest. The source code is available at https://github.com/ThreePoundUniverse/MACTN.
情感识别对于人机交互至关重要，脑电图 (EEG) 作为捕捉和反映人类情感的有力工具脱颖而出。在这项研究中，我们提出了一种名为混合注意力卷积和 Transformer 网络 (MACTN) 的分层混合模型。该模型旨在共同捕捉局部和全局时间信息，并受到神经科学研究对情绪时间动态的见解的启发。首先，我们引入深度时间卷积和可分离卷积来提取局部时间特征。然后，使用基于自注意力的 Transformer 来整合稀疏的全局情感特征。此外，设计了通道注意力机制来识别最相关的任务通道，促进不同通道和情感状态之间的关系捕捉。在离线和在线评估模式下，我们在三个公共数据集上进行了广泛的实验。在使用 THU-EP 数据集的多类跨主体在线评估中，与最先进的方法相比，MACTN 在 9 类情感识别精度方面提高了约 8%。在使用 DEAP 和 SEED 数据集的多类跨主体离线评估中，仅基于原始 EEG 信号即可实现可比性能，而无需在特征提取和学习过程中事先了解或进行迁移学习。此外，消融研究表明，集成自注意力和通道注意力机制可以提高分类性能。该方法在世界机器人竞赛的情感脑机接口竞赛中获得了最终冠军。源代码可在 https://github.com/ThreePoundUniverse/MACTN 获得。
