Si Xiaopeng, He Huang, Yu Jiayue, Ming Dong
Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, People's Republic of China.
Tianjin Key Laboratory of Brain Science and Neural Engineering, Tianjin University, Tianjin 300072, People's Republic of China.
Cyborg Bionic Syst. 2023 Jul 27;4:0045. doi: 10.34133/cbsystems.0045. eCollection 2023.
Functional near-infrared spectroscopy (fNIRS) is a noninvasive brain imaging technique that has gradually been applied in emotion recognition research due to its advantages of high spatial resolution, real time, and convenience. However, the current research on emotion recognition based on fNIRS is mainly limited to within-subject, and there is a lack of related work on emotion recognition across subjects. Therefore, in this paper, we designed an emotion evoking experiment with videos as stimuli and constructed the fNIRS emotion recognition database. On this basis, deep learning technology was introduced for the first time, and a dual-branch joint network (DBJNet) was constructed, creating the ability to generalize the model to new participants. The decoding performance obtained by the proposed model shows that fNIRS can effectively distinguish positive versus neutral versus negative emotions (accuracy is 74.8%, F1 score is 72.9%), and the decoding performance on the 2-category emotion recognition task of distinguishing positive versus neutral (accuracy is 89.5%, F1 score is 88.3%), negative versus neutral (accuracy is 91.7%, F1 score is 91.1%) proved fNIRS has a powerful ability to decode emotions. Furthermore, the results of the ablation study of the model structure demonstrate that the joint convolutional neural network branch and the statistical branch achieve the highest decoding performance. The work in this paper is expected to facilitate the development of fNIRS affective brain-computer interface.
功能近红外光谱技术（fNIRS）是一种非侵入性脑成像技术，因其具有高空间分辨率、实时性和便利性等优点，已逐渐应用于情绪识别研究。然而，目前基于fNIRS的情绪识别研究主要局限于个体内部，缺乏跨个体情绪识别的相关工作。因此，在本文中，我们设计了一个以视频为刺激的情绪诱发实验，并构建了fNIRS情绪识别数据库。在此基础上，首次引入深度学习技术，构建了一个双分支联合网络（DBJNet），使模型能够推广到新的参与者。所提模型获得的解码性能表明，fNIRS能够有效区分积极情绪、中性情绪和消极情绪（准确率为74.8%，F1分数为72.9%），在区分积极与中性的二分类情绪识别任务上的解码性能（准确率为89.5%，F1分数为{88.3%}）以及消极与中性的二分类情绪识别任务上的解码性能（准确率为91.7%，F1分数为91.1%）证明了fNIRS具有强大的情绪解码能力。此外，模型结构的消融研究结果表明，联合卷积神经网络分支和统计分支实现了最高的解码性能。本文的工作有望促进fNIRS情感脑机接口的发展。
