School of Optical-Electrical and Computer Engineer, University of Shanghai for Science and Technology, Shanghai 200093, China.
Sensors (Basel). 2023 Oct 23;23(20):8643. doi: 10.3390/s23208643.
In recent years, research on emotion recognition has become more and more popular, but there are few studies on emotion recognition based on cerebral blood oxygen signals. Since the electroencephalogram (EEG) is easily disturbed by eye movement and the portability is not high, this study uses a more comfortable and convenient functional near-infrared spectroscopy (fNIRS) system to record brain signals from participants while watching three different types of video clips. During the experiment, the changes in cerebral blood oxygen concentration in the 8 channels of the prefrontal cortex of the brain were collected and analyzed. We processed and divided the collected cerebral blood oxygen data, and used multiple classifiers to realize the identification of the three emotional states of joy, neutrality, and sadness. Since the classification accuracy of the convolutional neural network (CNN) in this research is not significantly superior to that of the XGBoost algorithm, this paper proposes a CNN-Transformer network based on the characteristics of time series data to improve the classification accuracy of ternary emotions. The network first uses convolution operations to extract channel features from multi-channel time series, then the features and the output information of the fully connected layer are input to the Transformer netork structure, and its multi-head attention mechanism is used to focus on different channel domain information, which has better spatiality. The experimental results show that the CNN-Transformer network can achieve 86.7% classification accuracy for ternary emotions, which is about 5% higher than the accuracy of CNN, and this provides some help for other research in the field of emotion recognition based on time series data such as fNIRS.
近年来，情感识别的研究越来越热门，但基于脑血氧信号的情感识别研究较少。由于脑电图（EEG）容易受到眼动的干扰，且便携性不高，本研究使用一种更舒适、方便的功能性近红外光谱（fNIRS）系统，在参与者观看三种不同类型的视频片段时记录大脑信号。在实验过程中，采集并分析了大脑前额叶 8 个通道的脑血氧浓度变化。我们对采集到的脑血氧数据进行了处理和划分，并使用多个分类器来实现对喜悦、中性和悲伤三种情绪状态的识别。由于卷积神经网络（CNN）在本研究中的分类准确率并不明显优于 XGBoost 算法，因此本文提出了一种基于时间序列数据特点的 CNN-Transformer 网络，以提高三分类情绪的分类准确率。该网络首先使用卷积操作从多通道时间序列中提取通道特征，然后将特征和全连接层的输出信息输入到 Transformer 网络结构中，其多头注意力机制用于关注不同通道域的信息，具有更好的空间性。实验结果表明，CNN-Transformer 网络对三分类情绪的分类准确率可达 86.7%，比 CNN 的准确率高出约 5%，这为基于 fNIRS 等时间序列数据的情感识别领域的其他研究提供了一些帮助。
