Department of Computer Science, University of North Carolina, Charlotte, NC 28223, USA.
Department of Health Outcomes and Biomedical Informatics, University of Florida, Gainesville, FL 32611, USA.
Sensors (Basel). 2024 Mar 15;24(6):1889. doi: 10.3390/s24061889.
Analysis of brain signals is essential to the study of mental states and various neurological conditions. The two most prevalent noninvasive signals for measuring brain activities are electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). EEG, characterized by its higher sampling frequency, captures more temporal features, while fNIRS, with a greater number of channels, provides richer spatial information. Although a few previous studies have explored the use of multimodal deep-learning models to analyze brain activity for both EEG and fNIRS, training-testing split analysis remains underexplored. The results of the subject-independent setting directly show the model's ability on unseen subjects, which is crucial for real-world applications. In this paper, we introduce , a new CNN-based multimodal deep-learning model. We evaluate EF-Net on an EEG-fNIRS word generation (WG) dataset on the mental state recognition task, primarily focusing on the subject-independent setting. For completeness, we report results in the subject-dependent and subject-semidependent settings as well. We compare our model with five baseline approaches, including three traditional machine learning methods and two deep learning methods. EF-Net demonstrates superior performance in both accuracy and F1 score, surpassing these baselines. Our model achieves F1 scores of 99.36%, 98.31%, and 65.05% in the subject-dependent, subject-semidependent, and subject-independent settings, respectively, surpassing the best baseline F1 scores by 1.83%, 4.34%, and 2.13% These results highlight EF-Net's capability to effectively learn and interpret mental states and brain activity across different and unseen subjects.
脑信号分析对于研究心理状态和各种神经状况至关重要。用于测量大脑活动的两种最常见的非侵入性信号是脑电图（EEG）和功能近红外光谱（fNIRS）。EEG 的特点是采样频率更高，能够捕捉更多的时间特征，而 fNIRS 则具有更多的通道，提供更丰富的空间信息。尽管之前有一些研究探索了使用多模态深度学习模型来分析 EEG 和 fNIRS 的大脑活动，但训练-测试分割分析仍未得到充分探索。独立于主体的设置的结果直接显示了模型在未见主体上的能力，这对于实际应用至关重要。在本文中，我们介绍了一种新的基于 CNN 的多模态深度学习模型 EF-Net。我们在心理状态识别任务上的 EEG-fNIRS 单词生成（WG）数据集上评估了 EF-Net，主要关注独立于主体的设置。为了完整性，我们还报告了依赖于主体和半依赖于主体的设置的结果。我们将我们的模型与五个基线方法进行比较，包括三种传统机器学习方法和两种深度学习方法。EF-Net 在准确性和 F1 得分方面都表现出了优越的性能，超过了这些基线。我们的模型在依赖于主体、半依赖于主体和独立于主体的设置中的 F1 得分分别为 99.36%、98.31%和 65.05%，超过了最佳基线 F1 得分 1.83%、4.34%和 2.13%。这些结果突出了 EF-Net 有效学习和解释不同和未见主体的心理状态和大脑活动的能力。
