Rabbani Md Hasin Raihan, Islam Sheikh Md Rabiul
Khulna University of Engineering and Technology, Khulna, Bangladesh.
Cogn Neurodyn. 2024 Aug;18(4):1489-1506. doi: 10.1007/s11571-023-09986-4. Epub 2023 Jun 30.
The detection of the cognitive tasks performed by a subject during data acquisition of a neuroimaging method has a wide range of applications: functioning of brain-computer interface (BCI), detection of neuronal disorders, neurorehabilitation for disabled patients, and many others. Recent studies show that the combination or fusion of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) demonstrates improved classification and detection performance compared to sole-EEG and sole-fNIRS. Deep learning (DL) networks are suitable for the classification of large volume time-series data like EEG and fNIRS. This study performs the decision fusion of EEG and fNIRS. The classification of EEG, fNIRS, and decision-fused EEG-fNIRSinto cognitive task labels is performed by DL networks. Two different open-source datasets of simultaneously recorded EEG and fNIRS are examined in this study. Dataset 01 is comprised of 26 subjects performing 3 cognitive tasks: n-back, discrimination or selection response (DSR), and word generation (WG). After data acquisition, fNIRS is converted to oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) in Dataset 01. Dataset 02 is comprised of 29 subjects who performed 2 tasks: motor imagery and mental arithmetic. The classification procedure of EEG and fNIRS (or HbO, HbR) are carried out by 7 DL classifiers: convolutional neural network (CNN), long short-term memory network (LSTM), gated recurrent unit (GRU), CNN-LSTM, CNN-GRU, LSTM-GRU, and CNN-LSTM-GRU. After the classification of single modalities, their prediction scores or decisions are combined to obtain the decision-fused modality. The classification performance is measured by overall accuracy and area under the ROC curve (AUC). The highest accuracy and AUC recorded in Dataset 01 are 96% and 100% respectively; both by the decision fusion modality using CNN-LSTM-GRU. For Dataset 02, the highest accuracy and AUC are 82.76% and 90.44% respectively; both by the decision fusion modality using CNN-LSTM. The experimental result shows that decision-fused EEG-HbO-HbR and EEG-fNIRSdeliver higher performances compared to their constituent unimodalities in most cases. For DL classifiers, CNN-LSTM-GRU in Dataset 01 and CNN-LSTM in Dataset 02 yield the highest performance.
脑机接口（BCI）的功能、神经元疾病的检测、残疾患者的神经康复等等。最近的研究表明，与单独的脑电图（EEG）和单独的功能近红外光谱（fNIRS）相比，脑电图（EEG）和功能近红外光谱（fNIRS）的组合或融合显示出更好的分类和检测性能。深度学习（DL）网络适用于对脑电图（EEG）和功能近红外光谱（fNIRS）等大量时间序列数据进行分类。本研究进行了脑电图（EEG）和功能近红外光谱（fNIRS）的决策融合。通过深度学习网络将脑电图（EEG）、功能近红外光谱（fNIRS）以及决策融合后的脑电图-功能近红外光谱（EEG-fNIRS）分类为认知任务标签。本研究考察了两个同时记录脑电图（EEG）和功能近红外光谱（fNIRS）的不同开源数据集。数据集01由26名执行3项认知任务的受试者组成：n-back任务、辨别或选择反应（DSR）任务以及单词生成（WG）任务。在数据采集后，数据集01中的功能近红外光谱（fNIRS）被转换为氧合血红蛋白（HbO）和脱氧血红蛋白（HbR）。数据集02由29名执行2项任务的受试者组成：运动想象任务和心算任务。脑电图（EEG）和功能近红外光谱（fNIRS）（或氧合血红蛋白（HbO）、脱氧血红蛋白（HbR））的分类过程由7个深度学习分类器进行：卷积神经网络（CNN）、长短期记忆网络（LSTM）、门控循环单元（GRU）、CNN-LSTM、CNN-GRU、LSTM-GRU以及CNN-LSTM-GRU。在对单模态进行分类后，将它们的预测分数或决策进行组合以获得决策融合模态。分类性能通过总体准确率和ROC曲线下面积（AUC）来衡量。在数据集01中记录的最高准确率和AUC分别为96%和100%；均来自使用CNN-LSTM-GRU的决策融合模态。对于数据集02，最高准确率和AUC分别为82.76%和90.44%；均来自使用CNN-LSTM的决策融合模态。实验结果表明，在大多数情况下，决策融合后的脑电图-氧合血红蛋白-脱氧血红蛋白（EEG-HbO-HbR）和脑电图-功能近红外光谱（EEG-fNIRS）比其组成的单模态具有更高的性能。对于深度学习分类器，数据集01中的CNN-LSTM-GRU和数据集02中的CNN-LSTM产生了最高性能。
