IEEE Trans Biomed Eng. 2022 Jun;69(6):1983-1994. doi: 10.1109/TBME.2021.3132861. Epub 2022 May 19.
Brain-computer interfaces (BCI) studies are increasingly leveraging different attributes of multiple signal modalities simultaneously. Bimodal data acquisition protocols combining the temporal resolution of electroencephalography (EEG) with the spatial resolution of functional near-infrared spectroscopy (fNIRS) require novel approaches to decoding.
We present an EEG-fNIRS Hybrid BCI that employs a new bimodal deep neural network architecture consisting of two convolutional sub-networks (subnets) to decode overt and imagined speech. Features from each subnet are fused before further feature extraction and classification. Nineteen participants performed overt and imagined speech in a novel cue-based paradigm enabling investigation of stimulus and linguistic effects on decoding.
Using the hybrid approach, classification accuracies (46.31% and 34.29% for overt and imagined speech, respectively (chance: 25%)) indicated a significant improvement on EEG used independently for imagined speech (p = 0.020) while tending towards significance for overt speech (p = 0.098). In comparison with fNIRS, significant improvements for both speech-types were achieved with bimodal decoding (p<0.001). There was a mean difference of ∼12.02% between overt and imagined speech with accuracies as high as 87.18% and 53%. Deeper subnets enhanced performance while stimulus effected overt and imagined speech in significantly different ways.
The bimodal approach was a significant improvement on unimodal results for several tasks. Results indicate the potential of multi-modal deep learning for enhancing neural signal decoding.
This novel architecture can be used to enhance speech decoding from bimodal neural signals.
脑机接口（BCI）研究越来越多地同时利用多种信号模态的不同属性。结合脑电图（EEG）的时间分辨率和功能近红外光谱（fNIRS）的空间分辨率的双模数据采集协议需要新的解码方法。
我们提出了一种 EEG-fNIRS 混合 BCI，它采用了一种新的双模深度神经网络架构，由两个卷积子网络（子网）组成，用于解码显性和想象中的语音。在进一步进行特征提取和分类之前，对每个子网的特征进行融合。19 名参与者在一种新的基于提示的范式中进行显性和想象中的语音，从而能够研究刺激和语言效应对解码的影响。
使用混合方法，分类准确率（显性和想象中的语音分别为 46.31%和 34.29%（机会：25%））表明，对于想象中的语音，独立使用 EEG 进行解码有显著提高（p = 0.020），而对于显性语音则有趋势（p = 0.098）。与 fNIRS 相比，双模解码在两种语音类型上都有显著提高（p<0.001）。显性和想象中的语音之间存在约 12.02%的平均差异，准确率高达 87.18%和 53%。更深的子网增强了性能，而刺激以显著不同的方式影响显性和想象中的语音。
对于几种任务，双模方法是对单模态结果的显著改进。结果表明，多模态深度学习有可能增强神经信号解码。
这种新架构可用于增强来自双模神经信号的语音解码。
