IEEE Trans Neural Syst Rehabil Eng. 2024;32:2492-2502. doi: 10.1109/TNSRE.2024.3424669. Epub 2024 Jul 11.
The process of reconstructing underlying cortical and subcortical electrical activities from Electroencephalography (EEG) or Magnetoencephalography (MEG) recordings is called Electrophysiological Source Imaging (ESI). Given the complementarity between EEG and MEG in measuring radial and tangential cortical sources, combined EEG/MEG is considered beneficial in improving the reconstruction performance of ESI algorithms. Traditional algorithms mainly emphasize incorporating predesigned neurophysiological priors to solve the ESI problem. Deep learning frameworks aim to directly learn the mapping from scalp EEG/MEG measurements to the underlying brain source activities in a data-driven manner, demonstrating superior performance compared to traditional methods. However, most of the existing deep learning approaches for the ESI problem are performed on a single modality of EEG or MEG, meaning the complementarity of these two modalities has not been fully utilized. How to fuse the EEG and MEG in a more principled manner under the deep learning paradigm remains a challenging question. This study develops a Multi-Modal Deep Fusion (MMDF) framework using Attention Neural Networks (ANN) to fully leverage the complementary information between EEG and MEG for solving the ESI inverse problem, which is termed as MMDF-ANN. Specifically, our proposed brain source imaging approach consists of four phases, including feature extraction, weight generation, deep feature fusion, and source mapping. Our experimental results on both synthetic dataset and real dataset demonstrated that using a fusion of EEG and MEG can significantly improve the source localization accuracy compared to using a single-modality of EEG or MEG. Compared to the benchmark algorithms, MMDF-ANN demonstrated good stability when reconstructing sources with extended activation areas and situations of EEG/MEG measurements with a low signal-to-noise ratio.
从脑电图（EEG）或脑磁图（MEG）记录中重建潜在的皮质和皮质下电活动的过程称为电生理源成像（ESI）。鉴于 EEG 和 MEG 在测量径向和切向皮质源方面的互补性，结合 EEG/MEG 被认为有利于提高 ESI 算法的重建性能。传统算法主要强调结合预先设计的神经生理学先验来解决 ESI 问题。深度学习框架旨在以数据驱动的方式直接学习从头皮 EEG/MEG 测量到潜在大脑源活动的映射，与传统方法相比表现出优越的性能。然而，现有的大多数用于 ESI 问题的深度学习方法都是在 EEG 或 MEG 的单一模态上进行的，这意味着这两种模态的互补性尚未得到充分利用。如何在深度学习范式下以更有原则的方式融合 EEG 和 MEG 仍然是一个具有挑战性的问题。本研究使用注意力神经网络（ANN）开发了一种多模态深度融合（MMDF）框架，以充分利用 EEG 和 MEG 之间的互补信息来解决 ESI 逆问题，这被称为 MMDF-ANN。具体来说，我们提出的脑源成像方法包括四个阶段，包括特征提取、权重生成、深度特征融合和源映射。我们在合成数据集和真实数据集上的实验结果表明，与使用单一模态的 EEG 或 MEG 相比，使用 EEG 和 MEG 的融合可以显著提高源定位精度。与基准算法相比，MMDF-ANN 在重建具有扩展激活区域的源和 EEG/MEG 测量具有低信噪比的情况下表现出良好的稳定性。
