Wei Xinxu, Zhao Kanhao, Jiao Yong, Carlisle Nancy B, Xie Hua, Fonzo Gregory A, Zhang Yu
Department of Electrical and Computer Engineering, Lehigh University, Bethlehem, PA 18015, USA.
Department of Bioengineering, Lehigh University, Bethlehem, PA 18015, USA.
Neural Netw. 2025 Apr;184:107066. doi: 10.1016/j.neunet.2024.107066. Epub 2024 Dec 24.
Neuroimaging techniques including functional magnetic resonance imaging (fMRI) and electroencephalogram (EEG) have shown promise in detecting functional abnormalities in various brain disorders. However, existing studies often focus on a single domain or modality, neglecting the valuable complementary information offered by multiple domains from both fMRI and EEG, which is crucial for a comprehensive representation of disorder pathology. This limitation poses a challenge in effectively leveraging the synergistic information derived from these modalities. To address this, we propose a Multi-modal Cross-domain Self-supervised Pre-training Model (MCSP), a novel approach that leverages self-supervised learning to synergize multi-modal information across spatial, temporal, and spectral domains. Our model employs cross-domain self-supervised loss that bridges domain differences by implementing domain-specific data augmentation and contrastive loss, enhancing feature discrimination. Furthermore, MCSP introduces cross-modal self-supervised loss to capitalize on the complementary information of fMRI and EEG, facilitating knowledge distillation within domains and maximizing cross-modal feature convergence. We constructed a large-scale pre-training dataset and pretrained MCSP model by leveraging proposed self-supervised paradigms to fully harness multimodal neuroimaging data. Through comprehensive experiments, we have demonstrated the superior performance and generalizability of our model on multiple classification tasks. Our study contributes a significant advancement in the fusion of fMRI and EEG, marking a novel integration of cross-domain features, which enriches the existing landscape of neuroimaging research, particularly within the context of mental disorder studies.
包括功能磁共振成像（fMRI）和脑电图（EEG）在内的神经成像技术在检测各种脑部疾病的功能异常方面已显示出前景。然而，现有研究往往集中在单一领域或模态上，忽略了fMRI和EEG多个领域提供的有价值的互补信息，而这些信息对于全面呈现疾病病理学至关重要。这种局限性给有效利用这些模态产生的协同信息带来了挑战。为了解决这个问题，我们提出了一种多模态跨域自监督预训练模型（MCSP），这是一种新颖的方法，利用自监督学习来整合跨空间、时间和频谱领域的多模态信息。我们的模型采用跨域自监督损失，通过实施特定领域的数据增强和对比损失来弥合领域差异，增强特征辨别力。此外，MCSP引入了跨模态自监督损失，以利用fMRI和EEG的互补信息，促进领域内的知识蒸馏并最大化跨模态特征收敛。我们通过利用提出的自监督范式构建了一个大规模预训练数据集并预训练了MCSP模型，以充分利用多模态神经成像数据。通过全面的实验，我们证明了我们的模型在多个分类任务上的卓越性能和通用性。我们的研究为fMRI和EEG的融合做出了重大进展，标志着跨域特征的一种新颖整合，丰富了现有的神经成像研究领域，特别是在精神障碍研究的背景下。
