Zhao Chongyue, Zhan Liang, Thompson Paul M, Huang Heng
Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, PA, USA.
Imaging Genetics Center, University of Southern California, Los Angeles, CA, USA.
Med Image Comput Comput Assist Interv. 2022 Sep;13431:346-355. doi: 10.1007/978-3-031-16431-6_33. Epub 2022 Sep 15.
Brain large-scale dynamics is constrained by the heterogeneity of intrinsic anatomical substrate. Little is known how the spatio-temporal dynamics adapt for the heterogeneous structural connectivity (SC). Modern neuroimaging modalities make it possible to study the intrinsic brain activity at the scale of seconds to minutes. Diffusion magnetic resonance imaging (dMRI) and functional MRI reveals the large-scale SC across different brain regions. Electrophysiological methods (i.e. MEG/EEG) provide direct measures of neural activity and exhibits complex neurobiological temporal dynamics which could not be solved by fMRI. However, most of existing multimodal analytical methods collapse the brain measurements either in space or time domain and fail to capture the spatio-temporal circuit dynamics. In this paper, we propose a novel spatio-temporal graph Transformer model to integrate the structural and functional connectivity in both spatial and temporal domain. The proposed method learns the heterogeneous node and graph representation via contrastive learning and multi-head attention based graph Transformer using multimodal brain data (i.e. fMRI, MRI, MEG and behavior performance). The proposed contrastive graph Transformer representation model incorporates the heterogeneity map constrained by T1-to-T2-weighted (T1w/T2w) to improve the model fit to structure-function interactions. The experimental results with multimodal resting state brain measurements demonstrate the proposed method could highlight the local properties of large-scale brain spatio-temporal dynamics and capture the dependence strength between functional connectivity and behaviors. In summary, the proposed method enables the complex brain dynamics explanation for different modal variants.
大脑的大规模动态受到内在解剖学基质异质性的限制。目前对于时空动态如何适应异质结构连接性（SC）知之甚少。现代神经成像技术使得在秒到分钟的时间尺度上研究大脑内在活动成为可能。扩散磁共振成像（dMRI）和功能磁共振成像揭示了不同脑区之间的大规模SC。电生理方法（即MEG/EEG）提供了神经活动的直接测量，并展现出复杂的神经生物学时间动态，这是功能磁共振成像无法解决的。然而，现有的大多数多模态分析方法在空间或时间域中都会压缩大脑测量数据，无法捕捉时空回路动态。在本文中，我们提出了一种新颖的时空图Transformer模型，以在空间和时间域中整合结构和功能连接性。所提出的方法通过对比学习和基于多头注意力的图Transformer，使用多模态大脑数据（即功能磁共振成像、磁共振成像、脑磁图和行为表现）来学习异质节点和图表示。所提出的对比图Transformer表示模型纳入了受T1加权到T2加权（T1w/T2w）约束的异质性图谱，以提高模型对结构 - 功能相互作用的拟合度。多模态静息态大脑测量的实验结果表明，所提出的方法可以突出大规模脑时空动态的局部特性，并捕捉功能连接性与行为之间的依赖强度。总之，所提出的方法能够对不同模态变体进行复杂的大脑动态解释。
