School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, China.
Department of Research and Development, Shanghai United Imaging Intelligence Co.,Ltd., China.
Comput Med Imaging Graph. 2023 Dec;110:102303. doi: 10.1016/j.compmedimag.2023.102303. Epub 2023 Sep 30.
Multimodal images such as magnetic resonance imaging (MRI) and positron emission tomography (PET) could provide complementary information about the brain and have been widely investigated for the diagnosis of neurodegenerative disorders such as Alzheimer's disease (AD). However, multimodal brain images are often incomplete in clinical practice. It is still challenging to make use of multimodality for disease diagnosis with missing data. In this paper, we propose a deep learning framework with the multi-level guided generative adversarial network (MLG-GAN) and multimodal transformer (Mul-T) for incomplete image generation and disease classification, respectively. First, MLG-GAN is proposed to generate the missing data, guided by multi-level information from voxels, features, and tasks. In addition to voxel-level supervision and task-level constraint, a feature-level auto-regression branch is proposed to embed the features of target images for an accurate generation. With the complete multimodal images, we propose a Mul-T network for disease diagnosis, which can not only combine the global and local features but also model the latent interactions and correlations from one modality to another with the cross-modal attention mechanism. Comprehensive experiments on three independent datasets (i.e., ADNI-1, ADNI-2, and OASIS-3) show that the proposed method achieves superior performance in the tasks of image generation and disease diagnosis compared to state-of-the-art methods.
多模态图像，如磁共振成像（MRI）和正电子发射断层扫描（PET），可以提供关于大脑的补充信息，并且已经广泛应用于阿尔茨海默病（AD）等神经退行性疾病的诊断。然而，多模态脑图像在临床实践中常常是不完整的。利用多模态数据进行疾病诊断仍然具有挑战性。在本文中，我们提出了一个具有多层次引导生成对抗网络（MLG-GAN）和多模态转换器（Mul-T）的深度学习框架，分别用于不完全图像生成和疾病分类。首先，提出了 MLG-GAN 来生成缺失数据，由来自体素、特征和任务的多层次信息引导。除了体素级监督和任务级约束外，还提出了一个特征级自回归分支，用于嵌入目标图像的特征，以实现准确的生成。有了完整的多模态图像，我们提出了一个 Mul-T 网络用于疾病诊断，它不仅可以结合全局和局部特征，还可以通过跨模态注意力机制对模态间的潜在相互作用和相关性进行建模。在三个独立数据集（即 ADNI-1、ADNI-2 和 OASIS-3）上的综合实验表明，与最先进的方法相比，所提出的方法在图像生成和疾病诊断任务中具有优越的性能。
