Guo Zhe, Li Xiang, Huang Heng, Guo Ning, Li Quanzheng
School of Information and Electronics, Beijing Institute of Technology, China.
Massachusetts General Hospital, USA.
IEEE Trans Radiat Plasma Med Sci. 2019 Mar;3(2):162-169. doi: 10.1109/trpms.2018.2890359. Epub 2019 Jan 1.
Multi-modality medical imaging techniques have been increasingly applied in clinical practice and research studies. Corresponding multi-modal image analysis and ensemble learning schemes have seen rapid growth and bring unique value to medical applications. Motivated by the recent success of applying deep learning methods to medical image processing, we first propose an algorithmic architecture for supervised multi-modal image analysis with cross-modality fusion at the feature learning level, classifier level, and decision-making level. We then design and implement an image segmentation system based on deep Convolutional Neural Networks (CNN) to contour the lesions of soft tissue sarcomas using multi-modal images, including those from Magnetic Resonance Imaging (MRI), Computed Tomography (CT) and Positron Emission Tomography (PET). The network trained with multi-modal images shows superior performance compared to networks trained with single-modal images. For the task of tumor segmentation, performing image fusion within the network (i.e. fusing at convolutional or fully connected layers) is generally better than fusing images at the network output (i.e. voting). This study provides empirical guidance for the design and application of multi-modal image analysis.
多模态医学成像技术已越来越多地应用于临床实践和研究中。相应的多模态图像分析和集成学习方案发展迅速，并为医学应用带来了独特价值。受近期将深度学习方法应用于医学图像处理取得成功的启发，我们首先提出一种算法架构，用于在特征学习层、分类器层和决策层进行跨模态融合的监督式多模态图像分析。然后，我们设计并实现了一个基于深度卷积神经网络（CNN）的图像分割系统，使用多模态图像（包括来自磁共振成像（MRI）、计算机断层扫描（CT）和正电子发射断层扫描（PET）的图像）勾勒软组织肉瘤的病变轮廓。与使用单模态图像训练的网络相比，使用多模态图像训练的网络表现出更优的性能。对于肿瘤分割任务，在网络内进行图像融合（即在卷积层或全连接层进行融合）通常比在网络输出端进行图像融合（即投票）效果更好。本研究为多模态图像分析的设计和应用提供了实证指导。
