IEEE J Biomed Health Inform. 2023 Jul;27(7):3489-3500. doi: 10.1109/JBHI.2023.3264819. Epub 2023 Jun 30.
Medical image fusion technology is an essential component of computer-aided diagnosis, which aims to extract useful cross-modality cues from raw signals to generate high-quality fused images. Many advanced methods focus on designing fusion rules, but there is still room for improvement in cross-modal information extraction. To this end, we propose a novel encoder-decoder architecture with three technical novelties. First, we divide the medical images into two attributes, namely pixel intensity distribution attributes and texture attributes, and thus design two self-reconstruction tasks to mine as many specific features as possible. Second, we propose a hybrid network combining a CNN and a transformer module to model both long-range and short-range dependencies. Moreover, we construct a self-adaptive weight fusion rule that automatically measures salient features. Extensive experiments on a public medical image dataset and other multimodal datasets show that the proposed method achieves satisfactory performance.
医学图像融合技术是计算机辅助诊断的重要组成部分，旨在从原始信号中提取有用的跨模态线索，以生成高质量的融合图像。许多先进的方法侧重于设计融合规则，但在跨模态信息提取方面仍有改进的空间。为此，我们提出了一种具有三个技术创新的新型编解码器架构。首先，我们将医学图像分为两个属性，即像素强度分布属性和纹理属性，并因此设计了两个自重建任务，以尽可能多地挖掘特定特征。其次，我们提出了一种结合卷积神经网络和变压器模块的混合网络，以建模长程和短程依赖关系。此外，我们构建了一种自适应权重融合规则，自动测量显著特征。在一个公共医学图像数据集和其他多模态数据集上的广泛实验表明，所提出的方法取得了令人满意的性能。
