Dharejo Fayaz Ali, Zawish Muhammad, Deeba Farah, Zhou Yuanchun, Dev Kapal, Khowaja Sunder Ali, Qureshi Nawab Muhammad Faseeh
IEEE/ACM Trans Comput Biol Bioinform. 2023 Jul-Aug;20(4):2420-2433. doi: 10.1109/TCBB.2022.3191387. Epub 2023 Aug 9.
Multimodal medical images are widely used by clinicians and physicians to analyze and retrieve complementary information from high-resolution images in a non-invasive manner. Loss of corresponding image resolution adversely affects the overall performance of medical image interpretation. Deep learning-based single image super resolution (SISR) algorithms have revolutionized the overall diagnosis framework by continually improving the architectural components and training strategies associated with convolutional neural networks (CNN) on low-resolution images. However, existing work lacks in two ways: i) the SR output produced exhibits poor texture details, and often produce blurred edges, ii) most of the models have been developed for a single modality, hence, require modification to adapt to a new one. This work addresses (i) by proposing generative adversarial network (GAN) with deep multi-attention modules to learn high-frequency information from low-frequency data. Existing approaches based on the GAN have yielded good SR results; however, the texture details of their SR output have been experimentally confirmed to be deficient for medical images particularly. The integration of wavelet transform (WT) and GANs in our proposed SR model addresses the aforementioned limitation concerning textons. While the WT divides the LR image into multiple frequency bands, the transferred GAN uses multi-attention and upsample blocks to predict high-frequency components. Additionally, we present a learning method for training domain-specific classifiers as perceptual loss functions. Using a combination of multi-attention GAN loss and a perceptual loss function results in an efficient and reliable performance. Applying the same model for medical images from diverse modalities is challenging, our work addresses (ii) by training and performing on several modalities via transfer learning. Using two medical datasets, we validate our proposed SR network against existing state-of-the-art approaches and achieve promising results in terms of structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR).
多模态医学图像被临床医生广泛用于以非侵入性方式从高分辨率图像中分析和检索补充信息。相应图像分辨率的损失会对医学图像解读的整体性能产生不利影响。基于深度学习的单图像超分辨率(SISR)算法通过不断改进与低分辨率图像上的卷积神经网络(CNN)相关的架构组件和训练策略,彻底改变了整体诊断框架。然而,现有工作存在两方面不足:i)生成的超分辨率输出纹理细节较差,且经常产生模糊边缘;ii)大多数模型是针对单一模态开发的,因此需要修改以适应新的模态。这项工作通过提出带有深度多注意力模块的生成对抗网络(GAN)来从低频数据中学习高频信息,从而解决了问题(i)。基于GAN的现有方法已经取得了良好的超分辨率结果;然而,通过实验证实,其超分辨率输出的纹理细节对于医学图像来说尤其不足。我们提出的超分辨率模型中,小波变换(WT)和GAN的整合解决了上述关于纹理基元的限制。WT将低分辨率图像划分为多个频带,而迁移的GAN使用多注意力和上采样块来预测高频分量。此外,我们提出了一种用于训练特定领域分类器作为感知损失函数的学习方法。使用多注意力GAN损失和感知损失函数的组合可实现高效且可靠的性能。将同一模型应用于不同模态的医学图像具有挑战性,我们的工作通过迁移学习在多个模态上进行训练和执行来解决问题(ii)。使用两个医学数据集,我们将提出的超分辨率网络与现有的最先进方法进行了验证,并在结构相似性指数(SSIM)和峰值信噪比(PSNR)方面取得了有希望的结果。