Xiao Chunlun, Zhu Anqi, Xia Chunmei, Qiu Zifeng, Liu Yuanlin, Zhao Cheng, Ren Weiwei, Wang Lifan, Dong Lei, Wang Tianfu, Guo Lehang, Lei Baiying
IEEE Trans Med Imaging. 2025 Jan;44(1):543-555. doi: 10.1109/TMI.2024.3450682. Epub 2025 Jan 2.
Skin lesion is one of the most common diseases, and most categories are highly similar in morphology and appearance. Deep learning models effectively reduce the variability between classes and within classes, and improve diagnostic accuracy. However, the existing multi-modal methods are only limited to the surface information of lesions in skin clinical and dermatoscopic modalities, which hinders the further improvement of skin lesion diagnostic accuracy. This requires us to further study the depth information of lesions in skin ultrasound. In this paper, we propose a novel skin lesion diagnosis network, which combines clinical and ultrasound modalities to fuse the surface and depth information of the lesion to improve diagnostic accuracy. Specifically, we propose an attention-guided learning (AL) module that fuses clinical and ultrasound modalities from both local and global perspectives to enhance feature representation. The AL module consists of two parts, attention-guided local learning (ALL) computes the intra-modality and inter-modality correlations to fuse multi-scale information, which makes the network focus on the local information of each modality, and attention-guided global learning (AGL) fuses global information to further enhance the feature representation. In addition, we propose a feature reconstruction learning (FRL) strategy which encourages the network to extract more discriminative features and corrects the focus of the network to enhance the model's robustness and certainty. We conduct extensive experiments and the results confirm the superiority of our proposed method. Our code is available at: https://github.com/XCL-hub/AGFnet.
皮肤病变是最常见的疾病之一，并且大多数类别在形态和外观上高度相似。深度学习模型有效地减少了类间和类内的变异性，并提高了诊断准确性。然而，现有的多模态方法仅局限于皮肤临床和皮肤镜检查模态中病变的表面信息，这阻碍了皮肤病变诊断准确性的进一步提高。这就要求我们进一步研究皮肤超声中病变的深度信息。在本文中，我们提出了一种新颖的皮肤病变诊断网络，它结合了临床和超声模态，融合病变的表面和深度信息以提高诊断准确性。具体来说，我们提出了一种注意力引导学习（AL）模块，该模块从局部和全局角度融合临床和超声模态以增强特征表示。AL模块由两部分组成，注意力引导局部学习（ALL）计算模态内和模态间的相关性以融合多尺度信息，这使得网络专注于每个模态的局部信息，而注意力引导全局学习（AGL）融合全局信息以进一步增强特征表示。此外，我们提出了一种特征重建学习（FRL）策略，该策略鼓励网络提取更具判别力的特征并校正网络的关注点，以增强模型的鲁棒性和确定性。我们进行了广泛的实验，结果证实了我们所提方法的优越性。我们的代码可在以下网址获取：https://github.com/XCL-hub/AGFnet。
