Department of Informatics and Munich School of BioEngineering, Technical University of Munich, Munich, Germany.
State Grid Henan Economic Research Institute, Zhengzhou, Henan 450052, China.
Med Image Anal. 2022 Feb;76:102307. doi: 10.1016/j.media.2021.102307. Epub 2021 Nov 22.
Skin disease is one of the most common diseases in the world. Deep learning-based methods have achieved excellent skin lesion recognition performance, most of which are based on only dermoscopy images. In recent works that use multi-modality data (patient's meta-data, clinical images, and dermoscopy images), the methods adopt a one-stage fusion approach and only optimize the information fusion at the feature level. These methods do not use information fusion at the decision level and thus cannot fully use the data of all modalities. This work proposes a novel two-stage multi-modal learning algorithm (FusionM4Net) for multi-label skin diseases classification. At the first stage, we construct a FusionNet, which exploits and integrates the representation of clinical and dermoscopy images at the feature level, and then uses a Fusion Scheme 1 to conduct the information fusion at the decision level. At the second stage, to further incorporate the patient's meta-data, we propose a Fusion Scheme 2, which integrates the multi-label predictive information from the first stage and patient's meta-data information to train an SVM cluster. The final diagnosis is formed by the fusion of the predictions from the first and second stages. Our algorithm was evaluated on the seven-point checklist dataset, a well-established multi-modality multi-label skin disease dataset. Without using the patient's meta-data, the proposed FusionM4Net's first stage (FusionM4Net-FS) achieved an average accuracy of 75.7% for multi-classification tasks and 74.9% for diagnostic tasks, which is more accurate than other state-of-the-art methods. By further fusing the patient's meta-data at FusionM4Net's second stage (FusionM4Net-SS), the entire FusionM4Net finally boosts the average accuracy to 77.0% and the diagnostic accuracy to 78.5%, which indicates its robust and excellent classification performance on the label-imbalanced dataset. The corresponding code is available at: https://github.com/pixixiaonaogou/MLSDR.
皮肤疾病是世界上最常见的疾病之一。基于深度学习的方法在皮肤病变识别方面取得了优异的性能，其中大多数方法仅基于皮肤镜图像。在最近使用多模态数据（患者的元数据、临床图像和皮肤镜图像）的工作中，这些方法采用了一种单阶段融合方法，仅在特征级别优化信息融合。这些方法没有在决策级别使用信息融合，因此不能充分利用所有模态的数据。本研究提出了一种新的两阶段多模态学习算法（FusionM4Net），用于多标签皮肤疾病分类。在第一阶段，我们构建了一个 FusionNet，它利用和整合了临床和皮肤镜图像的特征级表示，然后使用 Fusion Scheme 1 在决策级进行信息融合。在第二阶段，为了进一步纳入患者的元数据，我们提出了 Fusion Scheme 2，它将第一阶段的多标签预测信息和患者的元数据信息整合起来，以训练 SVM 聚类。最终的诊断是通过第一阶段和第二阶段的预测融合形成的。我们的算法在七点清单数据集上进行了评估，这是一个成熟的多模态多标签皮肤疾病数据集。在不使用患者元数据的情况下，所提出的 FusionM4Net 的第一阶段（FusionM4Net-FS）在多分类任务中达到了 75.7%的平均准确率，在诊断任务中达到了 74.9%的平均准确率，优于其他最先进的方法。通过在 FusionM4Net 的第二阶段进一步融合患者的元数据（FusionM4Net-SS），整个 FusionM4Net 最终将平均准确率提高到 77.0%，诊断准确率提高到 78.5%，这表明它在标签不平衡数据集上具有稳健和出色的分类性能。相应的代码可在：https://github.com/pixixiaonaogou/MLSDR 获得。
