Kim Ki Duk, Cho Kyungjin, Kim Mingyu, Lee Kyung Hwa, Lee Seungjun, Lee Sang Min, Lee Kyung Hee, Kim Namkug
Department of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul 05505, Republic of Korea.
Department of Biomedical Engineering, Asan Medical Institute of Convergence Science and Technology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea.
Comput Methods Programs Biomed. 2022 Jun;220:106705. doi: 10.1016/j.cmpb.2022.106705. Epub 2022 Feb 22.
The protocol for placing anatomical side markers (L/R markers) in chest radiographs varies from one hospital or department to another. However, the markers have strong signals that can be useful for deep learning-based classifier to predict diseases. We aimed to enhance the performance of a deep learning-based classifiers in multi-center datasets by inpainting the L/R markers.
The L/R marker was detected with using the EfficientDet detection network; only the detected regions were inpainted using a generative adversarial network (GAN). To analyze the effect of the inpainting in detail, deep learning-based classifiers were trained using original images, marker-inpainted images, and original images clipped using the min-max value of the marker-inpainted images. Binary classification, multi-class classification, and multi-task learning with segmentation and classification were developed and evaluated. Furthermore, the performances of the network on internal and external validation datasets were compared using DeLong's test for two correlated receiver operating characteristic (ROC) curves in binary classification and Stuart-Maxwell test for marginal homogeneity in multi-class classification and multi-task learning. In addition, the qualitative results of activation maps were evaluated using the gradient-class activation map (Grad-CAM).
Marker-inpainting preprocessing improved the classification performances. In the binary classification based on the internal validation, the area under the curves (AUCs) and accuracies were 0.950 and 0.900 for the model trained on the min-max clipped images and 0.911 and 0.850 for the model trained on the original images, respectively (P-value=0.006). In the external validation, the AUCs and accuracies were 0.858 and 0.677 for the model trained using the inpainted images and 0.723 and 0.568 for the model trained using the original images (P-value<0.001), respectively. In addition, the models trained using the marker inpainted images showed the best performance in multi-class classification and multi-task learning. Furthermore, the activation maps obtained using the Grad-CAM improved with the proposed method. The 5-fold validation results also showed improvement trend according to the preprocessing strategies.
Inpainting an L/R marker significantly enhanced the classifier's performance and robustness, especially in internal and external studies, which could be useful in developing a more robust and accurate deep learning-based classifier for multi-center trials. The code for detection is available at: https://github.com/mi2rl/MI2RLNet. And the code for inpainting is available at: https://github.com/mi2rl/L-R-marker-inpainting.
胸部X光片中放置解剖学侧方标记（左/右标记）的方案在不同医院或科室之间存在差异。然而，这些标记具有很强的信号，可用于基于深度学习的分类器来预测疾病。我们旨在通过修复左/右标记来提高基于深度学习的分类器在多中心数据集中的性能。
使用EfficientDet检测网络检测左/右标记；仅使用生成对抗网络（GAN）对检测到的区域进行修复。为了详细分析修复的效果，使用原始图像、标记修复后的图像以及使用标记修复后图像的最小-最大值裁剪的原始图像训练基于深度学习的分类器。开发并评估了二元分类、多类分类以及带有分割和分类的多任务学习。此外，在二元分类中使用DeLong检验比较两个相关的受试者工作特征（ROC）曲线，在多类分类和多任务学习中使用Stuart-Maxwell检验比较边缘同质性，以比较网络在内部和外部验证数据集上的性能。此外，使用梯度类激活映射（Grad-CAM）评估激活映射的定性结果。
标记修复预处理提高了分类性能。在基于内部验证的二元分类中，对于在最小-最大值裁剪图像上训练的模型，曲线下面积（AUC）和准确率分别为0.950和0.900，对于在原始图像上训练的模型，分别为0.911和0.850（P值 = 0.006）。在外部验证中，对于使用修复后图像训练的模型，AUC和准确率分别为0.858和0.677，对于使用原始图像训练的模型，分别为0.723和0.568（P值<0.001）。此外，使用标记修复后图像训练的模型在多类分类和多任务学习中表现最佳。此外，使用Grad-CAM获得的激活映射通过所提出的方法得到了改善。5折验证结果也显示出根据预处理策略的改善趋势。
修复左/右标记显著提高了分类器的性能和鲁棒性，特别是在内部和外部研究中，这对于为多中心试验开发更强大、准确的基于深度学习的分类器可能是有用的。检测代码可在：https://github.com/mi2rl/MI2RLNet获取。修复代码可在：https://github.com/mi2rl/L-R-marker-inpainting获取。
