Faculty of Medicine and Health, University of Sydney, Sydney, NSW, Australia.
Department of Otolaryngology, Head and Neck Surgery, Westmead Hospital, Sydney, NSW, Australia.
Sci Rep. 2023 Apr 1;13(1):5368. doi: 10.1038/s41598-023-31921-0.
To evaluate the generalizability of artificial intelligence (AI) algorithms that use deep learning methods to identify middle ear disease from otoscopic images, between internal to external performance. 1842 otoscopic images were collected from three independent sources: (a) Van, Turkey, (b) Santiago, Chile, and (c) Ohio, USA. Diagnostic categories consisted of (i) normal or (ii) abnormal. Deep learning methods were used to develop models to evaluate internal and external performance, using area under the curve (AUC) estimates. A pooled assessment was performed by combining all cohorts together with fivefold cross validation. AI-otoscopy algorithms achieved high internal performance (mean AUC: 0.95, 95%CI: 0.80-1.00). However, performance was reduced when tested on external otoscopic images not used for training (mean AUC: 0.76, 95%CI: 0.61-0.91). Overall, external performance was significantly lower than internal performance (mean difference in AUC: -0.19, p ≤ 0.04). Combining cohorts achieved a substantial pooled performance (AUC: 0.96, standard error: 0.01). Internally applied algorithms for otoscopy performed well to identify middle ear disease from otoscopy images. However, external performance was reduced when applied to new test cohorts. Further efforts are required to explore data augmentation and pre-processing techniques that might improve external performance and develop a robust, generalizable algorithm for real-world clinical applications.
为了评估使用深度学习方法从耳镜图像中识别中耳疾病的人工智能（AI）算法的泛化能力，对内部和外部性能进行了评估。从三个独立的来源收集了 1842 张耳镜图像：（a）Van，土耳其；（b）Santiago，智利；（c）Ohio，美国。诊断类别包括（i）正常或（ii）异常。使用曲线下面积（AUC）估计值，使用深度学习方法开发模型来评估内部和外部性能。通过将所有队列合并在一起并进行五重交叉验证，进行了汇总评估。AI 耳镜算法实现了较高的内部性能（平均 AUC：0.95，95%CI：0.80-1.00）。然而，当在未用于训练的外部耳镜图像上进行测试时，性能会降低（平均 AUC：0.76，95%CI：0.61-0.91）。总体而言，外部性能明显低于内部性能（AUC 差异平均值：-0.19，p≤0.04）。合并队列实现了较高的汇总性能（AUC：0.96，标准误差：0.01）。用于耳镜检查的内部应用算法在从耳镜图像中识别中耳疾病方面表现良好。然而，当应用于新的测试队列时，外部性能会降低。需要进一步努力探索数据增强和预处理技术，以提高外部性能并开发用于实际临床应用的稳健、可泛化的算法。
