CISTIB Centre for Computational Imaging and Simulation Technologies in Biomedicine, School of Computing.
University of Leeds, Leeds, LS2 9JT, United Kingdom.
Stud Health Technol Inform. 2022 Jun 6;290:679-683. doi: 10.3233/SHTI220164.
Since the emergence of COVID-19, deep learning models have been developed to identify COVID-19 from chest X-rays. With little to no direct access to hospital data, the AI community relies heavily on public data comprising numerous data sources. Model performance results have been exceptional when training and testing on open-source data, surpassing the reported capabilities of AI in pneumonia-detection prior to the COVID-19 outbreak. In this study impactful models are trained on a widely used open-source data and tested on an external test set and a hospital dataset, for the task of classifying chest X-rays into one of three classes: COVID-19, non-COVID pneumonia and no-pneumonia. Classification performance of the models investigated is evaluated through ROC curves, confusion matrices and standard classification metrics. Explainability modules are implemented to explore the image features most important to classification. Data analysis and model evalutions show that the popular open-source dataset COVIDx is not representative of the real clinical problem and that results from testing on this are inflated. Dependence on open-source data can leave models vulnerable to bias and confounding variables, requiring careful analysis to develop clinically useful/viable AI tools for COVID-19 detection in chest X-rays.
自 COVID-19 出现以来，已经开发出深度学习模型来从胸部 X 光片中识别 COVID-19。由于几乎无法直接访问医院数据，人工智能社区严重依赖由众多数据源组成的公共数据。在开源数据上进行训练和测试时，模型性能结果非常出色，超过了 COVID-19 爆发前 AI 在肺炎检测方面的报告能力。在这项研究中，有影响力的模型在广泛使用的开源数据上进行训练，并在外部测试集和医院数据集上进行测试，用于将胸部 X 光片分类为 COVID-19、非 COVID 肺炎和无肺炎三种类型之一。通过 ROC 曲线、混淆矩阵和标准分类指标评估所研究模型的分类性能。实现可解释性模块以探索对分类最重要的图像特征。数据分析和模型评估表明，流行的开源数据集 COVIDx 不能代表真实的临床问题，并且在该数据集上进行测试的结果被夸大了。对开源数据的依赖可能使模型容易受到偏差和混杂因素的影响，因此需要仔细分析，以便为胸部 X 光片 COVID-19 检测开发临床有用/可行的人工智能工具。
