Nguyen Ngoc Huy, Nguyen Ha Quy, Nguyen Nghia Trung, Nguyen Thang Viet, Pham Hieu Huy, Nguyen Tuan Ngoc-Minh
Phu Tho Department of Health, Viet Tri, Vietnam.
Medical Imaging Center, Vingroup Big Data Institute, Hanoi, Vietnam.
Front Digit Health. 2022 Jul 27;4:890759. doi: 10.3389/fdgth.2022.890759. eCollection 2022.
The purpose of this paper is to demonstrate a mechanism for deploying and validating an AI-based system for detecting abnormalities on chest X-ray scans at the Phu Tho General Hospital, Vietnam. We aim to investigate the performance of the system in real-world clinical settings and compare its effectiveness to the in-lab performance.
The AI system was directly integrated into the Hospital's Picture Archiving and Communication System (PACS) after being trained on a fixed annotated dataset from other sources. The system's performance was prospectively measured by matching and comparing the AI results with the radiology reports of 6,285 chest X-ray examinations extracted from the Hospital Information System (HIS) over the last 2 months of 2020. The normal/abnormal status of a radiology report was determined by a set of rules and served as the ground truth.
Our system achieves an F1 score-the harmonic average of the recall and the precision-of 0.653 (95% CI 0.635, 0.671) for detecting any abnormalities on chest X-rays. This corresponds to an accuracy of 79.6%, a sensitivity of 68.6%, and a specificity of 83.9%.
Computer-Aided Diagnosis (CAD) systems for chest radiographs using artificial intelligence (AI) have recently shown great potential as a second opinion for radiologists. However, the performances of such systems were mostly evaluated on a fixed dataset in a retrospective manner and, thus, far from the real performances in clinical practice. Despite a significant drop from the in-lab performance, our result establishes a reasonable level of confidence in applying such a system in real-life situations.
本文旨在展示在越南富寿综合医院部署和验证基于人工智能的胸部X光扫描异常检测系统的机制。我们旨在研究该系统在实际临床环境中的性能，并将其有效性与实验室性能进行比较。
在使用来自其他来源的固定注释数据集进行训练后，该人工智能系统被直接集成到医院的图像存档与通信系统（PACS）中。通过将人工智能结果与从医院信息系统（HIS）中提取的2020年最后两个月的6285份胸部X光检查的放射学报告进行匹配和比较，前瞻性地测量了该系统的性能。放射学报告的正常/异常状态由一组规则确定，并作为基本事实。
我们的系统在检测胸部X光片上的任何异常时，F1分数（召回率和精确率的调和平均值）达到0.653（95%置信区间0.635，0.671）。这对应于79.6%的准确率、68.6%的灵敏度和83.9%的特异性。
使用人工智能（AI）的胸部X光计算机辅助诊断（CAD）系统最近作为放射科医生的第二意见显示出巨大潜力。然而，此类系统的性能大多以回顾性方式在固定数据集上进行评估，因此与临床实践中的实际性能相差甚远。尽管与实验室性能相比有显著下降，但我们的结果为在实际情况中应用此类系统建立了合理的信心水平。
