López Alcolea Julia, Fernández Alfonso Ana, Cano Alonso Raquel, Álvarez Vázquez Ana, Díaz Moreno Alejandro, García Castellanos David, Sanabria Greciano Lucía, Hayoun Chawar, Recio Rodríguez Manuel, Andreu Vázquez Cristina, Thuissard Vasallo Israel John, Martínez de Vega Vicente
Hospital Universitario QuironSalud Madrid, 28223 Madrid, Spain.
Faculty of Biomedical and Health Science, Universidad Europea de Madrid, 28670 Madrid, Spain.
Diagnostics (Basel). 2024 Nov 18;14(22):2592. doi: 10.3390/diagnostics14222592.
The increasing integration of AI in chest X-ray evaluation holds promise for enhancing diagnostic accuracy and optimizing clinical workflows. However, understanding its performance in real-world clinical settings is essential.
In this study, we evaluated the sensitivity (Se) and specificity (Sp) of an AI-based software (Arterys MICA v29.4.0) alongside a radiology resident in interpreting chest X-rays referred from the emergency department (ED), using a senior radiologist's assessment as the gold standard (GS). We assessed the concordance between the AI system and the resident, noted the frequency of doubtful cases for each category, identified how many were considered positive by the GS, and assessed variables that AI was not trained to detect.
We conducted a retrospective observational study analyzing chest X-rays from a sample of 784 patients referred from the ED at our hospital. The AI system was trained to detect five categorical variables-pulmonary nodule, pulmonary opacity, pleural effusion, pneumothorax, and fracture-and assign each a confidence label ("positive", "doubtful", or "negative").
Sensitivity in detecting fractures and pneumothorax was high (100%) for both AI and the resident, moderate for pulmonary opacity (AI = 76%, resident = 71%), and acceptable for pleural effusion (AI = 60%, resident = 67%), with negative predictive values (NPV) above 95% and areas under the curve (AUC) exceeding 0.8. The resident showed moderate sensitivity (75%) for pulmonary nodules, while AI's sensitivity was low (33%). AI assigned a "doubtful" label to some diagnoses, most of which were deemed negative by the GS; the resident expressed doubt less frequently. The Kappa coefficient between the resident and AI was fair (0.3) across most categories, except for pleural effusion, where concordance was moderate (0.5). Our study highlighted additional findings not detected by AI, including 16% prevalence of mediastinal abnormalities, 20% surgical materials, and 20% other pulmonary findings.
Although AI demonstrated utility in identifying most primary findings-except for pulmonary nodules-its high NPV suggests it may be valuable for screening. Further training of the AI software and broadening its scope to identify additional findings could enhance its detection capabilities and increase its applicability in clinical practice.
人工智能在胸部X线评估中的日益融合有望提高诊断准确性并优化临床工作流程。然而，了解其在实际临床环境中的表现至关重要。
在本研究中，我们以一位资深放射科医生的评估作为金标准，评估了基于人工智能的软件（Arterys MICA v29.4.0）与放射科住院医师在解读急诊科转诊的胸部X线片时的敏感性（Se）和特异性（Sp）。我们评估了人工智能系统与住院医师之间的一致性，记录了每个类别中可疑病例的频率，确定了金标准认为阳性的病例数量，并评估了人工智能未训练检测的变量。
我们进行了一项回顾性观察研究，分析了我院急诊科转诊的784例患者的胸部X线片样本。人工智能系统经过训练以检测五个分类变量——肺结节、肺部实变、胸腔积液、气胸和骨折——并为每个变量分配一个置信标签（“阳性”、“可疑”或“阴性”）。
人工智能和住院医师检测骨折和气胸的敏感性都很高（100%），检测肺部实变的敏感性中等（人工智能 = 76%，住院医师 = 71%），检测胸腔积液的敏感性尚可（人工智能 = 60%，住院医师 = 67%），阴性预测值（NPV）高于95%，曲线下面积（AUC）超过0.8。住院医师检测肺结节的敏感性中等（75%），而人工智能的敏感性较低（33%）。人工智能为一些诊断分配了“可疑”标签，其中大多数被金标准判定为阴性；住院医师表达怀疑的频率较低。除胸腔积液的一致性为中等（0.5）外，住院医师与人工智能之间的Kappa系数在大多数类别中为一般（0.3）。我们的研究突出了人工智能未检测到的其他发现，包括16%的纵隔异常患病率、20%的手术材料和20%的其他肺部发现。
尽管人工智能在识别大多数主要发现方面显示出效用——除了肺结节——但其高阴性预测值表明它可能对筛查有价值。对人工智能软件进行进一步训练并扩大其识别其他发现的范围，可以提高其检测能力并增加其在临床实践中的适用性。
