Rudolph Jan, Huemmer Christian, Ghesu Florin-Cristian, Mansoor Awais, Preuhs Alexander, Fieselmann Andreas, Fink Nicola, Dinkel Julien, Koliogiannis Vanessa, Schwarze Vincent, Goller Sophia, Fischer Maximilian, Jörgens Maximilian, Ben Khaled Najib, Vishwanath Reddappagari Suryanarayana, Balachandran Abishek, Ingrisch Michael, Ricke Jens, Sabel Bastian Oliver, Rueckel Johannes
From the Department of Radiology, University Hospital, LMU Munich, Munich, Germany.
X-Ray Products, Siemens Healthineers, Forchheim, Germany.
Invest Radiol. 2022 Feb 1;57(2):90-98. doi: 10.1097/RLI.0000000000000813.
Chest radiographs (CXRs) are commonly performed in emergency units (EUs), but the interpretation requires radiology experience. We developed an artificial intelligence (AI) system (precommercial) that aims to mimic board-certified radiologists' (BCRs') performance and can therefore support non-radiology residents (NRRs) in clinical settings lacking 24/7 radiology coverage. We validated by quantifying the clinical value of our AI system for radiology residents (RRs) and EU-experienced NRRs in a clinically representative EU setting.
A total of 563 EU CXRs were retrospectively assessed by 3 BCRs, 3 RRs, and 3 EU-experienced NRRs. Suspected pathologies (pleural effusion, pneumothorax, consolidations suspicious for pneumonia, lung lesions) were reported on a 5-step confidence scale (sum of 20,268 reported pathology suspicions [563 images × 9 readers × 4 pathologies]) separately by every involved reader. Board-certified radiologists' confidence scores were converted into 4 binary reference standards (RFSs) of different sensitivities. The RRs' and NRRs' performances were statistically compared with our AI system (trained on nonpublic data from different clinical sites) based on receiver operating characteristics (ROCs) and operating point metrics approximated to the maximum sum of sensitivity and specificity (Youden statistics).
The NRRs lose diagnostic accuracy to RRs with increasingly sensitive BCRs' RFSs for all considered pathologies. Based on our external validation data set, the AI system/NRRs' consensus mimicked the most sensitive BCRs' RFSs with areas under ROC of 0.940/0.837 (pneumothorax), 0.953/0.823 (pleural effusion), and 0.883/0.747 (lung lesions), which were comparable to experienced RRs and significantly overcomes EU-experienced NRRs' diagnostic performance. For consolidation detection, the AI system performed on the NRRs' consensus level (and overcomes each individual NRR) with an area under ROC of 0.847 referenced to the BCRs' most sensitive RFS.
Our AI system matched RRs' performance, meanwhile significantly outperformed NRRs' diagnostic accuracy for most of considered CXR pathologies (pneumothorax, pleural effusion, and lung lesions) and therefore might serve as clinical decision support for NRRs.
胸部X光片（CXR）在急诊科（EU）中经常进行检查，但解读需要放射学经验。我们开发了一种人工智能（AI）系统（预商业化），旨在模仿获得委员会认证的放射科医生（BCR）的表现，因此可以在缺乏全天候放射科服务的临床环境中为非放射科住院医师（NRR）提供支持。我们通过量化我们的AI系统在具有临床代表性的急诊科环境中对放射科住院医师（RR）和有急诊科经验的NRR的临床价值进行了验证。
共有563份急诊科CXR由3名BCR、3名RR和3名有急诊科经验的NRR进行回顾性评估。每位参与的阅片者分别以5级置信度量表报告可疑病变（胸腔积液、气胸、可疑肺炎实变、肺部病变）（共20268份报告的病变可疑度[563张图像×9名阅片者×4种病变]）。BCR的置信度评分被转换为4种不同敏感度的二元参考标准（RFS）。基于接收器操作特征（ROC）和接近敏感度与特异度之和最大值的操作点指标（约登统计量），将RR和NRR的表现与我们的AI系统（基于来自不同临床地点的非公开数据进行训练）进行统计学比较。
对于所有考虑的病变，随着BCR的RFS敏感度增加，NRR相对于RR的诊断准确性降低。基于我们的外部验证数据集，AI系统/NRR的一致性模仿了最敏感的BCR的RFS，气胸的ROC曲线下面积为0.940/0.837，胸腔积液为0.953/0.823，肺部病变为0.88 /0.747，与经验丰富的RR相当，且显著优于有急诊科经验的NRR的诊断表现。对于实变检测，AI系统在NRR的一致性水平上表现（并超过了每个单独的NRR），相对于BCR最敏感的RFS，ROC曲线下面积为0.847。
我们的AI系统与RR的表现相当，同时在大多数考虑的CXR病变（气胸、胸腔积液和肺部病变）中显著优于NRR的诊断准确性，因此可能为NRR提供临床决策支持。
