UMass Chan Medical School, Worcester, Massachusetts.
Professor, Vice-Chair, Quality, Safety, and Process Improvement, and Interim Co-CMO, UMass Memorial Medical Center and Department of Radiology, UMass Chan Medical School, Worcester, Massachusetts.
J Am Coll Radiol. 2023 Dec;20(12):1225-1230. doi: 10.1016/j.jacr.2023.06.010. Epub 2023 Jul 8.
The aim of this study was to implement and evaluate a quality assurance (QA) workflow that leverages natural language processing to rapidly resolve inadvertent discordance between radiologists and an artificial intelligence (AI) decision support system (DSS) in the interpretation of high-acuity CT studies when the radiologist does not engage with AI DSS output.
All consecutive high-acuity adult CT examinations performed in a health system between March 1, 2020, and September 20, 2022, were interpreted alongside an AI DSS (Aidoc) for intracranial hemorrhage, cervical spine fracture, and pulmonary embolus. CT studies were flagged for this QA workflow if they met three criteria: (1) negative results by radiologist report, (2) a high probability of positive results by the AI DSS, and (3) unviewed AI DSS output. In these cases, an automated e-mail notification was sent to our quality team. If discordance was confirmed on secondary review-an initially missed diagnosis-addendum and communication documentation was performed.
Of 111,674 high-acuity CT examinations interpreted alongside the AI DSS over this 2.5-year time period, the frequency of missed diagnoses (intracranial hemorrhage, pulmonary embolus, and cervical spine fracture) uncovered by this workflow was 0.02% (n = 26). Of 12,412 CT studies prioritized as depicting positive findings by the AI DSS, 0.4% (n = 46) were discordant, unengaged, and flagged for QA. Among these discordant cases, 57% (26 of 46) were determined to be true positives. Addendum and communication documentation was performed within 24 hours of the initial report signing in 85% of these cases.
Inadvertent discordance between radiologists and the AI DSS occurred in a small number of cases. This QA workflow leveraged natural language processing to rapidly detect, notify, and resolve these discrepancies and prevent potential missed diagnoses.
本研究旨在实施和评估一种质量保证（QA）工作流程，该流程利用自然语言处理技术，在放射科医生不参与人工智能（AI）决策支持系统（DSS）输出的情况下，快速解决放射科医生与 AI DSS 在解读高风险 CT 研究时的意外不一致。
在 2020 年 3 月 1 日至 2022 年 9 月 20 日期间，对医疗系统中进行的所有连续高风险成人 CT 检查均与颅内出血、颈椎骨折和肺栓塞的 AI DSS（Aidoc）一起进行解读。如果 CT 研究符合以下三个标准，则将其标记为该 QA 工作流程：（1）放射科医生报告结果为阴性，（2）AI DSS 高度提示阳性结果，（3）未查看 AI DSS 输出。在这些情况下，将向我们的质量团队发送自动电子邮件通知。如果在二次审查中确认存在不一致-最初漏诊，则会进行补充和沟通记录。
在这段 2.5 年的时间里，有 111674 次高风险 CT 检查与 AI DSS 一起进行解读，通过该工作流程发现的漏诊率（颅内出血、肺栓塞和颈椎骨折）为 0.02%（n=26）。在 AI DSS 提示为阳性的 12412 次 CT 研究中，有 0.4%（n=46）被认为不一致、未参与且被标记为 QA。在这些不一致的病例中，57%（26 例）被确定为真正的阳性。在这些病例中，85%的病例在初始报告签署后 24 小时内完成了补充和沟通记录。
在放射科医生和 AI DSS 之间偶尔会出现不一致的情况。该 QA 工作流程利用自然语言处理技术快速检测、通知和解决这些差异，防止潜在的漏诊。
