Kunst Mara, Gupta Rajiv, Coombs Laura P, Delfino Jana G, Khan Amir, Berglar Inka, Kozak Benjamin, Small Juan E, Gillis Laura, Noonan Patrick, Fang Junyong, Pai Vinay, Tilkin Mike, Allen Bibb, Dreyer Keith, Wald Christoph
Neuroradiology Section Head, Lahey Hospital and Medical Center, Burlington, Massachusetts.
Neuroradiology Section Head, Massachusetts General Hospital, Boston, Massachusetts.
J Am Coll Radiol. 2024 Feb;21(2):329-340. doi: 10.1016/j.jacr.2023.04.003. Epub 2023 May 16.
To evaluate the real-world performance of two FDA-approved artificial intelligence (AI)-based computer-aided triage and notification (CADt) detection devices and compare them with the manufacturer-reported performance testing in the instructions for use.
Clinical performance of two FDA-cleared CADt large-vessel occlusion (LVO) devices was retrospectively evaluated at two separate stroke centers. Consecutive "code stroke" CT angiography examinations were included and assessed for patient demographics, scanner manufacturer, presence or absence of CADt result, CADt result, and LVO in the internal carotid artery (ICA), horizontal middle cerebral artery (MCA) segment (M1), Sylvian MCA segments after the bifurcation (M2), precommunicating part of cerebral artery, postcommunicating part of the cerebral artery, vertebral artery, basilar artery vessel segments. The original radiology report served as the reference standard, and a study radiologist extracted the above data elements from the imaging examination and radiology report.
At hospital A, the CADt algorithm manufacturer reports assessment of intracranial ICA and MCA with sensitivity of 97% and specificity of 95.6%. Real-world performance of 704 cases included 79 in which no CADt result was available. Sensitivity and specificity in ICA and M1 segments were 85.3% and 91.9%. Sensitivity decreased to 68.5% when M2 segments were included and to 59.9% when all proximal vessel segments were included. At hospital B the CADt algorithm manufacturer reports sensitivity of 87.8% and specificity of 89.6%, without specifying the vessel segments. Real-world performance of 642 cases included 20 cases in which no CADt result was available. Sensitivity and specificity in ICA and M1 segments were 90.7% and 97.9%. Sensitivity decreased to 76.4% when M2 segments were included and to 59.4% when all proximal vessel segments are included.
Real-world testing of two CADt LVO detection algorithms identified gaps in the detection and communication of potentially treatable LVOs when considering vessels beyond the intracranial ICA and M1 segments and in cases with absent and uninterpretable data.
评估两种获得美国食品药品监督管理局（FDA）批准的基于人工智能（AI）的计算机辅助分诊与通知（CADt）检测设备在实际应用中的性能，并将其与制造商在使用说明书中报告的性能测试结果进行比较。
在两个独立的卒中中心对两种获得FDA批准的用于检测大血管闭塞（LVO）的CADt设备的临床性能进行回顾性评估。纳入连续的“卒中代码”CT血管造影检查，并对患者人口统计学信息、扫描仪制造商、是否有CADt结果、CADt结果以及颈内动脉（ICA）、大脑中动脉水平段（M1）、大脑中动脉分叉后岛叶段（M2）、大脑动脉交通前段、大脑动脉交通后段、椎动脉、基底动脉血管段的LVO情况进行评估。原始放射学报告作为参考标准，一名研究放射科医生从影像检查和放射学报告中提取上述数据元素。
在医院A，CADt算法制造商报告对颅内ICA和MCA的评估灵敏度为97%，特异度为95.6%。704例病例的实际应用性能中，有79例没有CADt结果。ICA和M1段的灵敏度和特异度分别为85.3%和91.9%。当纳入M2段时，灵敏度降至68.5%；当纳入所有近端血管段时，灵敏度降至59.9%。在医院B，CADt算法制造商报告灵敏度为87.8%，特异度为89.6%，未指明血管段。642例病例的实际应用性能中，有20例没有CADt结果。ICA和M1段的灵敏度和特异度分别为90.7%和97.9%。当纳入M2段时，灵敏度降至76.4%；当纳入所有近端血管段时，灵敏度降至59.4%。
对两种CADt LVO检测算法的实际应用测试发现，在考虑颅内ICA和M1段以外的血管以及数据缺失和无法解读的情况下，在潜在可治疗的LVO的检测和沟通方面存在差距。
