Savardi Mattia, Signoroni Alberto, Benini Sergio, Vaccher Filippo, Alberti Matteo, Ciolli Pietro, Di Meo Nunzia, Falcone Teresa, Ramanzin Marco, Romano Barbara, Sozzi Federica, Farina Davide
Department of Medical and Surgical Specialties, Radiological Sciences and Public Health, University of Brescia, Brescia, Italy.
Department of Information Engineering, University of Brescia, Brescia, Italy.
Insights Imaging. 2025 Jan 29;16(1):23. doi: 10.1186/s13244-024-01893-4.
This article aims to evaluate the use and effects of an artificial intelligence system supporting a critical diagnostic task during radiology resident training, addressing a research gap in this field.
We involved eight residents evaluating 150 CXRs in three scenarios: no AI, on-demand AI, and integrated-AI. The considered task was the assessment of a multi-regional severity score of lung compromise in patients affected by COVID-19. The chosen artificial intelligence tool, fully integrated in the RIS/PACS, demonstrated superior performance in scoring compared to the average radiologist. Using quantitative metrics and questionnaires, we measured the 'upskilling' effects of using AI support and residents' resilience to 'deskilling,' i.e., their ability to overcome AI errors.
Residents required AI in 70% of cases when left free to choose. AI support significantly reduced severity score errors and increased inter-rater agreement by 22%. Residents were resilient to AI errors above an acceptability threshold. Questionnaires indicated high tool usefulness, reliability, and explainability, with a preference for collaborative AI scenarios.
With this work, we gathered quantitative and qualitative evidence of the beneficial use of a high-performance AI tool that is well integrated into the diagnostic workflow as a training aid for radiology residents.
Balancing educational benefits and deskilling risks is essential to exploit AI systems as effective learning tools in radiology residency programs. Our work highlights metrics for evaluating these aspects.
Insights into AI tools' effects in radiology resident training are lacking. Metrics were defined to observe residents using an AI tool in different settings. This approach is advisable for evaluating AI tools in radiology training.
本文旨在评估一种人工智能系统在放射科住院医师培训期间支持关键诊断任务的使用情况和效果，以填补该领域的研究空白。
我们让八名住院医师在三种场景下评估150份胸部X光片：无人工智能、按需使用人工智能和集成人工智能。所考虑的任务是评估感染新冠病毒患者肺部损伤的多区域严重程度评分。所选的人工智能工具完全集成在放射信息系统/图像存档与通信系统中，与普通放射科医生相比，在评分方面表现更优。我们使用定量指标和问卷，测量了使用人工智能支持的“技能提升”效果以及住院医师对“技能退化”的恢复能力，即他们克服人工智能错误的能力。
在可以自由选择的情况下，70%的病例中住院医师需要人工智能。人工智能支持显著减少了严重程度评分错误，并使评分者间的一致性提高了22%。住院医师对高于可接受阈值的人工智能错误具有恢复能力。问卷显示该工具具有很高的实用性、可靠性和可解释性，且住院医师更倾向于协作式人工智能场景。
通过这项研究，我们收集了定量和定性证据，证明了一种高性能人工智能工具作为放射科住院医师培训辅助工具，在诊断工作流程中得到良好整合后的有益应用。
在放射科住院医师培训项目中，平衡教育益处和技能退化风险对于将人工智能系统用作有效的学习工具至关重要。我们的研究突出了评估这些方面的指标。
目前缺乏关于人工智能工具在放射科住院医师培训中效果的见解。定义了指标以观察住院医师在不同环境下使用人工智能工具的情况。这种方法对于评估放射科培训中的人工智能工具是可取的。
