Gordon Emile B, Towbin Alexander J, Wingrove Peter, Shafique Umber, Haas Brian, Kitts Andrea B, Feldman Jill, Furlan Alessandro
Department of Radiology, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania; Clinical Associate, Department of Radiology, Duke University Medical Center, Department of Radiology, Durham, North Carolina.
Professor and Associate Chief, Department of Radiology (Clinical Operations and Informatics), Neil D. Johnson Chair of Radiology Informatics, University of Cincinnati, Cincinnati, Ohio.
J Am Coll Radiol. 2024 Feb;21(2):353-359. doi: 10.1016/j.jacr.2023.09.011. Epub 2023 Oct 18.
To assess ChatGPT's accuracy, relevance, and readability in answering patients' common imaging-related questions and examine the effect of a simple prompt.
A total of 22 imaging-related questions were developed from categories previously described as important to patients, as follows: safety, the radiology report, the procedure, preparation before imaging, meaning of terms, and medical staff. These questions were posed to ChatGPT with and without a short prompt instructing the model to provide an accurate and easy-to-understand response for the average person. Four board-certified radiologists evaluated the answers for accuracy, consistency, and relevance. Two patient advocates also reviewed responses for their utility for patients. Readability was assessed using the Flesch Kincaid Grade Level. Statistical comparisons were performed using χ and paired t tests.
A total of 264 answers were assessed for both unprompted and prompted questions. Unprompted responses were accurate 83% of the time (218 of 264), which did not significantly change for prompted responses (87% [229 of 264]; P = .2). The consistency of the responses increased from 72% (63 of 88) to 86% (76 of 88) when prompts were given (P = .02). Nearly all responses (99% [261 of 264]) were at least partially relevant for both question types. Fewer unprompted responses were considered fully relevant at 67% (176 of 264), although this increased significantly to 80% when prompts were given (210 of 264; P = .001). The average Flesch Kincaid Grade Level was high at 13.6 [CI, 12.9-14.2], unchanged with the prompt (13.0 [CI, 12.41-13.60], P = .2). None of the responses reached the eighth-grade readability level recommended for patient-facing materials.
ChatGPT demonstrates the potential to respond accurately, consistently, and relevantly to patients' imaging-related questions. However, imperfect accuracy and high complexity necessitate oversight before implementation. Prompts reduced response variability and yielded more-targeted information, but they did not improve readability. ChatGPT has the potential to increase accessibility to health information and streamline the production of patient-facing educational materials; however, its current limitations require cautious implementation and further research.
评估ChatGPT在回答患者常见的影像学相关问题时的准确性、相关性和可读性，并研究一个简单提示的效果。
从先前描述的对患者重要的类别中总共提出了22个影像学相关问题，如下：安全性、放射学报告、检查程序、成像前准备、术语含义和医务人员。这些问题分别在有和没有一个简短提示的情况下向ChatGPT提出，该提示指导模型为普通人提供准确且易于理解的回答。四位获得董事会认证的放射科医生评估答案的准确性、一致性和相关性。两位患者权益倡导者也审查回答对患者的实用性。使用弗莱什-金凯德年级水平评估可读性。使用χ检验和配对t检验进行统计比较。
总共对264个无提示和有提示问题的答案进行了评估。无提示回答的准确率为83%（264个中的218个），有提示回答的准确率没有显著变化（87%[264个中的229个]；P = 0.2）。给出提示后，回答的一致性从72%（88个中的63个）提高到86%（88个中的76个）（P = 0.02）。几乎所有回答（99%[264个中的261个]）对两种问题类型都至少部分相关。无提示回答中被认为完全相关的较少，为67%（264个中的176个），不过给出提示后这一比例显著提高到80%（264个中的210个；P = 0.001）。平均弗莱什-金凯德年级水平较高，为13.6[置信区间，12.9 - 14.2]，有提示时没有变化（13.0[置信区间，12.41 - 13.60]，P = 0.2）。没有一个回答达到面向患者材料推荐的八年级可读性水平。
ChatGPT展示了准确、一致且相关地回答患者影像学相关问题的潜力。然而，准确性不完善和复杂性高需要在实施前进行监督。提示减少了回答的变异性并产生了更有针对性的信息，但没有提高可读性。ChatGPT有潜力增加获取健康信息的机会并简化面向患者的教育材料的制作；然而，其当前的局限性需要谨慎实施和进一步研究。
