Sina Elliott M, Campbell Daniel J, Duffy Alexander, Mandloi Shreya, Benedict Peter, Farquhar Douglas, Unsal Aykut, Nyquist Gurston
Sidney Kimmel Medical College Thomas Jefferson University Philadelphia Pennsylvania USA.
Department of Otolaryngology Thomas Jefferson University Hospital Philadelphia Pennsylvania USA.
OTO Open. 2024 Sep 15;8(3):e70011. doi: 10.1002/oto2.70011. eCollection 2024 Jul-Sep.
While most patients with COVID-19-induced olfactory dysfunction (OD) recover spontaneously, those with persistent OD face significant physical and psychological sequelae. ChatGPT, an artificial intelligence chatbot, has grown as a tool for patient education. This study seeks to evaluate the quality of ChatGPT-generated responses for COVID-19 OD.
Quantitative observational study.
Publicly available online website.
ChatGPT (GPT-4) was queried 4 times with 30 identical questions. Prior to questioning, Chat-GPT was "prompted" to respond (1) to a patient, (2) to an eighth grader, (3) with references, and (4) no prompt. Answer accuracy was independently scored by 4 rhinologists using the Global Quality Score (GCS, range: 1-5). Proportions of responses at incremental score thresholds were compared using analysis. Flesch-Kincaid grade level was calculated for each answer. Relationship between prompt type and grade level was assessed via analysis of variance.
Across all graded responses (n = 480), 364 responses (75.8%) were "at least good" (GCS ≥ 4). Proportions of responses that were "at least good" ( < .0001) or "excellent" (GCS = 5) ( < .0001) differed by prompt; "at least moderate" (GCS ≥ 3) responses did not ( = .687). Eighth-grade level (14.06 ± 2.3) and patient-friendly (14.33 ± 2.0) responses were significantly lower mean grade level than no prompting ( < .0001).
ChatGPT provides appropriate answers to most questions on COVID-19 OD regardless of prompting. However, prompting influences response quality and grade level. ChatGPT responds at grade levels above accepted recommendations for presenting medical information to patients. Currently, ChatGPT offers significant potential for patient education as an adjunct to the conventional patient-physician relationship.
虽然大多数新冠病毒感染所致嗅觉功能障碍（OD）患者可自发恢复，但持续存在嗅觉功能障碍的患者会面临严重的身体和心理后遗症。人工智能聊天机器人ChatGPT已发展成为一种患者教育工具。本研究旨在评估ChatGPT生成的关于新冠病毒感染所致嗅觉功能障碍的回答质量。
定量观察性研究。
公开的在线网站。
用30个相同的问题对ChatGPT（GPT - 4）进行了4次询问。在提问之前，促使ChatGPT做出如下回应：（1）面向患者，（2）面向八年级学生，（3）提供参考文献，（4）不做提示。4位鼻科医生使用全球质量评分（GCS，范围：1 - 5）对回答的准确性进行独立评分。使用分析方法比较了不同分数阈值下回答的比例。计算每个回答的弗莱什 - 金凯德年级水平。通过方差分析评估提示类型与年级水平之间的关系。
在所有分级回答中（n = 480），364个回答（75.8%）“至少良好”（GCS≥4）。“至少良好”（P <.0001）或“优秀”（GCS = 5）（P <.0001）的回答比例因提示方式而异；“至少中等”（GCS≥3）的回答则没有差异（P =.687）。面向八年级学生水平（14.06±2.3）和患者友好型（14.33±2.0）的回答平均年级水平显著低于无提示的回答（P <.0001）。
无论是否有提示，ChatGPT都能为大多数关于新冠病毒感染所致嗅觉功能障碍的问题提供恰当答案。然而，提示会影响回答质量和年级水平。ChatGPT的回答年级水平高于向患者提供医学信息的公认推荐水平。目前，作为传统医患关系的辅助手段，ChatGPT在患者教育方面具有巨大潜力。
