School of Medicine, Department of Orthopaedics and Traumatology, Division of Hand Surgery, University of Mersin, Mersin, 33110, Turkey.
School of Medicine, Department of Orthopedics and Traumatology, Ömer Halisdemir University, Niğde, Turkey.
BMC Musculoskelet Disord. 2024 Nov 4;25(1):879. doi: 10.1186/s12891-024-07983-0.
BACKGROUND: This study aimed to assess the quality and readability of large language model-generated responses to frequently asked questions (FAQs) about Kienböck's disease (KD). METHODS: Nineteen FAQs about KD were selected, and the questions were divided into three categories: general knowledge, diagnosis, and treatment. The questions were inputted into the Chat Generative Pre-trained Transformer 4 (ChatGPT4) webpage using the zero-shot prompting method, and the responses were recorded. Hand surgeons with at least 5 years of experience and advanced English proficiency were individually contacted over instant WhatsApp messaging and requested to assess the responses. The quality of each response was analyzed by 33 experienced hand surgeons using the Global Quality Scale (GQS). The readability was assessed with the Flesch-Kincaid Grade Level (FKGL) and Flesch Reading Ease Score (FRES). RESULTS: The mean GQS score was 4.28 out of a maximum of 5 points. Most raters assessed the quality as good (270 of 627 responses; 43.1%) or excellent (260 of 627 responses; 41.5%). The mean FKGL was 15.5, and the mean FRES was 23.4, both of which are considered above the college graduate level. No statistically significant differences were found in the quality and readability of responses provided for questions related to general knowledge, diagnosis, and treatment. CONCLUSIONS: ChatGPT-4 provided high-quality responses to FAQs about KD. However, the primary drawback was the poor readability of these responses. By improving the readability of ChatGPT's output, we can transform it into a valuable information resource for individuals with KD. LEVEL OF EVIDENCE: Level IV, Observational study.
背景:本研究旨在评估大型语言模型生成的关于月骨骨软骨病(KD)常见问题(FAQ)解答的质量和可读性。
方法:选择了 19 个关于 KD 的 FAQ,问题分为三个类别:一般知识、诊断和治疗。使用零样本提示方法将问题输入 ChatGPT4 网页,记录回复。联系了至少有 5 年经验和高级英语水平的手外科医生,通过即时 WhatsApp 消息请求他们单独评估回复。33 名经验丰富的手外科医生使用全球质量量表(GQS)分析每个回复的质量。使用 Flesch-Kincaid 年级水平(FKGL)和 Flesch 阅读舒适度得分(FRES)评估可读性。
结果:GQS 的平均得分为 5 分制中的 4.28 分。大多数评分者将质量评估为良好(270/627 次回复;43.1%)或优秀(260/627 次回复;41.5%)。平均 FKGL 为 15.5,平均 FRES 为 23.4,均高于大学毕业水平。在与一般知识、诊断和治疗相关的问题提供的回复的质量和可读性方面,未发现统计学差异。
结论:ChatGPT-4 对 KD 的 FAQ 提供了高质量的回复。然而,主要缺点是这些回复的可读性差。通过提高 ChatGPT 输出的可读性,我们可以将其转化为 KD 患者的有价值的信息资源。
证据水平:IV 级,观察性研究。
BMC Musculoskelet Disord. 2024-11-4
Sci Rep. 2024-7-26
J Clin Med. 2024-8-28
Healthcare (Basel). 2023-10-20
Br J Psychiatry. 2024-2
J Med Internet Res. 2023-10-4
J Clin Sleep Med. 2023-12-1
J Bone Joint Surg Am. 2023-10-4