Lack Benjamin T, Mouhawasse Edwin, Childers Justin T, Jackson Garrett R, Daji Shay V, Yerke-Hansen Payton, Familiari Filippo, Knapik Derrick M, Sabesan Vani J
Charles E. Schmidt Florida Atlantic University College of Medicine, Boca Raton, FL, USA.
Department of Orthopaedic Surgery, University of Missouri, Columbia, MO 65212, USA.
J ISAKOS. 2024 Dec;9(6):100323. doi: 10.1016/j.jisako.2024.100323. Epub 2024 Sep 20.
In recent years, artificial intelligence (AI) has seen substantial progress in its utilization, with Chat Generated Pre-Trained Transformer (ChatGPT) is emerging as a popular language model. The purpose of this study was to test the accuracy and reliability of ChatGPT's responses to frequently asked questions (FAQ) pertaining to reverse shoulder arthroplasty (RSA).
The ten most common FAQs were queried from institution patient education websites. These ten questions were then input into the chatbot during a single session without additional contextual information. The responses were then critically analyzed by two orthopedic surgeons for clarity, accuracy, and the quality of evidence-based information using The Journal of the American Medical Association (JAMA) Benchmark criteria and the DISCERN score. The readability of the responses was analyzed using the Flesch-Kincaid Grade Level.
In response to the ten questions, the average DISCERN score was 44 (range 38-51). Seven responses were classified as fair and three were poor. The JAMA Benchmark criteria score was 0 for all responses. Furthermore, the average Flesch-Kincaid Grade Level was 14.35, which correlates to a college graduate reading level.
Overall, ChatGPT was able to provide fair responses to common patient questions. However, the responses were all written at a college graduate reading level and lacked reliable citations. The readability greatly limits its utility. Thus, adequate patient education should be done by orthopedic surgeons. This study underscores the need for patient education resources that are reliable, accessible, and comprehensible.
IV.
近年来，人工智能（AI）在其应用方面取得了重大进展，基于聊天生成的预训练变换器（ChatGPT）作为一种流行的语言模型正在兴起。本研究的目的是测试ChatGPT对与反肩关节置换术（RSA）相关的常见问题（FAQ）的回答的准确性和可靠性。
从机构患者教育网站查询了十个最常见的常见问题。然后在单个会话中将这十个问题输入聊天机器人，不提供额外的背景信息。然后由两名骨科医生使用《美国医学会杂志》（JAMA）基准标准和辨别分数对回答进行批判性分析，以评估清晰度、准确性和循证信息的质量。使用弗莱什-金凯德年级水平分析回答的可读性。
针对这十个问题，平均辨别分数为44（范围为38 - 51）。七个回答被归类为中等，三个回答较差。所有回答的JAMA基准标准分数均为0。此外，平均弗莱什-金凯德年级水平为14.35，这与大学毕业生的阅读水平相关。
总体而言，ChatGPT能够对常见的患者问题提供中等质量的回答。然而，这些回答都是以大学毕业生的阅读水平撰写的，并且缺乏可靠的引用。可读性极大地限制了其效用。因此，骨科医生应该进行充分的患者教育。本研究强调了对可靠、可获取且易于理解的患者教育资源的需求。
IV级。
