Johnson Christopher K, Mandalia Krishna, Corban Jason, Beall Kaley E, Shah Sarav S
Department of Orthopedic Surgery, New England Baptist Hospital, Boston, MA, USA.
Tufts University School of Medicine, Boston, MA, USA.
JSES Int. 2025 Feb 6;9(3):830-836. doi: 10.1016/j.jseint.2025.01.008. eCollection 2025 May.
Artificial intelligence (AI) large language models, such as ChatGPT, have numerous novel applications in medicine, one of which is patient education. Several studies in other specialties have investigated the adequacy of ChatGPT-generated responses to frequently asked questions (FAQs) by patients, with largely positive results. The purpose of this study is to evaluate the accuracy and clarity of ChatGPT-generated responses to website-derived FAQs relating to shoulder arthroplasty.
Ten questions regarding shoulder arthroplasty were compiled from the websites of 5 leading academic institutions. These questions were rated on a scale from 1 to 4, corresponding to "excellent response not requiring clarification," "satisfactory requiring minimal clarification," "satisfactory requiring moderate clarification," and "unsatisfactory requiring substantial clarification," respectively, by 2 orthopedic surgeons. A senior shoulder arthroplasty surgeon arbitrated disagreements. Cohen's Kappa coefficient was utilized to assess inter-rater agreement.
After arbitration, only one response was rated as "excellent response not requiring clarification." Nine of 10 responses required clarification. Four were rated as a "satisfactory requiring minimal clarification," 5 were rated as a "satisfactory requiring moderate clarification," and none were rated as "unsatisfactory requiring substantial clarification". The Kappa coefficient was 0.516 ( = .027), indicating moderate agreement between reviewers.
When queried with FAQs regarding shoulder arthroplasty, ChatGPT's responses were all deemed 'satisfactory', but most required clarification. This may be due to the nuances of anatomic vs. reverse shoulder replacement. Thus, patients may find benefit in using ChatGPT to guide whether or not they should seek medical attention, but are limited in the detail and accuracy of treatment-related questions. While a helpful tool to start provider-patient conversations, it does not appear that ChatGPT provides quality, verified, data-driven answers at this time, and should be used cautiously in conjunction to provider-patient discussions. Although the use of ChatGPT in answering FAQs is limited at the moment, orthopedic surgeons should continue to monitor the use of ChatGPT as a patient education tool, as well as the expanding use of AI as a possible adjunct in clinical decision-making.
人工智能（AI）大语言模型，如ChatGPT，在医学领域有众多新颖应用，其中之一是患者教育。其他专业的多项研究调查了ChatGPT生成的对患者常见问题（FAQ）的回答是否充分，结果大多是积极的。本研究的目的是评估ChatGPT生成的对源自网站的与肩关节置换术相关常见问题的回答的准确性和清晰度。
从5家领先学术机构的网站上收集了10个关于肩关节置换术的问题。由2名骨科医生根据1至4分的评分标准进行评分，分别对应“无需澄清的优秀回答”“只需最少澄清的满意回答”“需要适度澄清的满意回答”和“需要大量澄清的不满意回答”。一位资深肩关节置换术外科医生对分歧进行仲裁。使用科恩kappa系数评估评分者间的一致性。
仲裁后，只有一个回答被评为“无需澄清的优秀回答”。10个回答中有9个需要澄清。4个被评为“只需最少澄清即可满意”，5个被评为“需要适度澄清才能满意”，没有一个被评为“需要大量澄清的不满意回答”。kappa系数为0.516（P = 0.027），表明评审者之间有中度一致性。
当被问及与肩关节置换术相关的常见问题时，ChatGPT的回答都被认为“令人满意”，但大多数都需要澄清。这可能是由于解剖型与反式肩关节置换的细微差别。因此，患者使用ChatGPT来指导他们是否应该寻求医疗关注可能会有好处，但在与治疗相关问题的细节和准确性方面存在局限性。虽然它是开启医患对话的有用工具，但目前ChatGPT似乎并不能提供高质量、经过验证、数据驱动的答案，在医患讨论中应谨慎使用。尽管目前ChatGPT在回答常见问题方面的应用有限，但骨科医生应继续监测ChatGPT作为患者教育工具的使用情况，以及人工智能作为临床决策可能辅助手段的日益广泛应用。
