The First Clinical Medical College of Jinan University, The First Affiliated Hospital of Jinan University, Guangzhou, China.
Department of Joint Surgery and Sports Medicine, Zhuhai People's Hospital (Zhuhai Hospital Affiliated With Jinan University), Zhuhai, Guangdong, China.
J Med Internet Res. 2024 Aug 20;26:e57037. doi: 10.2196/57037.
ChatGPT is a natural language processing model developed by OpenAI, which can be iteratively updated and optimized to accommodate the changing and complex requirements of human verbal communication.
The study aimed to evaluate ChatGPT's accuracy in answering orthopedics-related multiple-choice questions (MCQs) and assess its short-term effects as a learning aid through a randomized controlled trial. In addition, long-term effects on student performance in other subjects were measured using final examination results.
We first evaluated ChatGPT's accuracy in answering MCQs pertaining to orthopedics across various question formats. Then, 129 undergraduate medical students participated in a randomized controlled study in which the ChatGPT group used ChatGPT as a learning tool, while the control group was prohibited from using artificial intelligence software to support learning. Following a 2-week intervention, the 2 groups' understanding of orthopedics was assessed by an orthopedics test, and variations in the 2 groups' performance in other disciplines were noted through a follow-up at the end of the semester.
ChatGPT-4.0 answered 1051 orthopedics-related MCQs with a 70.60% (742/1051) accuracy rate, including 71.8% (237/330) accuracy for A1 MCQs, 73.7% (330/448) accuracy for A2 MCQs, 70.2% (92/131) accuracy for A3/4 MCQs, and 58.5% (83/142) accuracy for case analysis MCQs. As of April 7, 2023, a total of 129 individuals participated in the experiment. However, 19 individuals withdrew from the experiment at various phases; thus, as of July 1, 2023, a total of 110 individuals accomplished the trial and completed all follow-up work. After we intervened in the learning style of the students in the short term, the ChatGPT group answered more questions correctly than the control group (ChatGPT group: mean 141.20, SD 26.68; control group: mean 130.80, SD 25.56; P=.04) in the orthopedics test, particularly on A1 (ChatGPT group: mean 46.57, SD 8.52; control group: mean 42.18, SD 9.43; P=.01), A2 (ChatGPT group: mean 60.59, SD 10.58; control group: mean 56.66, SD 9.91; P=.047), and A3/4 MCQs (ChatGPT group: mean 19.57, SD 5.48; control group: mean 16.46, SD 4.58; P=.002). At the end of the semester, we found that the ChatGPT group performed better on final examinations in surgery (ChatGPT group: mean 76.54, SD 9.79; control group: mean 72.54, SD 8.11; P=.02) and obstetrics and gynecology (ChatGPT group: mean 75.98, SD 8.94; control group: mean 72.54, SD 8.66; P=.04) than the control group.
ChatGPT answers orthopedics-related MCQs accurately, and students using it excel in both short-term and long-term assessments. Our findings strongly support ChatGPT's integration into medical education, enhancing contemporary instructional methods.
Chinese Clinical Trial Registry Chictr2300071774; https://www.chictr.org.cn/hvshowproject.html ?id=225740&v=1.0.
ChatGPT 是由 OpenAI 开发的自然语言处理模型，它可以进行迭代更新和优化，以适应人类言语交流不断变化和复杂的需求。
本研究旨在评估 ChatGPT 回答骨科相关多项选择题（MCQ）的准确性，并通过随机对照试验评估其作为学习辅助工具的短期效果。此外，还通过期末考试成绩来衡量其对其他学科学生成绩的长期影响。
我们首先评估了 ChatGPT 在回答各种问题格式的骨科 MCQ 方面的准确性。然后，129 名本科医学生参加了一项随机对照研究，其中 ChatGPT 组将 ChatGPT 用作学习工具，而对照组则禁止使用人工智能软件支持学习。经过两周的干预，通过骨科测试评估两组对骨科的理解，通过学期末的随访记录两组在其他学科表现的变化。
ChatGPT-4.0 回答了 1051 个骨科相关 MCQ，准确率为 70.60%（742/1051），包括 A1 MCQ 准确率为 71.8%（237/330），A2 MCQ 准确率为 73.7%（330/448），A3/4 MCQ 准确率为 70.2%（92/131），案例分析 MCQ 准确率为 58.5%（83/142）。截至 2023 年 4 月 7 日，共有 129 人参与了实验。然而，19 人在不同阶段退出了实验；因此，截至 2023 年 7 月 1 日，共有 110 人完成了试验并完成了所有随访工作。在我们干预学生的短期学习方式后，ChatGPT 组在骨科测试中答对的问题多于对照组（ChatGPT 组：平均 141.20，SD 26.68；对照组：平均 130.80，SD 25.56；P=.04），尤其是在 A1（ChatGPT 组：平均 46.57，SD 8.52；对照组：平均 42.18，SD 9.43；P=.01）、A2（ChatGPT 组：平均 60.59，SD 10.58；对照组：平均 56.66，SD 9.91；P=.047）和 A3/4 MCQ（ChatGPT 组：平均 19.57，SD 5.48；对照组：平均 16.46，SD 4.58；P=.002）。在学期末，我们发现 ChatGPT 组在手术（ChatGPT 组：平均 76.54，SD 9.79；对照组：平均 72.54，SD 8.11；P=.02）和妇产科（ChatGPT 组：平均 75.98，SD 8.94；对照组：平均 72.54，SD 8.66；P=.04）期末考试中的表现优于对照组。
ChatGPT 可以准确回答骨科相关的 MCQ，使用它的学生在短期和长期评估中都表现出色。我们的研究结果强烈支持将 ChatGPT 融入医学教育，增强当代教学方法。
中国临床试验注册中心 Chictr2300071774；https://www.chictr.org.cn/hvshowproject.html? id = 225740&v = 1.0.
