Baglivo Francesco, De Angelis Luigi, Casigliani Virginia, Arzilli Guglielmo, Privitera Gaetano Pierpaolo, Rizzo Caterina
Department of Translational Research and New Technologies in Medicine and Surgery, University of Pisa, Pisa (PI), Italy.
Training Office, National Institute of Health, Rome, Italy.
JMIR Med Educ. 2023 Nov 1;9:e51421. doi: 10.2196/51421.
Artificial intelligence (AI) is a rapidly developing field with the potential to transform various aspects of health care and public health, including medical training. During the "Hygiene and Public Health" course for fifth-year medical students, a practical training session was conducted on vaccination using AI chatbots as an educational supportive tool. Before receiving specific training on vaccination, the students were given a web-based test extracted from the Italian National Medical Residency Test. After completing the test, a critical correction of each question was performed assisted by AI chatbots.
The main aim of this study was to identify whether AI chatbots can be considered educational support tools for training in public health. The secondary objective was to assess the performance of different AI chatbots on complex multiple-choice medical questions in the Italian language.
A test composed of 15 multiple-choice questions on vaccination was extracted from the Italian National Medical Residency Test using targeted keywords and administered to medical students via Google Forms and to different AI chatbot models (Bing Chat, ChatGPT, Chatsonic, Google Bard, and YouChat). The correction of the test was conducted in the classroom, focusing on the critical evaluation of the explanations provided by the chatbot. A Mann-Whitney U test was conducted to compare the performances of medical students and AI chatbots. Student feedback was collected anonymously at the end of the training experience.
In total, 36 medical students and 5 AI chatbot models completed the test. The students achieved an average score of 8.22 (SD 2.65) out of 15, while the AI chatbots scored an average of 12.22 (SD 2.77). The results indicated a statistically significant difference in performance between the 2 groups (U=49.5, P<.001), with a large effect size (r=0.69). When divided by question type (direct, scenario-based, and negative), significant differences were observed in direct (P<.001) and scenario-based (P<.001) questions, but not in negative questions (P=.48). The students reported a high level of satisfaction (7.9/10) with the educational experience, expressing a strong desire to repeat the experience (7.6/10).
This study demonstrated the efficacy of AI chatbots in answering complex medical questions related to vaccination and providing valuable educational support. Their performance significantly surpassed that of medical students in direct and scenario-based questions. The responsible and critical use of AI chatbots can enhance medical education, making it an essential aspect to integrate into the educational system.
人工智能（AI）是一个快速发展的领域，有潜力改变医疗保健和公共卫生的各个方面，包括医学培训。在为五年级医学生开设的“卫生与公共卫生”课程中，使用人工智能聊天机器人作为教育辅助工具进行了一次关于疫苗接种的实践培训。在接受关于疫苗接种的具体培训之前，学生们进行了一次从意大利国家医学住院医师考试中提取的基于网络的测试。完成测试后，在人工智能聊天机器人的辅助下对每个问题进行了批判性纠正。
本研究的主要目的是确定人工智能聊天机器人是否可被视为公共卫生培训的教育辅助工具。次要目的是评估不同人工智能聊天机器人在意大利语复杂多项选择题上的表现。
使用目标关键词从意大利国家医学住院医师考试中提取了一份由15道关于疫苗接种的多项选择题组成的测试，并通过谷歌表单将其施测于医学生以及不同的人工智能聊天机器人模型（必应聊天、ChatGPT、Chatsonic、谷歌巴德和YouChat）。测试的批改在课堂上进行，重点是对聊天机器人提供的解释进行批判性评估。进行了曼-惠特尼U检验以比较医学生和人工智能聊天机器人的表现。在培训结束时匿名收集学生反馈。
共有36名医学生和5个人工智能聊天机器人模型完成了测试。学生们在15道题中平均得分为8.22（标准差2.65），而人工智能聊天机器人的平均得分为12.22（标准差2.77）。结果表明两组在表现上存在统计学显著差异（U = 49.5，P <.001），效应量较大（r = 0.69）。按问题类型（直接、基于情景和否定）划分时，在直接问题（P <.001）和基于情景的问题（P <.001）中观察到显著差异，但在否定问题中未观察到显著差异（P =.48）。学生们对教育体验的满意度较高（7.9/10），表示非常希望再次体验（7.6/10）。
本研究证明了人工智能聊天机器人在回答与疫苗接种相关的复杂医学问题以及提供有价值的教育支持方面的有效性。它们在直接和基于情景的问题上的表现明显超过医学生。负责任且批判性地使用人工智能聊天机器人可以加强医学教育，使其成为融入教育系统的一个重要方面。
