Liu Chiu-Liang, Ho Chien-Ta, Wu Tzu-Chi
Graduate Institute of Technology Management, National Chung-Hsing University, Taichung 402202, Taiwan.
College of Health Sciences, Central Taiwan University of Science and Technology, Taichung 406053, Taiwan.
Healthcare (Basel). 2024 Aug 30;12(17):1726. doi: 10.3390/healthcare12171726.
Given the widespread application of ChatGPT, we aim to evaluate its proficiency in the emergency medicine specialty written examination. Additionally, we compare the performance of GPT-3.5, GPT-4, GPTs, and GPT-4o. The research seeks to ascertain whether custom GPTs possess the essential capabilities and access to knowledge bases necessary for providing accurate information, and to explore the effectiveness and potential of personalized knowledge bases in supporting the education of medical residents. We evaluated the performance of ChatGPT-3.5, GPT-4, custom GPTs, and GPT-4o on the Emergency Medicine Specialist Examination in Taiwan. Two hundred single-choice exam questions were provided to these AI models, and their responses were recorded. Correct rates were compared among the four models, and the McNemar test was applied to paired model data to determine if there were significant changes in performance. Out of 200 questions, GPT-3.5, GPT-4, custom GPTs, and GPT-4o correctly answered 77, 105, 119, and 138 questions, respectively. GPT-4o demonstrated the highest performance, significantly better than GPT-4, which, in turn, outperformed GPT-3.5, while custom GPTs exhibited superior performance compared to GPT-4 but inferior performance compared to GPT-4o, with all < 0.05. In the emergency medicine specialty written exam, our findings highlight the value and potential of large language models (LLMs), and highlight their strengths and limitations, especially in question types and image-inclusion capabilities. Not only do GPT-4o and custom GPTs facilitate exam preparation, but they also elevate the evidence level in responses and source accuracy, demonstrating significant potential to transform educational frameworks and clinical practices in medicine.
鉴于ChatGPT的广泛应用，我们旨在评估其在急诊医学专业笔试中的能力。此外，我们比较了GPT-3.5、GPT-4、GPTs和GPT-4o的表现。该研究旨在确定定制的GPTs是否具备提供准确信息所需的基本能力和知识库访问权限，并探索个性化知识库在支持住院医师教育方面的有效性和潜力。我们评估了ChatGPT-3.5、GPT-4、定制GPTs和GPT-4o在台湾急诊医学专科考试中的表现。向这些人工智能模型提供了200道单项选择题，并记录了它们的答案。比较了四个模型的正确率，并对配对模型数据应用McNemar检验以确定表现是否有显著变化。在200道题中，GPT-3.5、GPT-4、定制GPTs和GPT-4o分别正确回答了77、105、119和138道题。GPT-4o表现最佳，明显优于GPT-4，而GPT-4又优于GPT-3.5，定制GPTs的表现优于GPT-4但劣于GPT-4o，所有p<0.05。在急诊医学专业笔试中，我们的研究结果突出了大语言模型（LLMs）的价值和潜力，并突出了它们的优势和局限性，特别是在题型和包含图像的能力方面。GPT-4o和定制GPTs不仅有助于考试准备，还提高了回答中的证据水平和来源准确性，显示出在改变医学教育框架和临床实践方面的巨大潜力。
