Roy Asitava Deb, Das Dipmala, Mondal Himel
Department of Pathology, Mata Gujri Memorial Medical College and Lions Seva Kendra Hospital, Kishanganj, Bihar, India.
Department Microbiology, Mata Gujri Memorial Medical College and Lions Seva Kendra Hospital, Kishanganj, Bihar, India.
J Educ Health Promot. 2024 Feb 7;13:22. doi: 10.4103/jehp.jehp_625_23. eCollection 2024.
Competency-based medical education (CBME) is a method of medical training that focuses on developing learners' competencies rather than simply assessing their knowledge and skills. Attitude, ethics, and communication (AETCOM) are important components of CBME, and the use of artificial intelligence (AI) tools such as ChatGPT for CBME has not been studied. Hence, we aimed to assess the capability of ChatGPT in solving AETCOM case scenarios used for CBME in India.
A total of 11 case scenarios were developed based on the AETCOM competencies. The scenarios were presented to ChatGPT, and the responses generated by ChatGPT were evaluated by three independent experts by awarding score ranging from 0 to 5. The scores were compared with a predefined score of 2.5 (50% accuracy) and 4 (80% accuracy) of a one-sample median test. Scores among the three raters were compared by the Kruskal-Wallis H test. The inter-rater reliability of the evaluations was assessed using the intraclass correlation coefficient (ICC).
The mean score of solution provided by ChatGPT was 3.88 ± 0.47 (out of 5), indicating an accuracy of approximately 78%. The responses evaluated by three raters were similar (Kruskal-Wallis H value 0.51), and the ICC value was 0.796, which indicates a relatively high level of agreement among the raters.
ChatGPT shows moderate capability in solving AETCOM case scenarios used for CBME in India. The inter-rater reliability of the evaluations suggests that ChatGPT's responses were consistent and reliable. Further studies are needed to explore the potential of ChatGPT and other AI tools in CBME and to determine the optimal use of these tools in medical education.
基于能力的医学教育（CBME）是一种医学培训方法，侧重于培养学习者的能力，而不仅仅是评估他们的知识和技能。态度、伦理和沟通（AETCOM）是CBME的重要组成部分，尚未对诸如ChatGPT等人工智能（AI）工具在CBME中的应用进行研究。因此，我们旨在评估ChatGPT解决印度用于CBME的AETCOM案例场景的能力。
基于AETCOM能力共开发了11个案例场景。将这些场景呈现给ChatGPT，由三位独立专家对ChatGPT生成的回复进行评分，评分范围为0至5分。将这些分数与单样本中位数检验中预定义的2.5分（50%准确率）和4分（80%准确率）进行比较。通过Kruskal-Wallis H检验比较三位评分者的分数。使用组内相关系数（ICC）评估评估的评分者间信度。
ChatGPT提供的解决方案的平均得分为3.88±0.47（满分5分），表明准确率约为78%。三位评分者评估的回复相似（Kruskal-Wallis H值为0.51），ICC值为0.796，这表明评分者之间的一致性水平较高。
ChatGPT在解决印度用于CBME的AETCOM案例场景方面表现出中等能力。评估的评分者间信度表明ChatGPT的回复是一致且可靠的。需要进一步研究以探索ChatGPT和其他AI工具在CBME中的潜力，并确定这些工具在医学教育中的最佳使用方式。
