Surapaneni Krishna Mohan, Rajajagadeesan Anusha, Goudhaman Lakshmi, Lakshmanan Shalini, Sundaramoorthi Saranya, Ravi Dineshkumar, Rajendiran Kalaiselvi, Swaminathan Porchelvan
Department of Biochemistry, Panimalar Medical College Hospital & Research Institute, Chennai, India.
Department of Medical Education, Panimalar Medical College Hospital & Research Institute, Chennai, India.
Biochem Mol Biol Educ. 2024 Mar-Apr;52(2):237-248. doi: 10.1002/bmb.21808. Epub 2023 Dec 19.
The emergence of ChatGPT as one of the most advanced chatbots and its ability to generate diverse data has given room for numerous discussions worldwide regarding its utility, particularly in advancing medical education and research. This study seeks to assess the performance of ChatGPT in medical biochemistry to evaluate its potential as an effective self-learning tool for medical students. This evaluation was carried out using the university examination question papers of both parts 1 and 2 of medical biochemistry which comprised theory and multiple choice questions (MCQs) accounting for a total of 100 in each part. The questions were used to interact with ChatGPT, and three raters independently reviewed and scored the answers to prevent bias in scoring. We conducted the inter-item correlation matrix and the interclass correlation between raters 1, 2, and 3. For MCQs, symmetric measures in the form of kappa value (a measure of agreement) were performed between raters 1, 2, and 3. ChatGPT generated relevant and appropriate answers to all questions along with explanations for MCQs. ChatGPT has "passed" the medical biochemistry university examination with an average score of 117 out of 200 (58%) in both papers. In Paper 1, ChatGPT has secured 60 ± 2.29 and 57 ± 4.36 in Paper 2. The kappa value for all the cross-analysis of Rater 1, Rater 2, and Rater 3 scores in MCQ was 1.000. The evaluation of ChatGPT as a self-learning tool in medical biochemistry has yielded important insights. While it is encouraging that ChatGPT has demonstrated proficiency in this area, the overall score of 58% indicates that there is work to be done. To unlock its full potential as a self-learning tool, ChatGPT must focus on generating not only accurate but also comprehensive and contextually relevant content.
ChatGPT作为最先进的聊天机器人之一的出现及其生成多样数据的能力，引发了全球范围内关于其效用的诸多讨论，尤其是在推进医学教育和研究方面。本研究旨在评估ChatGPT在医学生物化学方面的表现，以评估其作为医学生有效自学工具的潜力。这项评估使用了医学生物化学第1部分和第2部分的大学考试试卷，其中包括理论题和多项选择题（MCQ），每部分各占100分。这些问题用于与ChatGPT交互，三位评分者独立审查并对答案进行评分，以防止评分偏差。我们进行了项目间相关矩阵以及评分者1、2和3之间的组内相关性分析。对于多项选择题，在评分者1、2和3之间以kappa值（一致性度量）的形式进行对称度量。ChatGPT针对所有问题都给出了相关且恰当的答案，并对多项选择题进行了解释。ChatGPT在两份试卷中以平均200分中的117分（58%）“通过”了医学生物化学大学考试。在试卷1中，ChatGPT的得分是60±2.29分，在试卷2中是57±4.36分。评分者1、评分者2和评分者3在多项选择题评分中的所有交叉分析的kappa值为1.000。对ChatGPT作为医学生物化学自学工具的评估产生了重要见解。虽然ChatGPT在这一领域表现出了熟练程度令人鼓舞，但58%的总体得分表明仍有工作要做。为了释放其作为自学工具的全部潜力，ChatGPT必须专注于生成不仅准确而且全面且与上下文相关的内容。
