Roy R D, Gupta S D, Das D, Chowdhury P D
Department of Microbiology, Mata Gujri Medical College and L.S.K Hospital, Kishanganj, Bihar, India.
Department of Community Medicine, Mata Gujri Medical College and L.S.K Hospital, Kishanganj, Bihar, India.
J Postgrad Med. 2025 Jan 1;71(1):28-32. doi: 10.4103/jpgm.jpgm_775_24. Epub 2025 Mar 19.
Artificial intelligence (AI) platforms have achieved a noteworthy role in various fields of medical sciences, ranging from medical education to clinical diagnostics and treatment. ChatGPT, Gemini, and Meta AI are some large language models (LLMs) that have gained immense popularity among students for solving questions from different branches of education.
A cross-sectional study was conducted in the Department of Microbiology to assess the performance of ChatGPT, Gemini, and Meta AI in answering higher-order questions from various competencies of the microbiology curriculum (MI 1 to 8), according to CBME guidelines. Sixty higher-order questions were compiled from university question papers of two universities. Their responses were assessed by three faculty members from the department.
The mean rank scores of ChatGPT, Gemini, and Meta AI were found to be 102.76, 108.5, and 60.23 by Evaluator 1; 106.03, 88.5, and 76.95 by Evaluator 2; and 104.85, 85.6, and 81.04, respectively, indicating lowest overall mean rank score for Meta AI. ChatGPT had the highest mean score in MI 2,3,5,6,7, and 8 competencies, while Gemini had a higher score for MI 1 and 4 competencies. A qualitative assessment of the three platforms was also performed. ChatGPT provided elaborative responses, some responses from Gemini lacked certain significant points, and Meta AI gave answers in bullet points.
Both ChatGPT and Gemini have created vast databases to correctly respond to higher-order queries in medical microbiology in comparison to Meta AI. Our study is the first of its kind to compare these three popular LLM platforms for microbiology.
人工智能（AI）平台在医学科学的各个领域都发挥了重要作用，从医学教育到临床诊断与治疗。ChatGPT、Gemini和Meta AI是一些大型语言模型（LLMs），在学生中因能解答不同教育分支的问题而广受欢迎。
根据CBME指南，在微生物学系进行了一项横断面研究，以评估ChatGPT、Gemini和Meta AI在回答微生物学课程（MI 1至8）各能力方面高阶问题的表现。从两所大学的大学试卷中汇编了60个高阶问题。由该系的三名教员对它们的回答进行评估。
评估者1得出ChatGPT、Gemini和Meta AI的平均排名分数分别为102.76、108.5和60.23；评估者2得出的分数分别为106.03、88.5和76.95；评估者3得出的分数分别为104.85、85.6和81.04，这表明Meta AI的总体平均排名分数最低。ChatGPT在MI 2、3、5、6、7和8能力方面的平均得分最高，而Gemini在MI 1和4能力方面得分更高。还对这三个平台进行了定性评估。ChatGPT提供了详尽的回答，Gemini的一些回答缺少某些要点，而Meta AI以要点形式给出答案。
与Meta AI相比，ChatGPT和Gemini都创建了庞大的数据库，能够正确回答医学微生物学中的高阶问题。我们的研究是同类研究中首个比较这三个流行的微生物学大型语言模型平台的研究。
