Yao Zhong, Duan Liantan, Xu Shuo, Chi Lingyi, Sheng Dongfang
Department of Neurosurgery, Qilu Hospital of Shandong University, Jinan, China.
Cheeloo College of Medicine, Shandong University, Jinan, China.
JMIR Med Inform. 2025 Jun 27;13:e69485. doi: 10.2196/69485.
Research on large language models (LLMs) in the medical field has predominantly focused on models trained with English-language corpora, evaluating their performance within English-speaking contexts. The performances of models trained with non-English language corpora and their performance in non-English contexts remain underexplored.
This study aimed to evaluate the performances of LLMs trained on different languages corpora by using the Chinese National Medical Licensing Examination (CNMLE) as a benchmark and constructed analogous questions.
Under different prompt settings, we sequentially posed questions to 7 LLMs: 2 primarily trained on English-language corpora and 5 primarily on Chinese-language corpora. The models' responses were compared against standard answers to calculate the accuracy rate of each model. Further subgroup analyses were conducted by categorizing the questions based on various criteria. We also collected error sets to explore patterns of mistakes across different models.
Under the zero-shot setting, 6 out of 7 models exceeded the passing level, with the highest accuracy rate achieved by the Chinese LLM Baichuan (86.67%), followed by ChatGPT (83.83%). In the constructed questions, all 7 models exceeded the passing threshold, with Baichuan maintaining the highest accuracy rate (87.00%). In few-shot learning, all models exceeded the passing threshold. Baichuan, ChatGLM, and ChatGPT retained the highest accuracy. While Llama showed marked improvement over previous tests, the relative performance rankings of other models stayed similar to previous results. In subgroup analyses, English models demonstrated comparable or superior performance to Chinese models on questions related to ethics and policy. All models except Llama generally had higher accuracy rates for simple questions than for complex ones. The error set of ChatGPT was similar to those of other Chinese models. Multimodel cross-verification outperformed single model, particularly improving accuracy rate on simple questions. The implementation of dual-model and tri-model verification achieved accuracy rates of 94.17% and 96.33% respectively.
At the current level, LLMs trained primarily on English corpora and those trained mainly on Chinese corpora perform similarly well in CNMLE, with Chinese models still outperforming. The performance difference between ChatGPT and other Chinese LLMs are not solely due to communication barriers but are more likely influenced by disparities in the training data. By using a method of cross-verification with multiple LLMs, excellent performance can be achieved in medical examinations.
医学领域对大语言模型（LLMs）的研究主要集中在使用英语语料库训练的模型上，评估它们在英语语境中的表现。使用非英语语料库训练的模型的性能及其在非英语语境中的表现仍未得到充分探索。
本研究旨在以中国国家医师资格考试（CNMLE）为基准并构建类似问题，评估在不同语言语料库上训练的大语言模型的性能。
在不同的提示设置下，我们依次向7个大语言模型提出问题：2个主要在英语语料库上训练，5个主要在中文语料库上训练。将模型的回答与标准答案进行比较，以计算每个模型的准确率。通过根据各种标准对问题进行分类来进行进一步的亚组分析。我们还收集了错误集，以探索不同模型的错误模式。
在零样本设置下，7个模型中有6个超过了及格水平，其中中文大语言模型百川的准确率最高（86.67%），其次是ChatGPT（83.83%）。在构建的问题中，所有7个模型都超过了及格阈值，百川保持最高准确率（87.00%）。在少样本学习中，所有模型都超过了及格阈值。百川、ChatGLM和ChatGPT保持了最高准确率。虽然Llama比之前的测试有显著提高，但其他模型的相对性能排名与之前的结果相似。在亚组分析中，在与伦理和政策相关的问题上，英语模型表现出与中文模型相当或更优的性能。除Llama外，所有模型对简单问题的准确率通常高于复杂问题。ChatGPT的错误集与其他中文模型的相似。多模型交叉验证优于单模型，特别是提高了简单问题的准确率。双模型和三模型验证的准确率分别达到了94.17%和96.33%。
在当前水平下，主要在英语语料库上训练的大语言模型和主要在中文语料库上训练的大语言模型在CNMLE中的表现同样出色，中文模型仍然表现更优。ChatGPT与其他中文大语言模型之间的性能差异不仅仅是由于沟通障碍，更可能是受训练数据差异的影响。通过使用多个大语言模型进行交叉验证的方法，可以在医学考试中取得优异成绩。
