Li Yanqiu, Li Zhuojun, Li Jinze, Liu Long, Liu Yao, Zhu Bingbing, Shi Ke, Lu Yu, Li Yongqi, Zeng Xuanwei, Feng Ying, Wang Xianbo
Center for Integrative Medicine, Beijing Ditan Hospital, Capital Medical University, Beijing, China.
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China.
Int J Med Inform. 2025 Sep;201:105961. doi: 10.1016/j.ijmedinf.2025.105961. Epub 2025 May 5.
With the increasing prevalence of large language models (LLMs) in the medical field, patients are increasingly turning to advanced online resources for information related to liver cirrhosis due to its long-term management processes. Therefore, a comprehensive evaluation of real-world performance of LLMs in these specialized medical areas is necessary.
This study evaluates the performance of four mainstream LLMs (ChatGPT-4o, Claude-3.5 Sonnet, Gemini-1.5 Pro, and Llama-3.1) in answering 39 questions related to liver cirrhosis. The information quality, readability and accuracy were assessed using Ensuring Quality Information for Patients tool, Flesch-Kincaid metrics and consensus scoring. The simplification and their self-correction ability of LLMs were also assessed.
Significant performance differences were observed among the models. Gemini scored highest in providing high-quality information. While the readability of all four LLMs was generally low, requiring a college-level reading comprehension ability, they exhibited strong capabilities in simplifying complex information. ChatGPT performed best in terms of accuracy, with a "Good" rating of 80%, higher than Claude (72%), Gemini (49%), and Llama (64%). All models received high scores for comprehensiveness. Each of the four LLMs demonstrated some degree of self-correction ability, improving the accuracy of initial answers with simple prompts. ChatGPT's and Llama's accuracy improved by 100%, Claude's by 50% and Gemini's by 67%.
LLMs demonstrate excellent performance in generating health information related to liver cirrhosis, yet they exhibit differences in answer quality, readability and accuracy. Future research should enhance their value in healthcare, ultimately achieving reliable, accessible and patient-centered medical information dissemination.
随着大语言模型（LLMs）在医学领域的应用日益广泛，由于肝硬化的长期管理过程，患者越来越多地转向先进的在线资源获取相关信息。因此，有必要对大语言模型在这些专业医学领域的实际性能进行全面评估。
本研究评估了四种主流大语言模型（ChatGPT-4o、Claude-3.5 Sonnet、Gemini-1.5 Pro和Llama-3.1）回答39个与肝硬化相关问题的性能。使用“为患者确保信息质量”工具、弗莱什-金凯德指标和共识评分来评估信息质量、可读性和准确性。还评估了大语言模型的简化能力及其自我纠正能力。
各模型之间观察到显著的性能差异。Gemini在提供高质量信息方面得分最高。虽然所有四个大语言模型的可读性都要求具备大学水平的阅读理解能力，但它们在简化复杂信息方面表现出很强的能力。ChatGPT在准确性方面表现最佳，“良好”评级为80%，高于Claude（72%）、Gemini（49%）和Llama（64%）。所有模型在全面性方面都获得了高分。四个大语言模型都表现出一定程度的自我纠正能力，通过简单提示提高了初始答案的准确性。ChatGPT和Llama的准确性提高了100%，Claude提高了50%，Gemini提高了67%。
大语言模型在生成与肝硬化相关的健康信息方面表现出色，但在答案质量、可读性和准确性方面存在差异。未来的研究应提高它们在医疗保健中的价值，最终实现可靠、易获取且以患者为中心的医疗信息传播。
