Zhao Weilong, Lai Honghao, Pan Bei, Huang Jiajie, Xia Danni, Bai Chunyang, Liu Jiayi, Liu Jianing, Jin Yinghui, Shang Hongcai, Liu Jianping, Shi Nannan, Liu Jie, Chen Yaolong, Estill Janne, Ge Long
Department of Health Policy and Management, School of Public Health, Lanzhou University, Lanzhou, China.
Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China.
Front Pharmacol. 2025 Jul 25;16:1649041. doi: 10.3389/fphar.2025.1649041. eCollection 2025.
OBJECTIVE: Whether large language models (LLMs) can effectively facilitate CM knowledge acquisition remains uncertain. This study aims to assess the adherence of LLMs to Clinical Practice Guidelines (CPGs) in CM. METHODS: This cross-sectional study randomly selected ten CPGs in CM and constructed 150 questions across three categories: medication based on differential diagnosis (MDD), specific prescription consultation (SPC), and CM theory analysis (CTA). Eight LLMs (GPT-4o, Claude-3.5 Sonnet, Moonshot-v1, ChatGLM-4, DeepSeek-v3, DeepSeek-r1, Claude-4 sonnet, and Claude-4 sonnet thinking) were evaluated using both English and Chinese queries. The main evaluation metrics included accuracy, readability, and use of safety disclaimers. RESULTS: Overall, DeepSeek-v3 and DeepSeek-r1 demonstrated superior performance in both English (median 5.00, interquartile range (IQR) 4.00-5.00 vs. median 5.00, IQR 3.70-5.00) and Chinese (both median 5.00, IQR 4.30-5.00), significantly outperforming all other models. All models achieved significantly higher accuracy in Chinese versus English responses (all p < 0.05). Significant variations in accuracy were observed across the categories of questions, with MDD and SPC questions presenting more challenges than CTA questions. English responses had lower readability (mean flesch reading ease score 32.7) compared to Chinese responses. Moonshot-v1 provided the highest rate of safety disclaimers (98.7% English, 100% Chinese). CONCLUSION: LLMs showed varying degrees of potential for acquiring CM knowledge. The performance of DeepSeek-v3 and DeepSeek-r1 is satisfactory. Optimizing LLMs to become effective tools for disseminating CM information is an important direction for future development.
目的：大语言模型（LLMs）能否有效促进中医知识获取仍不确定。本研究旨在评估大语言模型在中医临床实践指南（CPGs）方面的遵循情况。 方法：这项横断面研究随机选取了十条中医临床实践指南，并构建了150个问题，分为三类：基于鉴别诊断的用药（MDD）、特定处方咨询（SPC）和中医理论分析（CTA）。使用英语和中文查询对八个大语言模型（GPT-4o、Claude-3.5 Sonnet、Moonshot-v1、ChatGLM-4、DeepSeek-v3、DeepSeek-r1、Claude-4 sonnet和Claude-4 sonnet thinking）进行评估。主要评估指标包括准确性、可读性和安全免责声明的使用情况。 结果：总体而言，DeepSeek-v3和DeepSeek-r1在英语（中位数5.00，四分位间距（IQR）4.00 - 5.00，而其他模型中位数5.00，IQR 3.70 - 5.00）和中文（两者中位数均为5.00，IQR 4.30 - 5.00）方面均表现出卓越性能，显著优于所有其他模型。与英语回答相比，所有模型在中文回答中均取得了显著更高的准确性（所有p < 0.05）。不同类别的问题在准确性上存在显著差异，MDD和SPC问题比CTA问题更具挑战性。与中文回答相比，英语回答的可读性较低（平均弗莱什易读性得分32.7）。Moonshot-v1提供安全免责声明的比例最高（英语为98.7%，中文为100%）。 结论：大语言模型在获取中医知识方面展现出不同程度的潜力。DeepSeek-v3和DeepSeek-r1的性能令人满意。优化大语言模型以成为传播中医信息的有效工具是未来发展的一个重要方向。
Zotero 插件
