Cao Yuyu, Lu Wei, Shi Runhan, Liu Fuying, Liu Steven, Xu Xinwei, Yang Jin, Rong Guangyu, Xin Changchang, Zhou Xujiao, Sun Xinghuai, Hong Jiaxu
Department of Ophthalmology, Eye & ENT Hospital, State Key Laboratory of Medical Neurobiology, Fudan University, Shanghai, China.
NHC Key Laboratory of Myopia and Related Eye Diseases Shanghai, China.
Adv Ophthalmol Pract Res. 2024 Dec 3;5(2):88-94. doi: 10.1016/j.aopr.2024.12.002. eCollection 2025 May-Jun.
The advent of chatbots based on large language models (LLMs), such as ChatGPT, has significantly transformed knowledge acquisition. However, the application of LLMs in glaucoma patient education remains elusive. In this study, we comprehensively compared the performance of four common LLMs - Qwen, Baichuan 2, ChatGPT-4.0, and PaLM 2 - in the context of glaucoma patient education.
Initially, senior ophthalmologists were asked with scoring responses generated by the LLMs, which were answers to the most frequent glaucoma-related questions posed by patients. The Chinese Readability Platform was employed to assess the recommended reading age and reading difficulty score of the four LLMs. Subsequently, optimized models were filtered, and 29 glaucoma patients participated in posing questions to the chatbots and scoring the answers within a real-world clinical setting. Attending ophthalmologists were also required to score the answers across five dimensions: correctness, completeness, readability, helpfulness, and safety. Patients, on the other hand, scored the answers based on three dimensions: satisfaction, readability, and helpfulness.
In the first stage, Baichuan 2 and ChatGPT-4.0 outperformed the other two models, though ChatGPT-4.0 had higher recommended reading age and reading difficulty scores. In the second stage, both Baichuan 2 and ChatGPT-4.0 demonstrated exceptional performance among patients and ophthalmologists, with no statistically significant differences observed.
Our research identifies Baichuan 2 and ChatGPT-4.0 as prominent LLMs, offering viable options for glaucoma education.
基于大语言模型（LLMs）的聊天机器人，如ChatGPT的出现，显著改变了知识获取方式。然而，大语言模型在青光眼患者教育中的应用仍不明确。在本研究中，我们在青光眼患者教育背景下全面比较了四种常见大语言模型——文心一言、百川2、ChatGPT-4.0和PaLM 2的性能。
首先，让资深眼科医生对大语言模型生成的回答进行评分，这些回答是针对患者提出的最常见青光眼相关问题的答案。使用中文可读性平台评估这四种大语言模型的推荐阅读年龄和阅读难度得分。随后，筛选出优化模型，29名青光眼患者参与在真实临床环境中向聊天机器人提问并对答案进行评分。主治眼科医生还需从正确性、完整性、可读性、实用性和安全性五个维度对答案进行评分。另一方面，患者则从满意度、可读性和实用性三个维度对答案进行评分。
在第一阶段，百川2和ChatGPT-4.0的表现优于其他两个模型，尽管ChatGPT-4.0的推荐阅读年龄和阅读难度得分更高。在第二阶段，百川2和ChatGPT-4.0在患者和眼科医生中均表现出色，未观察到统计学上的显著差异。
我们的研究确定百川2和ChatGPT-4.0为优秀的大语言模型，为青光眼教育提供了可行的选择。
