Department of Gastroenterology, Qilu Hospital of Shandong University, Jinan, Shandong, China.
Shandong Provincial Clinical Research Center for Digestive Disease, Qilu Hospital of Shandong University, Jinan, Shandong, China.
Helicobacter. 2024 Jul-Aug;29(4):e13115. doi: 10.1111/hel.13115.
Patient education contributes to improve public awareness of Helicobacter pylori. Large language models (LLMs) offer opportunities to revolutionize patient education transformatively. This study aimed to assess the quality of patient educational materials (PEMs) generated by LLMs and compared with physician sourced.
Unified instruction about composing a PEM about H. pylori at a sixth-grade reading level in both English and Chinese were given to physician and five LLMs (Bing Copilot, Claude 3 Opus, Gemini Pro, ChatGPT-4, and ERNIE Bot 4.0). The assessments of the completeness and comprehensibility of the Chinese PEMs were conducted by five gastroenterologists and 50 patients according to three-point Likert scale. Gastroenterologists were asked to evaluate both English and Chinese PEMs and determine the accuracy and safety. The accuracy was assessed by six-point Likert scale. The minimum acceptable scores were 4, 2, and 2 for accuracy, completeness, and comprehensibility, respectively. The Flesch-Kincaid and Simple Measure of Gobbledygook scoring systems were employed as readability assessment tools.
Accuracy and comprehensibility were acceptable for English PEMs of all sources, while completence was not satisfactory. Physician-sourced PEM had the highest accuracy mean score of 5.60 and LLM-generated English PEMs ranged from 4.00 to 5.40. The completeness score was comparable between physician-sourced PEM and LLM-generated PEMs in English. Chinese PEMs from LLMs proned to have lower score in accuracy and completeness assessment than English PEMs. The mean score for completeness of five LLM-generated Chinese PEMs was 1.82-2.70 in patients' perspective, which was higher than gastroenterologists' assessment. Comprehensibility was satisfactory for all PEMs. No PEM met the recommended sixth-grade reading level.
LLMs have potential in assisting patient education. The accuracy and comprehensibility of LLM-generated PEMs were acceptable, but further optimization on improving completeness and accounting for a variety of linguistic contexts are essential for enhancing the feasibility.
患者教育有助于提高公众对幽门螺杆菌的认识。大型语言模型 (LLM) 提供了彻底改变患者教育的机会。本研究旨在评估 LLM 生成的患者教育材料 (PEM) 的质量，并与医生来源的 PEM 进行比较。
向医生和五个 LLM（Bing Copilot、Claude 3 Opus、Gemini Pro、ChatGPT-4 和 ERNIE Bot 4.0）提供了关于用六年级阅读水平编写关于 H. pylori 的 PEM 的统一说明，用中文和英文。五位胃肠病学家和 50 名患者根据三点量表评估中文 PEM 的完整性和可理解性。胃肠病学家被要求评估英文和中文 PEM，并确定准确性和安全性。准确性评估采用六点量表。准确性、完整性和可理解性的最低可接受分数分别为 4、2 和 2。Flesch-Kincaid 和 Simple Measure of Gobbledygook 评分系统被用作可读性评估工具。
所有来源的英文 PEM 的准确性和可理解性都可以接受，但完整性不令人满意。医生来源的 PEM 的准确性平均得分为 5.60，而 LLM 生成的英文 PEM 的得分范围为 4.00 至 5.40。医生来源的 PEM 和英语 LLM 生成的 PEM 在完整性方面的得分相当。LLM 生成的中文 PEM 在准确性和完整性评估方面往往得分较低。从患者的角度来看，五个 LLM 生成的中文 PEM 的完整性平均得分为 1.82-2.70，高于胃肠病学家的评估。所有 PEM 的可理解性都令人满意。没有 PEM 符合推荐的六年级阅读水平。
LLM 具有辅助患者教育的潜力。LLM 生成的 PEM 的准确性和可理解性可以接受，但进一步优化提高完整性并考虑各种语言环境对于提高可行性至关重要。
