Leroy Gondy, Kauchak David, Harber Philip, Pal Ankit, Shukla Akash
University of Arizona, Tucson, AZ.
Pomona College, Claremont, CA.
AMIA Jt Summits Transl Sci Proc. 2024 May 31;2024:295-304. eCollection 2024.
Text and audio simplification to increase information comprehension are important in healthcare. With the introduction of ChatGPT, evaluation of its simplification performance is needed. We provide a systematic comparison of human and ChatGPT simplified texts using fourteen metrics indicative of text difficulty. We briefly introduce our online editor where these simplification tools, including ChatGPT, are available. We scored twelve corpora using our metrics: six text, one audio, and five ChatGPT simplified corpora (using five different prompts). We then compare these corpora with texts simplified and verified in a prior user study. Finally, a medical domain expert evaluated the user study texts and five, new ChatGPT simplified versions. We found that simple corpora show higher similarity with the human simplified texts. ChatGPT simplification moves metrics in the right direction. The medical domain expert's evaluation showed a preference for the ChatGPT style, but the text itself was rated lower for content retention.
在医疗保健领域，简化文本和音频以提高信息理解能力非常重要。随着ChatGPT的推出，需要对其简化性能进行评估。我们使用十四种表示文本难度的指标，对人工简化文本和ChatGPT简化文本进行了系统比较。我们简要介绍了我们的在线编辑器，在该编辑器中可以使用包括ChatGPT在内的这些简化工具。我们使用我们的指标对十二个语料库进行了评分：六个文本语料库、一个音频语料库和五个ChatGPT简化语料库（使用五个不同的提示）。然后，我们将这些语料库与之前用户研究中简化并验证过的文本进行比较。最后，一位医学领域专家对用户研究文本和五个新的ChatGPT简化版本进行了评估。我们发现，简单的语料库与人工简化文本具有更高的相似度。ChatGPT简化使各项指标朝着正确的方向发展。医学领域专家的评估显示，他们对ChatGPT的风格更青睐，但对文本内容保留率的评分较低。
