Chekuri Akhila, Johal Armaan S, Allen Matthew R, Ayers John W, Hogarth Michael, Farcas Emilia
University of California San Diego, La Jolla, CA Computer Science and Engineering.
University of California San Diego, La Jolla, CA Cognitive Science.
AMIA Annu Symp Proc. 2025 May 22;2024:232-241. eCollection 2024.
The volume of patient-provider messages is on the rise, and Large Language Models (LLMs) can potentially streamline the clinical messaging process, but their success hinges on triaging messages they can optimally address. In this study, we analyzed Electronic Health Records with over 4 million messages exchanged between patients and providers to characterize the utility of using LLMs for messages containing knowledge questions. We implemented a rule-based Syntactic Question Detector as a triage tool, and we evaluated it on 500 messages. The interrater reliability metrics and comparison with LLMs show the difficulty of detecting questions due to the informal text and implicit requests. Our results show that 25% of MyChart messages with questions do not have a response from the clinical team. This paper provides insights into the challenges of real-world data, highlights the importance and non-triviality of detecting questions, and suggests a pipeline for LLM use in healthcare.
患者与医疗服务提供者之间的消息量在不断增加，大语言模型（LLMs）有可能简化临床消息传递流程，但其成功取决于对它们能够最佳处理的消息进行分类。在本研究中，我们分析了包含患者与医疗服务提供者之间交换的超过400万条消息的电子健康记录，以描述使用大语言模型处理包含知识问题的消息的效用。我们实施了一个基于规则的句法问题检测器作为分类工具，并在500条消息上对其进行了评估。评分者间信度指标以及与大语言模型的比较表明，由于文本不规范和隐含请求，检测问题存在困难。我们的结果表明，25%带有问题的MyChart消息没有得到临床团队的回复。本文深入探讨了现实世界数据的挑战，强调了检测问题的重要性和复杂性，并提出了在医疗保健中使用大语言模型的流程。
