Liu Siru, Huang Sean S, McCoy Allison B, Wright Aileen P, Horst Sara, Wright Adam
Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee.
Department of Computer Science, Vanderbilt University, Nashville, Tennessee.
JAMA Netw Open. 2025 Sep 2;8(9):e2533277. doi: 10.1001/jamanetworkopen.2025.33277.
Optimizing order sets is vital to enhance clinical decision support and improve patient care. Manual review is resource intensive and cannot timely identify potential improvements in order sets.
To develop and evaluate the utility of a large language model (LLM)-powered multiagent system in optimizing order sets.
DESIGN, SETTING, AND PARTICIPANTS: A multiagent system was developed and evaluated between January 1, 2024, and December 31, 2024, which comprised agents for content critique, dynamic search, knowledge retrieval, medication verification, and suggestion summarization. A filter was developed to align suggestion usefulness scores with expert preferences. Experiment 1 evaluated 735 generated suggestions from a multiagent system developed for optimizing order sets, which were assessed by 3 physicians for 9 order sets and by 1 physician for 62 order sets. Experiment 2 implemented an LLM-as-a-judge approach to align generated suggestions with expert ratings and developed a filter to further refine the system's performance. The study was performed at Vanderbilt University Medical Center. A total of 735 suggestions for 71 order sets at VUMC were evaluated by 3 physicians.
The ratings of accuracy, usefulness, feasibility, and impact; interrater agreement; and alignment against historical ordering data.
In evaluation 1 of experiment 1, the median values for the number of suggestions scoring 4 or higher at the order set level were 5 (IQR, 5-6) for the metrics of accuracy, 2 (IQR, 1-4) for usefulness, 1 (IQR, 0-3) for feasibility, and 1 (IQR, 0-2) for impact. Of 96 suggestions, 44 (46%; 95% CI, 36%-56%) aligned with historical ordering patterns. In evaluation 2 of experiment 1, 639 suggestions were generated for 62 order sets; 52 order sets had at least 1 useful suggestion, with a median of 2 (IQR, 1-3) useful suggestions. Overall, 122 suggestions (19%; 95% CI, 16%-22%) were rated as useful. After expert alignment, Cohen κ improved from 0.06 to 0.41. Filtering using the aligned scores reduced total suggestions by 29% while retaining 92% of useful suggestions.
In this cohort study of an LLM-powered multiagent system for optimizing order sets, leveraging LLMs and multiagent systems provided a scalable approach. Alignment with a small set of expert ratings significantly enhanced the LLM evaluation. Future research could refine reasoning capabilities and integrate useful suggestions into electronic health records, while engaging end-users as artificial intelligence-supported reviewers.
优化医嘱集对于加强临床决策支持和改善患者护理至关重要。人工审查资源消耗大，且无法及时识别医嘱集中的潜在改进之处。
开发并评估一个由大语言模型（LLM）驱动的多智能体系统在优化医嘱集中的效用。
设计、设置和参与者：在2024年1月1日至2024年12月31日期间开发并评估了一个多智能体系统，该系统包括用于内容批判、动态搜索、知识检索、用药验证和建议总结的智能体。开发了一个过滤器，以使建议有用性得分与专家偏好相一致。实验1评估了为优化医嘱集而开发的多智能体系统生成的735条建议，这些建议由3名医生针对9个医嘱集进行评估，1名医生针对62个医嘱集进行评估。实验2采用LLM作为评判的方法，使生成的建议与专家评级相一致，并开发了一个过滤器以进一步优化系统性能。该研究在范德比尔特大学医学中心进行。3名医生对范德比尔特大学医学中心71个医嘱集的735条建议进行了评估。
准确性、有用性、可行性和影响的评级；评分者间一致性；以及与历史医嘱数据的一致性。
在实验1的评估1中，在医嘱集层面上得分4分及以上的建议数量的中位数，准确性指标为5条（四分位距，5 - 6），有用性指标为2条（四分位距，1 - 4），可行性指标为1条（四分位距，0 - 3），影响指标为1条（四分位距，0 - 2）。在96条建议中，44条（46%；95%置信区间，36% - 56%）与历史医嘱模式一致。在实验1的评估2中，为62个医嘱集生成了639条建议；52个医嘱集至少有1条有用建议，有用建议的中位数为2条（四分位距，1 - 3）。总体而言，122条建议（19%；95%置信区间，16% - 22%）被评为有用。经过专家校准后，科恩κ系数从0.06提高到了0.41。使用校准后的分数进行过滤，使总建议数减少了29%，同时保留了92%的有用建议。
在这项关于由LLM驱动的优化医嘱集多智能体系统的队列研究中，利用LLM和多智能体系统提供了一种可扩展的方法。与一小部分专家评级相一致显著增强了LLM评估。未来的研究可以完善推理能力，并将有用建议整合到电子健康记录中，同时让终端用户作为人工智能支持的审查者参与进来。
