Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37212, United States.
Department of Computer Science, Vanderbilt University, Nashville, TN 37212, United States.
J Am Med Inform Assoc. 2024 May 20;31(6):1388-1396. doi: 10.1093/jamia/ocae041.
To evaluate the capability of using generative artificial intelligence (AI) in summarizing alert comments and to determine if the AI-generated summary could be used to improve clinical decision support (CDS) alerts.
We extracted user comments to alerts generated from September 1, 2022 to September 1, 2023 at Vanderbilt University Medical Center. For a subset of 8 alerts, comment summaries were generated independently by 2 physicians and then separately by GPT-4. We surveyed 5 CDS experts to rate the human-generated and AI-generated summaries on a scale from 1 (strongly disagree) to 5 (strongly agree) for the 4 metrics: clarity, completeness, accuracy, and usefulness.
Five CDS experts participated in the survey. A total of 16 human-generated summaries and 8 AI-generated summaries were assessed. Among the top 8 rated summaries, five were generated by GPT-4. AI-generated summaries demonstrated high levels of clarity, accuracy, and usefulness, similar to the human-generated summaries. Moreover, AI-generated summaries exhibited significantly higher completeness and usefulness compared to the human-generated summaries (AI: 3.4 ± 1.2, human: 2.7 ± 1.2, P = .001).
End-user comments provide clinicians' immediate feedback to CDS alerts and can serve as a direct and valuable data resource for improving CDS delivery. Traditionally, these comments may not be considered in the CDS review process due to their unstructured nature, large volume, and the presence of redundant or irrelevant content. Our study demonstrates that GPT-4 is capable of distilling these comments into summaries characterized by high clarity, accuracy, and completeness. AI-generated summaries are equivalent and potentially better than human-generated summaries. These AI-generated summaries could provide CDS experts with a novel means of reviewing user comments to rapidly optimize CDS alerts both online and offline.
评估生成式人工智能（AI）在总结警示性注释方面的能力，并确定 AI 生成的摘要是否可用于改进临床决策支持（CDS）警示。
我们从范德比尔特大学医学中心 2022 年 9 月 1 日至 2023 年 9 月 1 日生成的警示中提取用户注释。对于 8 个警示子集，由 2 名医生独立生成注释摘要，然后由 GPT-4 分别生成。我们调查了 5 名 CDS 专家，让他们对 4 个指标（清晰度、完整性、准确性和有用性）对人类生成和 AI 生成的摘要进行 1（强烈不同意）到 5（强烈同意）的评分。
共有 5 名 CDS 专家参与了调查。共评估了 16 个人工生成的摘要和 8 个 AI 生成的摘要。在排名前 8 的摘要中，有 5 个是由 GPT-4 生成的。AI 生成的摘要在清晰度、准确性和有用性方面表现出很高的水平，与人工生成的摘要相似。此外，AI 生成的摘要在完整性和有用性方面明显高于人工生成的摘要（AI：3.4±1.2，人工：2.7±1.2，P=0.001）。
最终用户注释为 CDS 警示提供了临床医生的即时反馈，并且可以作为改进 CDS 交付的直接且有价值的数据资源。传统上，由于注释的非结构化性质、大量注释以及冗余或不相关内容的存在，这些注释可能不会在 CDS 审查过程中得到考虑。我们的研究表明，GPT-4 能够将这些注释提炼为具有高清晰度、准确性和完整性的摘要。AI 生成的摘要与人工生成的摘要相当，甚至可能更好。这些 AI 生成的摘要可以为 CDS 专家提供一种新的方法来快速在线和离线审查用户注释，以优化 CDS 警示。
