Department of Medicine, NYU Grossman School of Medicine, New York, NY, United States.
Medical Center Information Technology, NYU Langone Health, New York, NY, United States.
J Med Internet Res. 2021 Apr 9;23(4):e16651. doi: 10.2196/16651.
Clinical decision support (CDS) is a valuable feature of electronic health records (EHRs) designed to improve quality and safety. However, due to the complexities of system design and inconsistent results, CDS tools may inadvertently increase alert fatigue and contribute to physician burnout. A/B testing, or rapid-cycle randomized tests, is a useful method that can be applied to the EHR in order to rapidly understand and iteratively improve design choices embedded within CDS tools.
This paper describes how rapid randomized controlled trials (RCTs) embedded within EHRs can be used to quickly ascertain the superiority of potential CDS design changes to improve their usability, reduce alert fatigue, and promote quality of care.
A multistep process combining tools from user-centered design, A/B testing, and implementation science was used to understand, ideate, prototype, test, analyze, and improve each candidate CDS. CDS engagement metrics (alert views, acceptance rates) were used to evaluate which CDS version is superior.
To demonstrate the impact of the process, 2 experiments are highlighted. First, after multiple rounds of usability testing, a revised CDS influenza alert was tested against usual care CDS in a rapid (~6 weeks) RCT. The new alert text resulted in minimal impact on reducing firings per patients per day, but this failure triggered another round of review that identified key technical improvements (ie, removal of dismissal button and firings in procedural areas) that led to a dramatic decrease in firings per patient per day (23.1 to 7.3). In the second experiment, the process was used to test 3 versions (financial, quality, regulatory) of text supporting tobacco cessation alerts as well as 3 supporting images. Based on 3 rounds of RCTs, there was no significant difference in acceptance rates based on the framing of the messages or addition of images.
These experiments support the potential for this new process to rapidly develop, deploy, and rigorously evaluate CDS within an EHR. We also identified important considerations in applying these methods. This approach may be an important tool for improving the impact of and experience with CDS.
Flu alert trial: ClinicalTrials.gov NCT03415425; https://clinicaltrials.gov/ct2/show/NCT03415425. Tobacco alert trial: ClinicalTrials.gov NCT03714191; https://clinicaltrials.gov/ct2/show/NCT03714191.
临床决策支持（CDS）是电子健康记录（EHR）的一个有价值的功能，旨在提高质量和安全性。然而，由于系统设计的复杂性和不一致的结果，CDS 工具可能会无意中增加警报疲劳，并导致医生倦怠。A/B 测试或快速循环随机测试是一种有用的方法，可以应用于 EHR 中，以便快速了解并迭代改进 CDS 工具中嵌入的设计选择。
本文描述了如何在 EHR 中嵌入快速随机对照试验（RCT），以快速确定潜在 CDS 设计变更的优越性，以提高其可用性、减少警报疲劳并促进护理质量。
结合用户为中心的设计、A/B 测试和实施科学的工具，采用多步骤流程来理解、构思、原型设计、测试、分析和改进每个候选 CDS。使用 CDS 参与度指标（警报查看次数、接受率）来评估哪个 CDS 版本更优越。
为了展示该过程的影响，突出了 2 个实验。首先，经过多轮可用性测试后，在快速（约 6 周）RCT 中对修订后的流感警报 CDS 与常规护理 CDS 进行了测试。新的警报文本对减少每位患者每天的警报次数几乎没有影响，但这一失败引发了另一轮审查，确定了关键的技术改进（即，删除解雇按钮和程序区域中的警报），导致每位患者每天的警报次数大幅减少（从 23.1 减少到 7.3）。在第二个实验中，该过程用于测试支持烟草戒断警报的 3 种版本（财务、质量、监管）的文本以及 3 种支持图像。基于 3 轮 RCT，消息的框架或添加图像对接受率没有显著影响。
这些实验支持在 EHR 中快速开发、部署和严格评估 CDS 的新过程的潜力。我们还确定了在应用这些方法时需要考虑的重要因素。这种方法可能是提高 CDS 的影响和体验的重要工具。
流感警报试验：ClinicalTrials.gov NCT03415425；https://clinicaltrials.gov/ct2/show/NCT03415425。烟草警报试验：ClinicalTrials.gov NCT03714191；https://clinicaltrials.gov/ct2/show/NCT03714191。
