Department of Medicine, Division of General Internal Medicine, Mount Sinai School of Medicine, New York, NY, USA.
Int J Med Inform. 2012 Nov;81(11):761-72. doi: 10.1016/j.ijmedinf.2012.02.009. Epub 2012 Mar 27.
Usability evaluations can improve the usability and workflow integration of clinical decision support (CDS). Traditional usability testing using scripted scenarios with think-aloud protocol analysis provide a useful but incomplete assessment of how new CDS tools interact with users and clinical workflow. "Near-live" clinical simulations are a newer usability evaluation tool that more closely mimics clinical workflow and that allows for a complementary evaluation of CDS usability as well as impact on workflow.
This study employed two phases of testing a new CDS tool that embedded clinical prediction rules (an evidence-based medicine tool) into primary care workflow within a commercial electronic health record. Phase I applied usability testing involving "think-aloud" protocol analysis of 8 primary care providers encountering several scripted clinical scenarios. Phase II used "near-live" clinical simulations of 8 providers interacting with video clips of standardized trained patient actors enacting the clinical scenario. In both phases, all sessions were audiotaped and had screen-capture software activated for onscreen recordings. Transcripts were coded using qualitative analysis methods.
In Phase I, the impact of the CDS on navigation and workflow were associated with the largest volume of negative comments (accounting for over 90% of user raised issues) while the overall usability and the content of the CDS were associated with the most positive comments. However, usability had a positive-to-negative comment ratio of only 0.93 reflecting mixed perceptions about the usability of the CDS. In Phase II, the duration of encounters with simulated patients was approximately 12 min with 71% of the clinical prediction rules being activated after half of the visit had already elapsed. Upon activation, providers accepted the CDS tool pathway 82% of times offered and completed all of its elements in 53% of all simulation cases. Only 12.2% of encounter time was spent using the CDS tool. Two predominant clinical workflows, accounting for 75% of all cases simulations, were identified that characterized the sequence of provider interactions with the CDS. These workflows demonstrated a significant variation in temporal sequence of potential activation of the CDS.
This study successfully combined "think-aloud" protocol analysis with "near-live" clinical simulations in a usability evaluation of a new primary care CDS tool. Each phase of the study provided complementary observations on problems with the new onscreen tool and was used to refine both its usability and workflow integration. Synergistic use of "think-aloud" protocol analysis and "near-live" clinical simulations provide a robust assessment of how CDS tools would interact in live clinical environments and allows for enhanced early redesign to augment clinician utilization. The findings suggest the importance of using complementary testing methods before releasing CDS for live use.
可用性评估可以提高临床决策支持（CDS）的可用性和工作流程集成。使用脚本场景和出声思维协议分析的传统可用性测试提供了对新 CDS 工具与用户和临床工作流程交互方式的有用但不完整的评估。“近乎实时”的临床模拟是一种较新的可用性评估工具，它更能模拟临床工作流程，并允许对 CDS 的可用性以及对工作流程的影响进行补充评估。
本研究分两个阶段测试了一种新的 CDS 工具，该工具将临床预测规则（循证医学工具）嵌入商业电子病历中的初级保健工作流程中。第一阶段采用可用性测试，对 8 名初级保健提供者进行了出声思维协议分析，他们遇到了几个脚本化的临床场景。第二阶段使用 8 名提供者与标准化训练有素的患者演员表演临床场景的视频剪辑进行“近乎实时”的临床模拟。在两个阶段中，所有会议都进行了录音，并启用屏幕捕获软件以记录屏幕上的内容。使用定性分析方法对记录进行编码。
在第一阶段，CDS 对导航和工作流程的影响与最大数量的负面评论相关（占用户提出的问题的 90%以上），而整体可用性和 CDS 的内容与最积极的评论相关。然而，可用性的积极与消极评论比例仅为 0.93，反映出对 CDS 可用性的看法存在分歧。在第二阶段，模拟患者的就诊时间约为 12 分钟，在就诊已经过半的情况下，有 71%的临床预测规则被激活。在激活后，提供者接受 CDS 工具路径的比例为 82%，并在 53%的模拟病例中完成了其所有内容。仅 12.2%的就诊时间用于使用 CDS 工具。确定了两种主要的临床工作流程，占所有模拟病例的 75%，它们描述了提供者与 CDS 交互的顺序。这些工作流程显示了 CDS 潜在激活的时间顺序有显著差异。
本研究成功地将出声思维协议分析与新的初级保健 CDS 工具的“近乎实时”临床模拟相结合，进行了可用性评估。研究的每个阶段都对新屏幕工具的问题提供了互补性观察，并用于改进其可用性和工作流程集成。出声思维协议分析和“近乎实时”临床模拟的协同使用提供了对 CDS 工具在实际临床环境中如何交互的强大评估，并允许在发布 CDS 进行实际使用之前进行增强的早期重新设计以提高临床医生的利用率。研究结果表明，在将 CDS 投入实际使用之前，使用互补测试方法的重要性。
