Roy and Diana Vagelos Division of Biology and Biomedical Sciences, Washington University in St. Louis, St. Louis, MO 63110, United States.
Institute for Informatics, Data Science and Biostatistics (I2DB), Washington University School of Medicine, St. Louis, MO 63110, United States.
J Am Med Inform Assoc. 2024 Oct 1;31(10):2228-2235. doi: 10.1093/jamia/ocae171.
To develop and validate a novel measure, action entropy, for assessing the cognitive effort associated with electronic health record (EHR)-based work activities.
EHR-based audit logs of attending physicians and advanced practice providers (APPs) from four surgical intensive care units in 2019 were included. Neural language models (LMs) were trained and validated separately for attendings' and APPs' action sequences. Action entropy was calculated as the cross-entropy associated with the predicted probability of the next action, based on prior actions. To validate the measure, a matched pairs study was conducted to assess the difference in action entropy during known high cognitive effort scenarios, namely, attention switching between patients and to or from the EHR inbox.
Sixty-five clinicians performing 5 904 429 EHR-based audit log actions on 8956 unique patients were included. All attention switching scenarios were associated with a higher action entropy compared to non-switching scenarios (P < .001), except for the from-inbox switching scenario among APPs. The highest difference among attendings was for the from-inbox attention switching: Action entropy was 1.288 (95% CI, 1.256-1.320) standard deviations (SDs) higher for switching compared to non-switching scenarios. For APPs, the highest difference was for the to-inbox switching, where action entropy was 2.354 (95% CI, 2.311-2.397) SDs higher for switching compared to non-switching scenarios.
We developed a LM-based metric, action entropy, for assessing cognitive burden associated with EHR-based actions. The metric showed discriminant validity and statistical significance when evaluated against known situations of high cognitive effort (ie, attention switching). With additional validation, this metric can potentially be used as a screening tool for assessing behavioral action phenotypes that are associated with higher cognitive burden.
An LM-based action entropy metric-relying on sequences of EHR actions-offers opportunities for assessing cognitive effort in EHR-based workflows.
开发并验证一种新的衡量标准，即动作熵，用于评估与电子健康记录 (EHR) 相关的工作活动所涉及的认知努力。
纳入了来自 2019 年四个外科重症监护病房的主治医生和高级执业医师 (APP) 的基于 EHR 的审核日志。分别为主治医生和 APP 的动作序列训练和验证神经语言模型 (LM)。动作熵是根据先前的动作，基于预测下一个动作的概率的交叉熵来计算的。为了验证该衡量标准，进行了一项配对研究，以评估在已知高认知努力场景下，即患者之间和 EHR 收件箱之间的注意力切换以及到或从 EHR 收件箱的注意力切换过程中动作熵的差异。
共纳入了 65 名在 8956 名患者上执行了 5904429 次基于 EHR 的审核日志操作的临床医生。与非切换场景相比，所有注意力切换场景都与更高的动作熵相关（P < .001），除了 APP 中的从收件箱切换场景。主治医生之间的最大差异是从收件箱到注意力切换：与非切换场景相比，切换场景下的动作熵高 1.288（95%CI，1.256-1.320）个标准差。对于 APP，最高的差异是到收件箱切换，切换场景下的动作熵高 2.354（95%CI，2.311-2.397）个标准差。
我们开发了一种基于 LM 的指标，即动作熵，用于评估与基于 EHR 的操作相关的认知负担。当该指标针对高认知努力（即注意力切换）的已知情况进行评估时，它显示出了判别有效性和统计学意义。通过进一步验证，该指标可能可作为评估与更高认知负担相关的行为动作表型的筛查工具。
基于 LM 的动作熵指标——依赖于 EHR 操作的序列——为评估基于 EHR 的工作流程中的认知努力提供了机会。
