Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA.
Department of Pharmacotherapy, Skaggs College of Pharmacy, University of Utah, Salt Lake City, Utah, USA.
J Am Med Inform Assoc. 2022 Apr 13;29(5):891-899. doi: 10.1093/jamia/ocab292.
To evaluate the potential for machine learning to predict medication alerts that might be ignored by a user, and intelligently filter out those alerts from the user's view.
We identified features (eg, patient and provider characteristics) proposed to modulate user responses to medication alerts through the literature; these features were then refined through expert review. Models were developed using rule-based and machine learning techniques (logistic regression, random forest, support vector machine, neural network, and LightGBM). We collected log data on alerts shown to users throughout 2019 at University of Utah Health. We sought to maximize precision while maintaining a false-negative rate <0.01, a threshold predefined through discussion with physicians and pharmacists. We developed models while maintaining a sensitivity of 0.99. Two null hypotheses were developed: H1-there is no difference in precision among prediction models; and H2-the removal of any feature category does not change precision.
A total of 3,481,634 medication alerts with 751 features were evaluated. With sensitivity fixed at 0.99, LightGBM achieved the highest precision of 0.192 and less than 0.01 for the pre-defined maximal false-negative rate by subject-matter experts (H1) (P < 0.001). This model could reduce alert volume by 54.1%. We removed different combinations of features (H2) and found that not all features significantly contributed to precision. Removing medication order features (eg, dosage) most significantly decreased precision (-0.147, P = 0.001).
Machine learning potentially enables the intelligent filtering of medication alerts.
评估机器学习预测用户可能忽略的用药警示的潜力，并智能地将这些警示从用户视图中过滤掉。
我们通过文献确定了特征（如患者和提供者特征），这些特征被认为可以调节用户对用药警示的反应；然后通过专家审查对这些特征进行了细化。使用基于规则和机器学习技术（逻辑回归、随机森林、支持向量机、神经网络和 LightGBM）开发模型。我们收集了 2019 年在犹他大学健康中心向用户展示的警示的日志数据。我们试图在保持假阴性率<0.01（通过与医生和药剂师讨论预先定义的阈值）的同时，最大化精度。我们在保持灵敏度为 0.99 的情况下开发了模型。提出了两个零假设：H1-预测模型之间的精度没有差异；H2-删除任何特征类别都不会改变精度。
共评估了 3481634 次用药警示，其中有 751 个特征。在灵敏度固定为 0.99 的情况下，LightGBM 实现了最高的精度 0.192 和低于主题专家预定义的最大假阴性率 0.01（H1）（P<0.001）。该模型可以减少 54.1%的警示量。我们删除了不同的特征组合（H2），并发现并非所有特征都对精度有显著贡献。删除用药医嘱特征（如剂量）会显著降低精度（-0.147，P=0.001）。
机器学习有可能实现用药警示的智能过滤。
