Division of Hospital Medicine, Department of Medicine, Stanford University, Stanford, California, USA.
Program in Biomedical Informatics, Stanford University, Stanford, California, USA.
J Am Med Inform Assoc. 2020 Dec 9;27(12):1850-1859. doi: 10.1093/jamia/ocaa190.
To assess usability and usefulness of a machine learning-based order recommender system applied to simulated clinical cases.
43 physicians entered orders for 5 simulated clinical cases using a clinical order entry interface with or without access to a previously developed automated order recommender system. Cases were randomly allocated to the recommender system in a 3:2 ratio. A panel of clinicians scored whether the orders placed were clinically appropriate. Our primary outcome included the difference in clinical appropriateness scores. Secondary outcomes included total number of orders, case time, and survey responses.
Clinical appropriateness scores per order were comparable for cases randomized to the order recommender system (mean difference -0.11 order per score, 95% CI: [-0.41, 0.20]). Physicians using the recommender placed more orders (median 16 vs 15 orders, incidence rate ratio 1.09, 95%CI: [1.01-1.17]). Case times were comparable with the recommender system. Order suggestions generated from the recommender system were more likely to match physician needs than standard manual search options. Physicians used recommender suggestions in 98% of available cases. Approximately 95% of participants agreed the system would be useful for their workflows.
User testing with a simulated electronic medical record interface can assess the value of machine learning and clinical decision support tools for clinician usability and acceptance before live deployments.
Clinicians can use and accept machine learned clinical order recommendations integrated into an electronic order entry interface in a simulated setting. The clinical appropriateness of orders entered was comparable even when supported by automated recommendations.
评估基于机器学习的医嘱推荐系统在模拟临床病例中的可用性和实用性。
43 名医生使用具有或不具有先前开发的自动化医嘱推荐系统访问权限的临床医嘱输入界面为 5 个模拟临床病例输入医嘱。病例以 3:2 的比例随机分配给推荐系统。一组临床医生对所下医嘱的临床适宜性进行评分。我们的主要结果包括临床适宜性评分的差异。次要结果包括医嘱总数、病例时间和调查回复。
随机分配到医嘱推荐系统的病例的每条医嘱的临床适宜性评分相当（平均差异为 0.11 分/评分，95%CI：[-0.41, 0.20]）。使用推荐系统的医生下的医嘱更多（中位数 16 比 15 条医嘱，发病率比 1.09，95%CI：[1.01-1.17]）。病例时间与推荐系统相当。推荐系统生成的医嘱建议更符合医生的需求，而不是标准的手动搜索选项。在可用的病例中，医生使用推荐系统建议的比例约为 98%。约 95%的参与者认为该系统对他们的工作流程有用。
使用模拟电子病历界面进行用户测试可以在实际部署前评估机器学习和临床决策支持工具对临床医生可用性和接受程度的价值。
在模拟环境中，临床医生可以使用和接受集成到电子医嘱输入界面中的基于机器学习的临床医嘱推荐。即使有自动化推荐的支持，输入医嘱的临床适宜性也相当。
