Warren Alpert Medical School, Brown University, Providence, Rhode Island, United States.
Brown University Center for Biomedical Informatics, Providence, Rhode Island, United States.
Appl Clin Inform. 2022 Jan;13(1):56-66. doi: 10.1055/s-0041-1740923. Epub 2022 Feb 16.
Predictive analytic models, including machine learning (ML) models, are increasingly integrated into electronic health record (EHR)-based decision support tools for clinicians. These models have the potential to improve care, but are challenging to internally validate, implement, and maintain over the long term. Principles of ML operations (MLOps) may inform development of infrastructure to support the entire ML lifecycle, from feature selection to long-term model deployment and retraining.
This study aimed to present the conceptual prototypes for a novel predictive model management system and to evaluate the acceptability of the system among three groups of end users.
Based on principles of user-centered software design, human-computer interaction, and ethical design, we created graphical prototypes of a web-based MLOps interface to support the construction, deployment, and maintenance of models using EHR data. To assess the acceptability of the interface, we conducted semistructured user interviews with three groups of users (health informaticians, clinical and data stakeholders, chief information officers) and evaluated preliminary usability using the System Usability Scale (SUS). We subsequently revised prototypes based on user input and developed user case studies.
Our prototypes include design frameworks for feature selection, model training, deployment, long-term maintenance, visualization over time, and cross-functional collaboration. Users were able to complete 71% of prompted tasks without assistance. The average SUS score of the initial prototype was 75.8 out of 100, translating to a percentile range of 70 to 79, a letter grade of B, and an adjective rating of "good." We reviewed persona-based case studies that illustrate functionalities of this novel prototype.
The initial graphical prototypes of this MLOps system are preliminarily usable and demonstrate an unmet need within the clinical informatics landscape.
预测分析模型，包括机器学习（ML）模型，越来越多地被整合到基于电子健康记录（EHR）的决策支持工具中，为临床医生所用。这些模型有可能改善医疗服务，但在内部验证、长期实施和维护方面具有挑战性。机器学习运营（MLOps）原则可能为支持从特征选择到长期模型部署和再培训的整个 ML 生命周期的基础设施开发提供信息。
本研究旨在提出一种新的预测模型管理系统的概念原型，并评估该系统在三组终端用户中的可接受性。
基于用户为中心的软件设计、人机交互和伦理设计原则，我们创建了一个基于网络的 MLOps 界面的图形原型，以支持使用 EHR 数据构建、部署和维护模型。为了评估界面的可接受性，我们对三组用户（卫生信息学家、临床和数据利益相关者、首席信息官）进行了半结构化用户访谈，并使用系统可用性量表（SUS）评估了初步的可用性。随后，我们根据用户的输入修改了原型，并开发了用户案例研究。
我们的原型包括特征选择、模型训练、部署、长期维护、随时间的可视化以及跨职能协作的设计框架。用户能够在没有帮助的情况下完成 71%的提示任务。初始原型的平均 SUS 得分为 75.8 分（满分 100 分），相当于 70 到 79 分的百分位范围、B 级成绩和“良好”的形容词评级。我们审查了说明该新型原型功能的人物角色案例研究。
该 MLOps 系统的初始图形原型初步可用，并展示了临床信息学领域的一个未满足的需求。
