Lam Jonathan Y, Lu Xiaolei, Shashikumar Supreeth P, Lee Ye Sel, Miller Michael, Pour Hayden, Boussina Aaron E, Pearce Alex K, Malhotra Atul, Nemati Shamim
Department of Biomedical Informatics, University of California San Diego, La Jolla, CA 92093, United States.
Division of Pulmonary, Critical Care, and Sleep Medicine, University of California San Diego, La Jolla, CA 92093, United States.
JAMIA Open. 2024 Dec 11;7(4):ooae141. doi: 10.1093/jamiaopen/ooae141. eCollection 2024 Dec.
This study describes the development and deployment of a machine learning (ML) model called Vent.io to predict mechanical ventilation (MV).
We trained Vent.io using electronic health record data of adult patients admitted to the intensive care units (ICUs) of the University of California San Diego (UCSD) Health System. We prospectively deployed Vent.io using a real-time platform at UCSD and evaluated the performance of Vent.io for a 1-month period in silent mode and on the MIMIC-IV dataset. As part of deployment, we included a Predetermined Changed Control Plan (PCCP) for continuous model monitoring that triggers model fine-tuning if performance drops below a specified area under the receiver operating curve (AUC) threshold of 0.85.
The Vent.io model had a median AUC of 0.897 (IQR: 0.892-0.904) with specificity of 0.81 (IQR: 0.812-0.841) and positive predictive value (PPV) of 0.174 (IQR: 0.148-0.176) at a fixed sensitivity of 0.6 during 10-fold cross validation and an AUC of 0.908, sensitivity of 0.632, specificity of 0.849, and PPV of 0.235 during prospective deployment. Vent.io had an AUC of 0.73 on the MIMIC-IV dataset, triggering model fine-tuning per the PCCP as the AUC was below the minimum of 0.85. The fine-tuned Vent.io model achieved an AUC of 0.873.
Deterioration of model performance is a significant challenge when deploying ML models prospectively or at different sites. Implementation of a PCCP can help models adapt to new patterns in data and maintain generalizability.
Vent.io is a generalizable ML model that has the potential to improve patient care and resource allocation for ICU patients with need for MV.
本研究描述了一种名为Vent.io的机器学习（ML）模型的开发与应用，用于预测机械通气（MV）。
我们使用加利福尼亚大学圣地亚哥分校（UCSD）医疗系统重症监护病房（ICU）成年患者的电子健康记录数据对Vent.io进行训练。我们在UCSD通过一个实时平台对Vent.io进行前瞻性应用，并在静默模式下以及在MIMIC-IV数据集上对Vent.io的性能进行了为期1个月的评估。作为应用的一部分，我们纳入了一个预定变更控制计划（PCCP）用于持续的模型监测，如果性能下降到低于接收器操作曲线（AUC）阈值0.85的指定区域，则触发模型微调。
在10倍交叉验证期间，Vent.io模型在固定灵敏度为0.6时，中位数AUC为0.897（四分位距：0.892 - 0.904），特异性为0.81（四分位距：0.812 - 0.841），阳性预测值（PPV）为0.174（四分位距：0.148 - 0.176）；在前瞻性应用期间，AUC为0.908，灵敏度为0.632，特异性为0.849，PPV为0.235。Vent.io在MIMIC-IV数据集上的AUC为0.73，由于AUC低于最低值0.85，根据PCCP触发了模型微调。微调后的Vent.io模型AUC达到0.873。
在进行前瞻性应用或在不同地点应用ML模型时，模型性能的恶化是一个重大挑战。实施PCCP有助于模型适应数据中的新模式并保持通用性。
Vent.io是一个具有通用性的ML模型，有潜力改善对有MV需求的ICU患者的护理和资源分配。
