Division of Data-Driven and Digital Medicine, Department of Medicine, and The Charles Bronfman Institute of Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, New York (A.V., P.J.).
Division of Data-Driven and Digital Medicine, Department of Medicine; The Charles Bronfman Institute of Personalized Medicine; and Division of Hospital Medicine, Icahn School of Medicine at Mount Sinai, New York, New York (A.S.).
Ann Intern Med. 2023 Oct;176(10):1358-1369. doi: 10.7326/M23-0949. Epub 2023 Oct 10.
Substantial effort has been directed toward demonstrating uses of predictive models in health care. However, implementation of these models into clinical practice may influence patient outcomes, which in turn are captured in electronic health record data. As a result, deployed models may affect the predictive ability of current and future models.
To estimate changes in predictive model performance with use through 3 common scenarios: model retraining, sequentially implementing 1 model after another, and intervening in response to a model when 2 are simultaneously implemented.
Simulation of model implementation and use in critical care settings at various levels of intervention effectiveness and clinician adherence. Models were either trained or retrained after simulated implementation.
Admissions to the intensive care unit (ICU) at Mount Sinai Health System (New York, New York) and Beth Israel Deaconess Medical Center (Boston, Massachusetts).
130 000 critical care admissions across both health systems.
Across 3 scenarios, interventions were simulated at varying levels of clinician adherence and effectiveness.
Statistical measures of performance, including threshold-independent (area under the curve) and threshold-dependent measures.
At fixed 90% sensitivity, in scenario 1 a mortality prediction model lost 9% to 39% specificity after retraining once and in scenario 2 a mortality prediction model lost 8% to 15% specificity when created after the implementation of an acute kidney injury (AKI) prediction model; in scenario 3, models for AKI and mortality prediction implemented simultaneously, each led to reduced effective accuracy of the other by 1% to 28%.
In real-world practice, the effectiveness of and adherence to model-based recommendations are rarely known in advance. Only binary classifiers for tabular ICU admissions data were simulated.
In simulated ICU settings, a universally effective model-updating approach for maintaining model performance does not seem to exist. Model use may have to be recorded to maintain viability of predictive modeling.
National Center for Advancing Translational Sciences.
人们已经投入大量精力来证明预测模型在医疗保健中的应用。然而，这些模型在临床实践中的实施可能会影响患者的预后，而患者的预后又会反映在电子健康记录数据中。因此，已部署的模型可能会影响当前和未来模型的预测能力。
通过 3 种常见场景（模型重新训练、依次实施 1 个模型后再实施另 1 个模型，以及当同时实施 2 个模型时对模型进行干预）来估计模型使用过程中的预测模型性能变化。
在不同干预效果和临床医生依从性水平下，模拟重症监护环境中的模型实施和使用。模型在模拟实施后要么进行训练，要么进行重新训练。
西奈山卫生系统（纽约州纽约市）和贝斯以色列女执事医疗中心（马萨诸塞州波士顿市）的重症监护病房（ICU）的入院患者。
两个医疗系统共 130000 名重症监护入院患者。
在 3 种场景中，模拟了不同水平的临床医生依从性和效果的干预措施。
性能的统计度量，包括阈值独立（曲线下面积）和阈值依赖度量。
在固定的 90%灵敏度下，在场景 1 中，在进行一次重新训练后，死亡率预测模型的特异性损失了 9%至 39%；在场景 2 中，在实施急性肾损伤（AKI）预测模型后创建的死亡率预测模型的特异性损失了 8%至 15%；在场景 3 中，同时实施 AKI 和死亡率预测模型，每个模型的实施都会使另一个模型的有效准确性降低 1%至 28%。
在实际实践中，模型推荐的有效性和依从性很少事先知晓。仅对 ICU 入院数据的表格分类器进行了模拟。
在模拟的 ICU 环境中，似乎不存在一种普遍有效的模型更新方法来维持模型性能。为了保持预测模型的可行性，可能需要记录模型的使用情况。
美国国家转化医学推进中心。
