Department of Medicine, University of Minnesota, Minneapolis.
Institute for Health Informatics, University of Minnesota, Minneapolis.
JAMA Netw Open. 2023 Jul 3;6(7):e2324176. doi: 10.1001/jamanetworkopen.2023.24176.
The Deterioration Index (DTI), used by hospitals for predicting patient deterioration, has not been extensively validated externally, raising concerns about performance and equitable predictions.
To locally validate DTI performance and assess its potential for bias in predicting patient clinical deterioration.
DESIGN, SETTING, AND PARTICIPANTS: This retrospective prognostic study included 13 737 patients admitted to 8 heterogenous Midwestern US hospitals varying in size and type, including academic, community, urban, and rural hospitals. Patients were 18 years or older and admitted between January 1 and May 31, 2021.
DTI predictions made every 15 minutes.
Deterioration, defined as the occurrence of any of the following while hospitalized: mechanical ventilation, intensive care unit transfer, or death. Performance of the DTI was evaluated using area under the receiver operating characteristic curve (AUROC) and area under the precision recall curve (AUPRC). Bias measures were calculated across demographic subgroups.
A total of 5 143 513 DTI predictions were made for 13 737 patients across 14 834 hospitalizations. Among 13 918 encounters, the mean (SD) age of patients was 60.3 (19.2) years; 7636 (54.9%) were female, 11 345 (81.5%) were White, and 12 392 (89.0%) were of other ethnicity than Hispanic or Latino. The prevalence of deterioration was 10.3% (n = 1436). The DTI produced AUROCs of 0.759 (95% CI, 0.756-0.762) at the observation level and 0.685 (95% CI, 0.671-0.700) at the encounter level. Corresponding AUPRCs were 0.039 (95% CI, 0.037-0.040) at the observation level and 0.248 (95% CI, 0.227-0.273) at the encounter level. Bias measures varied across demographic subgroups and were 14.0% worse for patients identifying as American Indian or Alaska Native and 19.0% worse for those who chose not to disclose their ethnicity.
In this prognostic study, the DTI had modest ability to predict patient deterioration, with varying degrees of performance at the observation and encounter levels and across different demographic groups. Disparate performance across subgroups suggests the need for more transparency in model training data and reinforces the need to locally validate externally developed prediction models.
医院用于预测患者恶化的恶化指数（DTI）尚未在外部得到广泛验证，这引发了人们对其性能和公平预测的担忧。
对 DTI 的性能进行局部验证，并评估其预测患者临床恶化的潜在偏差。
设计、地点和参与者：这是一项回顾性预后研究，共纳入了来自美国中西部 8 家大小和类型各异的医院的 13737 名患者，包括学术型、社区型、城市型和农村型医院。患者年龄在 18 岁或以上，在 2021 年 1 月 1 日至 5 月 31 日期间入院。
每 15 分钟进行一次 DTI 预测。
恶化定义为住院期间发生以下任何一种情况：机械通气、转入重症监护病房或死亡。使用接收者操作特征曲线下面积（AUROC）和精度召回曲线下面积（AUPRC）评估 DTI 的性能。在人口统计学亚组中计算了偏差度量。
在 14834 次住院治疗中，共对 13737 名患者进行了 5143513 次 DTI 预测。在 13918 次就诊中，患者的平均（SD）年龄为 60.3（19.2）岁；7636 名（54.9%）为女性，11345 名（81.5%）为白人，12392 名（89.0%）为非西班牙裔或拉丁裔。恶化的患病率为 10.3%（n=1436）。DTI 在观察水平上的 AUROC 为 0.759（95%CI，0.756-0.762），在就诊水平上的 AUROC 为 0.685（95%CI，0.671-0.700）。相应的 AUPRC 在观察水平上为 0.039（95%CI，0.037-0.040），在就诊水平上为 0.248（95%CI，0.227-0.273）。偏倚指标在不同的人口统计学亚组之间存在差异，对于自称为美国印第安人或阿拉斯加原住民的患者，偏差为 14.0%，对于选择不透露其族裔的患者，偏差为 19.0%。
在这项预后研究中，DTI 对预测患者恶化具有中等能力，在观察水平和就诊水平以及不同的人口统计学群体中表现程度不同。亚组之间表现的差异表明，模型训练数据需要更加透明，并强调需要对外部开发的预测模型进行本地验证。
