Ogero Morris, Ndiritu John, Sarguta Rachel, Tuti Timothy, Akech Samuel
Department of Mathematics University of Nairobi Nairobi Kenya.
Department of Infectious Disease Epidemiology London School of Hygiene & Tropical Medicine London United Kingdom.
Health Sci Rep. 2023 Aug 27;6(8):e1433. doi: 10.1002/hsr2.1433. eCollection 2023 Aug.
Prognostic models provide evidence-based predictions and estimates of future outcomes, facilitating decision-making, patient care, and research. A few of these models have been externally validated, leading to uncertain reliability and generalizability. This study aims to externally validate four models to assess their transferability and usefulness in clinical practice. The models include the respiratory index of severity in children (RISC)-Malawi model and three other models by Lowlavaar et al.
The study used data from the Clinical Information Network (CIN) to validate the four models where the primary outcome was in-hospital mortality. 163,329 patients met eligibility criteria. Missing data were imputed, and the logistic function was used to compute predicted risk of in-hospital mortality. Models' discriminatory ability and calibration were determined using area under the curve (AUC), calibration slope, and intercept.
The RISC-Malawi model had 50,669 pneumonia patients who met the eligibility criteria, of which the case-fatality ratio was 4406 (8.7%). Its AUC was 0.77 (95% CI: 0.77-0.78), whereas the calibration slope was 1.04 (95% CI: 1.00 -1.06), and calibration intercept was 0.81 (95% CI: 0.77-0.84). Regarding the external validation of Lowlavaar et al. models, 10,782 eligible patients were included, with an in-hospital mortality rate of 5.3%. The primary model's AUC was 0.75 (95% CI: 0.72-0.77), the calibration slope was 0.78 (95% CI: 0.71-0.84), and the calibration intercept was 0.37 (95% CI: 0.28-0.46). All models markedly underestimated the risk of mortality.
All externally validated models exhibited either underestimation or overestimation of the risk as judged from calibration statistics. Hence, applying these models with confidence in settings other than their original development context may not be advisable. Our findings strongly suggest the need for recalibrating these model to enhance their generalizability.
预后模型为未来结果提供基于证据的预测和估计，有助于决策制定、患者护理及研究。其中一些模型已进行外部验证，但可靠性和通用性仍不确定。本研究旨在对四个模型进行外部验证，以评估其在临床实践中的可转移性和实用性。这些模型包括儿童严重程度呼吸指数（RISC）-马拉维模型以及Lowlavaar等人提出的其他三个模型。
本研究使用临床信息网络（CIN）的数据对这四个模型进行验证，主要结局为院内死亡率。163329名患者符合纳入标准。对缺失数据进行插补，并使用逻辑函数计算院内死亡的预测风险。通过曲线下面积（AUC）、校准斜率和截距来确定模型的区分能力和校准情况。
RISC-马拉维模型有50669名符合纳入标准的肺炎患者，其中病死率为4406例（8.7%）。其AUC为0.77（95%CI：0.77 - 0.78），校准斜率为1.04（95%CI：1.00 - 1.06），校准截距为0.81（95%CI：0.77 - 0.84）。关于Lowlavaar等人模型的外部验证，纳入了10782名符合条件的患者，院内死亡率为5.3%。主要模型的AUC为0.75（95%CI：0.72 - 0.77），校准斜率为0.78（95%CI：0.71 - 0.84），校准截距为0.37（95%CI：0.28 - 0.46）。所有模型均明显低估了死亡风险。
从校准统计数据判断，所有经过外部验证的模型均表现出对风险的低估或高估。因此，在其原始开发背景以外的环境中自信地应用这些模型可能并不可取。我们的研究结果强烈表明需要重新校准这些模型以提高其通用性。
