Kenya Medical Research Institute (KEMRI)-Wellcome Trust Research Programme, Nairobi, Kenya.
School of Mathematics, University of Nairobi, Nairobi, Kenya.
Paediatr Perinat Epidemiol. 2023 May;37(4):313-321. doi: 10.1111/ppe.12948. Epub 2023 Feb 6.
In an external validation study, model recalibration is suggested once there is evidence of poor model calibration but with acceptable discriminatory abilities. We identified four models, namely RISC-Malawi (Respiratory Index of Severity in Children) developed in Malawi, and three other predictive models developed in Uganda by Lowlaavar et al. (2016). These prognostic models exhibited poor calibration performance in the recent external validation study, hence the need for recalibration.
In this study, we aim to recalibrate these models using regression coefficients updating strategy and determine how much their performances improve.
We used data collected by the Clinical Information Network from paediatric wards of 20 public county referral hospitals. Missing data were multiply imputed using chained equations. Model updating entailed adjustment of the model's calibration performance while the discriminatory ability remained unaltered. We used two strategies to adjust the model: intercept-only and the logistic recalibration method.
Eligibility criteria for the RISC-Malawi model were met in 50,669 patients, split into two sets: a model-recalibrating set (n = 30,343) and a test set (n = 20,326). For the Lowlaavar models, 10,782 patients met the eligibility criteria, of whom 6175 were used to recalibrate the models and 4607 were used to test the performance of the adjusted model. The intercept of the recalibrated RISC-Malawi model was 0.12 (95% CI 0.07, 0.17), while the slope of the same model was 1.08 (95% CI 1.03, 1.13). The performance of the recalibrated models on the test set suggested that no model met the threshold of a perfectly calibrated model, which includes a calibration slope of 1 and a calibration-in-the-large/intercept of 0.
Even after model adjustment, the calibration performances of the 4 models did not meet the recommended threshold for perfect calibration. This finding is suggestive of models over/underestimating the predicted risk of in-hospital mortality, potentially harmful clinically. Therefore, researchers may consider other alternatives, such as ensemble techniques to combine these models into a meta-model to improve out-of-sample predictive performance.
在一项外部验证研究中，建议在模型校准效果不佳但判别能力尚可时进行模型重新校准。我们确定了四个模型，分别是马拉维的 RISC-Malawi（儿童呼吸严重程度指数）和由 Lowlaavar 等人在乌干达开发的另外三个预测模型。这些预后模型在最近的外部验证研究中表现出较差的校准性能，因此需要重新校准。
本研究旨在使用回归系数更新策略对这些模型进行重新校准，并确定其性能提高的程度。
我们使用临床信息网络从 20 家公立县转诊医院的儿科病房收集的数据。使用链式方程对缺失数据进行多重插补。模型更新包括调整模型的校准性能，而不改变其判别能力。我们使用两种策略来调整模型：仅调整截距和逻辑重新校准方法。
RISC-Malawi 模型的纳入标准符合 50669 名患者，将其分为两组：模型重新校准组（n=30343）和测试组（n=20326）。对于 Lowlaavar 模型，符合纳入标准的患者有 10782 名，其中 6175 名用于重新校准模型，4607 名用于测试调整后模型的性能。重新校准的 RISC-Malawi 模型的截距为 0.12（95%置信区间 0.07，0.17），而同一模型的斜率为 1.08（95%置信区间 1.03，1.13）。重新校准模型在测试集上的性能表明，没有一个模型符合完美校准模型的阈值，包括校准斜率为 1 和校准大/截距为 0。
即使在调整模型后，这 4 个模型的校准性能仍未达到推荐的完美校准阈值。这一发现表明，模型可能高估或低估住院死亡率的预测风险，这在临床上可能是有害的。因此，研究人员可能会考虑其他替代方案，例如集成技术，将这些模型组合成一个元模型，以提高样本外预测性能。
