Division of Chemical Pathology, Faculty of Health Sciences, National Health Laboratory Service (NHLS) and University of Stellenbosch, Cape Town, South Africa.
Non-Communicable Diseases Research Unit, South African Medical Research Council, Cape Town, South Africa.
PLoS One. 2019 Feb 7;14(2):e0211528. doi: 10.1371/journal.pone.0211528. eCollection 2019.
Prediction model updating methods are aimed at improving the prediction performance of a model in a new setting. This study sought to critically assess the impact of updating techniques when applying existent prevalent diabetes prediction models to a population different to the one in which they were developed, evaluating the performance in the mixed-ancestry population of South Africa.
The study sample consisted of 1256 mixed-ancestry individuals from the Cape Town Bellville-South cohort, of which 173 were excluded due to previously diagnosed diabetes and 162 individuals had undiagnosed diabetes. The primary outcome, undiagnosed diabetes, was based on an oral glucose tolerance test. Model updating techniques and prediction models were identified via recent systematic reviews. Model performance was assessed using the C-statistic and expected/observed (E/O) events rates ratio.
Intercept adjustment and logistic calibration improved calibration across all five models (Cambridge, Kuwaiti, Omani, Rotterdam and Simplified Finnish diabetes risk models). This was improved further by model revision, where likelihood ratio tests showed that the effect of body mass index, waist circumference and family history of diabetes required additional adjustment (Omani, Rotterdam and Finnish models). However, discrimination was poor following internal validation of these models. Re-estimation of the regression coefficients did not increase performance, while the addition of new variables resulted in the highest discriminatory and calibration performance combination for the models it was undertaken in.
While the discriminatory performance of the original existent models during external validation were higher, calibration was poor. The highest performing models, based on discrimination and calibration, were the Omani diabetes model following model revision, and the Cambridge diabetes risk model following the addition of waist circumference as a predictor. However, while more extensive methods incorporating development population information were superior over simpler methods, the increase in model performance was not great enough for recommendation.
预测模型更新方法旨在提高模型在新环境中的预测性能。本研究旨在批判性地评估在将现有的流行糖尿病预测模型应用于与模型开发人群不同的人群时，更新技术的影响，同时评估该模型在南非混合人群中的表现。
研究样本包括来自开普敦贝维尔-南部队列的 1256 名混合人群，其中 173 人因先前诊断出的糖尿病而被排除，162 人患有未诊断出的糖尿病。主要结局为未诊断出的糖尿病，基于口服葡萄糖耐量试验。通过最近的系统评价确定了模型更新技术和预测模型。使用 C 统计量和预期/观察（E/O）事件率比评估模型性能。
截距调整和逻辑校准提高了所有五个模型（剑桥、科威特、阿曼、鹿特丹和简化芬兰糖尿病风险模型）的校准度。通过模型修订进一步改善了校准度，似然比检验表明，体重指数、腰围和糖尿病家族史的影响需要进一步调整（阿曼、鹿特丹和芬兰模型）。然而，这些模型的内部验证显示出较差的区分度。重新估计回归系数并没有提高性能，而添加新变量则提高了对其进行评估的模型的区分度和校准性能的最佳组合。
虽然原始存在模型在外部验证中的区分性能更高，但校准度较差。基于区分度和校准度，表现最好的模型是在进行模型修订后的阿曼糖尿病模型，以及在添加腰围作为预测因子后的剑桥糖尿病风险模型。然而，尽管更广泛的方法结合了开发人群信息，但优于更简单的方法，但模型性能的提高不足以推荐使用。
