Wang Junfeng, Yuan Tanwei, Ling Xuemei, Li Quanmin, Tang Xiaoping, Cai Weiping, Zou Huachun, Li Linghua
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Universiteitsweg 100, 3584 CG, Utrecht, the Netherlands.
School of Public Health (Shenzhen), Sun Yat-sen University, Shenzhen, China.
Diagn Progn Res. 2020 Nov 25;4(1):19. doi: 10.1186/s41512-020-00088-x.
HIV/AIDS remains a leading cause of death worldwide. Recently, a model has been developed in Wenzhou, China, to predict the survival of people living with HIV/AIDS (PLWHA) who underwent antiretroviral therapy (ART). We aimed to evaluate the methodological quality and validate the model in an external population-based cohort.
Prediction Model Risk of Bias Assessment Tool (PROBAST) was used to assess the risk of bias of the Wenzhou model. Data were from the National Free Antiretroviral Treatment Program database. We included PLWHA treated between February 2004 and December 2019 in a tertiary hospital in Guangzhou city, China. The endpoint was all-cause deaths and assessed until January 2020. We assessed the discrimination performance of the model by Harrell's overall C-statistics and time-dependent C-statistics and calibration by comparing observed survival probabilities estimated with the Kaplan-Meier method versus predicted survival probabilities. To assess the potential prediction value of age and gender which were precluded in developing the Wenzhou model, we compared the discriminative ability of the original model with an extended model added with age and gender.
Based on PROBAST, the Wenzhou model was rated as high risk of bias in three out of the four domains (selection of participants, definition of outcome, and methods for statistical analysis) mainly because of the misuse of nested case-control design and propensity score matching. In the external validation analysis, 16758 patients were included, among whom 743 patients died (mortality rate 11.41 per 1000 person-years) during follow-up (median 3.41 years, interquartile range 1.64-5.62). The predictor of HIV viral load was missing in 14361 patients (85.7%). The discriminative ability of the Wenzhou model decreased in the external dataset, with the Harrell's overall C-statistics being 0.76, and time-dependent C-statistics dropping from 0.81 at 6 months to 0.48 at 10 years after ART initiation. The model consistently underestimated the survival, and the level was 6.23%, 10.02%, and 14.82% at 1, 2, and 3 years after ART initiation, respectively. The overall and time-dependent discriminative ability of the model improved after adding age and gender to the original model.
The Wenzhou prognostic model is at high risk of bias in model development, with inadequate model performance in external validation. Thereby, we could not confirm the validity and extended utility of the Wenzhou model. Future prediction model development and validation studies need to comply with the methodological standards and guidelines specifically developed for prediction models.
艾滋病毒/艾滋病仍然是全球主要的死亡原因。最近，中国温州开发了一种模型，用于预测接受抗逆转录病毒治疗（ART）的艾滋病毒/艾滋病感染者（PLWHA）的生存情况。我们旨在评估该模型的方法学质量，并在一个基于人群的外部队列中对其进行验证。
使用预测模型偏倚风险评估工具（PROBAST）来评估温州模型的偏倚风险。数据来自国家免费抗逆转录病毒治疗项目数据库。我们纳入了2004年2月至2019年12月在中国广州市一家三级医院接受治疗的PLWHA。终点为全因死亡，并评估至2020年1月。我们通过Harrell总体C统计量和时间依赖C统计量评估模型的判别性能，并通过比较用Kaplan-Meier方法估计的观察生存概率与预测生存概率来进行校准。为了评估在开发温州模型时未纳入的年龄和性别的潜在预测价值，我们比较了原始模型与添加了年龄和性别的扩展模型的判别能力。
基于PROBAST，温州模型在四个领域中的三个（参与者选择、结局定义和统计分析方法）被评为高偏倚风险，主要原因是嵌套病例对照设计和倾向得分匹配的误用。在外部验证分析中，纳入了16758例患者，其中743例患者在随访期间死亡（死亡率为每1000人年11.41例）（中位随访时间3.41年，四分位间距1.64 - 5.62）。14361例患者（85.7%）的艾滋病毒病毒载量预测指标缺失。温州模型在外部数据集中的判别能力下降，Harrell总体C统计量为0.76，时间依赖C统计量从ART开始后6个月时的0.81降至10年时的0.48。该模型持续低估生存情况，在ART开始后1年、2年和3年时低估水平分别为6.23%、10.02%和14.82%。在原始模型中添加年龄和性别后，模型的总体和时间依赖判别能力有所提高。
温州预后模型在模型开发中存在高偏倚风险，在外部验证中的模型性能不足。因此，我们无法确认温州模型的有效性和扩展效用。未来的预测模型开发和验证研究需要遵循专门为预测模型制定的方法学标准和指南。
