Department of Clinical Genetics/EMGO Institute for Health and Care Research, Section Community Genetics, VU University Medical Center, PO Box 7057 (BS7 A-529), 1007 MB, Amsterdam, The Netherlands.
Department of Clinical Genetics/EMGO Institute for Health and Care Research, Section Community Genetics, VU University Medical Center, PO Box 7057 (BS7 A-529), 1007 MB, Amsterdam, The Netherlands; Department of Epidemiology, Rollins School of Public Health, Emory University, 1518 Clifton Road NE, Atlanta, GA 30322, USA.
J Clin Epidemiol. 2016 Nov;79:159-164. doi: 10.1016/j.jclinepi.2016.07.002. Epub 2016 Jul 16.
Adding risk factors to a prediction model often increases the area under the receiver operating characteristic curve (AUC) only slightly, particularly when the AUC of the model was already high. We investigated whether a risk factor that minimally improves the AUC may nevertheless improve the predictive ability of the model, assessed by integrated discrimination improvement (IDI).
We simulated data sets with risk factors and event status for 100,000 hypothetical individuals and created prediction models with AUCs between 0.50 and 0.95. We added a single risk factor for which the effect was modeled as a certain odds ratio (OR 2, 4, 8) or AUC increment (ΔAUC 0.01, 0.02, 0.03).
Across all AUC values of the baseline model, for a risk factor with the same OR, both ΔAUC and IDI were lower when the AUC of the baseline model was higher. When the increment in AUC was small (ΔAUC 0.01), the IDI was also small, except when the AUC of the baseline model was >0.90.
When the addition of a risk factor shows minimal improvement in AUC, predicted risks generally show minimal changes too. Updating risk models with strong risk factors may be informative for a subgroup of individuals, but not at the population level. The AUC may not be as insensitive as is frequently argued.
在预测模型中添加风险因素通常只会略微提高受试者工作特征曲线（AUC）下的面积，尤其是当模型的 AUC 已经很高时。我们研究了风险因素是否可以最小程度地提高 AUC，但可以通过综合判别改善（IDI）来提高模型的预测能力。
我们模拟了具有风险因素和事件状态的 100,000 个假想个体的数据，并为 AUC 在 0.50 到 0.95 之间的预测模型创建了数据。我们添加了一个单一的风险因素，该因素的影响被建模为特定的比值比（OR2、4、8）或 AUC 增量（ΔAUC0.01、0.02、0.03）。
在基线模型的所有 AUC 值中，对于具有相同 OR 的风险因素，当基线模型的 AUC 较高时，ΔAUC 和 IDI 均较低。当 AUC 增量较小时（ΔAUC0.01），IDI 也较小，除非基线模型的 AUC>0.90。
当添加风险因素对 AUC 的改善很小，预测风险通常也会发生很小的变化。对于具有强风险因素的风险模型进行更新可能对个体的亚组有意义，但对人群水平没有意义。AUC 可能不像经常讨论的那样不敏感。
