Division of Biomedical Informatics, School of Medicine, University of California, San Diego, La Jolla, California 92093, USA.
J Am Med Inform Assoc. 2012 Mar-Apr;19(2):263-74. doi: 10.1136/amiajnl-2011-000291. Epub 2011 Oct 7.
Predictive models that generate individualized estimates for medically relevant outcomes are playing increasing roles in clinical care and translational research. However, current methods for calibrating these estimates lose valuable information. Our goal is to develop a new calibration method to conserve as much information as possible, and would compare favorably to existing methods in terms of important performance measures: discrimination and calibration.
We propose an adaptive technique that utilizes individualized confidence intervals (CIs) to calibrate predictions. We evaluate this new method, adaptive calibration of predictions (ACP), in artificial and real-world medical classification problems, in terms of areas under the ROC curves, the Hosmer-Lemeshow goodness-of-fit test, mean squared error, and computational complexity.
ACP compared favorably to other calibration methods such as binning, Platt scaling, and isotonic regression. In several experiments, binning, isotonic regression, and Platt scaling failed to improve the calibration of a logistic regression model, whereas ACP consistently improved the calibration while maintaining the same discrimination or even improving it in some experiments. In addition, the ACP algorithm is not computationally expensive.
The calculation of CIs for individual predictions may be cumbersome for certain predictive models. ACP is not completely parameter-free: the length of the CI employed may affect its results.
ACP can generate estimates that may be more suitable for individualized predictions than estimates that are calibrated using existing methods. Further studies are necessary to explore the limitations of ACP.
生成针对医学相关结果的个体化估计的预测模型在临床护理和转化研究中发挥着越来越重要的作用。然而，目前校准这些估计值的方法会丢失有价值的信息。我们的目标是开发一种新的校准方法，以尽可能多地保留信息，并在重要的性能指标（区分度和校准度）方面优于现有方法。
我们提出了一种利用个体化置信区间（CI）来校准预测的自适应技术。我们在人工和真实世界的医学分类问题中评估了这种新方法，即预测自适应校准（ACP），评估指标包括 ROC 曲线下面积、Hosmer-Lemeshow 拟合优度检验、均方误差和计算复杂度。
ACP 与其他校准方法（如分箱、Platt 缩放和单调回归）相比表现良好。在几个实验中，分箱、单调回归和 Platt 缩放未能改善逻辑回归模型的校准，而 ACP 则始终在保持相同区分度的情况下改善了校准，甚至在某些实验中还提高了区分度。此外，ACP 算法的计算复杂度不高。
对于某些预测模型，为个体预测计算 CI 可能很麻烦。ACP 并非完全无参数：所使用的 CI 长度可能会影响其结果。
ACP 可以生成比使用现有方法校准的估计值更适合个体化预测的估计值。需要进一步研究来探索 ACP 的局限性。
