König I R, Malley J D, Weimar C, Diener H-C, Ziegler A
Institut für Medizinische Biometrie und Statistik, Universität zu Lübeck, Lübeck, Germany.
Stat Med. 2007 Dec 30;26(30):5499-511. doi: 10.1002/sim.3069.
The validity of prognostic models is an important prerequisite for their applicability in practical clinical settings. Here, we report on a specific prognostic study on stroke patients and describe how we explored the prediction performance of our model. We considered two practically highly relevant generalization aspects, namely, the model's performance in patients recruited at a later time point (temporal transportability) and in medical centers different from those used for model building (geographic transportability). To estimate the accuracy of the model, we investigated classical internal validation techniques and leave-one-center-out cross validation (CV). Prognostic models predicting functional independence of stroke patients were developed in a training set using logistic regression, support vector machines, and random forests (RFs). Tenfold CV and leave-one-center-out CV were employed to estimate temporal and geographic transportability of the models. For temporal and external validation, the resulting models were used to classify patients from a later time point and from different clinics. When applying the regression model or the RFs, accuracy in the temporal validation data was well predicted from classical internal validation. However, when predicting geographic transportability all approaches had difficulties. We observed that the leave-one-center-out CV yielded better estimates than classical CV. On the basis of our results, we conclude that external validation in patients from different clinics is required before a prognostic model can be applied in practice. Even validating the model in patients recruited merely at a later time point does not suffice to predict how it may fare with regard to another clinic.
预后模型的有效性是其在实际临床环境中应用的重要前提。在此，我们报告一项针对中风患者的特定预后研究，并描述我们如何探索模型的预测性能。我们考虑了两个在实际中高度相关的泛化方面，即模型在较晚时间点招募的患者中的性能（时间可迁移性）以及在与用于模型构建的医疗中心不同的医疗中心中的性能（地理可迁移性）。为了估计模型的准确性，我们研究了经典的内部验证技术和留一中心交叉验证（CV）。使用逻辑回归、支持向量机和随机森林（RFs）在训练集中开发了预测中风患者功能独立性的预后模型。采用十折交叉验证和留一中心交叉验证来估计模型的时间和地理可迁移性。对于时间和外部验证，将所得模型用于对来自较晚时间点和不同诊所的患者进行分类。当应用回归模型或随机森林时，经典内部验证能很好地预测时间验证数据中的准确性。然而，在预测地理可迁移性时，所有方法都存在困难。我们观察到留一中心交叉验证比经典交叉验证能产生更好的估计。基于我们的结果，我们得出结论，在将预后模型应用于实践之前，需要在来自不同诊所的患者中进行外部验证。即使仅在较晚时间点招募的患者中验证模型，也不足以预测其在另一家诊所的表现。
