Department of Radiation Oncology (MAASTRO), GROW School for Oncology and Developmental Biology, Maastricht University Medical Centre+, Maastricht, 6062, NA, the Netherlands.
Department of Radiotherapy, Sacred Heart University Hospital, Rome, 00168, Italy.
Med Phys. 2017 Sep;44(9):4961-4967. doi: 10.1002/mp.12423. Epub 2017 Aug 8.
Multiple models have been developed to predict pathologic complete response (pCR) in locally advanced rectal cancer patients. Unfortunately, validation of these models normally omit the implications of cohort differences on prediction model performance. In this work, we will perform a prospective validation of three pCR models, including information whether this validation will target transferability or reproducibility (cohort differences) of the given models.
We applied a novel methodology, the cohort differences model, to predict whether a patient belongs to the training or to the validation cohort. If the cohort differences model performs well, it would suggest a large difference in cohort characteristics meaning we would validate the transferability of the model rather than reproducibility. We tested our method in a prospective validation of three existing models for pCR prediction in 154 patients.
Our results showed a large difference between training and validation cohort for one of the three tested models [Area under the Receiver Operating Curve (AUC) cohort differences model: 0.85], signaling the validation leans towards transferability. Two out of three models had a lower AUC for validation (0.66 and 0.58), one model showed a higher AUC in the validation cohort (0.70).
We have successfully applied a new methodology in the validation of three prediction models, which allows us to indicate if a validation targeted transferability (large differences between training/validation cohort) or reproducibility (small cohort differences).
已经开发出多种模型来预测局部晚期直肠癌患者的病理完全缓解（pCR）。不幸的是，这些模型的验证通常忽略了队列差异对预测模型性能的影响。在这项工作中，我们将对三种 pCR 模型进行前瞻性验证，包括验证的目的是转移性能还是再现性能（队列差异）。
我们应用了一种新的方法，即队列差异模型，来预测患者属于训练队列还是验证队列。如果队列差异模型表现良好，这表明队列特征存在较大差异，这意味着我们将验证模型的可转移性，而不是再现性。我们在 154 名患者中对三种现有的 pCR 预测模型进行了前瞻性验证，以测试我们的方法。
我们的结果表明，三个测试模型中的一个模型的训练队列和验证队列之间存在很大差异[受试者工作特征曲线（AUC）下的队列差异模型：0.85]，这表明验证更倾向于转移。三个模型中有两个在验证中的 AUC 较低（0.66 和 0.58），一个模型在验证队列中的 AUC 较高（0.70）。
我们已经成功地将一种新的方法应用于三种预测模型的验证中，这使我们能够表明验证的目的是转移性能（训练/验证队列之间的差异较大）还是再现性能（队列差异较小）。
