Validity of prognostic models: when is a model clinically useful?

Prognostic models combine patient characteristics to predict medical outcomes. Unfortunately, such models do not always perform as well for other patients as those from whose data the models were derived. Therefore, validity of prognostic models needs to be assessed in new patients. Predicted probabilities can be calculated with the model and compared with the actually observed outcomes. We may distinguish several aspects of validity: (1) agreement between predicted probabilities and observed probabilities (calibration), (2) ability of the model to distinguish subjects with different outcomes (discrimination), and (3) ability of the model to improve the decision-making process (clinical usefulness). We discuss those aspects and show some measures by using models for testicular and prostate cancer. We conclude that good calibration and discriminative ability are not sufficient for a model to be clinically useful. Application of a prognostic model is sensible, if the model is able to provide useful additional information for clinical decision making.