How well can physicians estimate mortality in a medical intensive care unit?

The accuracies of physicians' predictions of mortality for 523 patients in a medical intensive care unit were compared with estimates derived from a logistic model. The model utilized a popular severity-of-illness measure, the APACHE II. Accuracy was assessed through its components resolution (discrimination) and calibration. Physicians could better discriminate survivors from nonsurvivors, as measured by the area under the receiver operating characteristic curve (0.89 for physicians vs 0.83 for APACHE II model, p less than 0.001) and by resolution (0.103 for physicians vs 0.130 for APACHE II model, p less than 0.001). Overall, the APACHE II model was better calibrated (0.003 for APACHE II vs 0.021 for physicians, p less than 0.001). While the APACHE II model was better calibrated in the central probability ranges, physicians could more accurately identify those most likely to die. Decisions on withholding or withdrawing treatment are being made daily in intensive care units based on physicians' subjective prognostic estimates. At least for experienced physicians at a major medical center, these estimates are comparable in accuracy to quantitative models.