Receiver operating characteristic analysis of diagnostic skill.

In an effort to assess diagnostic skill, the authors collected estimates of the probability of positive results for 392 chest x-rays and 595 throat cultures from 56 pediatricians at different stages in training. Positivity rates were 28.6% for x-rays and 32.9% for throat cultures. The ability to predict the outcome of chest x-rays (pneumonia or not) improved with training on two measures: 1) "calibration," the accuracy of the guessed probability of positivity, and 2) "operating characteristic," the ability to sort patients with negative x-rays from patients with positive x-rays. Improvement was most marked in the ability to predict that a patient with a positive x-ray would have a positive x-ray. For throat culture results, no regular improvement was found with training either in calibration or in operating characteristic. Measurements of calibration and operating characteristic are useful for assessing the effect of training on diagnostic skill.