Receiver operating characteristic (ROC) methodology: the state of the art.

Receiver operating characteristic (ROC) methodology has been increasingly used in medical applications in the last 10 years. The text by Swets and Pickett has popularized the technique and the journal Medical Decision Making (1981--) provides a forum for further methodologic issues. In this article, I will (1) describe the nature of the data generated by ROC studies; (2) evaluate the choices of summary indices of performance (accuracy); (3) outline the data-analytic techniques used, and how to incorporate data from multiple observers and multiple "readings"; (4) review proposed alternatives to the commonly used binormal ROC model; and (5) discuss issues, such as verification bias, and challenges, such as multicenter comparative imaging studies and the difficulty of obtaining "truth data", which need to be addressed when adapting ROC methods to medical contexts.