The "differential diagnosis" for multiple diseases: comparison with the binary-truth state experiment in two empirical studies.

RATIONALE AND OBJECTIVES

In practice readers must often choose between multiple diagnoses. For assessing reader accuracy in these settings. Obuchowski et al have proposed the "differential diagnosis" method, which derives all pairwise estimates of accuracy for the various diagnoses, along with summary measures of accuracy. The current study assessed the correspondence between the differential diagnosis method and conventional binary-truth state experiments.

MATERIALS AND METHODS

Two empirical studies were conducted at two institutions with different readers and diagnostic tests. Readers used the differential diagnosis format to interpret a set of cases. In subsequent readings they interpreted the cases in binary-truth state experiments. Spearman rank correlation coefficients and the percentages of agreement in scores were computed, and the areas under the receiver operating characteristic curves were estimated and compared.

RESULTS

The between-format Spearman rank correlation coefficients were 0.697-0.718 and 0.750-0.780 for the two studies; the between-reader correlations were 0.417 and 0.792, respectively. The percentages of agreement between formats for the two studies were 50.0%-51.7% and 72.9%-78.8%; the percentages of agreement between readers were 45.0% and 80%, respectively. In the first study there were several significant differences in the areas under receiver operating characteristic curves; in the second study these differences were small.

CONCLUSION

The differences observed between the two formats can be attributed to within-reader variability and inherent differences in the questions posed to readers in the multiple-diagnoses versus binary-truth state reading sessions. The differential diagnosis format is useful for estimating accuracy when there are multiple possible diagnoses.