An exact test for detecting inconsistency in readers interpretation over time in screening mammograms.

Radiologists' interpretation on screening mammograms is measured by accuracy indices such as sensitivity and specificity. The hypothesis that radiologists' interpretation on screening mammograms is constant across time can be tested by measuring overdispersion. However, small sample sizes are problematic for the accuracy of asymptotic approaches. In this article, we propose an exact conditional distribution for testing overdispersion of the binomial assumption that is assumed for the accuracy indices. An exact p -value can be defined from the developed distribution. We also describe an algorithm for computing this exact test. This proposed method is applied to data from a study in reading screening mammograms in a population of US radiologists (Beam et al., 2003). The exact method is compared analytically with a currently available method based on large sample approximations.