The value of Bayes theorem in the interpretation of subjective diagnostic findings: what can we learn from agreement studies?

The Bayes theorem is advocated as the appropriate measure for the weight of evidence in medical decision making. It is based on the calculation of posttest probability as a function of the accuracy of the test and pretest probability. Nevertheless, for subjective diagnostic findings, there might be substantial variability in the accuracy among human observers, making the point estimate of posttest probability imprecise. Although there is limited evidence regarding the actual variability of accuracy among observers for the majority of diagnostic findings, classical observer agreement studies provide us with an indirect estimate of such variability. The aim of this work was to explicate the relationship between observer disagreement and variability of posttest probability. Using a random effects signal detection model with 3 stochastic components (between subject, between observer, and residual variations), the authors modeled diagnostic tests with various characteristics and calculated the expected between-observer disagreement and 95% interval of the observers' posttest probability. For the majority of simulated conditions, variation in posttest probability was surprisingly high, even in the presence of substantial agreement. Although the model is based on parametric assumptions, these results are a clue to a source of inaccuracy in the calculation of posttest probability. Practitioners should be aware of such variation in their clinical practice, and diagnostic studies need to develop strategies to address this uncertainty.