Comparing correlated kappas by resampling: is one level of agreement significantly different from another?

Researchers comparing diagnostic systems or screening tests frequently need to compare indices of agreement such as the kappa coefficient. While asymptotic methods exist for comparing kappas derived from independent samples, no satisfactory approach exists for comparing kappas derived from the same or related samples. Resampling methods for comparing kappa coefficients obtained from the same sample are presented. Easily undertaken using readily available software, these methods are illustrated by application to a small sample of psychiatric data, as well as to several thousand samples of simulated data. An acceptable type I error rate was exhibited. Resampling techniques-easily implemented and making few assumptions-deserve wider application in psychiatric research.