A design-independent method for measuring the reliability of psychiatric diagnosis.

The several different versions of the kappa coefficient currently in use can be seen as variants of a single basic computational method. This paper present general formulas for calculating kappa, both to measure the overall reliability of a set of diagnostic categories and to measure the specific reliability of individual categories. The formulas extend the range of situations in which kappa can be calculated to include multiple-rater, partially crossed designs of the sort that researchers frequently encounter.