Extension of the kappa coefficient.

An extension of the kappa coefficient is proposed which is appropriate for use with multiple observations per subject (not necessarily an equal number) and for multiple response choices per observation. Computational methods and nonasymptotic, nonnull distribution theory are discussed. The proposed method is applied to previously published data, not only to compare results with those in earlier methods, but to illustrate new approaches to difficult problems in evaluation of reliability. The effect of using an 'Other' response category is examined. Strategies to enhance reliability are evaluated, including empirical investigation of the Spearman-Brown formula as applied to nominal response measures.