BACKGROUND

Because inherent variability may exist in data collected by multiple reviewers or from potential difficulties with data abstraction tools, we developed a standardized method of evaluating interrater reliability (IRR) for clinical studies, HEDIS effectiveness of care measures, and onsite/medical record reviews.

OBJECTIVE

To demonstrate the ability of our standardized methods of data collection and analysis of results to determine the extent of agreement between multiple reviewers; identify areas for improvement in data collection procedures; and improve data reliability.

STUDY DESIGN

A prospective chart review with concurrent IRR.

METHODS

A subsample of patient records included in the Highmark Blue Cross Blue Shield/Keystone Health Plan West basic medical review for each HEDIS measure was selected for the IRR study. An experienced nurse ("gold standard") conducted a blinded concurrent review of these records. Using the kappa statistic (kappa) we evaluated interobserver agreement between results of the onsite reviewers and the "gold standard" from 1997 through 2000. Revised data collection methods and enhanced reviewer training were incorporated for measures showing areas for rater improvement.

RESULTS

Results across years showed excellent IRR for most measures; however, each year 1 or 2 measures showed areas for rater improvement (1997 Papanicolaou kappa = 0.50; 1998 well-child visits 3 to 6 years kappa = 0.37; 1999 comprehensive diabetes kappa = 0.73; high blood pressure kappa = 0.73). After reevaluating these measures, the results of the kappa showed excellent interrater agreement in subsequent years.

CONCLUSIONS

Standardized methods of data collection and evaluation of IRR results provides health plans increased confidence in data collection, statistical analyses, and in reaching conclusions and deriving relevant recommendations.