Discovering factors influencing examiner agreement for periodontal measures.

OBJECTIVES

Calibration studies are routinely performed to establish examiner reliability in clinical periodontal research. In these studies, each periodontal site is assessed in duplicate, enabling point and interval estimation of agreement measures. We show how these data can be used additionally to discover subgroups among the periodontal sites according to degree of agreement with true periodontal status and to identify factors associated with examiner bias.

METHODS

A Bayesian hierarchical model is developed that, for all examiners, links the examiner's recorded measurement with the site's true periodontal status, allowing for site-specific examiner effects on the recorded measurement. These site-specific examiner effects are modeled as arising from a Dirichlet process mixture, which yields a small number (relative to the number of sites) of distinct effects for each examiner. Hence, sites that share the same examiner effect form a subgroup for which that examiner exhibits consistent bias relative to truth. We fit this model to data from a pilot calibration study for probed pocket depth measurements and use the results to explore examiner-specific groupings of sites according to degree of agreement with true pocket depth. The discovered group assignments were then associated with characteristics of the site.

RESULTS

The Bayesian hierarchical modeling revealed that periodontal sites were grouped according to bias into three, two, and two subgroups, respectively, for each of the three study examiners. The magnitude of the bias was associated with tooth position and true depth of the pocket.

CONCLUSIONS

Our Bayesian hierarchical model enhances the utility of data obtained from calibration studies for periodontal pocket depth by facilitating discovery of subgroups of sites according to examiner bias. The results indicate that targeting specific tooth locations and pocket depths during examiner training, uniquely for each examiner, may reduce bias in periodontal pocket depth measurements, thereby enhancing the quality of oral epidemiologic research.