A practice oral examination rating scale--inter-observer reliability.

Three videotaped practice clinical, oral examinations were rated by 9, 9 and 11 raters respectively. Raters were all certified specialists engaged in training residents for the certification examinations. In two of the simulated examinations considerable differences in rating scores, to the extent of pass/fail disagreement, were found. The significance of the findings, including the possibility that one examiner may "contaminate" another, is discussed. Further work is essential to develop reliable instruments for rating certification examinations, whatever their format, as the Royal College policies evolve.