A surgery oral examination: interrater agreement and the influence of rater characteristics.

BACKGROUND

Poor interrater reliability is a common objection to the use of oral examinations.

METHOD

In 1990 the authors measured the agreement of 140 U.S. and Canadian surgical raters and the influences, if any, of age, years in practice, and experience as an examiner on individual oral examination scores. Eight actor examinees memorized transcripts of actual oral examinations and were videotaped using a single examiner. Examinee verbal style, dress, content of answers, and gender were purposefully adjusted. A repeated-measures analysis of variance was used for data analysis.

RESULTS

Three aspects of examinee performance influenced scores (verbal style, dress, and content of answers). No rater characteristic significantly affected scores. Raters showed high agreement (86%) when rating "good" performances but less agreement (67%) when rating "poor" performances.

CONCLUSION

The oral examination scores were not influenced by rater selection. The raters ranked good performances more consistently than poor performances. Therefore, more than one examiner appears necessary to confirm a poor performance during an examination.