Predicting performance on the Medical Council of Canada Qualifying Exam Part II.

BACKGROUND

Being able to predict which residents will likely be unsuccessful on high-stakes exams would allow residency programs to provide early intervention.

PURPOSE

To determine whether measures of clinical performance in clerkship (in-training evaluation reports) and first year of residency (program director ratings) predict pass-fail performance on the Medical Council of Canada Qualifying Exam Part II (MCCQE Part II).

METHODS

Residency program directors assessed the performance of our medical school graduates (Classes 2004-2007) at the end of the 1st postgraduate year. We subsequently collected clerkship in-training evaluation reports for these graduates. Using a neutral third party and unique codes, an anonymous dataset containing clerkship, residency, and MCCQE Part II performance scores was created for our use. Data were analyzed using descriptive statistics, correlations, receiver operating characteristics, and the Youdin index. Regression was also performed to further study the relationship among the variables.

RESULTS

Complete data were available for 78.6% of the graduates. Of these participants, 94% passed the licensing exam on their first attempt. Receiver operating characteristics revealed that the area under the curve for clerkship in-training evaluation reports was 0.67 (p<.05) and 0.66 (p<.05) for residency program directors assessments. Corresponding Youdin indices for in-training evaluation reports and residency program director assessments were 0.30 and 0.23, respectively.

CONCLUSIONS

Although clerkship in-training evaluation reports and residency program director ratings are significant predictors of pass-fail performance on the MCCQE Part II, the effectiveness of each one to predict pass-fail performance was relatively small. Reasons for these findings are discussed.