Are Otolaryngology Milestones Predictive of Otolaryngology Training Examination Scores?

IMPORTANCE

: Otolaryngology residents take the otolaryngology training examination (OTE) yearly to assess their fund of knowledge. The Accreditation Council for Graduate Medical Education (ACGME) milestone evaluations are also conducted semiannually. Accurate prediction of training examination performance allows identification of residents who are performing well and those who need targeted remediation. Prior studies in other specialties have attempted to use milestone evaluations to help predict in-training examination scores.

OBJECTIVE

: In this study, we aim to identify whether ACGME milestone evaluation scores predict OTE performance.

DESIGN

: Milestone ratings and OTE scores for residents at 2 US otolaryngology residency programs were collected. Multivariate analysis was achieved using linear mixed modeling. We considered a 2-tailed P value of ≤ .05 as statistically significant.

SETTING

: Two US otolaryngology residency programs.

PARTICIPANTS

: Forty-eight otolaryngology residents postgraduate years 2 to 5.

RESULTS

: Otolaryngology training examination scores and ACGME milestone evaluations were collected from 48 residents from postgraduate year 2 to 5 between the years 2014 and 2017. One hundred eight OTE scores were available. Linear mixed-effect models were constructed, and after adjusting for level of training and OTE year, the total milestone rating made a negligible impact in estimating OTE percentage correct (β = -.01, P = .9). Similarly, total milestone rating demonstrated a minimal contribution in approximating OTE national stanine score after adjusting for the level of training (β = -.003, P = .9).

CONCLUSIONS AND RELEVANCE

: In our study, ACGME milestone evaluations are not predictive of residents' OTE performance. What these milestone evaluation data mean and how they should be used continues to be an unanswered question. We should aim to identify the most effective applications of the milestone data collected yearly by otolaryngology programs.