Which Application Factors Are Associated With Outstanding Performance in Orthopaedic Surgery Residency?

BACKGROUND

Identifying ideal candidates for orthopaedic surgery residency is difficult. Data available for applicant selection are evolving; preclinical grades and the Alpha Omega Alpha (AOA) honors society are being phased out at some medical schools. Similarly, three-digit United States Medical Licensing Examination (USMLE) Step 1 scores have been eliminated. There is renewed interest in improving resident selection to provide a diverse, comprehensive educational opportunity that produces orthopaedic surgeons who are prepared for practice.

QUESTIONS/PURPOSES: We sought to identify whether (1) academic achievements, (2) letters of recommendation, (3) research activity, and (4) miscellaneous factors available on Electronic Residency Application Service (ERAS) applications were associated with outstanding residency performance.

METHODS

Ten faculty members (22% of all full-time faculty) with extensive educational involvement for at least 7 years, whose expertise covered all subspecialty departments at an urban, academic orthopaedic surgery residency program, were given an anonymous survey on the performance of the four most recent classes of residency graduates (24 residents). This survey was developed due to the lack of a validated residency outcomes tool or objective metrics for residency performance. The evaluated criteria were decided upon after discussion by a relatively large group of academic orthopaedic surgeons considering the factors most important for graduating orthopaedic residents. The faculty were selected based on their long-term knowledge of the residency, along with their diversity of specialty and backgrounds; there were no nonresponders. Faculty graded each resident on a scale from 1 to 10 (higher is better) on six criteria: surgical technical skills, research productivity, clinical knowledge, professionalism, personality, and fellowship match. The mean of the faculty ratings made by all faculty for all six criteria was calculated, producing the overall residency performance score. Factors available on each resident's ERAS application were then correlated with their overall residency performance score. Categorical ERAS factors, including AOA status, five or more honors in core clerkships, at least three exceptional letters of recommendation, collegiate athletics participation, expertise with a musical instrument, and research (6-year) track residents, were correlated with overall residency performance score via point biserial analysis. Continuous ERAS factors including USMLE Step 1 and Step 2 scores, number of publications before residency, number of research years before residency, medical school ranking, and number of volunteer experiences were correlated with overall residency performance score via Pearson correlation. USMLE Step 1 three-digit scores were evaluated despite their recent elimination because of their historic importance as a screening tool for residency interviews and for comparison to USMLE Step 2, which retains a three-digit score. Application factors with a p < 0.2 on univariate analysis (five or more honors in core clerkships, at least three exceptional letters of recommendation, research track residents) were included in a stepwise linear regression model with "overall residency performance score" as the outcome variable. All p values < 0.05 were considered significant.

RESULTS

The mean overall residency performance score was 7.9 ± 1.2. Applicants with at least five honors grades in core clerkships had overall residency performance scores 1.2 points greater than those of their peers (95% confidence interval (CI) 0.3 to 2.0; p = 0.01, Cohen ƒ 2 = 0.2, representing a small effect size). ERAS applications including at least three exceptional letters of recommendation were associated with a 0.9-point increase in residency performance (95% CI 0.02 to 1.7; p = 0.046, Cohen ƒ 2 = 0.1, representing a small effect size). Participation in the residency research (6-year) track was associated with a 1-point improvement in residency performance (95% CI 0.1 to 1.9; p = 0.03, Cohen ƒ 2 = 0.2, again, representing a small effect size). Together, these three factors accounted for 53% of the variance in overall residency performance score observed in this study.

CONCLUSION

Past clinical excellence, measured by core clerkship grades and exceptional letters of recommendation, is associated with slightly improved overall orthopaedic residency performance scores. Applicants meeting both criteria who also complete a research track residency may perform substantially better in residency than their counterparts, as these three factors accounted for half of all the variance observed in the current study. Although minimum requirements are necessary, traditionally used screening factors (such as USMLE scores, AOA status, medical school rank, and number of publications) may be of less utility in identifying successful future residents than previously thought.

LEVEL OF EVIDENCE

Level III, therapeutic study.