General Surgery Residency Application Evaluation in a USMLE Step 1 Pass/Fail World: A Retrospective Comparison.

Introduction The United States Medical Licensure Exam (USMLE) Step 1 has been used as both a licensing exam and a way for residency programs to evaluate applicants. It has had significant impact upon the match process over time. With the 2020 decision to make the exam pass/fail due to its unclear validity as an evaluation for future physician performance, programs will go through the match without the Step 1 score. We set out to better understand the effects of the exam score on our selection process, with the hypothesis that without the step 1 score, the ranking of our applicants would be significantly altered. Methods We performed a retrospective analysis of applications to a single General Surgery residency program with 4 categorial residents per year at a physician led, academic, tertiary care medical center from 2017-2020. Important applicant factors including USMLE Step 1 and 2, AOA status, science grades, clerkship scores, audition rotations, volunteer activities, research activities, letters of recommendation, and personal statements were given points and evaluated through our equation, the sum of which was used to create a rank list and offer interviews. The standard deviation of scores was calculated with and without Step 1, and the distribution of scores compared. The range and average of applicants' change in point scores were examined. Results The applications of 653 students were reviewed. After removal of USMLE step 1 points, 40% of all applicants decreased in rank, 35% remained the same, and 24% increased. Specifically, 18.8% of the top third dropped to the middle third, and 11.7% of the bottom third jumped to the middle third, while the middle third changed little (0.2% dropped and 0.9% jumped out of middle third). The points given for USMLE step 1 created a wider distribution of scores with a negative skewness, suggesting there were more applicants below the mean than above. After removing those points, applicants' scores had a narrower distribution and skewness closer to 0, showing fewer upper outliers and more applicants near the mean. Conclusions The USMLE Step 1 score significantly affected the evaluation of applicants, and the removal of it from the recruitment criteria tightened applicant rankings. The elimination of the USMLE Step 1 score in the assessment of applicants will allow for its replacement with variables that better reflect the core values of residency programs.