Do Program Directors of Anesthesiology Residency Programs Interpret Narrative Letters of Recommendation as Intended?

Background Letters of recommendation (LORs) are an important part of the application process for medical residency programs with most specialties preferring a narrative format. Given the inherent subjectivity of narrative LORs, the current study sought to determine whether the intended messages of narrative LORs written for applicants to anesthesiology residency programs are accurately interpreted by readers. Methodology Anonymous online surveys were sent via the Qualtrics platform to program directors (PDs) of the Accreditation Council for Graduate Medical Education-accredited anesthesiology residency programs in the Mid-Atlantic region as designated by the Electronic Residency Application Service, which consists of the states of New York, Pennsylvania, and New Jersey. Each PD participant received five surveys, each of which was attached to a de-identified LOR that was written by another PD located at an institution in the same region. Both the letter writer and study participants were asked to score LORs on a Likert-like scale. Participants were additionally asked whether the LORs, if received, would influence their decision to either offer an interview to the applicant or to rank the applicant. Finally, participants were asked to note any specific words or phrases within the LORs that they found to be particularly impactful. Results Overall, 10 of 34, 29.41%, PDs responded to the survey. There was a high correlation between the LOR intent and the respondents' assigned rating (Spearman's rho = 0.7973, p < 0.001). Responses were more accurate for "outstanding and excellent" LORs compared to the lower three categories. Results were unaffected after adjusting for respondents' years of experience as PDs. Additionally, 71.6% indicated that the LORs would influence the decision about offering an interview, and 56.5% stated that the LORs would influence a ranking decision. Conclusions Our results indicate that respondents' perception of LORs correlated strongly with the intent of the writer. Additionally, respondents seemed to value LORs for interview and ranking decisions.