Ortiz Alexander V, Feldman Michael J, Yengo-Kahn Aaron M, Roth Steven G, Dambrino Robert J, Chitale Rohan V, Chambless Lola B
1School of Medicine, Vanderbilt University; and.
2Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville, Tennessee.
J Neurosurg. 2022 Jul 8;138(2):559-566. doi: 10.3171/2022.5.JNS22558. Print 2023 Feb 1.
Narrative letters of recommendation (NLORs) are considered by neurosurgical program directors to be among the most important parts of the residency application. However, the utility of these NLORs in predicting match outcomes compared to objective measures has not been determined. In this study, the authors compare the performance of machine learning models trained on applicant NLORs and demographic data to predict match outcomes and investigate whether narrative language is predictive of standardized letter of recommendation (SLOR) rankings.
This study analyzed 1498 NLORs from 391 applications submitted to a single neurosurgery residency program over the 2020-2021 cycle. Applicant demographics and match outcomes were extracted from Electronic Residency Application Service applications and training program websites. Logistic regression models using least absolute shrinkage and selection operator were trained to predict match outcomes using applicant NLOR text and demographics. Another model was trained on NLOR text to predict SLOR rankings. Model performance was estimated using area under the curve (AUC).
Both the NLOR and demographics models were able to discriminate similarly between match outcomes (AUCs 0.75 and 0.80; p = 0.13). Words including "outstanding," "seamlessly," and "AOA" (Alpha Omega Alpha) were predictive of match success. This model was able to predict SLORs ranked in the top 5%. Words including "highest," "outstanding," and "best" were predictive of the top 5% SLORs.
NLORs and demographic data similarly discriminate whether applicants will or will not match into a neurosurgical residency program. However, NLORs potentially provide further insight regarding applicant fit. Because words used in NLORs are predictive of both match outcomes and SLOR rankings, continuing to include narrative evaluations may be invaluable to the match process.
神经外科住院医师项目主任认为，叙述性推荐信（NLORs）是住院医师申请中最重要的部分之一。然而，与客观指标相比，这些NLORs在预测匹配结果方面的效用尚未确定。在本研究中，作者比较了基于申请人NLORs和人口统计学数据训练的机器学习模型在预测匹配结果方面的表现，并研究叙述性语言是否能预测标准化推荐信（SLOR）的排名。
本研究分析了在2020 - 2021周期提交给单一神经外科住院医师项目的391份申请中的1498封NLORs。从电子住院医师申请服务系统的申请和培训项目网站中提取申请人的人口统计学信息和匹配结果。使用最小绝对收缩和选择算子的逻辑回归模型，通过申请人的NLOR文本和人口统计学信息来预测匹配结果。另一个模型基于NLOR文本进行训练，以预测SLOR排名。使用曲线下面积（AUC）评估模型性能。
NLOR模型和人口统计学模型在区分匹配结果方面的能力相似（AUC分别为0.75和0.80；p = 0.13）。包括“杰出的”“无缝地”和“AOA”（阿尔法欧米伽阿尔法）等词可预测匹配成功。该模型能够预测排名前5%的SLOR。包括“最高的”“杰出的”和“最好的”等词可预测排名前5%的SLOR。
NLORs和人口统计学数据在区分申请人是否能匹配到神经外科住院医师项目方面能力相似。然而，NLORs可能会提供关于申请人匹配度的进一步见解。由于NLORs中使用的词汇可预测匹配结果和SLOR排名，继续纳入叙述性评价对匹配过程可能非常有价值。
