Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA.
School of Medicine, Vanderbilt University, Nashville, Tennessee, USA.
Neurosurgery. 2021 Nov 18;89(6):1005-1011. doi: 10.1093/neuros/nyab346.
Letters of recommendation (LORs) are historically an important, though subjective, component of the neurosurgery residency application process. Standardized LORs (SLORs) were introduced during the 2020 to 2021 application cycle. The intent of SLORs is to allow objective comparison of applicants and to reduce bias.
To examine the utility of SLORs during this application cycle. We hypothesized that "grade inflation" and poor inter-rater reliability, as described by other specialties using SLORs, would limit the utility of SLORs in their current form.
This cross-sectional study analyzed all SLORs submitted to a single neurosurgery residency program over the 2020 to 2021 cycle. Data from 7 competency domains and the overall rating were recorded and stratified by academic category of letter writer. Inter-rater reliability was evaluated using Krippendorff's alpha.
One or more SLORs was submitted as part of 298 of 393 applications (76%). Approximately 58.3% of letters written by neurosurgery chairpersons rated a given applicant as being within the top 5% across all competencies. Approximately 44.4% of program director letters similarly rated applicants as amongst the top 5%, while 73.2% and 81.4% of letters by other neurosurgeons and general surgery evaluators, respectively, rated applicants in the top 5%. Inter-rater reliability was poor (<0.33) in all rating categories, including overall (α = 0.18).
The utility of the first iteration of SLORs in neurosurgery applications is undermined by significant "grade inflation" and poor inter-rater reliability. Improvements are necessary for SLORs if they are to provide meaningful information in future application cycles.
推荐信（LOR）是神经外科学住院医师申请过程中一个重要的、但主观的组成部分。标准化推荐信（SLOR）是在 2020 至 2021 申请周期中引入的。SLOR 的目的是允许对申请人进行客观比较，并减少偏见。
在本次申请周期中检查 SLOR 的实用性。我们假设，正如其他专业使用 SLOR 所描述的那样，“评分膨胀”和较差的评分者间可靠性将限制 SLOR 在其当前形式下的实用性。
这项横断面研究分析了在 2020 至 2021 周期内提交给单个神经外科学住院医师项目的所有 SLOR。记录了来自 7 个能力领域的数据和总体评分，并按信笺撰写者的学术类别进行分层。使用 Krippendorff's alpha 评估评分者间可靠性。
在 393 份申请中的 298 份（76%）提交了一份或多份 SLOR。大约 58.3%的神经外科主席所写的信件认为，在所有能力方面，某一特定申请人处于前 5%。大约 44.4%的项目主任信件同样将申请人评为前 5%，而分别有 73.2%和 81.4%的其他神经外科医生和普通外科评估者的信件将申请人评为前 5%。在所有评分类别中，包括总体评分（α=0.18），评分者间可靠性均较差（<0.33）。
在神经外科学申请中，第一代 SLOR 的实用性受到显著的“评分膨胀”和较差的评分者间可靠性的破坏。如果 SLOR 要在未来的申请周期中提供有意义的信息，则需要进行改进。
