Feldman Michael J, Ortiz Alexander V, Roth Steven G, Dambrino Robert J, Yengo-Kahn Aaron M, Chitale Rohan V, Chambless Lola B
1Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville; and.
2School of Medicine, Vanderbilt University, Nashville, Tennessee.
J Neurosurg. 2022 Sep 9;138(4):1132-1138. doi: 10.3171/2022.7.JNS221186. Print 2023 Apr 1.
Standardized letters of recommendation (SLORs) were introduced during the 2020-2021 neurosurgery residency application cycle, but suffered from poor interrater reliability and grade inflation. Changes were made to the SLOR template and utilization patterns in response to these shortcomings. The authors examined the second year of SLOR utilization. They hypothesized that grade inflation and interrater reliability would be improved from the first iteration. They also hypothesized that increased numbers of letters by single writers would correlate with broader rating distributions.
This cross-sectional study analyzed all SLORs submitted to a single neurosurgery residency program over the 2021-2022 cycle. Data from 7 competency domains and the overall rating were recorded and stratified by academic category of the letter writer. Interrater reliability was evaluated using Krippendorff's alpha. The frequency of letters written was evaluated using the Kruskal-Wallis H test.
Ninety percent of SLORs rated applicants among the top 25%, but there was a significant decrease in the usage of the top 1% and top 2%-5% ratings. Interrater reliability was poor across all competencies. Writers who completed 1 SLOR rated applicants higher and had a narrower range than those who completed multiple SLORs.
Changes in the format and subsequent utilization patterns of SLORs have slightly decreased grade inflation; however, interrater reliability remains poor. The most wide-ranging evaluators submitted the highest number of SLORs, suggesting that future evaluation and usage of SLORs should emphasize letter-writer characteristics and numbers of SLORs written. Overall, SLORs have been well and broadly accepted with subtle improvements in the second year of utilization.
标准化推荐信（SLORs）于2020 - 2021年神经外科住院医师申请周期引入，但存在评分者间信度差和分数膨胀的问题。针对这些不足，对SLOR模板和使用模式进行了修改。作者研究了SLOR使用的第二年情况。他们假设与第一次迭代相比，分数膨胀和评分者间信度会得到改善。他们还假设单个推荐人撰写的推荐信数量增加将与更广泛的评分分布相关。
这项横断面研究分析了在2021 - 2022周期提交给一个神经外科住院医师项目的所有SLOR。记录了来自7个能力领域的数据和总体评分，并按推荐人的学术类别进行分层。使用Krippendorff's alpha评估评分者间信度。使用Kruskal - Wallis H检验评估推荐信撰写频率。
90%的SLOR将申请者评为前25%，但前1%和前2% - 5%评分的使用显著减少。所有能力领域的评分者间信度都很差。完成1封SLOR的推荐人对申请者的评分更高，且评分范围比完成多封SLOR的推荐人更窄。
SLOR格式和后续使用模式的改变略微降低了分数膨胀；然而，评分者间信度仍然很差。范围最广的评估者提交的SLOR数量最多，这表明未来对SLOR的评估和使用应强调推荐人的特征和撰写的SLOR数量。总体而言，SLOR在使用的第二年得到了广泛接受且有细微改进。
