Gawad Nada, Younan Julia, Towaij Chelsea, Raiche Isabelle
is a Resident, Department of Surgery.
is Assistant Professor of Surgery, and Ottawa General Surgery CaRMS Director, Department of Surgery.
J Grad Med Educ. 2021 Apr;13(2):240-245. doi: 10.4300/JGME-D-20-00619.1. Epub 2021 Feb 17.
The residency selection process relies on subjective information in applications, as well as subjective assessment of applications by reviewers. This inherent subjectivity makes residency selection prone to poor reliability between those reviewing files.
We compared the interrater reliability of 2 assessment tools during file review: one rating applicant traits (ie, leadership, communication) and the other using a global rating of application elements (ie, curriculum vitae, reference letters).
Ten file reviewers were randomized into 2 groups, and each scored 7 general surgery applications from the 2019-2020 cycle. The first group used an element-based (EB) scoring tool, while the second group used a trait-based (TB) scoring tool. Feedback was collected, discrimination capacities were measured using variation in scores, and interrater reliability (IRR) was calculated using intraclass correlation (ICC) in a 2-way random effects model.
Both tools identified the same top-ranked and bottom-ranked applicants; however, discrepancies were noted for middle-ranked applicants. The score range for the 5 middle-ranked applicants was greater with the TB tool (6.43 vs 3.80), which also demonstrated fewer tie scores. The IRR for TB scoring was superior to EB scoring (ICC [2, 5] = 0.82 vs 0.55). The TB tool required only 2 raters to achieve an ICC ≥ 0.70.
Using a TB file review strategy can facilitate file review with improved reliability compared to EB, and a greater spread of candidate scores. TB file review potentially offers programs a feasible way to optimize and reflect their institution's core values in the process.
住院医师选拔过程依赖于申请材料中的主观信息，以及评审人员对申请材料的主观评估。这种固有的主观性使得住院医师选拔在评审人员之间的可靠性较差。
我们比较了两种评估工具在材料评审过程中的评分者间信度：一种是对申请人特质（即领导力、沟通能力）进行评分，另一种是对申请材料要素（即简历、推荐信）进行整体评分。
10名材料评审人员被随机分为两组，每组对2019 - 2020年度周期的7份普通外科申请材料进行评分。第一组使用基于要素的（EB）评分工具，而第二组使用基于特质的（TB）评分工具。收集反馈意见，通过分数差异测量区分能力，并在双向随机效应模型中使用组内相关系数（ICC）计算评分者间信度（IRR）。
两种工具都识别出了排名最高和最低的申请人；然而，对于排名中等的申请人存在差异。使用TB工具时，5名排名中等的申请人的分数范围更大（6.43对3.80），且平局分数更少。TB评分的IRR优于EB评分（ICC [2, 5] = 0.82对0.55）。TB工具仅需2名评分者就能达到ICC≥0.70。
与EB相比，使用TB材料评审策略可以提高评审的可靠性，并使候选人分数分布更广。TB材料评审可能为各项目提供一种可行的方法，以便在选拔过程中优化并体现其机构的核心价值观。
