Baylor College of Medicine, Houston, Texas; SurgWise Consulting, Houston, Texas.
SurgWise Consulting, Houston, Texas.
J Surg Educ. 2022 Nov-Dec;79(6):e12-e16. doi: 10.1016/j.jsurg.2022.05.013. Epub 2022 Jul 6.
We explored the impact of implementing structured interviews and associated interviewer education on interrater agreement within a large academic residency program.
Faculty and senior resident interviewers from a large academic residency program participated in a 3-hour structured interview course. Before and after the course, participants completed a 15-item assessment pertaining to the characteristics, logistics, and guidelines associated with structured interviews. Along with interviewer training, interview day logistics also changed from an unstructured format (no specific questions, one overall 1-9 rating scale) to a structured interview format, including incorporation of behavioral-based competency questions that would be asked of every applicant and behavioral anchored rating scales (1-10; 10 = highest). Interrater agreement was assessed via intraclass correlation coefficients (ICC1) for the 2 years before and 2 years after incorporation of the structured interview format.
A total of 45 faculty and resident interviewers participated in the course in 2018. Participant knowledge significantly increased from an average of 36% to 79% after the course (p < 0.01). Prior to the intervention, overall interrater agreement was "poor" to "fair," with an ICC1 of 0.51 in 2016 and 0.49 in 2017. After the structured interview intervention, overall agreement increased to the "good" level with an ICC1 of 0.71 in 2018 and 0.66 in 2019. The proportion of applicants who received interview scores with at least 2 ratings more than 2 points apart significantly decreased from 59% to 47% after the intervention (p < 0.01).
Incorporating an interviewer educational session and a structured interview format into residency selection can help increase agreement in ratings between interviewers. However, these data suggest that ongoing refresher trainings may be needed to maintain acceptable levels of interrater agreement.
本研究旨在探讨在大型学术住院医师培训项目中实施结构化面试及相关面试培训对评分者间一致性的影响。
来自大型学术住院医师培训项目的教师和高级住院医师面试考官参加了 3 小时的结构化面试课程。在课程开始前后，参与者完成了一项与结构化面试相关的 15 项评估，包括面试的特点、流程和准则。除了面试考官培训外，面试日流程也从非结构化格式（没有特定问题，只有一个整体 1-9 评分量表）转变为结构化面试格式，包括纳入将对每位申请人提出的基于行为的能力问题和行为锚定评分量表（1-10；10=最高）。采用组内相关系数（ICC1）评估结构化面试格式实施前 2 年和实施后 2 年的评分者间一致性。
2018 年共有 45 名教师和住院医师面试考官参加了该课程。课程结束后，参与者的知识水平显著提高，从平均 36%提高到 79%（p < 0.01）。在干预前，整体评分者间一致性为“差”到“中等”，2016 年的 ICC1 为 0.51，2017 年为 0.49。在结构化面试干预后，整体一致性提高到“良好”水平，2018 年的 ICC1 为 0.71，2019 年为 0.66。干预后，评分者间至少相差 2 分的申请人比例从 59%显著下降到 47%（p < 0.01）。
在住院医师选拔中纳入面试考官教育课程和结构化面试格式有助于提高评分者间的评分一致性。然而，这些数据表明，可能需要进行持续的复习培训以维持可接受的评分者间一致性水平。
