Department of Medicine, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada.
Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands.
Med Educ. 2017 Apr;51(4):401-410. doi: 10.1111/medu.13158. Epub 2017 Jan 16.
Interest is growing in the use of qualitative data for assessment. Written comments on residents' in-training evaluation reports (ITERs) can be reliably rank-ordered by faculty attendings, who are adept at interpreting these narratives. However, if residents do not interpret assessment comments in the same way, a valuable educational opportunity may be lost.
Our purpose was to explore residents' interpretations of written assessment comments using mixed methods.
Twelve internal medicine (IM) postgraduate year 2 (PGY2) residents were asked to rank-order a set of anonymised PGY1 residents (n = 48) from a previous year in IM based solely on their ITER comments. Each PGY1 was ranked by four PGY2s; generalisability theory was used to assess inter-rater reliability. The PGY2s were then interviewed separately about their rank-ordering process, how they made sense of the comments and how they viewed ITERs in general. Interviews were analysed using constructivist grounded theory.
Across four PGY2 residents, the G coefficient was 0.84; for a single resident it was 0.56. Resident rankings correlated extremely well with faculty member rankings (r = 0.90). Residents were equally adept at reading between the lines to construct meaning from the comments and used language cues in ways similarly reported in faculty attendings. Participants discussed the difficulties of interpreting vague language and provided perspectives on why they thought it occurs (time, discomfort, memorability and the permanency of written records). They emphasised the importance of face-to-face discussions, the relative value of comments over scores, staff-dependent variability of assessment and the perceived purpose and value of ITERs. They saw particular value in opportunities to review an aggregated set of comments.
Residents understood the 'hidden code' in assessment language and their ability to rank-order residents based on comments matched that of faculty. Residents seemed to accept staff-dependent variability as a reality. These findings add to the growing evidence that supports the use of narrative comments and subjectivity in assessment.
人们对使用定性数据进行评估的兴趣日益浓厚。住院医师培训评估报告（ITER）中的书面评语可以由带教教员进行可靠地排序，他们擅长解释这些叙述。然而，如果住院医师不以相同的方式解释评估意见，那么可能会失去宝贵的教育机会。
我们的目的是使用混合方法探讨住院医师对书面评估意见的解释。
我们要求 12 名内科住院医师培训 2 年级（PGY2）住院医师根据前一年的 ITER 评论，仅根据其 ITER 评论对一组匿名 PGY1 住院医师（n = 48）进行排序。每位 PGY1 由四位 PGY2 进行排序；使用概化理论评估评分者间可靠性。然后，PGY2 分别接受有关其排序过程、如何理解评论以及他们对 ITER 的总体看法的单独访谈。使用建构主义扎根理论对访谈进行分析。
在四名 PGY2 住院医师中，G 系数为 0.84；对于单个住院医师，其系数为 0.56。住院医师的排名与教员的排名非常相关（r = 0.90）。住院医师同样擅长从评论中阅读并构建意义，并以与教员相似的方式使用语言线索。参与者讨论了解释模糊语言的困难，并提供了他们认为出现这种情况的原因的观点（时间、不适、可记性和书面记录的永久性）。他们强调面对面讨论的重要性、评论相对于分数的相对价值、评估的人员依赖性可变性以及 ITER 的感知目的和价值。他们认为有机会查看一组汇总的评论具有特殊价值。
住院医师理解评估语言中的“隐藏代码”，他们根据评论对住院医师进行排序的能力与教员的能力相匹配。住院医师似乎接受了人员依赖性可变性是现实。这些发现增加了越来越多的支持使用叙述性评论和主观性进行评估的证据。
