Yeates Peter, Moreau Marc, Eva Kevin
P. Yeates is clinical lecturer in medical education, Centre for Respiratory Medicine and Allergy, Institute of Inflammation and Repair, University of Manchester, and specialist registrar, Respiratory and General Internal Medicine, Health Education North West, Manchester, United Kingdom. M. Moreau is assistant dean for admissions, Faculty of Medicine and Dentistry, and professor, Division of Orthopaedic Surgery, University of Alberta, Edmonton, Alberta, Canada. K. Eva is senior scientist, Centre for Health Education Scholarship, and professor and director of educational research and scholarship, Department of Medicine, University of British Columbia, Vancouver, British Columbia, Canada.
Acad Med. 2015 Jul;90(7):975-80. doi: 10.1097/ACM.0000000000000650.
Laboratory studies have shown that performance assessment judgments can be biased by "contrast effects." Assessors' scores become more positive, for example, when the assessed performance is preceded by relatively weak candidates. The authors queried whether this effect occurs in real, high-stakes performance assessments despite increased formality and behavioral descriptors.
Data were obtained for the 2011 United Kingdom Foundational Programme clinical assessment and the 2008 University of Alberta Multiple Mini Interview. Candidate scores were compared with scores for immediately preceding candidates and progressively distant candidates. In addition, average scores for the preceding three candidates were calculated. Relationships between these variables were examined using linear regression.
Negative relationships were observed between index scores and both immediately preceding and recent scores for all exam formats. Relationships were greater between index scores and the average of the three preceding scores. These effects persisted even when examiners had judged several performances, explaining up to 11% of observed variance on some occasions.
These findings suggest that contrast effects do influence examiner judgments in high-stakes performance-based assessments. Although the observed effect was smaller than observed in experimentally controlled laboratory studies, this is to be expected given that real-world data lessen the strength of the intervention by virtue of less distinct differences between candidates. Although it is possible that the format of circuital exams reduces examiners' susceptibility to these influences, the finding of a persistent effect after examiners had judged several candidates suggests that the potential influence on candidate scores should not be ignored.
实验室研究表明，绩效评估判断可能会受到“对比效应”的影响。例如，当被评估的绩效之前是相对较弱的候选人时，评估者的分数会变得更加积极。作者质疑这种效应在实际的、高风险的绩效评估中是否会出现，尽管评估形式更加正式且有行为描述。
获取了2011年英国基础项目临床评估和2008年阿尔伯塔大学多重迷你面试的数据。将候选人的分数与紧接其前的候选人以及逐渐疏远的候选人的分数进行比较。此外，计算了前三位候选人的平均分数。使用线性回归检查这些变量之间的关系。
在所有考试形式中，指标分数与紧接其前的分数以及最近的分数之间均观察到负相关关系。指标分数与前三位分数的平均值之间的关系更大。即使考官已经评判了多个绩效，这些效应仍然存在，在某些情况下可解释高达11%的观察方差。
这些发现表明，对比效应确实会影响高风险的基于绩效的评估中考官的判断。尽管观察到的效应小于在实验控制的实验室研究中观察到的效应，但考虑到现实世界的数据由于候选人之间的差异不那么明显而削弱了干预的强度，这是可以预期的。虽然巡回考试的形式可能会降低考官对这些影响的敏感度，但考官评判了多个候选人后仍存在持续效应的发现表明，对候选人分数的潜在影响不应被忽视。
