Northern Medical Program, University of Northern British Columbia, 3333 University Way, Prince George, BC, V2N 4Z9, Canada.
Keele University School of Medicine, Keele, Staffordshire, UK.
Adv Health Sci Educ Theory Pract. 2018 Dec;23(5):937-959. doi: 10.1007/s10459-018-9841-2. Epub 2018 Jul 6.
Recent literature places more emphasis on assessment comments rather than relying solely on scores. Both are variable, however, emanating from assessment judgements. One established source of variability is "contrast effects": scores are shifted away from the depicted level of competence in a preceding encounter. The shift could arise from an effect on the range-frequency of assessors' internal scales or the salience of performance aspects within assessment judgments. As these suggest different potential interventions, we investigated assessors' cognition by using the insight provided by "clusters of consensus" to determine whether any change in the salience of performance aspects was induced by contrast effects. A dataset from a previous experiment contained scores and comments for 3 encounters: 2 with significant contrast effects and 1 without. Clusters of consensus were identified using F-sort and latent partition analysis both when contrast effects were significant and non-significant. The proportion of assessors making similar comments only significantly differed when contrast effects were significant with assessors more frequently commenting on aspects that were dissimilar with the standard of competence demonstrated in the preceding performance. Rather than simply influencing range-frequency of assessors' scales, preceding performances may affect salience of performance aspects through comparative distinctiveness: when juxtaposed with the context some aspects are more distinct and selectively draw attention. Research is needed to determine whether changes in salience indicate biased or improved assessment information. The potential should be explored to augment existing benchmarking procedures in assessor training by cueing assessors' attention through observation of reference performances immediately prior to assessment.
近期文献更加重视评估意见，而不仅仅依赖于分数。然而，两者都是可变的，源自评估判断。一个已知的可变性来源是“对比效应”：分数会偏离前一次评估中所描绘的能力水平。这种偏移可能是由于评估者内部量表的范围-频率效应或评估判断中绩效方面的突出程度所致。由于这些因素暗示了不同的潜在干预措施，我们通过使用“共识集群”提供的洞察力来研究评估者的认知，以确定对比效应是否会引起绩效方面的突出程度发生变化。之前的实验数据集包含 3 次评估的分数和意见：2 次具有显著的对比效应，1 次没有。当对比效应显著和非显著时，使用 F 排序和潜在分区分析确定了共识集群。只有当对比效应显著时，做出类似意见的评估者的比例才会显著不同，评估者更频繁地评论与前一次表现中展示的能力标准不同的方面。先前的表现可能会通过比较独特性来影响绩效方面的突出程度，而不仅仅是简单地影响评估者量表的范围-频率：当与上下文并列时，某些方面更加突出，会选择性地吸引注意力。需要研究以确定突出程度的变化是否表示评估信息存在偏差或得到改善。应该通过在评估前立即观察参考表现来提示评估者的注意力，从而探索增强评估者培训中现有基准测试程序的潜力。
