Faculty of Education, The University of Hong Kong, Hong Kong, China.
Department of Special Education and Counselling, The Education University of Hong Kong, Hong Kong, China.
Behav Res Methods. 2022 Aug;54(4):1854-1868. doi: 10.3758/s13428-021-01699-y. Epub 2021 Nov 1.
Rater centrality, in which raters overuse middle scores for rating, is a common rater error which can affect test scores and subsequent decisions. Past studies on rater errors have focused on rater severity and inconsistency, neglecting rater centrality. This study proposes a new model within the hierarchical rater model framework to explicitly specify and directly estimate rater centrality in addition to rater severity and inconsistency. Simulations were conducted using the freeware JAGS to evaluate the parameter recovery of the new model and the consequences of ignoring rater centrality. The results revealed that the model had good parameter recovery with small bias, low root mean square errors, and high test score reliability, especially when a fully crossed linking design was used. Ignoring centrality yielded poor item difficulty estimates, person ability estimates, rater errors estimates, and underestimated reliability. We also showcase how the new model can be used, using an empirical example involving English essays in the Advanced Placement exam.
评分者中心化，即评分者过度使用中间分数进行评分，是一种常见的评分者误差，会影响考试分数和后续决策。过去关于评分者误差的研究主要集中在评分者严厉性和不一致性上，而忽略了评分者中心化。本研究在层级评分者模型框架内提出了一个新模型，除了评分者严厉性和不一致性之外，还可以明确指定和直接估计评分者中心化。使用免费软件 JAGS 进行模拟，以评估新模型的参数恢复情况以及忽略评分者中心化的后果。结果表明，该模型具有良好的参数恢复能力，偏差较小，均方根误差较低，测试分数可靠性较高，特别是在使用完全交叉链接设计时。忽略中心化会导致项目难度估计、个体能力估计、评分者误差估计不准确，并低估可靠性。我们还展示了如何使用新模型，使用涉及高级安置考试中英语作文的实证示例。
