Wang Jue, Engelhard George
University of Miami, Coral Gables, FL, USA.
University of Georgia, Athens, GA, USA.
Educ Psychol Meas. 2019 Aug;79(4):773-795. doi: 10.1177/0013164419827345. Epub 2019 Feb 5.
The purpose of this study is to explore the use of unfolding models for evaluating the quality of ratings obtained in rater-mediated assessments. Two different judgmental processes can be used to conceptualize ratings: impersonal judgments and personal preferences. Impersonal judgments are typically expected in rater-mediated assessments, and these ratings reflect a cumulative response process. However, raters may also be influenced by their personal preferences in providing ratings, and these ratings may reflect a noncumulative or unfolding response process. The goal of rater training in rater-mediated assessments is to stress impersonal judgments represented by scoring rubrics and to minimize the personal preferences that may represent construct-irrelevant variance in the assessment system. In this study, we explore the use of unfolding models as a framework for evaluating the quality of ratings in rater-mediated assessments. Data from a large-scale assessment of writing in the United States are used to illustrate our approach. The results suggest that unfolding models offer a useful way to evaluate rater-mediated assessments in order to initially explore the judgmental processes underlying the ratings. The data also indicate that there are significant relationships between some essay features (e.g., , and ) and essay orderings based on the personal preferences of raters. The implications of unfolding models for theory and practice in rater-mediated assessments are discussed.
本研究的目的是探索运用展开模型来评估评分者介导评估中所获评分的质量。可以使用两种不同的判断过程来概念化评分：客观判断和个人偏好。在评分者介导评估中通常期望的是客观判断，这些评分反映了一个累积的反应过程。然而，评分者在提供评分时也可能受到个人偏好的影响，这些评分可能反映了一个非累积或展开的反应过程。评分者介导评估中评分者培训的目标是强调由评分标准所代表的客观判断，并尽量减少可能在评估系统中代表与结构无关变异的个人偏好。在本研究中，我们探索运用展开模型作为评估评分者介导评估中评分质量的一个框架。来自美国一项大规模写作评估的数据被用于说明我们的方法。结果表明，展开模型为评估评分者介导评估提供了一种有用的方式，以便初步探索评分背后的判断过程。数据还表明，基于评分者的个人偏好，一些作文特征（例如， ，以及 ）与作文排序之间存在显著关系。讨论了展开模型对评分者介导评估中的理论和实践的启示。
