Wind Stefanie A, Engelhard George
The University of Alabama, Tuscaloosa, AL, USA.
The University of Georgia, Athens, GA, USA.
Educ Psychol Meas. 2016 Aug;76(4):685-706. doi: 10.1177/0013164415604704. Epub 2015 Sep 17.
Mokken scale analysis is a probabilistic nonparametric approach that offers statistical and graphical tools for evaluating the quality of social science measurement without placing potentially inappropriate restrictions on the structure of a data set. In particular, Mokken scaling provides a useful method for evaluating important measurement properties, such as invariance, in contexts where response processes are not well understood. Because rater-mediated assessments involve complex interactions among many variables, including assessment contexts, student artifacts, rubrics, individual rater characteristics, and others, rater-assigned scores are suitable candidates for Mokken scale analysis. The purposes of this study are to describe a suite of indices that can be used to explore the psychometric quality of data from rater-mediated assessments and to illustrate the substantive interpretation of Mokken-based statistics and displays in this context. Techniques that are commonly used in polytomous applications of Mokken scaling are adapted for use with rater-mediated assessments, with a focus on the substantive interpretation related to individual raters. Overall, the findings suggest that indices of rater monotonicity, rater scalability, and invariant rater ordering based on Mokken scaling provide diagnostic information at the level of individual raters related to the requirements for invariant measurement. These Mokken-based indices serve as an additional suite of diagnostic tools for exploring the quality of data from rater-mediated assessments that can supplement rating quality indices based on parametric models.
莫肯量表分析是一种概率非参数方法，它提供了统计和图形工具，用于评估社会科学测量的质量，而无需对数据集的结构施加可能不适当的限制。特别是，莫肯量表法提供了一种有用的方法，用于在响应过程不太清楚的情况下评估重要的测量属性，如不变性。由于评分者介导的评估涉及许多变量之间的复杂相互作用，包括评估背景、学生作品、评分标准、个体评分者特征等，评分者给出的分数是莫肯量表分析的合适候选对象。本研究的目的是描述一套可用于探索评分者介导评估数据的心理测量质量的指标，并说明在此背景下基于莫肯量表的统计数据和显示的实质性解释。莫肯量表法在多分类应用中常用的技术被改编用于评分者介导的评估，重点是与个体评分者相关的实质性解释。总体而言，研究结果表明，基于莫肯量表的评分者单调性、评分者可扩展性和不变评分者排序指标，在个体评分者层面提供了与不变测量要求相关的诊断信息。这些基于莫肯量表的指标可作为另一套诊断工具，用于探索评分者介导评估数据的质量，以补充基于参数模型的评分质量指标。
