Wind Stefanie A, Schumacker Randall E
The University of Alabama, Tuscaloosa, AL, USA.
Educ Psychol Meas. 2018 Oct;78(5):887-904. doi: 10.1177/0013164417724841. Epub 2017 Aug 4.
The interpretation of ratings from educational performance assessments assumes that rating scale categories are ordered as expected (i.e., higher ratings correspond to higher levels of judged student achievement). However, this assumption must be verified empirically using measurement models that do not impose ordering constraints on the rating scale category thresholds, such as item response theory models based on adjacent-categories probabilities. This study considers the application of an adjacent-categories formulation of polytomous Mokken scale analysis (ac-MSA) models as a method for evaluating the degree to which rating scale categories are ordered as expected for individual raters in performance assessments. Using simulated data, this study builds on the preliminary application of ac-MSA models to rater-mediated performance assessments, in which a real data analysis suggested that these models can be used to identify disordered rating scale categories. The results suggested that ac-MSA models are sensitive to disordered categories within individual raters. Implications are discussed as they relate to research, theory, and practice for rater-mediated educational performance assessments.
教育绩效评估中评分的解读假定评分量表类别按预期顺序排列（即，较高的评分对应较高水平的学生成绩评判）。然而，这一假设必须使用不对评分量表类别阈值施加排序约束的测量模型进行实证验证，例如基于相邻类别概率的项目反应理论模型。本研究考虑将多分类莫肯量表分析（ac-MSA）模型的相邻类别公式作为一种方法，用于评估在绩效评估中评分量表类别对于个体评分者按预期顺序排列的程度。利用模拟数据，本研究在ac-MSA模型对评分者介导的绩效评估的初步应用基础上展开，其中一项实际数据分析表明这些模型可用于识别无序的评分量表类别。结果表明，ac-MSA模型对个体评分者内的无序类别敏感。讨论了其与评分者介导的教育绩效评估的研究、理论和实践相关的意义。
