The University of Electro-Communications, Tokyo, Japan.
Behav Res Methods. 2023 Oct;55(7):3910-3928. doi: 10.3758/s13428-022-01997-z. Epub 2022 Oct 25.
Fair performance assessment requires consideration of the effects of rater severity on scoring. The many-facet Rasch model (MFRM), an item response theory model that incorporates rater severity parameters, has been widely used for this purpose. Although a typical MFRM assumes that rater severity does not change during the rating process, in actuality rater severity is known to change over time, a phenomenon called rater severity drift. To investigate this drift, several extensions of the MFRM have been proposed that incorporate time-specific rater severity parameters. However, these previous models estimate the severity parameters under the assumption of temporal independence. This introduces inefficiency into the parameter estimation because severities between adjacent time points tend to have temporal dependency in practice. To resolve this problem, we propose a Bayesian extension of the MFRM that incorporates time dependency for the rater severity parameters, based on a Markov modeling approach. The proposed model can improve the estimation accuracy of the time-specific rater severity parameters, resulting in improved estimation accuracy for the other rater parameters and for model fitting. We demonstrate the effectiveness of the proposed model through simulation experiments and application to actual data.
公平的绩效评估需要考虑评分者严厉程度对评分的影响。多方面 Rasch 模型（MFRM）是一种包含评分者严厉程度参数的项目反应理论模型，已被广泛用于此目的。尽管典型的 MFRM 假设评分者严厉程度在评分过程中不会改变，但实际上评分者严厉程度随着时间的推移而变化，这种现象称为评分者严厉程度漂移。为了研究这种漂移，已经提出了 MFRM 的几个扩展版本，这些扩展版本包含了特定于时间的评分者严厉程度参数。然而，这些先前的模型在时间独立性的假设下估计严厉程度参数。这会导致参数估计效率低下，因为实际上相邻时间点之间的严厉程度在实践中往往具有时间依赖性。为了解决这个问题，我们提出了一种基于马尔可夫建模方法的 MFRM 的贝叶斯扩展，该扩展为评分者严厉程度参数纳入了时间依赖性。所提出的模型可以提高特定于时间的评分者严厉程度参数的估计准确性，从而提高其他评分者参数和模型拟合的估计准确性。我们通过模拟实验和对实际数据的应用证明了所提出模型的有效性。
