A comparative analysis of the ratings in performance assessment using generalizability theory and the many-facet Rasch model.

The purpose of this study is to compare two different methods for modeling rater effects in performance assessment: Generalizability (G) Theory and the Many-facet Rasch Model (MFRM). The view that G theory and the MFRM are alternative solutions to the same measurement problem, in particular, rater effects, is seen to be only partially true. G theory provides a general summary including an estimation of the relative influence of each facet on a measure and the reliability of a decision based on the data. MFRM concentrates on the individual examinee or rater and provides as fair a measure as it is possible to derive from the data as well as summary information such as reliability indices and ways to express the relative influence of the facets. These conclusions are illustrated using data for ratings of student writing assessments.