When raters generalize: Examining sources of halo effects with mixture Rasch facets models.

Halo effects are commonly considered a cognitive or judgmental bias leading to rating error when raters assign scores to persons or performances on multiple criteria. Though a long tradition of research has pointed to possible sources of halo effects, measurement models for identifying these sources and detecting halo have been lacking. In the present research, we propose a general mixture Rasch facets model for halo effects (MRFM-H) and derive two more specific models, each assuming a different psychological mechanism. According to the first model, MRFM-H(GI), persons evoke general impressions that guide raters when assigning scores on conceptually distinct criteria. The second model, MRFM-H(ID), assumes that raters fail to discriminate adequately between the criteria. We adopted a Bayesian inference approach to implement these models, conducting two simulation studies and a real-data analysis. In the simulation studies, we found that (a) the number of raters and criteria determined the accuracy of classifying persons as inducing or not inducing halo; (b) 90% classification accuracy was achieved when at least 25 ratings were available for each rater-person combination; (c) ignoring halo caused by either mechanism (general impressions or inadequate criterion discrimination) biased the criterion parameter estimates while having a negligible impact on person and rater estimates; (d) Bayesian data-model fit statistics (WAIC and WBIC) reliably identified the true, data-generating model. The real-data analysis highlighted the models' practical utility for examining the likely source of halo effects. The discussion focuses on the models' application in various assessment contexts and points to directions for future research.