Pay no attention to the man behind the curtain: The overlooked issue of rater selection in psychotherapy observer ratings.

Psychotherapy process research relies heavily upon trained raters to identify and code therapist and client behaviors. Raters are often selected out of convenience or availability with little research to inform what qualities are desirable for this role. In this study, we evaluated several rater characteristics as potential predictors of raters' performance. We provided training to 39 undergraduate students to serve as raters in a psychotherapy process study. We assessed baseline characteristics, personality characteristics, executive functioning, and interviewers' assessment of students' rating potential as predictors of rater agreement with gold standard ratings (i.e., consensus judgements from a panel of CBT-trained clinicians). We also assessed these variables as predictors of the risk of raters dropping out prior to completing assigned ratings. Higher conscientiousness, higher neuroticism, and a lower year in college predicted greater agreement with the gold standard ratings. Lower year in college also predicted lower risk of dropout. These findings provide empirical support for key characteristics as predictors of rater performance. Such evidence raises the possibility of using these characteristics to select raters and thereby enhance the psychometric properties of psychotherapy ratings.