Zhou You, Sackett Paul R, Shen Winny, Beatty Adam S
Department of Psychology, University of Minnesota, Twin Cities.
Schulich School of Business, York University.
J Appl Psychol. 2024 Jun;109(6):949-970. doi: 10.1037/apl0001174. Epub 2024 Jan 25.
Given the centrality of the job performance construct to organizational researchers, it is critical to understand the reliability of the most common way it is operationalized in the literature. To this end, we conducted an updated meta-analysis on the interrater reliability of supervisory ratings of job performance ( = 132 independent samples) using a new meta-analytic procedure (i.e., the Morris estimator), which includes both within- and between-study variance in the calculation of study weights. An important benefit of this approach is that it prevents large-sample studies from dominating the results. In this investigation, we also examined different factors that may affect interrater reliability, including job complexity, managerial level, rating purpose, performance measure, and rater perspective. We found a higher interrater reliability estimate ( = .65) compared to previous meta-analyses on the topic, and our results converged with an important, but often neglected, finding from a previous meta-analysis by Conway and Huffcutt (1997), such that interrater reliability varies meaningfully by job type ( = .57 for managerial positions vs. = .68 for nonmanagerial positions). Given this finding, we advise against the use of an overall grand mean of interrater reliability. Instead, we recommend using job-specific or local reliabilities for making corrections for attenuation. (PsycInfo Database Record (c) 2024 APA, all rights reserved).
鉴于工作绩效结构对组织研究人员的核心重要性，了解文献中最常用的操作化方法的可靠性至关重要。为此，我们使用一种新的元分析程序（即莫里斯估计器）对工作绩效主管评分的评分者间信度进行了更新的元分析（(N = 132)个独立样本），该程序在计算研究权重时纳入了研究内和研究间的方差。这种方法的一个重要优点是它可以防止大样本研究主导结果。在这项调查中，我们还研究了可能影响评分者间信度的不同因素，包括工作复杂性、管理级别、评分目的、绩效衡量指标和评分者视角。与之前关于该主题的元分析相比，我们发现了更高的评分者间信度估计值（(r = 0.65)），并且我们的结果与康威和赫夫卡特（1997）之前的一项元分析中一个重要但经常被忽视的发现一致，即评分者间信度因工作类型而有显著差异（管理职位的(r = 0.57)，非管理职位的(r = 0.68)）。鉴于这一发现，我们建议不要使用评分者间信度的总体均值。相反，我们建议使用特定工作或局部信度来进行衰减校正。（PsycInfo数据库记录（c）2024美国心理学会，保留所有权利）
