Pier Elizabeth L, Raclaw Joshua, Kaatz Anna, Brauer Markus, Carnes Molly, Nathan Mitchell J, Ford Cecilia E
Center for Women's Health Research, University of Wisconsin-Madison, 700 Regent Street, Ste. 301, Madison, WI 53715, USA.
Department of Educational Psychology, University of Wisconsin-Madison, 1025 West Johnson Street, Madison, WI 53706, USA.
Res Eval. 2017 Jan;26(1):1-14. doi: 10.1093/reseval/rvw025. Epub 2017 Feb 14.
In scientific grant peer review, groups of expert scientists meet to engage in the collaborative decision-making task of evaluating and scoring grant applications. Prior research on grant peer review has established that inter-reviewer reliability is typically poor. In the current study, experienced reviewers for the National Institutes of Health (NIH) were recruited to participate in one of four constructed peer review panel meetings. Each panel discussed and scored the same pool of recently reviewed NIH grant applications. We examined the degree of intra-panel variability in panels' scores of the applications before versus after collaborative discussion, and the degree of inter-panel variability. We also analyzed videotapes of reviewers' interactions for instances of one particular form of discourse--as one factor influencing the variability we observe. Results suggest that although reviewers within a single panel agree more following collaborative discussion, different panels agree less after discussion, and Score Calibration Talk plays a pivotal role in scoring variability during peer review. We discuss implications of this variability for the scientific peer review process.
在科研基金同行评审中，专家科学家小组会齐聚一堂，共同完成评估科研基金申请并打分的协作决策任务。先前关于科研基金同行评审的研究表明，评审员之间的可靠性通常较差。在本研究中，招募了美国国立卫生研究院（NIH）的经验丰富的评审员，让他们参与四个构建好的同行评审小组会议之一。每个小组都对同一组近期评审过的NIH科研基金申请进行了讨论和打分。我们考察了小组在协作讨论前后对申请打分的小组内变异性程度，以及小组间变异性程度。我们还分析了评审员互动的录像，以寻找一种特定话语形式的实例——作为影响我们所观察到的变异性的一个因素。结果表明，尽管单个小组内的评审员在协作讨论后意见更趋一致，但不同小组在讨论后意见却更不一致，并且“分数校准谈话”在同行评审期间的打分变异性中起着关键作用。我们讨论了这种变异性对科学同行评审过程的影响。
