Scientific Peer Advisory and Review Services Division, American Institute of Biological Sciences, Herndon, Virginia, United States of America.
Department of Psychology, Washington State University, Vancouver, Washington, United States of America.
PLoS One. 2022 Aug 26;17(8):e0273813. doi: 10.1371/journal.pone.0273813. eCollection 2022.
Peer review, commonly used in grant funding decisions, relies on scientists' ability to evaluate research proposals' quality. Such judgments are sometimes beyond reviewers' discriminatory power and could lead to a reliance on subjective biases, including preferences for lower risk, incremental projects. However, peer reviewers' risk tolerance has not been well studied. We conducted a cross-sectional experiment of peer reviewers' evaluations of mock primary reviewers' comments in which the level and sources of risks and weaknesses were manipulated. Here we show that proposal risks more strongly predicted reviewers' scores than proposal strengths based on mock proposal evaluations. Risk tolerance was not predictive of scores but reviewer scoring leniency was predictive of overall and criteria scores. The evaluation of risks dominates reviewers' evaluation of research proposals and is a source of inter-reviewer variability. These results suggest that reviewer scoring variability may be attributed to the interpretation of proposal risks, and could benefit from intervention to improve the reliability of reviews. Additionally, the valuation of risk drives proposal evaluations and may reduce the chances that risky, but highly impactful science, is supported.
同行评议常用于拨款资助决策，依赖于科学家评估研究提案质量的能力。这种判断有时超出了评审员的辨别能力，可能导致依赖于主观偏见，包括对低风险、渐进式项目的偏好。然而，同行评审员的风险承受能力尚未得到很好的研究。我们进行了一项同行评审员对模拟主要评审员评论的评估的横断面实验，其中操纵了风险和弱点的水平和来源。在这里，我们表明，基于模拟提案评估，提案风险比提案优势更能预测评审员的分数。风险承受能力不能预测分数，但评审员评分宽松度可预测总体和标准分数。风险评估主导着评审员对研究提案的评估，是评审员间变异性的一个来源。这些结果表明，评审员评分的可变性可能归因于对提案风险的解释，并且可以通过干预来提高审查的可靠性从中受益。此外，风险评估驱动提案评估，并可能降低支持高风险但极具影响力的科学的机会。
