Schmaling Karen B, Gallo Stephen A
Department of Psychology, Washington State University, Vancouver, Washington, United States of America.
Scientific Peer Advisory and Review Services, American Institute of Biological Sciences, Herndon, Virginia, United States of America.
PLoS One. 2024 Dec 17;19(12):e0315567. doi: 10.1371/journal.pone.0315567. eCollection 2024.
Peer review is a decisive factor in selecting research grant proposals for funding. The usefulness of peer review depends in part on the agreement of multiple reviewers' judgments of the same proposal, and on each reviewer's consistency in judging proposals. Peer reviewers are also instructed to disregard characteristics that are not among the evaluation criteria. However, for example, the gender identity-of the investigator or reviewer-may be associated with differing evaluations. This experiment sought to characterize the psychometric properties of peer review among 605 experienced peer reviewers and to examine possible differences in peer review judgments based on peer reviewer and investigator gender. Participants evaluated National Institutes of Health-style primary reviewers' overall impact statements that summarized the study's purpose, its overall evaluation, and its strengths and weaknesses in five criterion areas: significance, approach, investigator, innovation, and environment. Evaluations were generally consistent between reviewers and within reviewers over a two-week period. However, there was less consistency in judging proposals with weaknesses. Regarding gender differences, women reviewers tended to provide more positive evaluations, and women investigators received better overall evaluations. Unsuccessful grant applicants use reviewer feedback to improve their proposals, which could be made more challenging with inconsistent reviews. Peer reviewer training and calibration could increase reviewer consistency, which is especially relevant for proposals with weaknesses according to this study's results. Evidence of systematic differences in proposal scores based on investigator and reviewer gender may also indicate the usefulness of calibration and training. For example, peer reviewers could score practice proposals and discuss differences prior to independently scoring assigned proposals.
同行评审是决定研究资助提案是否获得资金的关键因素。同行评审的有效性部分取决于多位评审员对同一提案判断的一致性，以及每位评审员在评判提案时的连贯性。同行评审员也被要求忽略那些不在评估标准之列的特征。然而，例如，研究者或评审员的性别认同可能与不同的评估结果相关。本实验旨在描述605名经验丰富的同行评审员的同行评审心理测量特性，并研究基于同行评审员和研究者性别的同行评审判断中可能存在的差异。参与者对美国国立卫生研究院风格的主要评审员的总体影响陈述进行评估，这些陈述总结了研究目的、总体评估以及在五个标准领域的优势和不足：重要性、方法、研究者、创新性和环境。在两周时间内，评审员之间以及评审员自身的评估总体上是一致的。然而，在评判存在不足的提案时，一致性较低。关于性别差异，女性评审员倾向于给出更积极的评价，女性研究者获得的总体评价更好。未成功获得资助的申请者利用评审员的反馈来改进他们的提案，而评审意见不一致可能会使改进变得更具挑战性。同行评审员培训和校准可以提高评审员的一致性，根据本研究结果，这对于存在不足的提案尤为重要。基于研究者和评审员性别的提案分数存在系统差异的证据，也可能表明校准和培训的有效性。例如，同行评审员可以对练习提案进行评分，并在独立评分分配的提案之前讨论差异。
