Clinical and Health Informatics Research Group, McGill University, Montreal, Canada.
Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montreal, Canada.
PLoS One. 2023 Oct 5;18(10):e0292306. doi: 10.1371/journal.pone.0292306. eCollection 2023.
The allocation of public funds for research has been predominantly based on peer review where reviewers are asked to rate an application on some form of ordinal scale from poor to excellent. Poor reliability and bias of peer review rating has led funding agencies to experiment with different approaches to assess applications. In this study, we compared the reliability and potential sources of bias associated with application rating with those of application ranking in 3,156 applications to the Canadian Institutes of Health Research. Ranking was more reliable than rating and less susceptible to the characteristics of the review panel, such as level of expertise and experience, for both reliability and potential sources of bias. However, both rating and ranking penalized early career investigators and favoured older applicants. Sex bias was only evident for rating and only when the applicant's H-index was at the lower end of the H-index distribution. We conclude that when compared to rating, ranking provides a more reliable assessment of the quality of research applications, is not as influenced by reviewer expertise or experience, and is associated with fewer sources of bias. Research funding agencies should consider adopting ranking methods to improve the quality of funding decisions in health research.
公共研究资金的分配主要基于同行评审，评审员被要求在某种有序尺度上对申请进行评分，从差到优。同行评审评分的可靠性和偏差导致资助机构尝试了不同的方法来评估申请。在这项研究中，我们比较了与应用程序排名相关的可靠性和潜在偏差来源，以及在加拿大卫生研究院的 3156 个申请中的应用程序排名。排名比评分更可靠，并且对于审查小组的特征（如专业知识和经验水平）的可靠性和潜在偏差来源都不太敏感。然而，评分和排名都对初级研究人员不利，并且更倾向于年龄较大的申请人。性别偏差仅在评分时明显，并且仅在申请人的 H 指数处于 H 指数分布的低端时才明显。我们的结论是，与评分相比，排名提供了对研究申请质量的更可靠评估，受评审员专业知识或经验的影响较小，并且偏差来源更少。研究资助机构应考虑采用排名方法来提高健康研究资助决策的质量。
