Mayo Nancy E, Brophy James, Goldberg Mark S, Klein Marina B, Miller Sydney, Platt Robert W, Ritchie Judith
Division of Clinical Epidemiology R4.29, McGill University Health Center, RVH Site, 687 Pine Avenue West, Montreal, H3A 1A1, Canada.
J Clin Epidemiol. 2006 Aug;59(8):842-8. doi: 10.1016/j.jclinepi.2005.12.007. Epub 2006 Mar 27.
There is a persistent degree of uncertainty and dissatisfaction with the peer review process underlining the need to validate the current grant awarding procedures. This study compared the CLassic Structured Scientific In-depth two reviewer critique (CLASSIC) with an all panel members' independent ranking method (RANKING). Eleven reviewers, reviewed 32 applications for a pilot project competition at a major university medical center.
The degree of agreement between the two methods was poor (kappa = 0.36). The top rated project in each stream would have failed the funding cutoff with a frequency of 9 and 35%, depending on which pair of reviewers had been selected. Four of the top 10 projects identified by RANKING had a greater than 50% of not being funded by the CLASSIC ranking. Ten reviewers provided optimal consistency for the RANKING method.
This study found that there is a considerable amount of chance associated with funding decisions under the traditional method of assigning the grant to two main reviewers. We recommend using the all reviewer ranking procedure to arrive at decisions about grant applications as this removes the impact of extreme reviews.
同行评审过程中一直存在一定程度的不确定性和不满，这凸显了验证当前资助授予程序的必要性。本研究将经典结构化科学深度双评审批评法（CLASSIC）与所有评审小组成员的独立排名法（RANKING）进行了比较。11名评审人员对一所主要大学医学中心的一个试点项目竞赛的32份申请进行了评审。
两种方法之间的一致程度较差（kappa = 0.36）。每个类别中排名最高的项目，根据所选的评审人员对不同，未通过资金截止标准的频率分别为9%和35%。RANKING法确定的前10个项目中有4个在CLASSIC排名中获得资助的可能性小于50%。10名评审人员对RANKING法表现出了最佳的一致性。
本研究发现，在将资助分配给两名主要评审人员的传统方法下，资助决策存在相当大的偶然性。我们建议采用所有评审人员的排名程序来做出关于资助申请的决策，因为这消除了极端评审的影响。
