Coveney John, Herbert Danielle L, Hill Kathy, Mow Karen E, Graves Nicholas, Barnett Adrian
1College of Nursing and Health Sciences, Flinders University, Adelaide, Australia.
2School of Public Health, Social Work & Institute of Health and Biomedical Innovation, Queensland University of Technology, Brisbane, Australia.
Res Integr Peer Rev. 2017 Dec 4;2:19. doi: 10.1186/s41073-017-0043-x. eCollection 2017.
In Australia, the peer review process for competitive funding is usually conducted by a peer review group in conjunction with prior assessment from external assessors. This process is quite mysterious to those outside it. The purpose of this research was to throw light on grant review panels (sometimes called the 'black box') through an examination of the impact of panel procedures, panel composition and panel dynamics on the decision-making in the grant review process. A further purpose was to compare experience of a simplified review process with more conventional processes used in assessing grant proposals in Australia.
This project was one aspect of a larger study into the costs and benefits of a simplified peer review process. The Queensland University of Technology (QUT)-simplified process was compared with the National Health and Medical Research Council's (NHMRC) more complex process. Grant review panellists involved in both processes were interviewed about their experience of the decision-making process that assesses the excellence of an application. All interviews were recorded and transcribed. Each transcription was de-identified and returned to the respondent for review. Final transcripts were read repeatedly and coded, and similar codes were amalgamated into categories that were used to build themes. Final themes were shared with the research team for feedback.
Two major themes arose from the research: (1) assessing grant proposals and (2) factors influencing the fairness, integrity and objectivity of review. Issues such as the quality of writing in a grant proposal, comparison of the two review methods, the purpose and use of the rebuttal, assessing the financial value of funded projects, the importance of the experience of the panel membership and the role of track record and the impact of group dynamics on the review process were all discussed. The research also examined the influence of research culture on decision-making in grant review panels. One of the aims of this study was to compare a simplified review process with more conventional processes. Generally, participants were supportive of the simplified process.
Transparency in the grant review process will result in better appreciation of the outcome. Despite the provision of clear guidelines for peer review, reviewing processes are likely to be subjective to the extent that different reviewers apply different rules. The peer review process will come under more scrutiny as funding for research becomes even more competitive. There is justification for further research on the process, especially of a kind that taps more deeply into the 'black box' of peer review.
在澳大利亚，竞争性科研经费的同行评审过程通常由同行评审小组结合外部评审员的预先评估来进行。这个过程对于外部人员来说相当神秘。本研究的目的是通过考察评审小组程序、评审小组构成和评审小组动态对科研经费评审过程中决策的影响，来揭示资助评审小组（有时被称为“黑匣子”）的情况。另一个目的是将简化评审过程的经验与澳大利亚评估科研经费申请时使用的更传统的过程进行比较。
本项目是一项关于简化同行评审过程的成本与效益的更大规模研究的一个方面。将昆士兰科技大学（QUT）的简化过程与澳大利亚国家卫生与医学研究委员会（NHMRC）更复杂的过程进行了比较。参与这两个过程的科研经费评审小组成员就他们对评估申请卓越性的决策过程的经验接受了访谈。所有访谈都进行了录音和转录。每份转录稿都经过匿名处理后返还给受访者进行审核。最终的转录稿被反复阅读并编码，相似的代码被合并为类别，用于构建主题。最终主题与研究团队分享以获取反馈。
研究产生了两个主要主题：（1）评估科研经费申请，以及（2）影响评审公正性、完整性和客观性的因素。讨论了诸如科研经费申请中的写作质量、两种评审方法的比较、反驳的目的和用途、评估获批项目的财务价值、评审小组成员经验的重要性以及过往记录的作用，以及群体动态对评审过程的影响等问题。该研究还考察了研究文化对科研经费评审小组决策的影响。本研究的目标之一是将简化评审过程与更传统的过程进行比较。总体而言，参与者对简化过程表示支持。
科研经费评审过程的透明度将使人们对评审结果有更好的理解。尽管为同行评审提供了明确的指导方针，但评审过程在不同评审员应用不同规则的程度上可能仍然是主观的。随着科研经费竞争愈发激烈，同行评审过程将受到更多审查。有理由对该过程进行进一步研究，尤其是那种更深入探究同行评审“黑匣子”的研究。
