Just One Giant Lab, Paris, France.
University of Warwick, Coventry, UK.
F1000Res. 2023 Apr 18;11:1440. doi: 10.12688/f1000research.125886.2. eCollection 2022.
Resource allocation is essential to selection and implementation of innovative projects in science and technology. Current "winner-take-all" models for grant applications require significant researcher time in writing extensive project proposals, and rely on the availability of a few time-saturated volunteer experts. Such processes usually carry over several months, resulting in high effective costs compared to expected benefits. We devised an agile "community review" system to allocate micro-grants for the fast prototyping of innovative solutions. Here we describe and evaluate the implementation of this community review across 147 projects from the "Just One Giant Lab's OpenCOVID19 initiative" and "Helpful Engineering" open research communities. The community review process uses granular review forms and requires the participation of grant applicants in the review process. Within a year, we organised 7 rounds of review, resulting in 614 reviews from 201 reviewers, and the attribution of 48 micro-grants of up to 4,000 euros. The system is fast, with a median process duration of 10 days, scalable, with a median of 4 reviewers per project independent of the total number of projects, and fair, with project rankings highly preserved after the synthetic removal of reviewers. Regarding potential bias introduced by involving applicants in the process, we find that review scores from both applicants and non-applicants have a similar correlation of r=0.28 with other reviews within a project, matching traditional approaches. Finally, we find that the ability of projects to apply to several rounds allows to foster the further implementation of successful early prototypes, as well as provide a pathway to constructively improve an initially failing proposal in an agile manner. Overall, this study quantitatively highlights the benefits of a frugal, community review system acting as a due diligence for rapid and agile resource allocation in open research and innovation programs, with implications for decentralised communities.
资源分配对于科学技术领域创新项目的选择和实施至关重要。目前，针对拨款申请的“胜者通吃”模式要求研究人员花费大量时间撰写详尽的项目提案，并依赖少数时间充裕的志愿专家。这样的流程通常需要几个月的时间，与预期收益相比，其有效成本较高。我们设计了一种敏捷的“社区评审”系统，用于快速原型制作创新解决方案的小额拨款分配。在这里，我们描述并评估了该社区评审在“Just One Giant Lab 的 OpenCOVID19 倡议”和“Helpful Engineering”开放研究社区的 147 个项目中的实施情况。社区评审过程使用细粒度的评审表，并要求拨款申请人参与评审过程。在一年内，我们组织了 7 轮评审，来自 201 位评审员的 614 次评审，以及 48 项小额拨款，每项拨款金额高达 4000 欧元。该系统速度快，中位数评审周期为 10 天，可扩展，中位数每个项目有 4 位评审员，与项目总数无关，公平公正，在综合去除评审员后，项目排名高度保留。关于在评审过程中让申请人参与可能带来的潜在偏见，我们发现，申请人和非申请人的评审得分与项目内其他评审的相关性相似，r 值为 0.28，与传统方法一致。最后，我们发现项目申请多轮评审的能力可以促进成功早期原型的进一步实施，并以敏捷的方式为建设性地改进最初失败的提案提供途径。总体而言，这项研究从定量角度强调了节俭的社区评审系统的优势，该系统在开放研究和创新计划中充当了快速、敏捷资源分配的尽职调查，对分散式社区具有启示意义。
