MetaMelb Lab, University of Melbourne, Melbourne, Victoria, Australia.
Quantitative & Applied Ecology Group, University of Melbourne, Melbourne, Victoria, Australia.
PLoS One. 2023 Jan 26;18(1):e0274429. doi: 10.1371/journal.pone.0274429. eCollection 2023.
As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach based on a modified Delphi technique applied to the evaluation of research claims in social and behavioural sciences. The utility of processes to predict replicability is their capacity to test scientific claims without the costs of full replication. Experimental data supports the validity of this process, with a validation study producing a classification accuracy of 84% and an Area Under the Curve of 0.94, meeting or exceeding the accuracy of other techniques used to predict replicability. The repliCATS process provides other benefits. It is highly scalable, able to be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period through an online elicitation platform, having been used to assess 3000 research claims over an 18 month period. It is available to be implemented in a range of ways and we describe one such implementation. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to provide insight in understanding the limits of generalizability of scientific claims. The primary limitation of the repliCATS process is its reliance on human-derived predictions with consequent costs in terms of participant fatigue although careful design can minimise these costs. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.
由于单个研究的复制工作需要耗费大量资源，因此需要预测其可复制性的技术。我们引入了 repliCATS（可信科学协作评估）流程，这是一种新的方法，可以让专家对研究的可复制性进行预测。该流程是一种基于改良 Delphi 技术的结构化专家启发式方法，适用于评估社会和行为科学中的研究主张。预测可复制性的过程的实用性在于，它们能够在无需进行全面复制的情况下检验科学主张。实验数据支持该过程的有效性，一项验证研究的分类准确率达到 84%，曲线下面积为 0.94，达到或超过了用于预测可复制性的其他技术的准确率。repliCATS 流程还有其他好处。它具有高度可扩展性，能够快速评估少量主张，也能够通过在线启发式平台对大量主张进行长时间评估，在 18 个月的时间里，该平台已经对 3000 项研究主张进行了评估。它可以以多种方式实施，我们描述了其中一种实施方式。repliCATS 流程的一个重要优势是，它收集了定性数据，有可能深入了解科学主张的可推广性的局限性。repliCATS 流程的主要局限性在于其依赖于人类预测，因此存在参与者疲劳的成本，尽管精心设计可以最小化这些成本。repliCATS 流程在替代同行评审和复制研究工作的分配方面具有潜在应用。
