Massey University, Auckland, New Zealand.
George Mason University, Fairfax, VA, United States of America.
PLoS One. 2021 Apr 14;16(4):e0248780. doi: 10.1371/journal.pone.0248780. eCollection 2021.
The reproducibility of published research has become an important topic in science policy. A number of large-scale replication projects have been conducted to gauge the overall reproducibility in specific academic fields. Here, we present an analysis of data from four studies which sought to forecast the outcomes of replication projects in the social and behavioural sciences, using human experts who participated in prediction markets and answered surveys. Because the number of findings replicated and predicted in each individual study was small, pooling the data offers an opportunity to evaluate hypotheses regarding the performance of prediction markets and surveys at a higher power. In total, peer beliefs were elicited for the replication outcomes of 103 published findings. We find there is information within the scientific community about the replicability of scientific findings, and that both surveys and prediction markets can be used to elicit and aggregate this information. Our results show prediction markets can determine the outcomes of direct replications with 73% accuracy (n = 103). Both the prediction market prices, and the average survey responses are correlated with outcomes (0.581 and 0.564 respectively, both p < .001). We also found a significant relationship between p-values of the original findings and replication outcomes. The dataset is made available through the R package "pooledmaRket" and can be used to further study community beliefs towards replications outcomes as elicited in the surveys and prediction markets.
已发表研究的可重复性已成为科学政策中的一个重要议题。已经开展了一些大规模的复制项目，以评估特定学术领域的整体可重复性。在这里，我们分析了四项研究的数据，这些研究试图利用参与预测市场和回答调查的人类专家来预测社会和行为科学中的复制项目的结果。由于每个单独研究中复制和预测的发现数量较少，因此汇总数据为评估关于预测市场和调查在更高功效下的表现的假设提供了机会。总共为 103 项已发表发现的复制结果征集了同行的意见。我们发现科学界对科学发现的可重复性有一定的了解，并且可以使用调查和预测市场来获取和汇总这些信息。我们的研究结果表明，预测市场可以以 73%的准确率（n=103）确定直接复制的结果。预测市场的价格以及平均调查回复与结果相关（分别为 0.581 和 0.564，两者均 p<.001）。我们还发现原始发现的 p 值与复制结果之间存在显著关系。该数据集通过 R 包“pooledmaRket”提供，可用于进一步研究社区对调查和预测市场中得出的复制结果的看法。
