Viganola Domenico, Buckles Grant, Chen Yiling, Diego-Rosell Pablo, Johannesson Magnus, Nosek Brian A, Pfeiffer Thomas, Siegel Adam, Dreber Anna
World Bank Group, Washington D.C., USA.
Gallup Inc, Washington, District of Columbia, USA.
R Soc Open Sci. 2021 Jul 14;8(7):181308. doi: 10.1098/rsos.181308. eCollection 2021 Jul.
There is evidence that prediction markets are useful tools to aggregate information on researchers' beliefs about scientific results including the outcome of replications. In this study, we use prediction markets to forecast the results of novel experimental designs that test established theories. We set up prediction markets for hypotheses tested in the Defense Advanced Research Projects Agency's (DARPA) Next Generation Social Science (NGS2) programme. Researchers were invited to bet on whether 22 hypotheses would be supported or not. We define support as a test result in the same direction as hypothesized, with a Bayes factor of at least 10 (i.e. a likelihood of the observed data being consistent with the tested hypothesis that is at least 10 times greater compared with the null hypothesis). In addition to betting on this binary outcome, we asked participants to bet on the expected effect size (in Cohen's ) for each hypothesis. Our goal was to recruit at least 50 participants that signed up to participate in these markets. While this was the case, only 39 participants ended up actually trading. Participants also completed a survey on both the binary result and the effect size. We find that neither prediction markets nor surveys performed well in predicting outcomes for NGS2.
有证据表明，预测市场是汇总研究人员对科学结果（包括复制结果）信念信息的有用工具。在本研究中，我们使用预测市场来预测检验既定理论的新实验设计的结果。我们为美国国防高级研究计划局（DARPA）下一代社会科学（NGS2）计划中所检验的假设设立了预测市场。邀请研究人员就22个假设是否会得到支持进行投注。我们将支持定义为与假设方向相同的测试结果，贝叶斯因子至少为10（即观察到的数据与检验假设一致的可能性至少是与零假设相比的10倍）。除了对这个二元结果进行投注外，我们还要求参与者对每个假设的预期效应大小（以科恩d值表示）进行投注。我们的目标是招募至少50名报名参与这些市场的参与者。虽然情况如此，但最终只有39名参与者实际进行了交易。参与者还完成了一份关于二元结果和效应大小的调查。我们发现，预测市场和调查在预测NGS2的结果方面都表现不佳。
