Muradchanian Jasmine, Hoekstra Rink, Kiers Henk, van Ravenzwaaij Don
Behavioural and Social Sciences, University of Groningen, Groningen, the Netherlands.
PLoS One. 2024 Dec 11;19(12):e0308495. doi: 10.1371/journal.pone.0308495. eCollection 2024.
The importance of replication in the social and behavioural sciences has been emphasized for decades. Various frequentist and Bayesian approaches have been proposed to qualify a replication study as successful or unsuccessful. One of them is meta-analysis. The focus of the present study is on the way meta-analysis functions as a replication success metric. To investigate this, original and replication studies that are part of two large-scale replication projects were used. For each original study, the probability of replication success was calculated using meta-analysis under different assumptions of the underlying population effect when replication results were unknown. The accuracy of the predicted overall replication success was evaluated once replication results became available using adjusted Brier scores.
Our results showed that meta-analysis performed poorly when used as a replication success metric. In many cases, quantifying replication success using meta-analysis resulted in the conclusion where the replication was deemed a success regardless of the results of the replication study.
We conclude that when using meta-analysis as a replication success metric, it has a relatively high probability of finding evidence in favour of a non-zero population effect even when it is zero. This behaviour largely results from the significance of the original study. Furthermore, we argue that there are fundamental reasons against using meta-analysis as a metric for replication success.
几十年来，复制在社会科学和行为科学中的重要性一直得到强调。已经提出了各种频率主义和贝叶斯方法来判定一项复制研究是成功还是失败。其中之一是元分析。本研究的重点是元分析作为复制成功度量的作用方式。为了对此进行调查，我们使用了两个大规模复制项目中的原始研究和复制研究。对于每项原始研究，在复制结果未知的情况下，根据潜在总体效应的不同假设，使用元分析计算复制成功的概率。一旦复制结果可用，使用调整后的布里尔分数评估预测的总体复制成功的准确性。
我们的结果表明，元分析作为复制成功度量时表现不佳。在许多情况下，使用元分析量化复制成功会得出这样的结论：无论复制研究的结果如何，都认为复制是成功的。
我们得出结论，当使用元分析作为复制成功度量时，即使总体效应为零，它也有相对较高的概率找到支持非零总体效应的证据。这种行为很大程度上源于原始研究的显著性。此外，我们认为有根本原因反对使用元分析作为复制成功的度量。
