Department of Marketing, Bocconi University.
ESADE Business School, Ramon Llull University.
J Exp Psychol Gen. 2019 Sep;148(9):1628-1639. doi: 10.1037/xge0000663.
Several researchers have relied on, or advocated for, internal meta-analysis, which involves statistically aggregating multiple studies in a paper to assess their overall evidential value. Advocates of internal meta-analysis argue that it provides an efficient approach to increasing statistical power and solving the file-drawer problem. Here we show that the validity of internal meta-analysis rests on the assumption that no studies or analyses were selectively reported. That is, the technique is only valid if (a) all conducted studies were included (i.e., an empty file drawer), and (b) for each included study, exactly one analysis was attempted (i.e., there was no p-hacking). We show that even very small doses of selective reporting invalidate internal meta-analysis. For example, the kind of minimal p-hacking that increases the false-positive rate of 1 study to just 8% increases the false-positive rate of a 10-study internal meta-analysis to 83%. If selective reporting is approximately zero, but not exactly zero, then internal meta-analysis is invalid. To be valid, (a) an internal meta-analysis would need to contain exclusively studies that were properly preregistered, (b) those preregistrations would have to be followed in all essential aspects, and (c) the decision of whether to include a given study in an internal meta-analysis would have to be made before any of those studies are run. (PsycINFO Database Record (c) 2019 APA, all rights reserved).
几位研究人员依赖或倡导内部荟萃分析，这种方法涉及到在一篇论文中对多项研究进行统计学汇总，以评估其整体证据价值。内部荟萃分析的支持者认为，这种方法提供了一种有效的方法来提高统计能力并解决文件抽屉问题。在这里，我们表明内部荟萃分析的有效性取决于没有选择性报告研究或分析的假设。也就是说，只有在以下情况下，该技术才是有效的：（a）所有进行的研究都被包括在内（即，没有空的文件抽屉）；（b）对于每个被纳入的研究，只尝试了一个分析（即，不存在 p-值操纵）。我们表明，即使是很小剂量的选择性报告也会使内部荟萃分析无效。例如，将 1 项研究的假阳性率提高到 8%的最小程度的 p 值操纵，会将 10 项研究的内部荟萃分析的假阳性率提高到 83%。如果选择性报告接近于零，但不是完全为零，那么内部荟萃分析就是无效的。为了使内部荟萃分析有效，（a）内部荟萃分析必须只包含经过适当预注册的研究；（b）这些预注册必须在所有重要方面都得到遵守；（c）决定是否将某项研究纳入内部荟萃分析必须在进行任何研究之前做出。（PsycINFO 数据库记录（c）2019 APA，保留所有权利）。
