Hempel Susanne, Miles Jeremy N V, Booth Marika J, Wang Zhen, Morton Sally C, Shekelle Paul G
RAND Corporation, Santa Monica, CA 90407, USA.
Syst Rev. 2013 Nov 28;2:107. doi: 10.1186/2046-4053-2-107.
There are both theoretical and empirical reasons to believe that design and execution factors are associated with bias in controlled trials. Statistically significant moderator effects, such as the effect of trial quality on treatment effect sizes, are rarely detected in individual meta-analyses, and evidence from meta-epidemiological datasets is inconsistent. The reasons for the disconnect between theory and empirical observation are unclear. The study objective was to explore the power to detect study level moderator effects in meta-analyses.
We generated meta-analyses using Monte-Carlo simulations and investigated the effect of number of trials, trial sample size, moderator effect size, heterogeneity, and moderator distribution on power to detect moderator effects. The simulations provide a reference guide for investigators to estimate power when planning meta-regressions.
The power to detect moderator effects in meta-analyses, for example, effects of study quality on effect sizes, is largely determined by the degree of residual heterogeneity present in the dataset (noise not explained by the moderator). Larger trial sample sizes increase power only when residual heterogeneity is low. A large number of trials or low residual heterogeneity are necessary to detect effects. When the proportion of the moderator is not equal (for example, 25% 'high quality', 75% 'low quality' trials), power of 80% was rarely achieved in investigated scenarios. Application to an empirical meta-epidemiological dataset with substantial heterogeneity (I(2) = 92%, τ(2) = 0.285) estimated >200 trials are needed for a power of 80% to show a statistically significant result, even for a substantial moderator effect (0.2), and the number of trials with the less common feature (for example, few 'high quality' studies) affects power extensively.
Although study characteristics, such as trial quality, may explain some proportion of heterogeneity across study results in meta-analyses, residual heterogeneity is a crucial factor in determining when associations between moderator variables and effect sizes can be statistically detected. Detecting moderator effects requires more powerful analyses than are employed in most published investigations; hence negative findings should not be considered evidence of a lack of effect, and investigations are not hypothesis-proving unless power calculations show sufficient ability to detect effects.
有理论和实证依据表明，设计和执行因素与对照试验中的偏倚相关。在个体荟萃分析中，很少能检测到具有统计学意义的调节效应，例如试验质量对治疗效应大小的影响，并且来自元流行病学数据集的证据也不一致。理论与实证观察之间脱节的原因尚不清楚。本研究的目的是探讨在荟萃分析中检测研究水平调节效应的效能。
我们使用蒙特卡洛模拟生成荟萃分析，并研究试验数量、试验样本量、调节效应大小、异质性以及调节变量分布对检测调节效应效能的影响。这些模拟为研究者在计划进行元回归时估计效能提供了参考指南。
在荟萃分析中检测调节效应的效能，例如研究质量对效应大小的影响，很大程度上取决于数据集中存在的残余异质性程度（未由调节变量解释的噪声）。只有当残余异质性较低时，更大的试验样本量才会提高效能。检测效应需要大量试验或低残余异质性。当调节变量的比例不相等时（例如，25%为“高质量”试验，75%为“低质量”试验），在所研究的情形中很少能达到80%的效能。应用于一个具有大量异质性的实证元流行病学数据集（I(2)=92%，τ(2)=0.285）时，估计即使对于较大的调节效应（0.2），要达到80%的效能以显示具有统计学意义的结果也需要超过200项试验，并且具有较少见特征的试验数量（例如，很少有“高质量”研究）会广泛影响效能。
尽管研究特征，如试验质量，可能在一定程度上解释荟萃分析中研究结果之间的异质性，但残余异质性是决定何时能够在统计学上检测到调节变量与效应大小之间关联的关键因素。检测调节效应需要比大多数已发表研究中所采用的更强大的分析方法；因此，阴性结果不应被视为缺乏效应的证据，并且除非效能计算显示有足够的能力检测效应，否则研究并非是在证明假设。
