Faculty of Psychology and Educational Sciences, KU Leuven, University of Leuven, Etienne Sabbelaan 51, 8500, Kortrijk, Belgium.
ITEC, an Imec research group at KU Leuven, University of Leuven, Leuven, Belgium.
Behav Res Methods. 2021 Apr;53(2):702-717. doi: 10.3758/s13428-020-01459-4.
In meta-analysis, primary studies often include multiple, dependent effect sizes. Several methods address this dependency, such as the multivariate approach, three-level models, and the robust variance estimation (RVE) method. As for today, most simulation studies that explore the performance of these methods have focused on the estimation of the overall effect size. However, researchers are sometimes interested in obtaining separate effect size estimates for different types of outcomes. A recent simulation study (Park & Beretvas, 2019) has compared the performance of the three-level approach and the RVE method in estimating outcome-specific effects when several effect sizes are reported for different types of outcomes within studies. The goal of this paper is to extend that study by incorporating additional simulation conditions and by exploring the performance of additional models, such as the multivariate model, a three-level model that specifies different study-effects for each type of outcome, a three-level model that specifies a common study-effect for all outcomes, and separate three-level models for each type of outcome. Additionally, we also tested whether the a posteriori application of the RV correction improves the standard error estimates and the 95% confidence intervals. Results show that the application of separate three-level models for each type of outcome is the only approach that consistently gives adequate standard error estimates. Also, the a posteriori application of the RV correction results in correct 95% confidence intervals in all models, even if they are misspecified, meaning that Type I error rate is adequate when the RV correction is implemented.
在荟萃分析中，原始研究通常包含多个相关的效应量。有几种方法可以解决这种相关性，例如多变量方法、三级模型和稳健方差估计（RVE）方法。截至目前，大多数探索这些方法性能的模拟研究都集中在总体效应量的估计上。然而，研究人员有时有兴趣为不同类型的结果获得单独的效应量估计。最近的一项模拟研究（Park & Beretvas，2019）比较了三级方法和 RVE 方法在估计研究中不同类型结果的特定效应量时的性能，当研究中为不同类型的结果报告了多个效应量时。本文的目的是通过纳入更多的模拟条件和探索更多模型的性能来扩展该研究，例如多变量模型、为每种类型的结果指定不同研究效应的三级模型、为所有结果指定共同研究效应的三级模型以及为每种类型的结果分别指定三级模型。此外，我们还测试了 RV 校正的后验应用是否可以改善标准误估计和 95%置信区间。结果表明，为每种类型的结果分别使用三级模型是唯一一种始终能给出适当标准误估计的方法。此外，即使模型存在错误指定，RV 校正的后验应用也会导致所有模型中的正确 95%置信区间，这意味着当实施 RV 校正时，I 型错误率是适当的。
