Department of Educational Studies, University of Nevada, Reno, Reno, NV, USA.
Department of Psychology, National Taiwan University, Taipei, Taiwan.
Behav Res Methods. 2024 Jan;56(1):379-405. doi: 10.3758/s13428-022-02035-8. Epub 2023 Jan 17.
What Works Clearinghouse (WWC, 2022) recommends a design-comparable effect size (D-CES; i.e., g) to gauge an intervention in single-case experimental design (SCED) studies, or to synthesize findings in meta-analysis. So far, no research has examined g's performance under non-normal distributions. This study expanded Pustejovsky et al. (2014) to investigate the impact of data distributions, number of cases (m), number of measurements (N), within-case reliability or intra-class correlation (ρ), ratio of variance components (λ), and autocorrelation (ϕ) on g in multiple-baseline (MB) design. The performance of g was assessed by relative bias (RB), relative bias of variance (RBV), MSE, and coverage rate of 95% CIs (CR). Findings revealed that g was unbiased even under non-normal distributions. g's variance was generally overestimated, and its 95% CI was over-covered, especially when distributions were normal or nearly normal combined with small m and N. Large imprecision of g occurred when m was small and ρ was large. According to the ANOVA results, data distributions contributed to approximately 49% of variance in RB and 25% of variance in both RBV and CR. m and ρ each contributed to 34% of variance in MSE. We recommend g for MB studies and meta-analysis with N ≥ 16 and when either (1) data distributions are normal or nearly normal, m = 6, and ρ = 0.6 or 0.8, or (2) data distributions are mildly or moderately non-normal, m ≥ 4, and ρ = 0.2, 0.4, or 0.6. The paper concludes with a discussion of g's applicability and design-comparability, and sound reporting practices of ES indices.
What Works Clearinghouse (WWC, 2022) 推荐使用设计可比效应量 (D-CES；即 g) 来衡量单病例实验设计 (SCED) 研究中的干预措施，或在荟萃分析中综合研究结果。到目前为止，还没有研究检验 g 在非正态分布下的表现。本研究扩展了 Pustejovsky 等人（2014）的研究，以调查数据分布、案例数 (m)、测量次数 (N)、病例内信度或组内相关系数 (ρ)、方差分量比 (λ) 和自相关 (ϕ) 对多基线 (MB) 设计中 g 的影响。通过相对偏差 (RB)、方差相对偏差 (RBV)、均方误差 (MSE) 和 95%置信区间 (CI) 的覆盖率 (CR) 评估 g 的性能。研究结果表明，即使在非正态分布下，g 也是无偏的。g 的方差通常被高估，其 95%CI 被过度覆盖，尤其是在分布正常或接近正常且 m 和 N 较小的情况下。当 m 较小时，ρ 较大时，g 的精度会大幅降低。根据方差分析结果，数据分布对 RB 的方差贡献约为 49%，对 RBV 和 CR 的方差贡献各约为 25%。m 和 ρ 分别对 MSE 的方差贡献 34%。我们建议在 MB 研究和荟萃分析中使用 g，当 N≥16 且 (1) 数据分布正常或接近正态，m=6，ρ=0.6 或 0.8，或 (2) 数据分布轻度或中度非正态，m≥4，且 ρ=0.2、0.4 或 0.6 时。本文最后讨论了 g 的适用性和设计可比性，以及 ES 指数的合理报告实践。
