Evolution & Ecology Research Centre and School of Biological, Earth and Environmental Sciences, University of New South Wales, Sydney, New South Wales, Australia.
Department of Biosystems Engineering, Zhejiang University, Hangzhou, China.
Glob Chang Biol. 2022 Feb;28(3):969-989. doi: 10.1111/gcb.15972. Epub 2021 Dec 10.
Field studies are essential to reliably quantify ecological responses to global change because they are exposed to realistic climate manipulations. Yet such studies are limited in replicates, resulting in less power and, therefore, potentially unreliable effect estimates. Furthermore, while manipulative field experiments are assumed to be more powerful than non-manipulative observations, it has rarely been scrutinized using extensive data. Here, using 3847 field experiments that were designed to estimate the effect of environmental stressors on ecosystems, we systematically quantified their statistical power and magnitude (Type M) and sign (Type S) errors. Our investigations focused upon the reliability of field experiments to assess the effect of stressors on both ecosystem's response magnitude and variability. When controlling for publication bias, single experiments were underpowered to detect response magnitude (median power: 18%-38% depending on effect sizes). Single experiments also had much lower power to detect response variability (6%-12% depending on effect sizes) than response magnitude. Such underpowered studies could exaggerate estimates of response magnitude by 2-3 times (Type M errors) and variability by 4-10 times. Type S errors were comparatively rare. These observations indicate that low power, coupled with publication bias, inflates the estimates of anthropogenic impacts. Importantly, we found that meta-analyses largely mitigated the issues of low power and exaggerated effect size estimates. Rather surprisingly, manipulative experiments and non-manipulative observations had very similar results in terms of their power, Type M and S errors. Therefore, the previous assumption about the superiority of manipulative experiments in terms of power is overstated. These results call for highly powered field studies to reliably inform theory building and policymaking, via more collaboration and team science, and large-scale ecosystem facilities. Future studies also require transparent reporting and open science practices to approach reproducible and reliable empirical work and evidence synthesis.
野外研究对于可靠地量化全球变化对生态系统的响应至关重要，因为它们会受到真实气候的影响。然而，此类研究的重复次数有限，导致其统计效力较低，因此潜在的效应估计结果可能不可靠。此外，虽然操纵性野外实验被认为比非操纵性观测更有力，但很少使用广泛的数据对其进行严格审查。在这里，我们使用了 3847 个野外实验来估计环境胁迫对生态系统的影响，系统地量化了它们的统计效力和幅度（M 型）以及符号（S 型）误差。我们的研究重点是野外实验评估胁迫对生态系统响应幅度和变异性的可靠性。在控制出版偏倚的情况下，单个实验检测响应幅度的效力不足（中位数效力：取决于效应大小，18%-38%）。单个实验检测响应变异性的效力也明显低于响应幅度（取决于效应大小，6%-12%）。这种效力不足的研究可能会使响应幅度的估计值夸大 2-3 倍（M 型误差），变异性夸大 4-10 倍。S 型误差相对较少。这些观察结果表明，低效力加上出版偏倚会夸大人为影响的估计值。重要的是，我们发现元分析在很大程度上减轻了效力低和效应大小估计值夸大的问题。相当令人惊讶的是，操纵性实验和非操纵性观测在效力、M 型和 S 型误差方面具有非常相似的结果。因此，以前关于操纵性实验在效力方面具有优越性的假设被夸大了。这些结果呼吁通过更多的合作和团队科学以及大规模生态系统设施，开展高效力的野外研究，以可靠地为理论构建和决策制定提供信息。未来的研究还需要透明的报告和开放科学实践，以实现可重现和可靠的实证工作和证据综合。
