Rocky Mountain Mental Illness Research, Education, and Clinical Center (MIRECC), Department of Veterans Affairs, Aurora, CO, USA; Department of Physical Medicine and Rehabilitation, University of Colorado Anschutz Medical Campus, Aurora, CO, USA.
New England Geriatric Research Education and Clinical Center, VA Boston Healthcare System, Boston, MA, USA.
Ann Phys Rehabil Med. 2022 Jan;65(1):101503. doi: 10.1016/j.rehab.2021.101503. Epub 2021 Nov 14.
Relying solely on null hypothesis significance testing to investigate rehabilitation interventions may result in researchers erroneously concluding the presence of a treatment effect.
We sought to quantify the strength of evidence in favour of rehabilitation treatment effects by calculating Bayes factors (BFs) for significant findings. Additionally, we sought to examine associations between BFs, P-values, and Cohen's d effect sizes.
We searched the Cochrane Database of Systematic Reviews for meta-analyses with "rehabilitation" as a keyword that evaluated a rehabilitation intervention. We extracted means, standard deviations, and sample sizes for treatment and comparison groups from individual findings within 175 meta-analyses. Investigators independently classified the interventions according to the Rehabilitation Treatment Specification System. We calculated t-statistics, P-values, effect sizes, and BFs for each finding. We isolated statistically significant findings (P≤0.05); applied evidential categories to BFs, P-values, and effect sizes; and examined relationships descriptively.
We analysed 1935 rehabilitation findings. Across intervention types, 25% of significant findings offered only anecdotal evidence in favour of a treatment effect; only 48% indicated strong evidence. This pattern persisted within intervention types and when conducting robustness analyses. Smaller P-values and larger effect sizes were associated with stronger evidence in favour of a treatment effect. However, a notable portion of findings with P-value 0.01 to 0.05 (63%) or a large effect size (18%) offered anecdotal evidence in favour of an effect.
For a substantial portion of statistically significant rehabilitation findings, the data neither support nor refute the presence of a treatment effect. This was the case among a notable portion of large treatment effects and for most findings with P-value>0.01. Rehabilitation evidence would be improved by researchers adopting more conservative levels of significance, complementing the use of null hypothesis significance testing with Bayesian techniques and reporting effect sizes.
仅依靠零假设显著性检验来研究康复干预措施，可能会导致研究人员错误地得出存在治疗效果的结论。
我们旨在通过计算显著发现的贝叶斯因子（BFs）来量化支持康复治疗效果的证据力度。此外，我们还试图研究 BF、P 值和 Cohen's d 效应大小之间的关联。
我们在 Cochrane 系统评价数据库中搜索了以“康复”为关键词的元分析，评估了康复干预措施。我们从 175 项元分析中的每个发现中提取了治疗组和对照组的均值、标准差和样本量。研究者根据康复治疗规范系统独立对干预措施进行分类。我们为每个发现计算了 t 统计量、P 值、效应大小和 BF。我们分离出具有统计学意义的发现（P≤0.05）；应用证据类别对 BF、P 值和效应大小进行分类；并进行描述性分析。
我们分析了 1935 项康复发现。在各种干预类型中，25%的显著发现仅提供了支持治疗效果的轶事证据；只有 48%的发现表明有强有力的证据。这种模式在干预类型内和进行稳健性分析时仍然存在。较小的 P 值和较大的效应大小与更有力的支持治疗效果的证据相关。然而，有相当一部分 P 值为 0.01 到 0.05（63%）或效应量较大（18%）的发现提供了支持效果的轶事证据。
对于相当一部分具有统计学意义的康复发现，数据既不支持也不反驳治疗效果的存在。这在相当一部分大的治疗效果中以及大多数 P 值>0.01 的发现中都是如此。研究人员采用更保守的显著性水平，并将零假设显著性检验与贝叶斯技术相结合，报告效应大小，这将提高康复证据的质量。
