von Klinggraeff Lauren, Pfledderer Chris D, Burkart Sarah, Ramey Kaitlyn, Smith Michal, McLain Alexander C, Armstrong Bridget, Weaver R Glenn, Okely Anthony, Lubans David, Ioannidis John P A, Jago Russell, Turner-McGrievy Gabrielle, Thrasher James, Li Xiaoming, Beets Michael W
Augusta University, Augusta University.
University of Texas Health Science Center at Houston.
Res Sq. 2024 Feb 26:rs.3.rs-3897976. doi: 10.21203/rs.3.rs-3897976/v1.
Preliminary studies (e.g., pilot/feasibility studies) can result in misleading evidence that an intervention is ready to be evaluated in a large-scale trial when it is not. Risk of Generalizability Biases (RGBs, a set of external validity biases) represent study features that influence estimates of effectiveness, often inflating estimates in preliminary studies which are not replicated in larger-scale trials. While RGBs have been empirically established in interventions targeting obesity, the extent to which RGBs generalize to other health areas is unknown. Understanding the relevance of RGBs across health behavior intervention research can inform organized efforts to reduce their prevalence.
The purpose of our study was to examine whether RGBs generalize outside of obesity-related interventions.
A systematic review identified health behavior interventions across four behaviors unrelated to obesity that follow a similar intervention development framework of preliminary studies informing larger-scale trials (i.e., tobacco use disorder, alcohol use disorder, interpersonal violence, and behaviors related to increased sexually transmitted infections). To be included, published interventions had to be tested in a preliminary study followed by testing in a larger trial (the two studies thus comprising a study pair). We extracted health-related outcomes and coded the presence/absence of RGBs. We used meta-regression models to estimate the impact of RGBs on the change in standardized mean difference (ΔSMD) between the preliminary study and larger trial.
We identified sixty-nine study pairs, of which forty-seven were eligible for inclusion in the analysis (k = 156 effects), with RGBs identified for each behavior. For pairs where the RGB was present in the preliminary study but removed in the larger trial the treatment effect decreased by an average of ΔSMD=-0.38 (range - 0.69 to -0.21). This provides evidence of larger drop in effectiveness for studies containing RGBs relative to study pairs with no RGBs present (treatment effect decreased by an average of ΔSMD =-0.24, range - 0.19 to -0.27).
RGBs may be associated with higher effect estimates across diverse areas of health intervention research. These findings suggest commonalities shared across health behavior intervention fields may facilitate introduction of RGBs within preliminary studies, rather than RGBs being isolated to a single health behavior field.
初步研究（如试点/可行性研究）可能会产生误导性证据，表明一项干预措施已准备好在大规模试验中进行评估，而实际上并非如此。可推广性偏差风险（RGBs，一组外部效度偏差）代表影响效果估计的研究特征，在初步研究中常常夸大估计值，而在大规模试验中却无法重现。虽然RGBs在针对肥胖症的干预措施中已得到实证验证，但RGBs在其他健康领域的普遍程度尚不清楚。了解RGBs在健康行为干预研究中的相关性可为有组织地努力降低其发生率提供参考。
我们研究的目的是检验RGBs是否能推广到与肥胖症无关的干预措施之外。
一项系统评价确定了四种与肥胖症无关的行为的健康行为干预措施，这些措施遵循类似的干预发展框架，即初步研究为大规模试验提供信息（即烟草使用障碍、酒精使用障碍、人际暴力以及与性传播感染增加相关的行为）。要纳入研究，已发表的干预措施必须先在初步研究中进行测试，然后在更大规模的试验中进行测试（这两项研究因此构成一对研究）。我们提取了与健康相关的结果，并对RGBs的存在与否进行编码。我们使用元回归模型来估计RGBs对初步研究和更大规模试验之间标准化平均差变化（ΔSMD）的影响。
我们确定了69对研究，其中47对符合纳入分析的条件（k = 156个效应），每种行为都识别出了RGBs。对于那些在初步研究中存在RGBs但在更大规模试验中消除了RGBs的研究对，治疗效果平均下降了ΔSMD = -0.38（范围为-0.69至-0.21）。这提供了证据，表明相对于不存在RGBs的研究对，包含RGBs的研究的效果下降幅度更大（治疗效果平均下降了ΔSMD = -0.24，范围为-0.19至-0.27）。
RGBs可能与不同健康干预研究领域中较高的效应估计值相关。这些发现表明，健康行为干预领域共有的共性可能有助于在初步研究中引入RGBs，而不是RGBs仅局限于单一的健康行为领域。
