Social and Behavioral Interventions Program, Department of International Health, Johns Hopkins Bloomberg School of Public Health, 615 North Wolfe Street, Room E5547, Baltimore, MD, 21205, USA.
Division of Global and Community Health, Department of Psychiatry and Behavioral Sciences, Medical University of South Carolina, 176 Croghan Spur Road, Suite 104, Charleston, SC, 29407, USA.
Syst Rev. 2019 Jan 3;8(1):3. doi: 10.1186/s13643-018-0925-0.
Different tools exist for assessing risk of bias of intervention studies for systematic reviews. We present a tool for assessing risk of bias across both randomized and non-randomized study designs. The tool was developed by the Evidence Project, which conducts systematic reviews and meta-analyses of behavioral interventions for HIV in low- and middle-income countries.
We present the eight items of the tool and describe considerations for each and for the tool as a whole. We then evaluate reliability of the tool by presenting inter-rater reliability for 125 selected studies from seven published reviews, calculating a kappa for each individual item and a weighted kappa for the total count of items.
The tool includes eight items, each of which is rated as being present (yes) or not present (no) and, for some items, not applicable or not reported. The items include (1) cohort, (2) control or comparison group, (3) pre-post intervention data, (4) random assignment of participants to the intervention, (5) random selection of participants for assessment, (6) follow-up rate of 80% or more, (7) comparison groups equivalent on sociodemographics, and (8) comparison groups equivalent at baseline on outcome measures. Together, items (1)-(3) summarize the study design, while the remaining items consider other common elements of study rigor. Inter-rater reliability was moderate to substantial for all items, ranging from 0.41 to 0.80 (median κ = 0.66). Agreement between raters on the total count of items endorsed was also substantial (κ = 0.66).
Strengths of the tool include its applicability to a range of study designs, from randomized trials to various types of observational and quasi-experimental studies. It is relatively easy to use and interpret and can be applied to a range of review topics without adaptation, facilitating comparability across reviews. Limitations include the lack of potentially relevant items measured in other tools and potential threats to validity of some items. To date, the tool has been applied in over 30 reviews. We believe it is a practical option for assessing risk of bias in systematic reviews of interventions that include a range of study designs.
评估系统评价中干预研究偏倚风险的工具多种多样。我们提出了一种适用于随机和非随机研究设计的偏倚风险评估工具。该工具由 Evidence Project 开发，该项目对中低收入国家的 HIV 行为干预进行系统评价和荟萃分析。
我们介绍了工具的八项内容，并描述了每项内容以及整个工具的考虑因素。然后，我们通过呈现来自七个已发表综述的 125 项选定研究的评分者间可靠性，计算每个单项的kappa 值和总项目计数的加权 kappa 值来评估工具的可靠性。
该工具包括八项内容，每项内容均评为存在（是）或不存在（否），某些项目为不适用或未报告。这些项目包括（1）队列，（2）对照组或比较组，（3）干预前后数据，（4）参与者随机分配至干预组，（5）参与者随机选择进行评估，（6）随访率达到 80%或更高，（7）在社会人口统计学方面比较组相当，以及（8）在结局测量方面比较组基线相当。项目（1）-（3）共同总结了研究设计，而其余项目则考虑了研究严谨性的其他常见要素。所有项目的评分者间可靠性均为中度至高度，范围为 0.41 至 0.80（中位数 κ=0.66）。评分者对所认可的项目总数的一致性也很高（κ=0.66）。
该工具的优点包括适用于从随机试验到各种类型的观察性和准实验研究的一系列研究设计。它相对易于使用和解释，无需适应即可应用于一系列综述主题，促进了综述之间的可比性。局限性包括其他工具中未测量的潜在相关项目以及某些项目有效性的潜在威胁。迄今为止，该工具已应用于 30 多个综述中。我们认为，对于包括各种研究设计的干预措施系统评价的偏倚风险评估，这是一种实用的选择。
