Department of Epidemiology, Lazio Regional Health Service, Rome, Italy; Laboratory of Clinical Research Methodology, Istituto di Ricerche Farmacologiche Mario Negri IRCCS, Milan, Italy.
Laboratory of Clinical Research Methodology, Istituto di Ricerche Farmacologiche Mario Negri IRCCS, Milan, Italy.
J Clin Epidemiol. 2020 Oct;126:37-44. doi: 10.1016/j.jclinepi.2020.06.015. Epub 2020 Jun 18.
The objective of the study is to assess the interrater reliability (IRR) and usability of the revised Cochrane risk of bias tool for randomized trials (RoB 2).
This is a cross-sectional study. Four raters independently applied RoB 2 on the primary outcome of a random sample of individually randomized parallel-group trials (randomized controlled trials (RCTs)). We calculated the Fleiss' kappa for multiple raters, the time needed to complete the tool, and discussed the application of RoB 2 to identify difficulties and reasons for disagreement.
A total of 70 outcomes from 70 RCTs were included. IRR was slight for overall judgment (IRR 0.16, 95% confidence interval (CI) 0.08-0.24); individual domain analysis gave IRR as moderate for "randomization process" (IRR 0.45, 95% CI 0.37-0.53), slight for "deviations from intended intervention" for RCTs assessing the effect of the assignment to an intervention (IRR 0.04, 95% CI -0.06 to 0.14), fair for those assessing the effect of adhering (IRR 0.21, 95% CI 0.11-0.31), and fair for the other domains, ranging from 0.22 (95% CI 0.14-0.30) for "missing outcome data" to 0.30 (95% CI 0.22-0.38) for "selection of reported results". Mean time to apply the tool was 28 minutes (standard deviation 13.4) per study outcome. The main difficulties were due to poor knowledge of the subject matter of primary studies, new terminology, different approaches for some domains compared with the previous tool, and way of formulating signaling questions.
RoB 2 is a detailed and comprehensive tool but difficult and demanding, even for raters with substantial expertise in systematic reviews. Calibration exercises and intensive training are needed before its application, to improve reliability.
本研究旨在评估修订后的 Cochrane 偏倚风险工具（RoB 2）用于随机试验的评分者间信度（IRR）和可用性。
这是一项横断面研究。四名评分者独立应用 RoB 2 于个体随机平行组试验（随机对照试验（RCT））的随机抽样主要结局。我们计算了多位评分者的 Fleiss' kappa，完成工具所需的时间，并讨论了应用 RoB 2 以识别困难和意见分歧的原因。
共纳入 70 项 RCT 的 70 个结局。整体判断的 IRR 为轻度（IRR 0.16，95%置信区间（CI）0.08-0.24）；个别领域分析表明，“随机化过程”的 IRR 为中度（IRR 0.45，95%CI 0.37-0.53），对于评估分配到干预措施效果的 RCT，“偏离意向干预”的 IRR 为轻度（IRR 0.04，95%CI -0.06 至 0.14），对于评估遵守情况效果的 RCT，IRR 为适度（IRR 0.21，95%CI 0.11-0.31），其他领域的 IRR 为适度，范围从“缺失结局数据”的 0.22（95%CI 0.14-0.30）到“选择报告结果”的 0.30（95%CI 0.22-0.38）。每项研究结局应用工具的平均时间为 28 分钟（标准差 13.4）。主要困难归因于对原始研究主题的知识不足、新术语、与前一工具相比，某些领域的方法不同以及信号问题的表述方式。
RoB 2 是一种详细全面的工具，但即使对于系统评价方面有丰富专业知识的评分者来说，也具有难度和要求。在应用之前需要进行校准练习和强化培训，以提高可靠性。
