Faculty of Health, School of Medicine, Institute for Research in Operative Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, 51109 Cologne, Germany.
Department of Surgery, University Hospital Split, Spinciceva 1, Split, Croatia.
J Clin Epidemiol. 2020 Apr;120:25-32. doi: 10.1016/j.jclinepi.2019.12.016. Epub 2019 Dec 19.
The objective of the study was to measure the level of agreement between Cochrane reviews of overlapping randomized controlled trials (RCTs) regarding risk-of-bias (RoB) judgments.
On November 5, 2017, the Cochrane Database of Systematic Reviews was searched for Cochrane reviews on tobacco. Reviews that included overlapping RCTs were included. RoB judgments were extracted from RoB tables using automated data scraping with manual verification and adjustments. Agreement between the reviews was calculated using Conger's generalized kappa coefficient (κ) and raw agreement (a).
We included 53 Cochrane reviews of 376 RCTs. For the RoB domain "random sequence generation," the level of agreement between the reviews was substantial with κ = 0.63 (95% confidence interval: 0.56 to 0.71; a = 0.80). There was slight-to-moderate agreement between the reviews regarding the domains "allocation concealment": κ = 0.51 (0.41 to 0.61), a = 0.75; "blinding": κ = 0.19 (0.02 to 0.37), a = 0.52; "blinding of outcome assessment": κ = 0.43 (0.14 to 0.72) a = 0.67; and "incomplete outcome data": κ = 0.15 (-0.03 to 0.32), a = 0.64. For "blinding of participants and personnel" and "selective reporting", κ could not be calculated. The raw agreement was 0.40 and 0.42, respectively.
The level of agreement between Cochrane reviews regarding RoB judgments ranged from slight to substantial depending on the RoB domain. Further investigations regarding reasons for variation and interventions to improve agreement are needed.
本研究旨在衡量重叠随机对照试验（RCT）的 Cochrane 评价在偏倚风险（RoB）判断方面的一致性程度。
2017 年 11 月 5 日，对 Cochrane 系统评价数据库中的烟草主题文献进行检索，纳入包含重叠 RCT 的 Cochrane 评价。采用自动化数据抓取并辅以人工验证和调整，从 RoB 表中提取 RoB 判断结果。使用 Conger 广义kappa 系数（κ）和原始一致性（a）计算评价间的一致性。
共纳入 53 项 Cochrane 评价，包含 376 项 RCT。在“随机序列生成”RoB 领域，评价间的一致性为高度一致，κ=0.63（95%置信区间：0.56 至 0.71；a=0.80）。在“分配隐藏”RoB 领域，评价间存在轻度至中度一致，κ=0.51（0.41 至 0.61），a=0.75；“盲法”κ=0.19（0.02 至 0.37），a=0.52；“结局评估盲法”κ=0.43（0.14 至 0.72），a=0.67；“结局数据不全”κ=0.15（-0.03 至 0.32），a=0.64。“参与者和实施者设盲”和“选择性报告”领域的 κ 无法计算，原始一致性分别为 0.40 和 0.42。
在 RoB 领域，Cochrane 评价间的一致性程度从轻度至高度不等。需要进一步研究变异的原因和改善一致性的干预措施。
