Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK.
Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK.
J Clin Epidemiol. 2024 Jul;171:111370. doi: 10.1016/j.jclinepi.2024.111370. Epub 2024 Apr 24.
To review the findings of studies that have evaluated the design and/or usability of key risk of bias (RoB) tools for the assessment of RoB in primary studies, as categorized by the Library of Assessment Tools and InsTruments Used to assess Data validity in Evidence Synthesis Network (a searchable library of RoB tools for evidence synthesis): Prediction model Risk Of Bias ASessment Tool (PROBAST) , Risk of Bias-2 (RoB2), Risk Of Bias In Non-randomised Studies of Interventions (ROBINS-I), Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2), Quality Assessment of Diagnostic Accuracy Studies-Comparative (QUADAS-C), Quality Assessment of Prognostic Accuracy Studies (QUAPAS), Risk Of Bias in Non-randomised Studies of Exposures (ROBINS-E), and the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) RoB checklist.
Systematic review of methodological studies. We conducted a forward citation search from the primary report of each tool, to identify primary studies that aimed to evaluate the design and/or usability of the tool. Two reviewers assessed studies for inclusion. We extracted tool features into Microsoft Word and used NVivo for document analysis, comprising a mix of deductive and inductive approaches. We summarized findings within each tool and explored common findings across tools.
We identified 13 tool evaluations meeting our inclusion criteria: PROBAST (3), RoB2 (3), ROBINS-I (4), and QUADAS-2 (3). We identified no evaluations for the other tools. Evaluations varied in clinical topic area, methodology, approach to bias assessment, and tool user background. Some had limitations affecting generalizability. We identified common findings across tools for 6/14 themes: (1) challenging items (eg, RoB2/ROBINS-I "deviations from intended interventions" domain), (2) overall RoB judgment (concerns with overall risk calculation in PROBAST/ROBINS-I), (3) tool usability (concerns about complexity), (4) time to complete tool (varying demands on time, eg, depending on number of outcomes assessed), (5) user agreement (varied across tools), and (6) recommendations for future use (eg, piloting) and development (add intermediate domain answer to QUADAS-2/PROBAST; provide clearer guidance for all tools). Of the other eight themes, seven only had findings for the QUADAS-2 tool, limiting comparison across tools, and one ("reorganization of questions") had no findings.
Evaluations of key RoB tools have posited common challenges and recommendations for tool use and development. These findings may be helpful to people who use or develop RoB tools. Guidance is necessary to support the design and implementation of future RoB tool evaluations.
回顾评估初级研究中偏倚风险（RoB）的关键风险评估工具（ROB）设计和/或可用性的研究结果，这些工具按评估数据有效性的工具和仪器库（用于证据综合的 RoB 工具的可搜索库）进行分类：预测模型风险评估工具（PROBAST）、风险偏倚-2（RoB2）、干预措施非随机研究中的风险偏倚（ROBINS-I）、诊断准确性研究的质量评估-2（QUADAS-2）、诊断准确性研究的质量评估-比较（QUADAS-C）、预后准确性研究的质量评估（QUAPAS）、暴露非随机研究中的风险偏倚（ROBINS-E）和基于共识的健康测量仪器选择标准（COSMIN）RoB 检查表。
系统评价方法学研究。我们从每个工具的主要报告中进行了正向引用搜索，以确定旨在评估工具设计和/或可用性的初步研究。两名审查员评估了纳入标准的研究。我们将工具特征提取到 Microsoft Word 中，并使用 NVivo 进行文档分析，包括演绎和归纳方法的混合。我们总结了每个工具中的发现，并探讨了工具之间的共同发现。
我们确定了 13 项符合纳入标准的工具评估：PROBAST（3）、RoB2（3）、ROBINS-I（4）和 QUADAS-2（3）。我们没有发现其他工具的评估。评估在临床主题领域、方法学、偏差评估方法和工具使用者背景方面存在差异。其中一些评估存在影响推广性的局限性。我们在 6/14 个主题中确定了工具之间的共同发现：（1）具有挑战性的项目（例如，RoB2/ROBINS-I“偏离预期干预”领域）；（2）整体 RoB 判断（PROBAST/ROBINS-I 中对整体风险计算的关注）；（3）工具可用性（对复杂性的关注）；（4）完成工具的时间（时间要求不同，例如，取决于评估的结果数量）；（5）用户协议（工具之间存在差异）；（6）未来使用建议（例如，试点）和开发（向 QUADAS-2/PROBAST 添加中间域答案；为所有工具提供更清晰的指导）。其他八个主题中，七个仅对 QUADAS-2 工具存在发现，限制了工具之间的比较，一个主题（“问题重组”）没有发现。
对关键 RoB 工具的评估提出了工具使用和开发的共同挑战和建议。这些发现可能对使用或开发 RoB 工具的人有帮助。有必要提供指导，以支持未来 RoB 工具评估的设计和实施。
