Institute for Research in Operative Medicine, Faculty of Health, School of Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, 51109, Cologne, Germany.
Department of Periodontology and Operative Dentistry, Faculty of Dentistry, University Hospital Münster, Münster, Germany.
BMC Med Res Methodol. 2021 Jul 29;21(1):157. doi: 10.1186/s12874-021-01342-6.
To develop and test an approach to test reproducibility of SRs.
Case study. We have developed an approach to test reproducibility retrospectively while focusing on the whole conduct of an SR instead of single steps of it. We replicated the literature searches and drew a 25% random sample followed by study selection, data extraction, and risk of bias (ROB) assessments performed by two reviewers independently. These results were compared narratively with the original review.
We were not able to fully reproduce the original search resulting in minor differences in the number of citations retrieved. The biggest disagreements were found in study selection. The most difficult section to be reproduced was the RoB assessment due to the lack of reporting clear criteria to support the judgement of RoB ratings, although agreement was still found to be satisfactory.
Our approach as well as other approaches needs to undergo testing and comparison in the future as the area of testing for reproducibility of SRs is still in its infancy.
开发并测试一种系统评价可重复性测试方法。
案例研究。我们开发了一种回顾性测试可重复性的方法，重点关注整个系统评价的实施过程，而不是单个步骤。我们复制了文献检索，并抽取了 25%的随机样本，然后由两名评审员独立进行研究选择、数据提取和偏倚风险（ROB）评估。这些结果与原始综述进行了叙述性比较。
我们无法完全复制原始检索，导致检索到的引文数量存在细微差异。最大的分歧出现在研究选择上。由于缺乏明确的报告标准来支持 ROB 评估的判断，因此可重复性最差的部分是 ROB 评估，尽管仍发现一致性令人满意。
我们的方法以及其他方法需要在未来进行测试和比较，因为系统评价可重复性测试领域仍处于起步阶段。
