一种用于检测更新系统评价信号的方法的评估

Assessment of a method to detect signals for updating systematic reviews.

作者信息

Shekelle Paul G, Motala Aneesa, Johnsen Breanne, Newberry Sydne J

机构信息

RAND Corporation, RAND Health, 1776 Main Street, Santa Monica, CA 90407, USA.

出版信息

Syst Rev. 2014 Feb 14;3:13. doi: 10.1186/2046-4053-3-13.

DOI:10.1186/2046-4053-3-13

PMID:24529068

原文链接:https://pmc.ncbi.nlm.nih.gov/articles/PMC3937021/

Abstract

BACKGROUND

Systematic reviews are a cornerstone of evidence-based medicine but are useful only if up-to-date. Methods for detecting signals of when a systematic review needs updating have face validity, but no proposed method has had an assessment of predictive validity performed.

METHODS

The AHRQ Comparative Effectiveness Review program had produced 13 comparative effectiveness reviews (CERs), a subcategory of systematic reviews, by 2009, 11 of which were assessed in 2009 using a surveillance system to determine the degree to which individual conclusions were out of date and to assign a priority for updating each report. Four CERs were judged to be a high priority for updating, four CERs were judged to be medium priority for updating, and three CERs were judged to be low priority for updating. AHRQ then commissioned full update reviews for 9 of these 11 CERs. Where possible, we matched the original conclusions with their corresponding conclusions in the update reports, and compared the congruence between these pairs with our original predictions about which conclusions in each CER remained valid. We then classified the concordance of each pair as good, fair, or poor. We also made a summary determination of the priority for updating each CER based on the actual changes in conclusions in the updated report, and compared these determinations with the earlier assessments of priority.

RESULTS

The 9 CERs included 149 individual conclusions, 84% with matches in the update reports. Across reports, 83% of matched conclusions had good concordance, and 99% had good or fair concordance. The one instance of poor concordance was partially attributable to the publication of new evidence after the surveillance signal searches had been done. Both CERs originally judged as being low priority for updating had no substantive changes to their conclusions in the actual updated report. The agreement on overall priority for updating between prediction and actual changes to conclusions was Kappa = 0.74.

CONCLUSIONS

These results provide some support for the validity of a surveillance system for detecting signals indicating when a systematic review needs updating.

摘要

背景

系统评价是循证医学的基石，但只有及时更新才有用。检测系统评价何时需要更新的信号的方法具有表面效度，但尚无任何提议的方法进行过预测效度评估。

方法

到2009年，美国医疗保健研究与质量局（AHRQ）比较效果评价项目已产生了13项比较效果评价（CER），这是系统评价的一个子类别，其中11项在2009年使用一个监测系统进行了评估，以确定各个结论过时的程度，并为更新每份报告确定优先级。4项CER被判定为更新的高优先级，4项CER被判定为更新的中优先级，3项CER被判定为更新的低优先级。然后，AHRQ委托对这11项CER中的9项进行全面更新评价。在可能的情况下，我们将原始结论与其在更新报告中的相应结论进行匹配，并将这些配对之间的一致性与我们对每项CER中哪些结论仍然有效的原始预测进行比较。然后，我们将每对的一致性分类为好、一般或差。我们还根据更新报告中结论的实际变化对每项CER的更新优先级进行了总结判定，并将这些判定与早期的优先级评估进行了比较。

结果

这9项CER包含149个单独结论，其中84%在更新报告中有匹配项。在各报告中，83%的匹配结论具有良好的一致性，99%具有良好或一般的一致性。一致性差的一个实例部分归因于在监测信号搜索完成后新证据的发表。最初被判定为更新低优先级的两项CER在实际更新报告中的结论没有实质性变化。预测与结论实际变化之间在更新总体优先级上的一致性为Kappa = 0.74。