Netcompany A/S, Aarhus C, Denmark.
Health Services, Policy and Practice, Brown University, Providence, RI, USA.
Res Synth Methods. 2017 Sep;8(3):366-386. doi: 10.1002/jrsm.1252. Epub 2017 Jul 4.
Systematic reviews are increasingly used to inform health care decisions, but are expensive to produce. We explore the use of crowdsourcing (distributing tasks to untrained workers via the web) to reduce the cost of screening citations. We used Amazon Mechanical Turk as our platform and 4 previously conducted systematic reviews as examples. For each citation, workers answered 4 or 5 questions that were equivalent to the eligibility criteria. We aggregated responses from multiple workers into an overall decision to include or exclude the citation using 1 of 9 algorithms and compared the performance of these algorithms to the corresponding decisions of trained experts. The most inclusive algorithm (designating a citation as relevant if any worker did) identified 95% to 99% of the citations that were ultimately included in the reviews while excluding 68% to 82% of irrelevant citations. Other algorithms increased the fraction of irrelevant articles excluded at some cost to the inclusion of relevant studies. Crowdworkers completed screening in 4 to 17 days, costing $460 to $2220, a cost reduction of up to 88% compared to trained experts. Crowdsourcing may represent a useful approach to reducing the cost of identifying literature for systematic reviews.
系统评价越来越多地被用于为医疗保健决策提供信息，但制作成本很高。我们探讨了使用众包（通过网络将任务分配给未经培训的工人）来降低筛选引文的成本。我们使用亚马逊 Mechanical Turk 作为我们的平台，并以 4 项先前进行的系统评价为例。对于每个引文，工人回答 4 到 5 个相当于资格标准的问题。我们使用 9 种算法之一，将来自多个工人的回答汇总为纳入或排除引文的总体决定，并将这些算法的性能与经过培训的专家的相应决策进行比较。最具包容性的算法（如果有任何工人认为引用相关，则将其指定为相关）确定了 95%到 99%的最终纳入评价的引文，同时排除了 68%到 82%的不相关引文。其他算法以增加相关研究纳入的代价，排除了更多不相关的文章。众包工人在 4 到 17 天内完成筛选，成本为 460 至 2220 美元，与经过培训的专家相比，成本降低了高达 88%。众包可能是一种有用的方法，可以降低系统评价中识别文献的成本。
