Noel-Storr Anna, Gartlehner Gerald, Dooley Gordon, Persad Emma, Nussbaumer-Streit Barbara
Cochrane Dementia and Cognitive Improvement Group, Radcliffe Department of Medicine, University of Oxford, Oxford, UK.
Department for Evidence-Based Medicine and Evaluation, Danube University Krems, Krems an der Donau, Austria.
Res Synth Methods. 2022 Sep;13(5):585-594. doi: 10.1002/jrsm.1559. Epub 2022 Apr 25.
Utilisation of crowdsourcing within evidence synthesis has increased over the last decade. Crowdsourcing platform Cochrane Crowd has engaged a global community of 22,000 people from 170 countries. The COVID-19 pandemic presented an opportunity to engage the community and keep up with the exponential output of COVID-19 research.
To test whether a crowd could accurately assess study eligibility for reviews under time constraints.
time taken to complete each task, time to produce required training modules, crowd sensitivity, specificity and crowd consensus.
We created four crowd tasks, corresponding to four Cochrane COVID-19 Rapid Reviews. The search results of each were uploaded and an interactive training module was developed for each task. Contributors who had participated in another COVID-19 task were invited to participate. Each task was live for 48-h. The final inclusion and exclusion decisions made by the core author team were used as the reference standard.
Across all four reviews 14,299 records were screened by 101 crowd contributors. The crowd completed each screening task within 48-h for three reviews and in 52 h for one. Sensitivity ranged from 94% to 100%. Four studies, out of a total of 109, were incorrectly rejected by the crowd. However, their absence ultimately would not have altered the conclusions of the reviews. Crowd consensus ranged from 71% to 92% across the four reviews.
Crowdsourcing can play a valuable role in study identification and offers willing contributors the opportunity to help identify COVID-19 research for rapid evidence syntheses.
在过去十年中，众包在证据综合中的应用有所增加。众包平台Cochrane Crowd吸引了来自170个国家的22000人的全球社区。新冠疫情提供了一个让该社区参与并跟上新冠研究指数级产出的机会。
测试一群人在时间限制下能否准确评估纳入综述的研究的合格性。
完成每项任务所需时间、制作所需培训模块的时间、人群敏感性、特异性和人群共识。
我们创建了四项众包任务，对应四项Cochrane新冠快速综述。上传每项任务的检索结果，并为每项任务开发了一个交互式培训模块。邀请参与过另一项新冠任务的贡献者参与。每项任务持续48小时。核心作者团队做出的最终纳入和排除决定用作参考标准。
在所有四项综述中，101名众包贡献者筛选了14299条记录。人群在48小时内完成了三项综述的每项筛选任务，一项在52小时内完成。敏感性范围为94%至100%。在总共109项研究中，有四项被人群错误拒绝。然而，它们的缺失最终不会改变综述的结论。四项综述的人群共识范围为71%至92%。
众包在研究识别中可以发挥重要作用，并为愿意的贡献者提供机会，帮助识别用于快速证据综合的新冠研究。
