National Centre for Text Mining, School of Computer Science, University of Manchester, Manchester, UK.
Cochrane Switzerland, Institute of Social and Preventive Medicine, Lausanne University Hospital, Lausanne, Switzerland.
Res Synth Methods. 2018 Sep;9(3):470-488. doi: 10.1002/jrsm.1311. Epub 2018 Jul 30.
Screening references is a time-consuming step necessary for systematic reviews and guideline development. Previous studies have shown that human effort can be reduced by using machine learning software to prioritise large reference collections such that most of the relevant references are identified before screening is completed. We describe and evaluate RobotAnalyst, a Web-based software system that combines text-mining and machine learning algorithms for organising references by their content and actively prioritising them based on a relevancy classification model trained and updated throughout the process. We report an evaluation over 22 reference collections (most are related to public health topics) screened using RobotAnalyst with a total of 43 610 abstract-level decisions. The number of references that needed to be screened to identify 95% of the abstract-level inclusions for the evidence review was reduced on 19 of the 22 collections. Significant gains over random sampling were achieved for all reviews conducted with active prioritisation, as compared with only two of five when prioritisation was not used. RobotAnalyst's descriptive clustering and topic modelling functionalities were also evaluated by public health analysts. Descriptive clustering provided more coherent organisation than topic modelling, and the content of the clusters was apparent to the users across a varying number of clusters. This is the first large-scale study using technology-assisted screening to perform new reviews, and the positive results provide empirical evidence that RobotAnalyst can accelerate the identification of relevant studies. The results also highlight the issue of user complacency and the need for a stopping criterion to realise the work savings.
筛选参考文献是系统评价和指南制定的一个耗时步骤。先前的研究表明，可以使用机器学习软件来优先处理大型参考文献集，从而在完成筛选之前确定大多数相关参考文献，从而减少人力投入。我们描述并评估了 RobotAnalyst，这是一个基于网络的软件系统，它结合了文本挖掘和机器学习算法，根据内容对参考文献进行组织，并根据在整个过程中训练和更新的相关性分类模型积极对其进行优先级排序。我们报告了在使用 RobotAnalyst 进行筛选的 22 个参考文献集（大多数与公共卫生主题相关）上进行的评估，共进行了 43610 次摘要级别的决策。在 22 个集合中的 19 个集合中，减少了需要筛选的参考文献数量，以识别证据综述中 95%的摘要级别的纳入内容。与不使用优先级排序时的 5 次中的 2 次相比，所有使用主动优先级排序进行的综述都取得了显著的收益。公共卫生分析师还评估了 RobotAnalyst 的描述性聚类和主题建模功能。描述性聚类提供了比主题建模更连贯的组织，并且用户可以在不同数量的聚类中明显看出聚类的内容。这是第一项使用技术辅助筛选来进行新综述的大规模研究，积极的结果提供了经验证据，证明 RobotAnalyst 可以加速相关研究的识别。结果还突出了用户自满的问题，需要一个停止标准来实现工作节省。
