Center for Bioinformatics and Computational Biology, University of Delaware, Newark, DE 19711, USA.
Database (Oxford). 2013 Jan 17;2013:bas056. doi: 10.1093/database/bas056. Print 2013.
In many databases, biocuration primarily involves literature curation, which usually involves retrieving relevant articles, extracting information that will translate into annotations and identifying new incoming literature. As the volume of biological literature increases, the use of text mining to assist in biocuration becomes increasingly relevant. A number of groups have developed tools for text mining from a computer science/linguistics perspective, and there are many initiatives to curate some aspect of biology from the literature. Some biocuration efforts already make use of a text mining tool, but there have not been many broad-based systematic efforts to study which aspects of a text mining tool contribute to its usefulness for a curation task. Here, we report on an effort to bring together text mining tool developers and database biocurators to test the utility and usability of tools. Six text mining systems presenting diverse biocuration tasks participated in a formal evaluation, and appropriate biocurators were recruited for testing. The performance results from this evaluation indicate that some of the systems were able to improve efficiency of curation by speeding up the curation task significantly (∼1.7- to 2.5-fold) over manual curation. In addition, some of the systems were able to improve annotation accuracy when compared with the performance on the manually curated set. In terms of inter-annotator agreement, the factors that contributed to significant differences for some of the systems included the expertise of the biocurator on the given curation task, the inherent difficulty of the curation and attention to annotation guidelines. After the task, annotators were asked to complete a survey to help identify strengths and weaknesses of the various systems. The analysis of this survey highlights how important task completion is to the biocurators' overall experience of a system, regardless of the system's high score on design, learnability and usability. In addition, strategies to refine the annotation guidelines and systems documentation, to adapt the tools to the needs and query types the end user might have and to evaluate performance in terms of efficiency, user interface, result export and traditional evaluation metrics have been analyzed during this task. This analysis will help to plan for a more intense study in BioCreative IV.
在许多数据库中，生物注释主要涉及文献注释，通常包括检索相关文章、提取可转化为注释的信息以及识别新的文献。随着生物文献量的增加，使用文本挖掘来辅助生物注释变得越来越重要。许多团队从计算机科学/语言学的角度开发了文本挖掘工具，并且有许多倡议从文献中注释生物学的某个方面。一些生物注释工作已经使用了文本挖掘工具，但很少有基于广泛的系统努力来研究文本挖掘工具的哪些方面有助于其完成注释任务。在这里，我们报告了一项努力，即将文本挖掘工具开发人员和数据库生物注释人员聚集在一起，以测试工具的实用性和可用性。六个呈现不同生物注释任务的文本挖掘系统参与了正式评估，并招募了适当的生物注释人员进行测试。该评估的性能结果表明，一些系统能够通过显著提高注释效率（手动注释的约 1.7-2.5 倍）来加速注释任务。此外，与手动注释集的性能相比，一些系统能够提高注释准确性。在注释者之间的一致性方面，一些系统产生显著差异的因素包括注释人员在给定注释任务上的专业知识、注释的固有难度和对注释指南的关注。任务完成后，注释人员被要求完成一项调查，以帮助识别各种系统的优缺点。对该调查的分析突出了任务完成对生物注释人员对系统整体体验的重要性，而不论系统在设计、可学习性和可用性方面的得分如何。此外，还分析了细化注释指南和系统文档、使工具适应最终用户可能具有的需求和查询类型以及根据效率、用户界面、结果导出和传统评估指标评估性能的策略。这一分析将有助于在 BioCreative IV 中进行更深入的研究。
