Crichton Gamal, Pyysalo Sampo, Chiu Billy, Korhonen Anna
Language Technology Laboratory, DTAL, University of Cambridge, 9 West Road, Cambridge, CB39DB, UK.
BMC Bioinformatics. 2017 Aug 15;18(1):368. doi: 10.1186/s12859-017-1776-8.
Named Entity Recognition (NER) is a key task in biomedical text mining. Accurate NER systems require task-specific, manually-annotated datasets, which are expensive to develop and thus limited in size. Since such datasets contain related but different information, an interesting question is whether it might be possible to use them together to improve NER performance. To investigate this, we develop supervised, multi-task, convolutional neural network models and apply them to a large number of varied existing biomedical named entity datasets. Additionally, we investigated the effect of dataset size on performance in both single- and multi-task settings.
We present a single-task model for NER, a Multi-output multi-task model and a Dependent multi-task model. We apply the three models to 15 biomedical datasets containing multiple named entities including Anatomy, Chemical, Disease, Gene/Protein and Species. Each dataset represent a task. The results from the single-task model and the multi-task models are then compared for evidence of benefits from Multi-task Learning. With the Multi-output multi-task model we observed an average F-score improvement of 0.8% when compared to the single-task model from an average baseline of 78.4%. Although there was a significant drop in performance on one dataset, performance improves significantly for five datasets by up to 6.3%. For the Dependent multi-task model we observed an average improvement of 0.4% when compared to the single-task model. There were no significant drops in performance on any dataset, and performance improves significantly for six datasets by up to 1.1%. The dataset size experiments found that as dataset size decreased, the multi-output model's performance increased compared to the single-task model's. Using 50, 25 and 10% of the training data resulted in an average drop of approximately 3.4, 8 and 16.7% respectively for the single-task model but approximately 0.2, 3.0 and 9.8% for the multi-task model.
Our results show that, on average, the multi-task models produced better NER results than the single-task models trained on a single NER dataset. We also found that Multi-task Learning is beneficial for small datasets. Across the various settings the improvements are significant, demonstrating the benefit of Multi-task Learning for this task.
命名实体识别（NER）是生物医学文本挖掘中的一项关键任务。准确的NER系统需要特定任务的、人工标注的数据集，而开发这些数据集成本高昂，因此规模有限。由于此类数据集包含相关但不同的信息，一个有趣的问题是，是否有可能将它们一起使用以提高NER性能。为了研究这一点，我们开发了有监督的多任务卷积神经网络模型，并将其应用于大量多样的现有生物医学命名实体数据集。此外，我们还研究了数据集大小在单任务和多任务设置中对性能的影响。
我们提出了一个用于NER的单任务模型、一个多输出多任务模型和一个依赖多任务模型。我们将这三个模型应用于15个生物医学数据集，这些数据集包含多个命名实体，包括解剖学、化学物质、疾病、基因/蛋白质和物种。每个数据集代表一项任务。然后比较单任务模型和多任务模型的结果，以证明多任务学习的益处。使用多输出多任务模型时，与单任务模型相比，我们观察到平均F值从78.4%的平均基线提高了0.8%。虽然在一个数据集上性能有显著下降，但在五个数据集上性能显著提高，最高可达6.3%。对于依赖多任务模型，与单任务模型相比，我们观察到平均提高了0.4%。在任何数据集上性能都没有显著下降，并且在六个数据集上性能显著提高，最高可达1.1%。数据集大小实验发现，随着数据集大小的减小，与单任务模型相比，多输出模型的性能有所提高。使用50%、25%和10%的训练数据时，单任务模型的平均下降分别约为3.4%、8%和16.7%，而多任务模型的平均下降分别约为0.2%、3.0%和9.8%。
我们的结果表明，平均而言，多任务模型比在单个NER数据集上训练的单任务模型产生了更好的NER结果。我们还发现多任务学习对小数据集有益。在各种设置下，改进都很显著，证明了多任务学习对这项任务的益处。
