Computational Health Informatics Program, Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts, USA.
School of Information, University of Arizona, Tucson, Arizona, USA.
J Am Med Inform Assoc. 2020 Apr 1;27(4):584-591. doi: 10.1093/jamia/ocaa001.
Classifying whether concepts in an unstructured clinical text are negated is an important unsolved task. New domain adaptation and transfer learning methods can potentially address this issue.
We examine neural unsupervised domain adaptation methods, introducing a novel combination of domain adaptation with transformer-based transfer learning methods to improve negation detection. We also want to better understand the interaction between the widely used bidirectional encoder representations from transformers (BERT) system and domain adaptation methods.
We use 4 clinical text datasets that are annotated with negation status. We evaluate a neural unsupervised domain adaptation algorithm and BERT, a transformer-based model that is pretrained on massive general text datasets. We develop an extension to BERT that uses domain adversarial training, a neural domain adaptation method that adds an objective to the negation task, that the classifier should not be able to distinguish between instances from 2 different domains.
The domain adaptation methods we describe show positive results, but, on average, the best performance is obtained by plain BERT (without the extension). We provide evidence that the gains from BERT are likely not additive with the gains from domain adaptation.
Our results suggest that, at least for the task of clinical negation detection, BERT subsumes domain adaptation, implying that BERT is already learning very general representations of negation phenomena such that fine-tuning even on a specific corpus does not lead to much overfitting.
Despite being trained on nonclinical text, the large training sets of models like BERT lead to large gains in performance for the clinical negation detection task.
对非结构化临床文本中的概念进行否定分类是一个尚未解决的重要问题。新的领域自适应和迁移学习方法可能会解决这个问题。
我们研究了神经无监督领域自适应方法，引入了一种新的领域自适应与基于转换器的迁移学习方法的组合，以提高否定检测的效果。我们还希望更好地理解广泛使用的双向转换器表示（BERT）系统和领域自适应方法之间的相互作用。
我们使用 4 个具有否定状态注释的临床文本数据集。我们评估了一种神经无监督领域自适应算法和 BERT，这是一种基于转换器的模型，在大规模通用文本数据集上进行了预训练。我们开发了 BERT 的扩展版本，使用了对抗训练的领域自适应方法，该方法为否定任务添加了一个目标，即分类器不应能够区分来自 2 个不同领域的实例。
我们描述的领域自适应方法显示出了积极的结果，但平均而言，最好的性能是由普通的 BERT（没有扩展）获得的。我们提供的证据表明，BERT 的收益与领域自适应的收益并非相加的关系。
我们的结果表明，至少对于临床否定检测任务，BERT 包含了领域自适应，这意味着 BERT 已经在学习否定现象的非常通用的表示，因此即使在特定语料库上进行微调也不会导致过度拟合。
尽管 BERT 是在非临床文本上进行训练的，但像 BERT 这样的大型训练集的模型在临床否定检测任务中表现出了很大的性能提升。
