Department of Computer Science, Aalto University, Espoo, 00076, Finland.
Comput Biol Med. 2021 Dec;139:104998. doi: 10.1016/j.compbiomed.2021.104998. Epub 2021 Oct 30.
Unsupervised pretraining is an integral part of many natural language processing systems, and transfer learning with language models has achieved remarkable results in downstream tasks. In the clinical application of medical code assignment, diagnosis and procedure codes are inferred from lengthy clinical notes such as hospital discharge summaries. However, it is not clear if pretrained models are useful for medical code prediction without further architecture engineering. This paper conducts a comprehensive quantitative analysis of various contextualized language models' performances, pretrained in different domains, for medical code assignment from clinical notes. We propose a hierarchical fine-tuning architecture to capture interactions between distant words and adopt label-wise attention to exploit label information. Contrary to current trends, we demonstrate that a carefully trained classical CNN outperforms attention-based models on a MIMIC-III subset with frequent codes. Our empirical findings suggest directions for building robust medical code assignment models.
无监督预训练是许多自然语言处理系统的一个组成部分，语言模型的迁移学习在下游任务中取得了显著的成果。在医疗编码分配的临床应用中，诊断和程序代码是从医院出院总结等冗长的临床记录中推断出来的。然而，在没有进一步进行架构工程的情况下，预训练模型对于医疗编码预测是否有用尚不清楚。本文对不同领域预训练的各种上下文语言模型在从临床记录中分配医疗代码方面的性能进行了全面的定量分析。我们提出了一种分层微调架构，以捕获远距离单词之间的交互，并采用标签分类注意力机制来利用标签信息。与当前的趋势相反，我们证明了在包含频繁代码的 MIMIC-III 子集中，经过精心训练的经典 CNN 优于基于注意力的模型。我们的实证研究结果为构建稳健的医疗编码分配模型提供了方向。
