Department of Emergency Medicine, Yale School of Medicine, New Haven, Connecticut, USA.
Section for Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, Connecticut, USA.
Acad Emerg Med. 2024 Jun;31(6):599-610. doi: 10.1111/acem.14883. Epub 2024 Apr 3.
Natural language processing (NLP) tools including recently developed large language models (LLMs) have myriad potential applications in medical care and research, including the efficient labeling and classification of unstructured text such as electronic health record (EHR) notes. This opens the door to large-scale projects that rely on variables that are not typically recorded in a structured form, such as patient signs and symptoms.
This study is designed to acquaint the emergency medicine research community with the foundational elements of NLP, highlighting essential terminology, annotation methodologies, and the intricacies involved in training and evaluating NLP models. Symptom characterization is critical to urinary tract infection (UTI) diagnosis, but identification of symptoms from the EHR has historically been challenging, limiting large-scale research, public health surveillance, and EHR-based clinical decision support. We therefore developed and compared two NLP models to identify UTI symptoms from unstructured emergency department (ED) notes.
The study population consisted of patients aged ≥ 18 who presented to an ED in a northeastern U.S. health system between June 2013 and August 2021 and had a urinalysis performed. We annotated a random subset of 1250 ED clinician notes from these visits for a list of 17 UTI symptoms. We then developed two task-specific LLMs to perform the task of named entity recognition: a convolutional neural network-based model (SpaCy) and a transformer-based model designed to process longer documents (Clinical Longformer). Models were trained on 1000 notes and tested on a holdout set of 250 notes. We compared model performance (precision, recall, F1 measure) at identifying the presence or absence of UTI symptoms at the note level.
A total of 8135 entities were identified in 1250 notes; 83.6% of notes included at least one entity. Overall F1 measure for note-level symptom identification weighted by entity frequency was 0.84 for the SpaCy model and 0.88 for the Longformer model. F1 measure for identifying presence or absence of any UTI symptom in a clinical note was 0.96 (232/250 correctly classified) for the SpaCy model and 0.98 (240/250 correctly classified) for the Longformer model.
The study demonstrated the utility of LLMs and transformer-based models in particular for extracting UTI symptoms from unstructured ED clinical notes; models were highly accurate for detecting the presence or absence of any UTI symptom on the note level, with variable performance for individual symptoms.
自然语言处理 (NLP) 工具包括最近开发的大型语言模型 (LLM)，在医疗保健和研究中有无数潜在的应用，包括对电子健康记录 (EHR) 等非结构化文本进行高效的标记和分类。这为依赖于通常不以结构化形式记录的变量的大型项目打开了大门，例如患者的体征和症状。
本研究旨在使急诊医学研究界了解 NLP 的基础要素，重点介绍基本术语、注释方法以及培训和评估 NLP 模型所涉及的复杂性。症状特征对于尿路感染 (UTI) 的诊断至关重要，但从 EHR 中识别症状一直具有挑战性，限制了大规模研究、公共卫生监测和基于 EHR 的临床决策支持。因此，我们开发并比较了两种 NLP 模型，以从非结构化急诊 (ED) 记录中识别 UTI 症状。
研究人群包括 2013 年 6 月至 2021 年 8 月期间在美国东北部医疗系统就诊的年龄≥18 岁的患者，并进行了尿液分析。我们为这些就诊的 1250 名临床医生的随机亚组注释了一份 17 种 UTI 症状列表。然后，我们开发了两种专门用于执行命名实体识别任务的特定于任务的 LLM：基于卷积神经网络的模型 (SpaCy) 和专为处理较长文档而设计的基于转换器的模型 (Clinical Longformer)。模型在 1000 条记录上进行训练，并在 250 条保留记录上进行测试。我们比较了模型在识别笔记级 UTI 症状存在或不存在时的性能（精度、召回率、F1 度量）。
在 1250 条记录中总共识别出 8135 个实体；83.6%的记录至少包含一个实体。基于实体频率加权的笔记级症状识别的整体 F1 度量对于 SpaCy 模型为 0.84，对于 Longformer 模型为 0.88。SpaCy 模型在临床记录中识别任何 UTI 症状存在或不存在的 F1 度量为 0.96（232/250 正确分类），Longformer 模型为 0.98（240/250 正确分类）。
该研究证明了 LLM 和基于转换器的模型在提取非结构化 ED 临床记录中的 UTI 症状方面的实用性；模型在检测记录中任何 UTI 症状的存在或不存在方面非常准确，个别症状的性能存在差异。
