Optum Insight, Optum, Eden Prairie, MN.
Departments of Neurology and Population Health, New York University Grossman School of Medicine, New York, NY.
JCO Clin Cancer Inform. 2024 Sep;8:e2300099. doi: 10.1200/CCI.23.00099.
Limited studies have used natural language processing (NLP) in the context of non-small cell lung cancer (NSCLC). This study aimed to validate the application of an NLP model to an NSCLC cohort by extracting NSCLC concepts from free-text medical notes and converting them to structured, interpretable data.
Patients with a lung neoplasm, NSCLC histology, and treatment information in their notes were selected from a repository of over 27 million patients. From these, 200 were randomly selected for this study with the longest and the most recent note included for each patient. An NLP model developed and validated on a large solid and blood cancer oncology cohort was applied to this NSCLC cohort. Two certified tumor registrars and a curator abstracted concepts from the notes: neoplasm, histology, stage, TNM values, and metastasis sites. This manually abstracted gold standard was compared with the NLP model output. Precision and recall scores were calculated.
The NLP model extracted the NSCLC concepts with excellent precision and recall with the following scores, respectively: Lung neoplasm 100% and 100%, NSCLC histology 99% and 88%, histology correctly linked to neoplasm 98% and 79%, stage value 98.8% and 92%, stage TNM value 93% and 98%, and metastasis site 97% and 89%. High precision is related to a low number of false positives, and therefore, extracted concepts are likely accurate. High recall indicates that the model captured most of the desired concepts.
This study validates that Optum's oncology NLP model has high precision and recall with clinical real-world data and is a reliable model to support research studies and clinical trials. This validation study shows that our nonspecific solid tumor and blood cancer oncology model is generalizable to successfully extract clinical information from specific cancer cohorts.
已有少量研究在非小细胞肺癌（NSCLC）背景下使用自然语言处理（NLP）。本研究旨在通过从医疗记录的自由文本中提取 NSCLC 概念并将其转换为结构化、可解释的数据，验证 NLP 模型在 NSCLC 队列中的应用。
从超过 2700 万患者的存储库中选择记录有肺部肿瘤、NSCLC 组织学和治疗信息的患者。在此基础上，本研究随机选择了 200 名患者，每位患者都包含最长和最近的记录。将在大型实体瘤和血液肿瘤队列上开发和验证的 NLP 模型应用于该 NSCLC 队列。两位经过认证的肿瘤登记员和一位管理员从记录中提取概念：肿瘤、组织学、分期、TNM 值和转移部位。将此手动提取的金标准与 NLP 模型输出进行比较。计算精度和召回率得分。
NLP 模型以优异的精度和召回率提取 NSCLC 概念，分别为：肺部肿瘤 100%和 100%、非小细胞肺癌组织学 99%和 88%、组织学正确链接到肿瘤 98%和 79%、分期值 98.8%和 92%、分期 TNM 值 93%和 98%、转移部位 97%和 89%。高精确度与低假阳性率有关，因此提取的概念很可能是准确的。高召回率表明模型捕获了大部分所需的概念。
本研究验证了 Optum 的肿瘤 NLP 模型在临床真实世界数据中具有高精度和高召回率，是支持研究和临床试验的可靠模型。该验证研究表明，我们的非特异性实体瘤和血液肿瘤肿瘤学模型具有通用性，可以成功地从特定癌症队列中提取临床信息。
