Wang Liqin, Novoa-Laurentiev John, Cook Claire, Srivatsan Shruthi, Hua Yining, Yang Jie, Miloslavsky Eli, Choi Hyon K, Zhou Li, Wallace Zachary S
Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, USA.
Rheumatology and Allergy Clinical Epidemiology Research Center and Division of Rheumatology, Allergy, and Immunology, and Mongan Institute, Department of Medicine, Massachusetts General Hospital, Boston, MA, USA.
medRxiv. 2024 Jun 10:2024.06.09.24308603. doi: 10.1101/2024.06.09.24308603.
ANCA-associated vasculitis (AAV) is a rare but serious disease. Traditional case-identification methods using claims data can be time-intensive and may miss important subgroups. We hypothesized that a deep learning model analyzing electronic health records (EHR) can more accurately identify AAV cases.
We examined the Mass General Brigham (MGB) repository of clinical documentation from 12/1/1979 to 5/11/2021, using expert-curated keywords and ICD codes to identify a large cohort of potential AAV cases. Three labeled datasets (I, II, III) were created, each containing note sections. We trained and evaluated a range of machine learning and deep learning algorithms for note-level classification, using metrics like positive predictive value (PPV), sensitivity, F-score, area under the receiver operating characteristic curve (AUROC), and area under the precision and recall curve (AUPRC). The deep learning model was further evaluated for its ability to classify AAV cases at the patient-level, compared with rule-based algorithms in 2,000 randomly chosen samples.
Datasets I, II, and III comprised 6,000, 3,008, and 7,500 note sections, respectively. Deep learning achieved the highest AUROC in all three datasets, with scores of 0.983, 0.991, and 0.991. The deep learning approach also had among the highest PPVs across the three datasets (0.941, 0.954, and 0.800, respectively). In a test cohort of 2,000 cases, the deep learning model achieved a PPV of 0.262 and an estimated sensitivity of 0.975. Compared to the best rule-based algorithm, the deep learning model identified six additional AAV cases, representing 13% of the total.
The deep learning model effectively classifies clinical note sections for AAV diagnosis. Its application to EHR notes can potentially uncover additional cases missed by traditional rule-based methods.
抗中性粒细胞胞浆抗体相关性血管炎（AAV）是一种罕见但严重的疾病。使用索赔数据的传统病例识别方法可能耗时较长，并且可能遗漏重要的亚组。我们假设，分析电子健康记录（EHR）的深度学习模型能够更准确地识别AAV病例。
我们检查了1979年12月1日至2021年5月11日期间麻省总医院布莱根分院（MGB）的临床文档库，使用专家整理的关键词和国际疾病分类代码来识别一大批潜在的AAV病例。创建了三个标记数据集（I、II、III），每个数据集都包含注释部分。我们使用阳性预测值（PPV）、灵敏度、F值、受试者工作特征曲线下面积（AUROC）以及精度和召回率曲线下面积（AUPRC）等指标，对一系列用于注释级分类的机器学习和深度学习算法进行了训练和评估。在2000个随机选择的样本中，将深度学习模型在患者层面分类AAV病例的能力与基于规则的算法进行比较，对其进行了进一步评估。
数据集I、II和III分别包含6000个、3008个和7500个注释部分。深度学习在所有三个数据集中的AUROC最高，得分分别为0.983、0.991和0.991。深度学习方法在三个数据集中的PPV也处于最高水平（分别为0.941、0.954和0.800）。在一个2000例的测试队列中，深度学习模型的PPV为0.262，估计灵敏度为0.975。与最佳的基于规则的算法相比，深度学习模型额外识别出6例AAV病例，占总数的13%。
深度学习模型有效地对用于AAV诊断的临床注释部分进行分类。将其应用于EHR注释有可能发现传统基于规则的方法遗漏的其他病例。
