使用医生记录来识别适合癫痫手术的候选者的机器学习模型的前瞻性验证。
Prospective validation of a machine learning model that uses provider notes to identify candidates for resective epilepsy surgery.
机构信息
Department of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio.
Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio.
出版信息
Epilepsia. 2020 Jan;61(1):39-48. doi: 10.1111/epi.16398. Epub 2019 Nov 29.
OBJECTIVE
Delay to resective epilepsy surgery results in avoidable disease burden and increased risk of mortality. The objective was to prospectively validate a natural language processing (NLP) application that uses provider notes to assign epilepsy surgery candidacy scores.
METHODS
The application was trained on notes from (1) patients with a diagnosis of epilepsy and a history of resective epilepsy surgery and (2) patients who were seizure-free without surgery. The testing set included all patients with unknown surgical candidacy status and an upcoming neurology visit. Training and testing sets were updated weekly for 1 year. One- to three-word phrases contained in patients' notes were used as features. Patients prospectively identified by the application as candidates for surgery were manually reviewed by two epileptologists. Performance metrics were defined by comparing NLP-derived surgical candidacy scores with surgical candidacy status from expert chart review.
RESULTS
The training set was updated weekly and included notes from a mean of 519 ± 67 patients. The area under the receiver operating characteristic curve (AUC) from 10-fold cross-validation was 0.90 ± 0.04 (range = 0.83-0.96) and improved by 0.002 per week (P < .001) as new patients were added to the training set. Of the 6395 patients who visited the neurology clinic, 4211 (67%) were evaluated by the model. The prospective AUC on this test set was 0.79 (95% confidence interval [CI] = 0.62-0.96). Using the optimal surgical candidacy score threshold, sensitivity was 0.80 (95% CI = 0.29-0.99), specificity was 0.77 (95% CI = 0.64-0.88), positive predictive value was 0.25 (95% CI = 0.07-0.52), and negative predictive value was 0.98 (95% CI = 0.87-1.00). The number needed to screen was 5.6.
SIGNIFICANCE
An electronic health record-integrated NLP application can accurately assign surgical candidacy scores to patients in a clinical setting.
目的
延迟癫痫手术会导致可避免的疾病负担增加和死亡率上升。本研究旨在前瞻性验证一种使用医生记录分配癫痫手术候选资格评分的自然语言处理（NLP）应用。
方法
该应用程序在以下两种情况下的记录中进行了训练：（1）有癫痫诊断和癫痫切除术史的患者；（2）无癫痫发作且未接受手术的患者。测试集包括所有手术候选状态未知且即将进行神经科就诊的患者。训练和测试集每周更新一次，持续一年。应用程序中包含的患者记录中的 1 到 3 个单词的短语被用作特征。应用程序前瞻性识别为手术候选者的患者由两名癫痫专家进行手动审查。通过比较 NLP 得出的手术候选评分与专家图表审查的手术候选状态来定义性能指标。
结果
训练集每周更新，包括平均 519 ± 67 名患者的记录。10 倍交叉验证的接收器操作特征曲线（AUC）面积为 0.90 ± 0.04（范围为 0.83-0.96），随着新患者被添加到训练集中，每周增加 0.002（P < 0.001）。在神经科诊所就诊的 6395 名患者中，有 4211 名（67%）接受了模型评估。该测试集的前瞻性 AUC 为 0.79（95%置信区间[CI] = 0.62-0.96）。使用最佳手术候选评分阈值，敏感性为 0.80（95%CI = 0.29-0.99），特异性为 0.77（95%CI = 0.64-0.88），阳性预测值为 0.25（95%CI = 0.07-0.52），阴性预测值为 0.98（95%CI = 0.87-1.00）。筛查所需人数为 5.6。
意义
电子健康记录集成的 NLP 应用程序可以在临床环境中准确地为患者分配手术候选评分。
Zotero 插件