Prieto José Tomás, Scott Kenneth, McEwen Dean, Podewils Laura J, Al-Tayyib Alia, Robinson James, Edwards David, Foldy Seth, Shlay Judith C, Davidson Arthur J
Division of Scientific Education and Professional Development, Centers for Disease Control and Prevention, Atlanta, GA, United States.
Denver Public Health, Denver Health and Hospital Authority, Denver, CO, United States.
J Med Internet Res. 2020 Jan 3;22(1):e15645. doi: 10.2196/15645.
Timely, precise, and localized surveillance of nonfatal events is needed to improve response and prevention of opioid-related problems in an evolving opioid crisis in the United States. Records of naloxone administration found in prehospital emergency medical services (EMS) data have helped estimate opioid overdose incidence, including nonhospital, field-treated cases. However, as naloxone is often used by EMS personnel in unconsciousness of unknown cause, attributing naloxone administration to opioid misuse and heroin use (OM) may misclassify events. Better methods are needed to identify OM.
This study aimed to develop and test a natural language processing method that would improve identification of potential OM from paramedic documentation.
First, we searched Denver Health paramedic trip reports from August 2017 to April 2018 for keywords naloxone, heroin, and both combined, and we reviewed narratives of identified reports to determine whether they constituted true cases of OM. Then, we used this human classification as reference standard and trained 4 machine learning models (random forest, k-nearest neighbors, support vector machines, and L1-regularized logistic regression). We selected the algorithm that produced the highest area under the receiver operating curve (AUC) for model assessment. Finally, we compared positive predictive value (PPV) of the highest performing machine learning algorithm with PPV of searches of keywords naloxone, heroin, and combination of both in the binary classification of OM in unseen September 2018 data.
In total, 54,359 trip reports were filed from August 2017 to April 2018. Approximately 1.09% (594/54,359) indicated naloxone administration. Among trip reports with reviewer agreement regarding OM in the narrative, 57.6% (292/516) were considered to include information revealing OM. Approximately 1.63% (884/54,359) of all trip reports mentioned heroin in the narrative. Among trip reports with reviewer agreement, 95.5% (784/821) were considered to include information revealing OM. Combined results accounted for 2.39% (1298/54,359) of trip reports. Among trip reports with reviewer agreement, 77.79% (907/1166) were considered to include information consistent with OM. The reference standard used to train and test machine learning models included details of 1166 trip reports. L1-regularized logistic regression was the highest performing algorithm (AUC=0.94; 95% CI 0.91-0.97) in identifying OM. Tested on 5983 unseen reports from September 2018, the keyword naloxone inaccurately identified and underestimated probable OM trip report cases (63 cases; PPV=0.68). The keyword heroin yielded more cases with improved performance (129 cases; PPV=0.99). Combined keyword and L1-regularized logistic regression classifier further improved performance (146 cases; PPV=0.99).
A machine learning application enhanced the effectiveness of finding OM among documented paramedic field responses. This approach to refining OM surveillance may lead to improved first-responder and public health responses toward prevention of overdoses and other opioid-related problems in US communities.
在美国不断演变的阿片类药物危机中，需要及时、精确且本地化地监测非致命事件，以改进对阿片类药物相关问题的应对和预防措施。在院前急救医疗服务（EMS）数据中发现的纳洛酮使用记录有助于估计阿片类药物过量发生率，包括非医院现场救治的病例。然而，由于EMS人员经常在不明原因的昏迷情况下使用纳洛酮，将纳洛酮的使用归因于阿片类药物滥用和海洛因使用（OM）可能会对事件进行错误分类。因此需要更好的方法来识别OM。
本研究旨在开发并测试一种自然语言处理方法，以改进从护理人员记录中识别潜在OM的能力。
首先，我们在丹佛健康护理人员2017年8月至2018年4月的出车报告中搜索纳洛酮、海洛因以及两者组合的关键词，并审查已识别报告的叙述内容，以确定它们是否构成真正的OM病例。然后，我们将这种人工分类作为参考标准，训练4种机器学习模型（随机森林、k近邻、支持向量机和L1正则化逻辑回归）。我们选择在接收器操作曲线（AUC）下面积最高的算法进行模型评估。最后，我们在2018年9月未见过的数据的OM二元分类中，将表现最佳的机器学习算法的阳性预测值（PPV）与纳洛酮、海洛因及两者组合关键词搜索的PPV进行比较。
2017年8月至2018年4月共提交了54359份出车报告。约1.09%（594/54359）表明使用了纳洛酮。在叙述中经审核员认定为OM的出车报告中，57.6%（292/516）被认为包含揭示OM的信息。所有出车报告中约1.63%（884/54359）在叙述中提到了海洛因。在经审核员认定的出车报告中，95.5%（784/821）被认为包含揭示OM的信息。综合结果占出车报告的2.39%（1298/54359）。在经审核员认定的出车报告中，77.79%（907/1166）被认为包含与OM一致的信息。用于训练和测试机器学习模型的参考标准包括1166份出车报告的详细信息。L1正则化逻辑回归是识别OM时表现最佳的算法（AUC = 0.94；95% CI 0.91 - 0.97））。在对2018年9月的5983份未见过的报告进行测试时，关键词纳洛酮错误识别并低估了可能的OM出车报告病例（63例；PPV = 0.68）。关键词海洛因产生了更多病例，性能有所提高（129例；PPV = 0.99）。关键词与L1正则化逻辑回归分类器相结合进一步提高了性能（146例；PPV = 0.99）。
机器学习应用提高了在护理人员现场记录中查找OM的有效性。这种改进OM监测的方法可能会导致急救人员和公共卫生部门对美国社区预防过量用药和其他阿片类药物相关问题的应对得到改善。
