Department of Emergency Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts.
Firearm Injury & Policy Research Program, University of Washington, Seattle.
JAMA Netw Open. 2023 Apr 3;6(4):e235870. doi: 10.1001/jamanetworkopen.2023.5870.
International Classification of Diseases-coded hospital discharge data do not accurately reflect whether firearm injuries were caused by assault, unintentional injury, self-harm, legal intervention, or were of undetermined intent. Applying natural language processing (NLP) and machine learning (ML) techniques to electronic health record (EHR) narrative text could be associated with improved accuracy of firearm injury intent data.
To assess the accuracy with which an ML model identified firearm injury intent.
DESIGN, SETTING, AND PARTICIPANTS: A cross-sectional retrospective EHR review was conducted at 3 level I trauma centers, 2 from health care institutions in Boston, Massachusetts, and 1 from Seattle, Washington, between January 1, 2000, and December 31, 2019; data analysis was performed from January 18, 2021, to August 22, 2022. A total of 1915 incident cases of firearm injury in patients presenting to emergency departments at the model development institution and 769 from the external validation institution with a firearm injury code assigned according to International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) or International Statistical Classification of Diseases and Related Health Problems, 10th Revision, Clinical Modification (ICD-10-CM), in discharge data were included.
Classification of firearm injury intent.
Intent classification accuracy by the NLP model was compared with ICD codes assigned by medical record coders in discharge data. The NLP model extracted intent-relevant features from narrative text that were then used by a gradient-boosting classifier to determine the intent of each firearm injury. Classification accuracy was evaluated against intent assigned by the research team. The model was further validated using an external data set.
The NLP model was evaluated in 381 patients presenting with firearm injury at the model development site (mean [SD] age, 39.2 [13.0] years; 348 [91.3%] men) and 304 patients at the external development site (mean [SD] age, 31.8 [14.8] years; 263 [86.5%] men). The model proved more accurate than medical record coders in assigning intent to firearm injuries at the model development site (accident F-score, 0.78 vs 0.40; assault F-score, 0.90 vs 0.78). The model maintained this improvement on an external validation set from a second institution (accident F-score, 0.64 vs 0.58; assault F-score, 0.88 vs 0.81). While the model showed some degradation between institutions, retraining the model using data from the second institution further improved performance on that site's records (accident F-score, 0.75; assault F-score, 0.92).
The findings of this study suggest that NLP ML can be used to improve the accuracy of firearm injury intent classification compared with ICD-coded discharge data, particularly for cases of accident and assault intents (the most prevalent and commonly misclassified intent types). Future research could refine this model using larger and more diverse data sets.
国际疾病分类编码的医院出院数据不能准确反映火器伤是由攻击、意外伤害、自残、法律干预还是意图不明造成的。应用自然语言处理（NLP）和机器学习（ML）技术于电子健康记录（EHR）叙述文本可能与提高火器伤意图数据的准确性有关。
评估机器学习模型识别火器伤意图的准确性。
设计、设置和参与者：这是一项在三个一级创伤中心进行的横断面回顾性电子健康记录研究，其中两个来自马萨诸塞州波士顿的医疗机构，一个来自华盛顿州西雅图；研究时间为 2000 年 1 月 1 日至 2019 年 12 月 31 日；数据分析于 2021 年 1 月 18 日至 2022 年 8 月 22 日进行。纳入了在模型开发机构因火器伤就诊的 1915 例火器伤患者的 1915 例事件病例和 769 例来自外部验证机构的火器伤患者（根据国际疾病分类，第九修订版，临床修正版（ICD-9-CM）或国际疾病分类和相关健康问题，第十修订版，临床修正版（ICD-10-CM）在出院数据中分配了火器伤代码）。
火器伤意图分类。
将 NLP 模型的意图分类准确性与出院数据中病历编码员分配的 ICD 代码进行比较。NLP 模型从叙述文本中提取出与意图相关的特征，然后由梯度提升分类器使用这些特征来确定每个火器伤的意图。使用研究团队分配的意图对分类准确性进行评估。该模型还使用外部数据集进行了进一步验证。
在模型开发现场（平均[标准差]年龄，39.2[13.0]岁；348[91.3%]名男性）和外部开发现场（平均[标准差]年龄，31.8[14.8]岁；263[86.5%]名男性）评估了 NLP 模型在 381 名火器伤患者和 304 名患者中的表现。与病历编码员相比，该模型在分配火器伤意图方面更准确（意外 F 分数，0.78 与 0.40；攻击 F 分数，0.90 与 0.78）。该模型在来自第二家机构的外部验证集中保持了这一改进（意外 F 分数，0.64 与 0.58；攻击 F 分数，0.88 与 0.81）。虽然模型在不同机构之间显示出一些降级，但使用第二家机构的数据重新训练模型进一步提高了该机构记录的性能（意外 F 分数，0.75；攻击 F 分数，0.92）。
这项研究的结果表明，与 ICD 编码的出院数据相比，NLP ML 可用于提高火器伤意图分类的准确性，特别是对意外和攻击意图（最常见和最常被错误分类的意图类型）。未来的研究可以使用更大和更多样化的数据集来改进该模型。
