Lin Pei-Chen, Chen Kuo-Tai, Chen Huan-Chieh, Islam Md Mohaimenul, Lin Ming-Chin
Graduate Institute of Biomedical Informatics, College of Medicine Science and Technology, Taipei Medical University, Taipei 106, Taiwan.
Emergency Department, Taoyuan General Hospital, Ministry of Health and Welfare, Taoyuan 330, Taiwan.
J Pers Med. 2021 Oct 21;11(11):1055. doi: 10.3390/jpm11111055.
Accurate stratification of sepsis can effectively guide the triage of patient care and shared decision making in the emergency department (ED). However, previous research on sepsis identification models focused mainly on ICU patients, and discrepancies in model performance between the development and external validation datasets are rarely evaluated. The aim of our study was to develop and externally validate a machine learning model to stratify sepsis patients in the ED. We retrospectively collected clinical data from two geographically separate institutes that provided a different level of care at different time periods. The Sepsis-3 criteria were used as the reference standard in both datasets for identifying true sepsis cases. An eXtreme Gradient Boosting (XGBoost) algorithm was developed to stratify sepsis patients and the performance of the model was compared with traditional clinical sepsis tools; quick Sequential Organ Failure Assessment (qSOFA) and Systemic Inflammatory Response Syndrome (SIRS). There were 8296 patients (1752 (21%) being septic) in the development and 1744 patients (506 (29%) being septic) in the external validation datasets. The mortality of septic patients in the development and validation datasets was 13.5% and 17%, respectively. In the internal validation, XGBoost achieved an area under the receiver operating characteristic curve (AUROC) of 0.86, exceeding SIRS (0.68) and qSOFA (0.56). The performance of XGBoost deteriorated in the external validation (the AUROC of XGBoost, SIRS and qSOFA was 0.75, 0.57 and 0.66, respectively). Heterogeneity in patient characteristics, such as sepsis prevalence, severity, age, comorbidity and infection focus, could reduce model performance. Our model showed good discriminative capabilities for the identification of sepsis patients and outperformed the existing sepsis identification tools. Implementation of the ML model in the ED can facilitate timely sepsis identification and treatment. However, dataset discrepancies should be carefully evaluated before implementing the ML approach in clinical practice. This finding reinforces the necessity for future studies to perform external validation to ensure the generalisability of any developed ML approaches.
脓毒症的准确分层能够有效指导急诊科患者护理的分诊以及共同决策。然而，先前关于脓毒症识别模型的研究主要集中在重症监护病房患者，且很少评估模型在开发数据集和外部验证数据集之间的性能差异。我们研究的目的是开发并在外部验证一个用于急诊科脓毒症患者分层的机器学习模型。我们回顾性收集了来自两个地理位置不同的机构在不同时间段提供不同护理水平的临床数据。在两个数据集中均使用脓毒症 - 3标准作为识别真正脓毒症病例的参考标准。开发了一种极端梯度提升（XGBoost）算法对脓毒症患者进行分层，并将该模型的性能与传统临床脓毒症工具（快速序贯器官衰竭评估（qSOFA）和全身炎症反应综合征（SIRS））进行比较。开发数据集中有8296例患者（1752例（21%）为脓毒症患者），外部验证数据集中有1744例患者（506例（29%）为脓毒症患者）。开发数据集和验证数据集中脓毒症患者的死亡率分别为13.5%和17%。在内部验证中，XGBoost的受试者工作特征曲线下面积（AUROC）为0.86，超过了SIRS（0.68）和qSOFA（0.56）。XGBoost在外部验证中的性能有所下降（XGBoost、SIRS和qSOFA的AUROC分别为0.75、0.57和0.66）。患者特征的异质性，如脓毒症患病率、严重程度、年龄、合并症和感染部位，可能会降低模型性能。我们的模型在识别脓毒症患者方面显示出良好的判别能力，并且优于现有的脓毒症识别工具。在急诊科实施机器学习模型可以促进脓毒症的及时识别和治疗。然而，在临床实践中实施机器学习方法之前，应仔细评估数据集差异。这一发现强化了未来研究进行外部验证以确保任何开发的机器学习方法具有通用性的必要性。
