van der Meijden Siri L, van Boekel Anna M, Schinkelshoek Laurens J, van Goor Harry, Steyerberg Ewout W, Nelissen Rob G H H, Mesotten Dieter, Geerts Bart F, de Boer Mark G J, Arbous M Sesmu
Intensive Care Unit, Leiden University Medical Centre, Leiden, the Netherlands.
Healthplus.ai B.V., Amsterdam, the Netherlands.
Lancet Reg Health Eur. 2024 Dec 5;49:101163. doi: 10.1016/j.lanepe.2024.101163. eCollection 2025 Feb.
Postoperative infections significantly impact patient outcomes and costs, exacerbated by late diagnoses, yet early reliable predictors are scarce. Existing artificial intelligence (AI) models for postoperative infection prediction often lack external validation or perform poorly in local settings when validated. We aimed to develop locally valid models as part of the PERISCOPE AI system to enable early detection, safer discharge, and more timely treatment of patients.
We developed and validated XGBoost models to predict postoperative infections within 7 and 30 days of surgery. Using retrospective pre-operative and intra-operative electronic health record data from 2014 to 2023 across various surgical specialities, the models were developed at Hospital A and validated and updated at Hospitals B and C in the Netherlands and Belgium. Model performance was evaluated before and after updating using the two most recent years of data as temporal validation datasets. Main outcome measures were model discrimination (area under the receiver operating characteristic curve (AUROC)), calibration (slope, intercept, and plots), and clinical utility (decision curve analysis with net benefit).
The study included 253,010 surgical procedures with 23,903 infections within 30-days. Discriminative performance, calibration properties, and clinical utility significantly improved after updating. Final AUROCs after updating for Hospitals A, B, and C were 0.82 (95% confidence interval (CI) 0.81-0.83), 0.82 (95% CI 0.81-0.83), and 0.91 (95% CI 0.90-0.91) respectively for 30-day predictions on the temporal validation datasets (2022-2023). Calibration plots demonstrated adequate correspondence between observed outcomes and predicted risk. All local models were deemed clinically useful as the net benefit was higher than default strategies (treat all and treat none) over a wide range of clinically relevant decision thresholds.
PERISCOPE can accurately predict overall postoperative infections within 7- and 30-days post-surgery. The robust performance implies potential for improving clinical care in diverse clinical target populations. This study supports the need for approaches to local updating of AI models to account for domain shifts in patient populations and data distributions across different clinical settings.
This study was funded by a REACT EU grant from European Regional Development Fund (ERDF) and Kansen voor West.
术后感染对患者的治疗结果和费用有重大影响，而延迟诊断会使情况恶化，但早期可靠的预测指标却很匮乏。现有的用于预测术后感染的人工智能（AI）模型往往缺乏外部验证，或者在本地环境中进行验证时表现不佳。我们旨在开发作为PERISCOPE AI系统一部分的本地有效模型，以便能够早期发现、更安全地让患者出院并更及时地进行治疗。
我们开发并验证了XGBoost模型，以预测手术后7天和30天内的术后感染情况。利用2014年至2023年来自各个外科专科的术前和术中回顾性电子健康记录数据，这些模型在A医院开发，并在荷兰和比利时的B医院和C医院进行验证和更新。使用最近两年的数据作为时间验证数据集，在更新前后评估模型性能。主要结局指标包括模型辨别力（受试者工作特征曲线下面积（AUROC））、校准（斜率、截距和图表）以及临床实用性（通过净效益进行决策曲线分析）。
该研究纳入了253,010例外科手术，其中30天内有23,903例感染。更新后，辨别性能、校准特性和临床实用性均有显著改善。在时间验证数据集（2022 - 2023年）上，更新后A医院、B医院和C医院对于30天预测的最终AUROC分别为0.82（95%置信区间（CI）0.81 - 0.83）、0.82（95% CI 0.81 - 0.83）和0.91（95% CI 0.90 - 0.91）。校准图显示观察到的结果与预测风险之间具有充分的一致性。所有本地模型在广泛的临床相关决策阈值范围内，因其净效益高于默认策略（全部治疗和全部不治疗）而被认为具有临床实用性。
PERISCOPE能够准确预测手术后7天和30天内的总体术后感染情况。其强大的性能意味着在不同临床目标人群中改善临床护理的潜力。本研究支持需要采用对AI模型进行本地更新的方法，以应对不同临床环境中患者群体和数据分布的领域变化。
本研究由欧洲区域发展基金（ERDF）的REACT EU资助以及Kansen voor West资助。
