Intelligent Critical Care Center, University of Florida, Gainesville.
Division of Nephrology, Hypertension, and Renal Transplantation, Department of Medicine, University of Florida, Gainesville.
JAMA Netw Open. 2022 May 2;5(5):e2211973. doi: 10.1001/jamanetworkopen.2022.11973.
Predicting postoperative complications has the potential to inform shared decisions regarding the appropriateness of surgical procedures, targeted risk-reduction strategies, and postoperative resource use. Realizing these advantages requires that accurate real-time predictions be integrated with clinical and digital workflows; artificial intelligence predictive analytic platforms using automated electronic health record (EHR) data inputs offer an intriguing possibility for achieving this, but there is a lack of high-level evidence from prospective studies supporting their use.
To examine whether the MySurgeryRisk artificial intelligence system has stable predictive performance between development and prospective validation phases and whether it is feasible to provide automated outputs directly to surgeons' mobile devices.
DESIGN, SETTING, AND PARTICIPANTS: In this prognostic study, the platform used automated EHR data inputs and machine learning algorithms to predict postoperative complications and provide predictions to surgeons, previously through a web portal and currently through a mobile device application. All patients 18 years or older who were admitted for any type of inpatient surgical procedure (74 417 total procedures involving 58 236 patients) between June 1, 2014, and September 20, 2020, were included. Models were developed using retrospective data from 52 117 inpatient surgical procedures performed between June 1, 2014, and November 27, 2018. Validation was performed using data from 22 300 inpatient surgical procedures collected prospectively from November 28, 2018, to September 20, 2020.
Algorithms for generalized additive models and random forest models were developed and validated using real-time EHR data. Model predictive performance was evaluated primarily using area under the receiver operating characteristic curve (AUROC) values.
Among 58 236 total adult patients who received 74 417 major inpatient surgical procedures, the mean (SD) age was 57 (17) years; 29 226 patients (50.2%) were male. Results reported in this article focus primarily on the validation cohort. The validation cohort included 22 300 inpatient surgical procedures involving 19 132 patients (mean [SD] age, 58 [17] years; 9672 [50.6%] male). A total of 2765 patients (14.5%) were Black or African American, 14 777 (77.2%) were White, 1235 (6.5%) were of other races (including American Indian or Alaska Native, Asian, Native Hawaiian or Pacific Islander, and multiracial), and 355 (1.9%) were of unknown race because of missing data; 979 patients (5.1%) were Hispanic, 17 663 (92.3%) were non-Hispanic, and 490 (2.6%) were of unknown ethnicity because of missing data. A greater number of input features was associated with stable or improved model performance. For example, the random forest model trained with 135 input features had the highest AUROC values for predicting acute kidney injury (0.82; 95% CI, 0.82-0.83); cardiovascular complications (0.81; 95% CI, 0.81-0.82); neurological complications, including delirium (0.87; 95% CI, 0.87-0.88); prolonged intensive care unit stay (0.89; 95% CI, 0.88-0.89); prolonged mechanical ventilation (0.91; 95% CI, 0.90-0.91); sepsis (0.86; 95% CI, 0.85-0.87); venous thromboembolism (0.82; 95% CI, 0.81-0.83); wound complications (0.78; 95% CI, 0.78-0.79); 30-day mortality (0.84; 95% CI, 0.82-0.86); and 90-day mortality (0.84; 95% CI, 0.82-0.85), with accuracy similar to surgeons' predictions. Compared with the original web portal, the mobile device application allowed efficient fingerprint login access and loaded data approximately 10 times faster. The application output displayed patient information, risk of postoperative complications, top 3 risk factors for each complication, and patterns of complications for individual surgeons compared with their colleagues.
In this study, automated real-time predictions of postoperative complications with mobile device outputs had good performance in clinical settings with prospective validation, matching surgeons' predictive accuracy.
重要性：预测术后并发症有可能为手术程序的适当性、有针对性的降低风险策略以及术后资源使用提供信息共享决策。要实现这些优势，需要将准确的实时预测与临床和数字工作流程相结合；使用自动化电子健康记录（EHR）数据输入的人工智能预测分析平台为此提供了一种有趣的可能性，但缺乏来自前瞻性研究的高级别的证据支持其使用。
目的：研究 MySurgeryRisk 人工智能系统在开发和前瞻性验证阶段的预测性能是否稳定，以及是否可以直接将自动化输出提供给外科医生的移动设备。
设计、地点和参与者：在这项预测研究中，该平台使用自动化 EHR 数据输入和机器学习算法来预测术后并发症，并向外科医生提供预测，以前是通过网络门户，现在是通过移动设备应用程序。纳入 2014 年 6 月 1 日至 2020 年 9 月 20 日期间接受任何类型的住院手术（总共 74417 例手术，涉及 58236 名患者）的 18 岁及以上的所有患者。模型使用 2014 年 6 月 1 日至 2018 年 11 月 27 日期间进行的 52117 例住院手术的回顾性数据进行开发。2018 年 11 月 28 日至 2020 年 9 月 20 日期间收集的 22300 例住院手术前瞻性数据进行验证。
主要结果和措施：使用实时 EHR 数据开发和验证了广义加性模型和随机森林模型的算法。主要使用接收者操作特征曲线（AUROC）值评估模型预测性能。
结果：在总共接受 74417 例主要住院手术的 58236 名成年患者中，平均（SD）年龄为 57（17）岁；29226 名患者（50.2%）为男性。本文主要关注验证队列。验证队列包括 22300 例住院手术，涉及 19132 名患者（平均[SD]年龄为 58[17]岁；506%为男性）。2765 名患者（14.5%）为黑人或非裔美国人，14777 名（77.2%）为白人，1235 名（6.5%）为其他种族（包括美国印第安人或阿拉斯加原住民、亚洲人、夏威夷原住民或太平洋岛民和多种族），355 名（1.9%）因数据缺失而无法确定种族；979 名患者（5.1%）为西班牙裔，17663 名（92.3%）为非西班牙裔，490 名（2.6%）因数据缺失而无法确定种族。更多的输入特征与稳定或改善的模型性能相关。例如，用 135 个输入特征训练的随机森林模型在预测急性肾损伤（0.82；95%CI，0.82-0.83）、心血管并发症（0.81；95%CI，0.81-0.82）、包括谵妄在内的神经并发症（0.87；95%CI，0.87-0.88）、重症监护病房延长停留时间（0.89；95%CI，0.88-0.89）、机械通气延长时间（0.91；95%CI，0.90-0.91）、脓毒症（0.86；95%CI，0.85-0.87）、静脉血栓栓塞症（0.82；95%CI，0.81-0.83）、伤口并发症（0.78；95%CI，0.78-0.79）、30 天死亡率（0.84；95%CI，0.82-0.86）和 90 天死亡率（0.84；95%CI，0.82-0.85）方面的表现与外科医生的预测相当。与原始网络门户相比，移动设备应用程序允许通过指纹登录访问，并且加载数据的速度快约 10 倍。应用程序输出显示患者信息、术后并发症风险、每种并发症的前 3 个风险因素以及与同事相比单个外科医生的并发症模式。
结论：在这项前瞻性验证研究中，具有移动设备输出的自动化实时术后并发症预测在临床环境中具有良好的性能，与外科医生的预测准确性相匹配。
