Machine Learning Refinement of the NSQIP Risk Calculator: Who Survives the "Hail Mary" Case?

BACKGROUND

The American College of Surgeons (ACS) NSQIP risk calculator helps guide operative decision making. In patients with significant surgical risk, it may be unclear whether to proceed with "Hail Mary"-type interventions. To refine predictions, a local interpretable model-agnostic explanations machine (LIME) learning algorithm was explored to determine weighted patient-specific factors' contribution to mortality.

STUDY DESIGN

The ACS-NSQIP database was queried for all surgical patients with mortality probability greater than 50% between 2012 and 2019. Preoperative factors (n = 38) were evaluated using stepwise logistic regression; 26 significant factors were used in gradient boosted machine (GBM) modeling. Data were divided into training and testing sets, and model performance was substantiated with 10-fold cross validation. LIME provided individual subject mortality. The GBM-trained model was interpolated to LIME, and predictions were made using the test dataset.

RESULTS

There were 6,483 deaths (53%) among 12,248 admissions. GBM modeling displayed good performance (area under the curve = 0.65, 95% CI 0.636-0.671). The top 5 factors (% contribution) to mortality included: septic shock (27%), elevated International Normalized Ratio (22%), ventilator-dependence (14%), thrombocytopenia (14%), and elevated serum creatinine (5%). LIME modeling subset personalized patients by factors and weights on survival. In the entire cohort, mortality positive predictive value with 2 factor combinations was 53.5% (specificity 0.713), 3 combinations 64.2% (specificity 0.835), 4 combinations 72.1% (specificity 0.943), and all 5 combinations 77.9% (specificity 0.993). Conversely, mortality positive predictive value fell to 34% in the absence of 4 factors.

CONCLUSIONS

Through the application of machine learning algorithms (GBM and LIME), our model individualized predicted mortality and contributing factors with substantial ACS-NSQIP predicted mortality. USE of machine learning techniques may better inform operative decisions and family conversations in cases of significant surgical risk.