A new approach for interpretability and reliability in clinical risk prediction: Acute coronary syndrome scenario.

INTRODUCTION

The risk prediction of the occurrence of a clinical event is often based on conventional statistical procedures, through the implementation of risk score models. Recently, approaches based on more complex machine learning (ML) methods have been developed. Despite the latter usually have a better predictive performance, they obtain little approval from the physicians, as they lack interpretability and, therefore, clinical confidence. One clinical issue where both types of models have received great attention is the mortality risk prediction after acute coronary syndromes (ACS).

OBJECTIVE

We intend to create a new risk assessment methodology that combines the best characteristics of both risk score and ML models. More specifically, we aim to develop a method that, besides having a good performance, offers a personalized model and outcome for each patient, presents high interpretability, and incorporates an estimation of the prediction reliability which is not usually available. By combining these features in the same approach we expect that it can boost the confidence of physicians to use such a tool in their daily activity.

METHODS

In order to achieve the mentioned goals, a three-step methodology was developed: several rules were created by dichotomizing risk factors; such rules were trained with a machine learning classifier to predict the acceptance degree of each rule (the probability that the rule is correct) for each patient; that information was combined and used to compute the risk of mortality and the reliability of such prediction. The methodology was applied to a dataset of 1111 patients admitted with any type of ACS (myocardial infarction and unstable angina) in two Portuguese hospitals, to assess the 30-days all-cause mortality risk, being validated through a Monte-Carlo cross-validation technique. The performance was compared with state-of-the-art approaches: logistic regression (LR), artificial neural network (ANN), and clinical risk score model (namely the Global Registry of Acute Coronary Events - GRACE).

RESULTS

For the scenario being analyzed, the performance of the proposed approach and the comparison models was assessed through discrimination and calibration. The ability to rank the patients was evaluated through the area under the ROC curve (AUC), and the ability to stratify the patients into low or high-risk groups was determined using the geometric mean (GM) of specificity and sensitivity, the negative predictive value (NPV) and the positive predictive value (PPV). The validation calibration curves were also inspected. The proposed approach (AUC = 81%, GM = 74%, PPV = 17%, NPV = 99%) achieved testing results identical to the standard LR model (AUC = 83%, GM = 73%, PPV = 16%, NPV=99%), but offers superior interpretability and personalization; it also significantly outperforms the GRACE risk model (AUC = 79%, GM = 47%, PPV = 13%, NPV = 98%) and the standard ANN model (AUC = 78%, GM = 70%, PPV = 13%, NPV = 98%). The calibration curve also suggests a very good generalization ability of the obtained model as it approaches the ideal curve (slope = 0.96). Finally, the reliability estimation of individual predictions presented a great correlation with the misclassifications rate.

CONCLUSION

We developed and described a new tool that showed great potential to guide the clinical staff in the risk assessment and decision-making process, and to obtain their wide acceptance due to its interpretability and reliability estimation properties. The methodology presented a good performance when applied to ACS events, but those properties may have a beneficial application in other clinical scenarios as well.