Ribeirão Preto Medical School, University of São Paulo, Ribeirão Preto, São Paulo, Brazil.
Predicting Health GmbH, Graz, Austria.
PLoS One. 2024 Oct 11;19(10):e0311719. doi: 10.1371/journal.pone.0311719. eCollection 2024.
Studies of cardiovascular disease risk prediction by machine learning algorithms often do not assess their ability to generalize to other populations and few of them include an analysis of the interpretability of individual predictions. This manuscript addresses the development and validation, both internal and external, of predictive models for the assessment of risks of major adverse cardiovascular events (MACE). Global and local interpretability analyses of predictions were conducted towards improving MACE's model reliability and tailoring preventive interventions.
The models were trained and validated on a retrospective cohort with the use of data from Ribeirão Preto Medical School (RPMS), University of São Paulo, Brazil. Data from Beth Israel Deaconess Medical Center (BIDMC), USA, were used for external validation. A balanced sample of 6,000 MACE cases and 6,000 non-MACE cases from RPMS was created for training and internal validation and an additional one of 8,000 MACE cases and 8,000 non-MACE cases from BIDMC was employed for external validation. Eight machine learning algorithms, namely Penalized Logistic Regression, Random Forest, XGBoost, Decision Tree, Support Vector Machine, k-Nearest Neighbors, Naive Bayes, and Multi-Layer Perceptron were trained to predict a 5-year risk of major adverse cardiovascular events and their predictive performance was evaluated regarding accuracy, ROC curve (receiver operating characteristic), and AUC (area under the ROC curve). LIME and Shapley values were applied towards insights about model interpretability.
Random Forest showed the best predictive performance in both internal validation (AUC = 0.871 (0.859-0.882); Accuracy = 0.794 (0.782-0.808)) and external one (AUC = 0.786 (0.778-0.792); Accuracy = 0.710 (0.704-0.717)). Compared to LIME, Shapley values suggest more consistent explanations on exploratory analysis and importance of features.
Among the machine learning algorithms evaluated, Random Forest showed the best generalization ability, both internally and externally. Shapley values for local interpretability were more informative than LIME ones, which is in line with our exploratory analysis and global interpretation of the final model. Machine learning algorithms with good generalization and accompanied by interpretability analyses are recommended for assessments of individual risks of cardiovascular diseases and development of personalized preventive actions.
机器学习算法进行心血管疾病风险预测的研究通常并未评估其对其他人群的泛化能力，且很少对个体预测的可解释性进行分析。本文旨在开发和验证用于评估主要不良心血管事件（MACE）风险的预测模型，并对其进行内部和外部验证。通过对预测结果进行全局和局部可解释性分析，提高 MACE 模型的可靠性并为预防干预措施提供参考。
本研究使用来自巴西圣保罗大学里贝朗普雷图医学院（RPMS）的数据，对回顾性队列进行模型训练和验证。使用来自美国贝斯以色列女执事医疗中心（BIDMC）的数据进行外部验证。从 RPMS 创建了一个平衡的样本，包含 6000 例 MACE 病例和 6000 例非 MACE 病例，用于训练和内部验证，并从 BIDMC 额外创建了一个包含 8000 例 MACE 病例和 8000 例非 MACE 病例的样本，用于外部验证。训练了八种机器学习算法，包括 Penalized Logistic Regression、Random Forest、XGBoost、Decision Tree、Support Vector Machine、k-Nearest Neighbors、Naive Bayes 和 Multi-Layer Perceptron，以预测 5 年的主要不良心血管事件风险，并评估其准确性、ROC 曲线（接收者操作特征）和 AUC（ROC 曲线下面积）。应用 LIME 和 Shapley 值来深入了解模型的可解释性。
Random Forest 在内部验证（AUC = 0.871（0.859-0.882）；准确度 = 0.794（0.782-0.808））和外部验证（AUC = 0.786（0.778-0.792）；准确度 = 0.710（0.704-0.717））中均表现出最佳的预测性能。与 LIME 相比，Shapley 值在探索性分析和特征重要性方面提供了更一致的解释。
在所评估的机器学习算法中，Random Forest 表现出最佳的泛化能力，无论是在内部还是外部验证中。局部可解释性的 Shapley 值比 LIME 值提供了更有信息量的解释，这与我们的探索性分析和最终模型的全局解释一致。建议对个体的心血管疾病风险进行评估，并制定个性化的预防措施时，使用具有良好泛化能力并结合可解释性分析的机器学习算法。
