Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main St W, Hamilton, ON, L8S 4L8, Canada.
ICES, Hamilton, ON, Canada.
BMC Med Res Methodol. 2024 Mar 27;24(1):77. doi: 10.1186/s12874-024-02189-3.
SARS-CoV-2 vaccines are effective in reducing hospitalization, COVID-19 symptoms, and COVID-19 mortality for nursing home (NH) residents. We sought to compare the accuracy of various machine learning models, examine changes to model performance, and identify resident characteristics that have the strongest associations with 30-day COVID-19 mortality, before and after vaccine availability.
We conducted a population-based retrospective cohort study analyzing data from all NH facilities across Ontario, Canada. We included all residents diagnosed with SARS-CoV-2 and living in NHs between March 2020 and July 2021. We employed five machine learning algorithms to predict COVID-19 mortality, including logistic regression, LASSO regression, classification and regression trees (CART), random forests, and gradient boosted trees. The discriminative performance of the models was evaluated using the area under the receiver operating characteristic curve (AUC) for each model using 10-fold cross-validation. Model calibration was determined through evaluation of calibration slopes. Variable importance was calculated by repeatedly and randomly permutating the values of each predictor in the dataset and re-evaluating the model's performance.
A total of 14,977 NH residents and 20 resident characteristics were included in the model. The cross-validated AUCs were similar across algorithms and ranged from 0.64 to 0.67. Gradient boosted trees and logistic regression had an AUC of 0.67 pre- and post-vaccine availability. CART had the lowest discrimination ability with an AUC of 0.64 pre-vaccine availability, and 0.65 post-vaccine availability. The most influential resident characteristics, irrespective of vaccine availability, included advanced age (≥ 75 years), health instability, functional and cognitive status, sex (male), and polypharmacy.
The predictive accuracy and discrimination exhibited by all five examined machine learning algorithms were similar. Both logistic regression and gradient boosted trees exhibit comparable performance and display slight superiority over other machine learning algorithms. We observed consistent model performance both before and after vaccine availability. The influence of resident characteristics on COVID-19 mortality remained consistent across time periods, suggesting that changes to pre-vaccination screening practices for high-risk individuals are effective in the post-vaccination era.
SARS-CoV-2 疫苗可有效降低疗养院（NH）居民住院、COVID-19 症状和 COVID-19 死亡率。我们旨在比较各种机器学习模型的准确性，检查模型性能的变化，并在疫苗接种前后，确定与 30 天 COVID-19 死亡率关联最强的居民特征。
我们进行了一项基于人群的回顾性队列研究，分析了加拿大安大略省所有 NH 设施的数据。我们纳入了所有在 2020 年 3 月至 2021 年 7 月期间被诊断患有 SARS-CoV-2 并居住在 NH 的居民。我们采用了五种机器学习算法来预测 COVID-19 死亡率，包括逻辑回归、LASSO 回归、分类和回归树（CART）、随机森林和梯度提升树。使用 10 折交叉验证评估了每个模型的接收者操作特征曲线（ROC）下面积（AUC），以评估模型的判别性能。通过评估校准斜率来确定模型的校准。通过反复随机置换数据集每个预测器的值并重新评估模型的性能，计算变量的重要性。
共纳入了 14977 名 NH 居民和 20 个居民特征。跨算法的交叉验证 AUC 相似，范围在 0.64 至 0.67 之间。疫苗接种前后，梯度提升树和逻辑回归的 AUC 均为 0.67。疫苗接种前，CART 的判别能力最低，AUC 为 0.64；疫苗接种后，AUC 为 0.65。无论疫苗是否可用，最具影响力的居民特征包括高龄（≥75 岁）、健康不稳定、功能和认知状态、性别（男性）和多种药物治疗。
所有五种检查的机器学习算法的预测准确性和判别能力相似。逻辑回归和梯度提升树的表现相当，略优于其他机器学习算法。我们观察到疫苗接种前后模型性能一致。居民特征对 COVID-19 死亡率的影响在不同时间段保持一致，这表明在接种疫苗后，对高危人群的疫苗接种前筛查实践的改变是有效的。
