Lopes Ricardo R, Yordanov Tsvetan T R, Ravelli Anita A C J, Houterman Saskia, Vis Marije, de Mol Bas A J M, Marquering Henk, Abu-Hanna Ameen
Amsterdam UMC Location University of Amsterdam, Biomedical Engineering and Physics, Meibergdreef 9, Amsterdam, the Netherlands.
Amsterdam UMC Location University of Amsterdam, Radiology and Nuclear Medicine, Meibergdreef 9, Amsterdam, the Netherlands.
Heliyon. 2023 Jun 10;9(6):e17139. doi: 10.1016/j.heliyon.2023.e17139. eCollection 2023 Jun.
Various mortality prediction models for Transcatheter Aortic Valve Implantation (TAVI) have been developed in the past years. The effect of time on the performance of such models, however, is unclear given the improvements in the procedure and changes in patient selection, potentially jeopardizing the usefulness of the prediction models in clinical practice. We aim to explore how time affects the performance and stability of different types of prediction models of 30-day mortality after TAVI.
We developed both parametric (Logistic Regression) and non-parametric (XGBoost) models to predict 30-day mortality after TAVI using data from the Netherlands Heart Registration. The models were trained with data from 2013 to the beginning of 2016 and pre-control charts from Statistical Process Control were used to analyse how time affects the models' performance on independent data from the mid of 2016 to the end of 2019. The area under the Receiver Operating Characteristics curve (AUC) was used to evaluate the models in terms of discrimination and the Brier Score (BS), which is related to calibration, in terms of accuracy of the predicted probabilities. To understand the extent to which refitting the models contribute to the models' stability, we also allowed the models to be updated over time.
We included data from 11,291 consecutive TAVI patients from hospitals in the Netherlands. The parametric model without re-training had a median AUC of 0.64 (IQR 0.54-0.73) and BS of 0.028 (IQR 0.021-0.035). For the non-parametric model, the median AUC was 0.63 (IQR 0.48-0.68) and BS was 0.027 (IQR 0.021-0.036). Over time, the developed parametric model was stable in terms of AUC and unstable in terms of BS. The non-parametric model was considered unstable in both AUC and BS. Repeated model refitting resulted in stable models in terms of AUC and decreased the variability of BS, although BS was still unstable. The refitted parametric model had a median AUC of 0.66 (IQR 0.57-0.73) and BS of 0.027 (IQR 0.020-0.035) while the non-parametric model had a median AUC of 0.66 (IQR 0.57-0.74) and BS of 0.027 (IQR 0.023-0.035).
The temporal validation of the TAVI 30-day mortality prediction models showed that the models refitted over time are more stable and accurate when compared to the frozen models. This highlights the importance of repeatedly refitted models over time to improve or at least maintain their performance stability. The non-parametric approach did not show improvement over the parametric approach.
在过去几年中，已经开发了各种用于经导管主动脉瓣植入术（TAVI）的死亡率预测模型。然而，鉴于该手术的改进和患者选择的变化，时间对这些模型性能的影响尚不清楚，这可能会危及预测模型在临床实践中的实用性。我们旨在探讨时间如何影响TAVI术后30天死亡率不同类型预测模型的性能和稳定性。
我们使用来自荷兰心脏注册的数据，开发了参数模型（逻辑回归）和非参数模型（XGBoost）来预测TAVI术后30天死亡率。这些模型使用2013年至2016年初的数据进行训练，并使用统计过程控制的预控制图来分析时间如何影响模型对2016年年中至2019年底独立数据的性能。受试者操作特征曲线（AUC）下的面积用于评估模型的辨别能力，而Brier评分（BS）与校准相关，用于评估预测概率的准确性。为了了解重新拟合模型对模型稳定性的贡献程度，我们还允许模型随时间更新。
我们纳入了来自荷兰医院的11291例连续TAVI患者的数据。未经重新训练的参数模型的AUC中位数为0.64（IQR 0.54 - 0.73），BS为0.028（IQR 0.021 - 0.035）。对于非参数模型，AUC中位数为0.63（IQR 0.48 - 0.68），BS为0.027（IQR 0.021 - 0.036）。随着时间的推移，所开发的参数模型在AUC方面是稳定的，而在BS方面是不稳定的。非参数模型在AUC和BS方面均被认为是不稳定的。重复模型重新拟合导致模型在AUC方面稳定，并且降低了BS的变异性，尽管BS仍然不稳定。重新拟合的参数模型的AUC中位数为0.66（IQR 0.57 - 0.73），BS为0.027（IQR 0.020 - 0.035），而非参数模型的AUC中位数为0.66（IQR 0.57 - 0.74），BS为0.027（IQR 0.023 - 0.035）。
TAVI术后30天死亡率预测模型的时间验证表明，与固定模型相比，随时间重新拟合的模型更稳定、更准确。这突出了随着时间重复重新拟合模型以改善或至少维持其性能稳定性的重要性。非参数方法并未显示出优于参数方法的改进。
