Mathematical Institute Leiden University, Niels Bohrweg 1, 2333 CA Leiden, Netherlands.
Department of Biomedical Data Sciences, Section Medical Statistics, Leiden University Medical Center (LUMC), Albinusdreef 2, 2333 ZA Leiden, Netherlands.
Comput Math Methods Med. 2021 Nov 28;2021:2160322. doi: 10.1155/2021/2160322. eCollection 2021.
Studies focusing on prediction models are widespread in medicine. There is a trend in applying machine learning (ML) by medical researchers and clinicians. Over the years, multiple ML algorithms have been adapted to censored data. However, the choice of methodology should be motivated by the real-life data and their complexity. Here, the predictive performance of ML techniques is compared with statistical models in a simple clinical setting (small/moderate sample size and small number of predictors) with Monte-Carlo simulations.
Synthetic data (250 or 1000 patients) were generated that closely resembled 5 prognostic factors preselected based on a European Osteosarcoma Intergroup study (MRC BO06/EORTC 80931). Comparison was performed between 2 partial logistic artificial neural networks (PLANNs) and Cox models for 20, 40, 61, and 80% censoring. Survival times were generated from a log-normal distribution. Models were contrasted in terms of the C-index, Brier score at 0-5 years, integrated Brier score (IBS) at 5 years, and miscalibration at 2 and 5 years (usually neglected). The endpoint of interest was overall survival.
PLANNs original/extended were tuned based on the IBS at 5 years and the C-index, achieving a slightly better performance with the IBS. Comparison with Cox models showed that PLANNs can reach similar predictive performance on simulated data for most scenarios with respect to the C-index, Brier score, or IBS. However, Cox models were frequently less miscalibrated. Performance was robust in scenario data where censored patients were removed before 2 years or curtailing at 5 years was performed (on training data).
Survival neural networks reached a comparable predictive performance with Cox models but were generally less well calibrated. All in all, researchers should be aware of burdensome aspects of ML techniques such as data preprocessing, tuning of hyperparameters, and computational intensity that render them disadvantageous against conventional regression models in a simple clinical setting.
医学领域中专注于预测模型的研究非常普遍。医学研究人员和临床医生倾向于应用机器学习（ML）。多年来，已经有多种 ML 算法适用于删失数据。然而，方法的选择应该基于实际数据及其复杂性。在这里，通过蒙特卡罗模拟，在简单的临床环境（小/中等样本量和少量预测因子）中比较了 ML 技术的预测性能与统计模型。
生成了与基于欧洲骨肉瘤协作组研究（MRC BO06/EORTC 80931）预选的 5 个预后因素密切相似的合成数据（250 或 1000 例患者）。对于 20%、40%、61%和 80%的删失情况，比较了 2 个部分逻辑人工神经网络（PLANN）和 Cox 模型。生存时间来自对数正态分布。根据 5 年时的 C 指数、0-5 年时的 Brier 评分、5 年时的综合 Brier 评分（IBS）和 2 年和 5 年时的校准不良（通常被忽视）来对比模型。关注的终点是总生存。
PLANNs 根据 5 年时的 IBS 和 C 指数进行了原始/扩展调整，在 IBS 方面获得了稍好的性能。与 Cox 模型的比较表明，PLANNs 可以在大多数情况下，在 C 指数、Brier 评分或 IBS 方面，在模拟数据上达到相似的预测性能。然而，Cox 模型的校准不良情况通常较少。在删失患者在 2 年前被移除或在 5 年前截尾（在训练数据上）的情况下，情景数据中的性能具有稳健性。
生存神经网络与 Cox 模型达到了相当的预测性能，但通常校准效果较差。总而言之，研究人员应该意识到 ML 技术的繁琐方面，例如数据预处理、超参数调整和计算强度，这些方面使它们在简单的临床环境中相对于传统回归模型处于不利地位。
