Lynam Anita L, Dennis John M, Owen Katharine R, Oram Richard A, Jones Angus G, Shields Beverley M, Ferrat Lauric A
Institute of Biomedical and Clinical Science, College of Medicine and Health, University of Exeter, Exeter, EX2 5DW UK.
Oxford Centre for Diabetes Endocrinology and Metabolism, University of Oxford, Churchill Hospital, Oxford, OX3 7LE UK.
Diagn Progn Res. 2020 Jun 4;4:6. doi: 10.1186/s41512-020-00075-2. eCollection 2020.
There is much interest in the use of prognostic and diagnostic prediction models in all areas of clinical medicine. The use of machine learning to improve prognostic and diagnostic accuracy in this area has been increasing at the expense of classic statistical models. Previous studies have compared performance between these two approaches but their findings are inconsistent and many have limitations. We aimed to compare the discrimination and calibration of seven models built using logistic regression and optimised machine learning algorithms in a clinical setting, where the number of potential predictors is often limited, and externally validate the models.
We trained models using logistic regression and six commonly used machine learning algorithms to predict if a patient diagnosed with diabetes has type 1 diabetes (versus type 2 diabetes). We used seven predictor variables (age, BMI, GADA islet-autoantibodies, sex, total cholesterol, HDL cholesterol and triglyceride) using a UK cohort of adult participants (aged 18-50 years) with clinically diagnosed diabetes recruited from primary and secondary care ( = 960, 14% with type 1 diabetes). Discrimination performance (ROC AUC), calibration and decision curve analysis of each approach was compared in a separate external validation dataset ( = 504, 21% with type 1 diabetes).
Average performance obtained in internal validation was similar in all models (ROC AUC ≥ 0.94). In external validation, there were very modest reductions in discrimination with AUC ROC remaining ≥ 0.93 for all methods. Logistic regression had the numerically highest value in external validation (ROC AUC 0.95). Logistic regression had good performance in terms of calibration and decision curve analysis. Neural network and gradient boosting machine had the best calibration performance. Both logistic regression and support vector machine had good decision curve analysis for clinical useful threshold probabilities.
Logistic regression performed as well as optimised machine algorithms to classify patients with type 1 and type 2 diabetes. This study highlights the utility of comparing traditional regression modelling to machine learning, particularly when using a small number of well understood, strong predictor variables.
临床医学各领域对预后和诊断预测模型的应用都极为关注。利用机器学习提高该领域的预后和诊断准确性的做法日益增多，而经典统计模型的应用则有所减少。以往研究比较了这两种方法的性能，但结果并不一致，且许多研究存在局限性。我们旨在比较在临床环境中使用逻辑回归和优化的机器学习算法构建的七个模型的区分度和校准度，临床环境中潜在预测变量的数量通常有限，并对模型进行外部验证。
我们使用逻辑回归和六种常用的机器学习算法训练模型，以预测被诊断为糖尿病的患者是否患有1型糖尿病（与2型糖尿病相对）。我们使用了七个预测变量（年龄、体重指数、谷氨酸脱羧酶胰岛自身抗体、性别、总胆固醇、高密度脂蛋白胆固醇和甘油三酯），研究对象为来自英国初级和二级医疗机构招募的成年参与者（年龄在18至50岁之间）的队列，这些参与者均患有临床诊断的糖尿病（n = 960，14%为1型糖尿病）。在一个单独的外部验证数据集中（n = 504，21%为1型糖尿病）比较了每种方法的区分性能（ROC曲线下面积）、校准度和决策曲线分析。
内部验证中所有模型获得的平均性能相似（ROC曲线下面积≥0.94）。在外部验证中，所有方法的区分度虽有非常小的降低，但ROC曲线下面积仍≥0.93。逻辑回归在外部验证中的数值最高（ROC曲线下面积为0.95）。逻辑回归在校准度和决策曲线分析方面表现良好。神经网络和梯度提升机具有最佳的校准性能。逻辑回归和支持向量机在临床有用阈值概率的决策曲线分析方面均表现良好。
在对1型和2型糖尿病患者进行分类时，逻辑回归的表现与优化的机器学习算法相当。本研究强调了将传统回归建模与机器学习进行比较的实用性，特别是在使用少量易于理解的强预测变量时。
