Higgins Oliver, Wilson Rhonda L, Chalup Stephan K
School of Health and Biomedical Sciences, RMIT University, Melbourne, Australia.
Department of Mental Health, Central Coast Local Health District, Gosford, NSW, Australia.
Digit Health. 2024 Nov 11;10:20552076241287364. doi: 10.1177/20552076241287364. eCollection 2024 Jan-Dec.
The objective of this study was to assess the predictability of admissions to a MH inpatient ward using ML models, based on routine data collected during triage in EDs. This research sought to identify the most effective ML model for this purpose while considering the practical implications of model interpretability for clinical use.
The study utilised existing data from January 2016 to December 2021. After data pre-processing, an exploratory analysis revealed the non-linear nature of the dataset. Six different ML models were tested: Random Forest, XGBoost, CatBoost, k-Nearest Neighbours (kNN), Explainable Boosting Machine (EBM) using InterpretML, and Support Vector Machine using Support Vector Classification (SVC). The performance of these models was evaluated using various metrics including the Matthews Correlation Coefficient (MCC).
Among the models evaluated, the CatBoost model achieved the highest MCC score of 0.1952, demonstrating superior balanced accuracy and predictive power, particularly in correctly identifying positive cases. The InterpretML model also performed well, with an MCC score of 0.1914. While CatBoost showed strong predictive capabilities, its complexity poses challenges for clinical interpretation. Conversely, the InterpretML model, though slightly less powerful, offers better transparency and is more practical for clinical use.
The findings suggest that the CatBoost model is a compelling choice for scenarios prioritising the detection of positive cases. However, the InterpretML model's ease of interpretation makes it more suitable for clinical application. Integrating explanation methods like SHAP with non-linear models could enhance model transparency and foster clinician trust. Further research is recommended to refine non-linear models within decision support systems, explore multi-source data integration, understand clinician attitudes towards ML, and develop real-time data collection systems. This study highlights the potential of ML in predicting MH admissions from ED data while stressing the importance of interpretability, ethical considerations, and ongoing validation for successful clinical implementation.
本研究的目的是基于急诊科分诊期间收集的常规数据，使用机器学习模型评估精神科住院病房入院的可预测性。本研究旨在确定为此目的最有效的机器学习模型，同时考虑模型可解释性对临床应用的实际影响。
该研究利用了2016年1月至2021年12月的现有数据。经过数据预处理，探索性分析揭示了数据集的非线性性质。测试了六种不同的机器学习模型：随机森林、XGBoost、CatBoost、k近邻（kNN）、使用InterpretML的可解释增强机（EBM）以及使用支持向量分类（SVC）的支持向量机。使用包括马修斯相关系数（MCC）在内的各种指标评估这些模型的性能。
在评估的模型中，CatBoost模型的MCC得分最高，为0.1952，显示出卓越的平衡准确性和预测能力，尤其是在正确识别阳性病例方面。InterpretML模型也表现良好，MCC得分为0.1914。虽然CatBoost显示出强大的预测能力，但其复杂性给临床解释带来了挑战。相反，InterpretML模型虽然能力稍弱，但具有更好的透明度，在临床应用中更实用。
研究结果表明，在优先检测阳性病例的情况下，CatBoost模型是一个有吸引力的选择。然而，InterpretML模型易于解释，使其更适合临床应用。将SHAP等解释方法与非线性模型相结合可以提高模型透明度并增强临床医生的信任。建议进一步开展研究，以完善决策支持系统中的非线性模型，探索多源数据整合，了解临床医生对机器学习的态度，并开发实时数据收集系统。本研究强调了机器学习在根据急诊科数据预测精神科住院方面的潜力，同时强调了可解释性、伦理考量以及持续验证对于成功临床实施的重要性。
