Department of Urology and Urological Surgery, University Medical Center Mannheim, University of Heidelberg, Theodor-Kutzer-Ufer 1-3, 68167, Mannheim, Germany.
Department of Urology, University of Leipzig, Leipzig, Germany.
World J Urol. 2024 Jul 10;42(1):396. doi: 10.1007/s00345-024-05078-y.
To investigate and implement semiautomated screening for meta-analyses (MA) in urology under consideration of class imbalance.
Machine learning algorithms were trained on data from three MA with detailed information of the screening process. Different methods to account for class imbalance (Sampling (up- and downsampling, weighting and cost-sensitive learning), thresholding) were implemented in different machine learning (ML) algorithms (Random Forest, Logistic Regression with Elastic Net Regularization, Support Vector Machines). Models were optimized for sensitivity. Besides metrics such as specificity, receiver operating curves, total missed studies, and work saved over sampling were calculated.
During training, models trained after downsampling achieved the best results consistently among all algorithms. Computing time ranged between 251 and 5834 s. However, when evaluated on the final test data set, the weighting approach performed best. In addition, thresholding helped to improve results as compared to the standard of 0.5. However, due to heterogeneity of results no clear recommendation can be made for a universal sample size. Misses of relevant studies were 0 for the optimized models except for one review.
It will be necessary to design a holistic methodology that implements the presented methods in a practical manner, but also takes into account other algorithms and the most sophisticated methods for text preprocessing. In addition, the different methods of a cost-sensitive learning approach can be the subject of further investigations.
在考虑类别不平衡的情况下，研究并实施泌尿外科半自动荟萃分析（MA）筛选。
在具有详细筛选过程信息的三个 MA 数据上对机器学习算法进行训练。实现了不同的方法来解决类别不平衡问题（采样（上采样和下采样、加权和代价敏感学习）、阈值），并在不同的机器学习（ML）算法（随机森林、具有弹性网正则化的逻辑回归、支持向量机）中实现。对模型进行了优化以提高敏感性。除了特异性、接收者操作曲线、总漏检研究和过采样节省的工作等指标外，还计算了其他指标。
在训练过程中，在所有算法中，经过下采样后训练的模型始终能取得最佳效果。计算时间在 251 到 5834 秒之间。然而，当在最终的测试数据集上进行评估时，加权方法的效果最佳。此外，与 0.5 的标准相比，阈值有助于提高结果。但是，由于结果的异质性，无法为通用的样本量推荐一种明确的方法。除了一篇综述外，优化后的模型都没有遗漏相关的研究。
需要设计一种整体方法，以实际的方式实现所提出的方法，同时还考虑其他算法和最复杂的文本预处理方法。此外，代价敏感学习方法的不同方法可以进一步研究。
