Passemiers Antoine, Folco Pietro, Raimondi Daniele, Birolo Giovanni, Moreau Yves, Fariselli Piero
ESAT-STADIUS, KU Leuven, Leuven, Belgium.
Department of Medical Sciences, University of Torino, Torino, Italy.
Sci Rep. 2024 Dec 28;14(1):31180. doi: 10.1038/s41598-024-82583-5.
Classification and regression problems can be challenging when the relevant input features are diluted in noisy datasets, in particular when the sample size is limited. Traditional Feature Selection (FS) methods address this issue by relying on some assumptions such as the linear or additive relationship between features. Recently, a proliferation of Deep Learning (DL) models has emerged to tackle both FS and prediction at the same time, allowing non-linear modeling of the selected features. In this study, we systematically assess the performance of DL-based feature selection methods on synthetic datasets of varying complexity, and benchmark their efficacy in uncovering non-linear relationships between features. We also use the same settings to benchmark the reliability of gradient-based feature attribution techniques for Neural Networks (NNs), such as Saliency Maps (SM). A quantitative evaluation of the reliability of these approaches is currently missing. Our analysis indicates that even simple synthetic datasets can significantly challenge most of the DL-based FS and SM methods, while Random Forests, TreeShap, mRMR and LassoNet are the best performing FS methods. Our conclusion is that when quantifying the relevance of a few non linearly-entangled predictive features diluted in a large number of irrelevant noisy variables, DL-based FS and SM interpretation methods are still far from being reliable.
当相关输入特征在噪声数据集中被稀释时，尤其是样本量有限的情况下，分类和回归问题可能具有挑战性。传统的特征选择（FS）方法通过依赖一些假设（如特征之间的线性或加性关系）来解决这个问题。最近，涌现出大量深度学习（DL）模型，旨在同时处理特征选择和预测，从而能够对所选特征进行非线性建模。在本研究中，我们系统地评估了基于深度学习的特征选择方法在不同复杂程度的合成数据集上的性能，并对它们在揭示特征之间非线性关系方面的有效性进行了基准测试。我们还使用相同的设置来评估基于梯度的神经网络（NN）特征归因技术（如显著性图（SM））的可靠性。目前缺少对这些方法可靠性的定量评估。我们的分析表明，即使是简单的合成数据集也能对大多数基于深度学习的FS和SM方法构成重大挑战，而随机森林、TreeShap、mRMR和LassoNet是性能最佳的FS方法。我们的结论是，在量化少数在大量无关噪声变量中被稀释的非线性纠缠预测特征的相关性时，基于深度学习的FS和SM解释方法仍远不可靠。
