Tosti Guerra Francesco, Napoletano Andrea, Zaccaria Andrea
Department of Physics, Sapienza University of Rome, 00185 Rome, Italy.
Istituto dei Sistemi Complessi (ISC-CNR), UOS Sapienza, 00185 Rome, Italy.
Entropy (Basel). 2025 Apr 18;27(4):440. doi: 10.3390/e27040440.
In this work, we propose to study the collective behavior of different ensembles of neural networks. These sets define and live on complex manifolds that evolve through training. Each manifold is characterized by its intrinsic dimension, a measure of the variability of the ensemble and, as such, a measure of the impact of the different training strategies. Indeed, higher intrinsic dimension values imply higher variability among the networks and a larger parameter space coverage. Here, we quantify how much the training choices allow the exploration of the parameter space, finding that a random initialization of the parameters is a stronger source of variability than, progressively, data distortion, dropout, and batch shuffle. We then investigate the combinations of these strategies, the parameters involved, and the impact on the accuracy of the predictions, shedding light on the often-underestimated consequences of these training choices.
在这项工作中，我们提议研究不同神经网络集合的集体行为。这些集合定义并存在于通过训练而演化的复杂流形上。每个流形都由其内在维度来表征，内在维度是集合变异性的一种度量，因此也是不同训练策略影响的一种度量。实际上，更高的内在维度值意味着网络之间更高的变异性以及更大的参数空间覆盖范围。在此，我们量化训练选择在多大程度上允许对参数空间进行探索，发现参数的随机初始化是比数据失真、随机失活和批次洗牌逐渐更强的变异性来源。然后，我们研究这些策略的组合、所涉及的参数以及对预测准确性的影响，揭示这些训练选择常常被低估的后果。
