Tartaglione Enzo, Bragagnolo Andrea, Odierna Francesco, Fiandrotti Attilio, Grangetto Marco
IEEE Trans Neural Netw Learn Syst. 2022 Dec;33(12):7237-7250. doi: 10.1109/TNNLS.2021.3084527. Epub 2022 Nov 30.
Deep neural networks include millions of learnable parameters, making their deployment over resource-constrained devices problematic. Sensitivity-based regularization of neurons (SeReNe) is a method for learning sparse topologies with a structure, exploiting neural sensitivity as a regularizer. We define the sensitivity of a neuron as the variation of the network output with respect to the variation of the activity of the neuron. The lower the sensitivity of a neuron, the less the network output is perturbed if the neuron output changes. By including the neuron sensitivity in the cost function as a regularization term, we are able to prune neurons with low sensitivity. As entire neurons are pruned rather than single parameters, practical network footprint reduction becomes possible. Our experimental results on multiple network architectures and datasets yield competitive compression ratios with respect to state-of-the-art references.
深度神经网络包含数百万个可学习参数，这使得它们在资源受限设备上的部署存在问题。基于神经元敏感度的正则化（SeReNe）是一种通过利用神经敏感度作为正则化项来学习具有特定结构的稀疏拓扑的方法。我们将神经元的敏感度定义为网络输出相对于神经元活动变化的变化率。神经元的敏感度越低，如果神经元输出发生变化，网络输出受到的扰动就越小。通过将神经元敏感度作为正则化项包含在代价函数中，我们能够修剪敏感度低的神经元。由于修剪的是整个神经元而非单个参数，因此切实可行地减少网络占用空间成为可能。我们在多个网络架构和数据集上的实验结果相对于现有技术参考产生了具有竞争力的压缩率。
