MIIT Key Laboratory of Dynamics and Control of Complex Systems, Xi'an 710072, China; School of Mathematics and Statistics, Northwestern Polytechnical University, Xi'an 710072, China.
MIIT Key Laboratory of Dynamics and Control of Complex Systems, Xi'an 710072, China; School of Mechanics, Civil Engineering and Architecture, Northwestern Polytechnical University, Xi'an 710072, China.
Neural Netw. 2024 Nov;179:106534. doi: 10.1016/j.neunet.2024.106534. Epub 2024 Jul 14.
As Deep Neural Networks (DNNs) continue to grow in complexity and size, leading to a substantial computational burden, weight pruning techniques have emerged as an effective solution. This paper presents a novel method for dynamic regularization-based pruning, which incorporates the Alternating Direction Method of Multipliers (ADMM). Unlike conventional methods that employ simple and abrupt threshold processing, the proposed method introduces a reweighting mechanism to assign importance to the weights in DNNs. Compared to other ADMM-based methods, the new method not only achieves higher accuracy but also saves considerable time thanks to the reduced number of necessary hyperparameters. The method is evaluated on multiple architectures, including LeNet-5, ResNet-32, ResNet-56, and ResNet-50, using the MNIST, CIFAR-10, and ImageNet datasets, respectively. Experimental results demonstrate its superior performance in terms of compression ratios and accuracy compared to state-of-the-art pruning methods. In particular, on the LeNet-5 model for the MNIST dataset, it achieves compression ratios of 355.9× with a slight improvement in accuracy; on the ResNet-50 model trained with the ImageNet dataset, it achieves compression ratios of 4.24× without sacrificing accuracy.
随着深度神经网络 (DNN) 的复杂性和规模不断扩大，导致计算负担很大，权重剪枝技术已经成为一种有效的解决方案。本文提出了一种新的基于动态正则化的剪枝方法，该方法结合了交替方向乘子法 (ADMM)。与传统的采用简单和突然的阈值处理的方法不同，所提出的方法引入了一种重新加权机制，为 DNN 中的权重分配重要性。与其他基于 ADMM 的方法相比，该新方法不仅实现了更高的准确性，而且由于所需超参数数量的减少，还节省了相当多的时间。该方法在多个架构上进行了评估，包括 LeNet-5、ResNet-32、ResNet-56 和 ResNet-50，分别使用 MNIST、CIFAR-10 和 ImageNet 数据集。实验结果表明，与最先进的剪枝方法相比，该方法在压缩比和准确性方面具有优越的性能。特别是在 MNIST 数据集上的 LeNet-5 模型上，它实现了 355.9×的压缩比，同时准确性略有提高；在使用 ImageNet 数据集训练的 ResNet-50 模型上，它实现了 4.24×的压缩比，而不牺牲准确性。
