Tian Danhe, Yamagiwa Shinichi, Wada Koichi
Doctoral Program in Computer Science, University of Tsukuba, 1-1-1 Tennodai, Tsukuba 305-8573, Ibaraki, Japan.
JST, SPRING, 4-1-8 Honcho, Kawaguchi 332-0012, Saitama, Japan.
Sensors (Basel). 2022 Aug 5;22(15):5874. doi: 10.3390/s22155874.
Network pruning techniques have been widely used for compressing computational and memory intensive deep learning models through removing redundant components of the model. According to the pruning granularity, network pruning can be categorized into structured and unstructured methods. The structured pruning removes the large components in a model such as channels or layers, which might reduce the accuracy. The unstructured pruning directly removes mainly the parameters in a model as well as the redundant channels or layers, which might result in an inadequate pruning. To address the limitations of the pruning methods, this paper proposes a heuristic method for minimizing model size. This paper implements an algorithm to combine both the structured and the unstructured pruning methods while maintaining the target accuracy that is configured by its application. We use network slimming for the structured pruning method and deep compression for the unstructured one. Our method achieves a higher compression ratio than the case when the individual pruning method is applied. To show the effectiveness of our proposed method, this paper evaluates our proposed method with actual state-of-the-art CNN models of VGGNet, ResNet and DenseNet under the CIFAR-10 dataset. This paper discusses the performance of the proposed method with the cases of individual usage of the structured and unstructured pruning methods and then proves that our method achieves better performance with higher compression ratio. In the best case of the VGGNet, our method results in a 13× reduction ratio in the model size, and also gives a 15× reduction ratio regarding the pruning time compared with the brute-force search method.
网络剪枝技术已被广泛用于通过去除模型的冗余组件来压缩计算和内存密集型深度学习模型。根据剪枝粒度，网络剪枝可分为结构化和非结构化方法。结构化剪枝会去除模型中的大型组件，如通道或层，这可能会降低准确性。非结构化剪枝主要直接去除模型中的参数以及冗余通道或层，这可能会导致剪枝不充分。为了解决剪枝方法的局限性，本文提出了一种用于最小化模型大小的启发式方法。本文实现了一种算法，将结构化和非结构化剪枝方法相结合，同时保持由其应用配置的目标精度。我们将网络瘦身用于结构化剪枝方法，将深度压缩用于非结构化剪枝方法。我们的方法比单独应用剪枝方法时实现了更高的压缩率。为了展示我们提出的方法的有效性，本文在CIFAR-10数据集下，使用VGGNet、ResNet和DenseNet等实际的先进CNN模型对我们提出的方法进行了评估。本文讨论了结构化和非结构化剪枝方法单独使用时所提出方法的性能，然后证明了我们的方法在更高压缩率下实现了更好的性能。在VGGNet的最佳情况下，我们的方法使模型大小减少了13倍，与暴力搜索方法相比，剪枝时间也减少了15倍。
