Liu Jing, Zhuang Bohan, Chen Peng, Shen Chunhua, Cai Jianfei, Tan Mingkui
IEEE Trans Pattern Anal Mach Intell. 2023 Oct;45(10):12459-12473. doi: 10.1109/TPAMI.2023.3275159. Epub 2023 Sep 5.
Network pruning and quantization are proven to be effective ways for deep model compression. To obtain a highly compact model, most methods first perform network pruning and then conduct quantization based on the pruned model. However, this strategy may ignore that the pruning and quantization would affect each other and thus performing them separately may lead to sub-optimal performance. To address this, performing pruning and quantization jointly is essential. Nevertheless, how to make a trade-off between pruning and quantization is non-trivial. Moreover, existing compression methods often rely on some pre-defined compression configurations (i.e., pruning rates or bitwidths). Some attempts have been made to search for optimal configurations, which however may take unbearable optimization cost. To address these issues, we devise a simple yet effective method named Single-path Bit Sharing (SBS) for automatic loss-aware model compression. To this end, we consider the network pruning as a special case of quantization and provide a unified view for model pruning and quantization. We then introduce a single-path model to encode all candidate compression configurations, where a high bitwidth value will be decomposed into the sum of a lowest bitwidth value and a series of re-assignment offsets. Relying on the single-path model, we introduce learnable binary gates to encode the choice of configurations and learn the binary gates and model parameters jointly. More importantly, the configuration search problem can be transformed into a subset selection problem, which helps to significantly reduce the optimization difficulty and computation cost. In this way, the compression configurations of each layer and the trade-off between pruning and quantization can be automatically determined. Extensive experiments on CIFAR-100 and ImageNet show that SBS significantly reduces computation cost while achieving promising performance. For example, our SBS compressed MobileNetV2 achieves 22.6× Bit-Operation (BOP) reduction with only 0.1% drop in the Top-1 accuracy.
网络剪枝和量化被证明是深度模型压缩的有效方法。为了获得高度紧凑的模型，大多数方法首先进行网络剪枝，然后基于剪枝后的模型进行量化。然而，这种策略可能忽略了剪枝和量化会相互影响，因此分别执行它们可能会导致次优性能。为了解决这个问题，联合执行剪枝和量化至关重要。然而，如何在剪枝和量化之间进行权衡并非易事。此外，现有的压缩方法通常依赖于一些预定义的压缩配置（即剪枝率或比特宽度）。已经有人尝试寻找最优配置，然而这可能会带来难以承受的优化成本。为了解决这些问题，我们设计了一种简单而有效的方法，名为单路径比特共享（SBS），用于自动损失感知模型压缩。为此，我们将网络剪枝视为量化的一种特殊情况，并为模型剪枝和量化提供统一的视角。然后，我们引入一个单路径模型来编码所有候选压缩配置，其中高比特宽度值将被分解为最低比特宽度值与一系列重新分配偏移量的总和。依靠单路径模型，我们引入可学习的二进制门来编码配置选择，并联合学习二进制门和模型参数。更重要的是，配置搜索问题可以转化为子集选择问题，这有助于显著降低优化难度和计算成本。通过这种方式，可以自动确定每一层的压缩配置以及剪枝和量化之间的权衡。在CIFAR - 100和ImageNet上进行的大量实验表明，SBS在显著降低计算成本的同时实现了良好的性能。例如，我们的SBS压缩的MobileNetV2实现了22.6倍的比特运算（BOP）减少，而Top - 1准确率仅下降0.1%。
