Department of Computer Science, Ben-Gurion University, Beer Sheva 8410501, Israel.
Sensors (Basel). 2022 Dec 13;22(24):9772. doi: 10.3390/s22249772.
Quantized neural networks (QNNs) are among the main approaches for deploying deep neural networks on low-resource edge devices. Training QNNs using different levels of precision throughout the network (mixed-precision quantization) typically achieves superior trade-offs between performance and computational load. However, optimizing the different precision levels of QNNs can be complicated, as the values of the bit allocations are discrete and difficult to differentiate for. Moreover, adequately accounting for the dependencies between the bit allocation of different layers is not straightforward. To meet these challenges, in this work, we propose GradFreeBits: a novel joint optimization scheme for training mixed-precision QNNs, which alternates between gradient-based optimization for the weights and gradient-free optimization for the bit allocation. Our method achieves a better or on par performance with the current state-of-the-art low-precision classification networks on CIFAR10/100 and ImageNet, semantic segmentation networks on Cityscapes, and several graph neural networks benchmarks. Furthermore, our approach can be extended to a variety of other applications involving neural networks used in conjunction with parameters that are difficult to optimize for.
量化神经网络 (QNN) 是在资源有限的边缘设备上部署深度神经网络的主要方法之一。在整个网络中使用不同精度级别（混合精度量化）训练 QNN 通常可以在性能和计算负载之间实现更好的权衡。然而，优化 QNN 的不同精度级别可能很复杂，因为位分配的值是离散的，并且难以区分。此外，充分考虑不同层之间的位分配之间的依赖关系并不简单。为了应对这些挑战，在这项工作中，我们提出了 GradFreeBits：一种用于训练混合精度 QNN 的新联合优化方案，它交替使用基于梯度的权重优化和基于梯度的位分配优化。我们的方法在 CIFAR10/100 和 ImageNet 上的分类网络、Cityscapes 上的语义分割网络以及几个图神经网络基准测试中实现了与当前最先进的低精度分类网络相当或更好的性能。此外，我们的方法可以扩展到其他各种涉及与难以优化的参数一起使用的神经网络的应用。
