Wang Bao, Ye Qiang
IEEE Trans Neural Netw Learn Syst. 2024 Sep;35(9):12288-12300. doi: 10.1109/TNNLS.2023.3255783. Epub 2024 Sep 3.
Momentum is crucial in stochastic gradient-based optimization algorithms for accelerating or improving training deep neural networks (DNNs). In deep learning practice, the momentum is usually weighted by a well-calibrated constant. However, tuning the hyperparameter for momentum can be a significant computational burden. In this article, we propose a novel adaptive momentum for improving DNNs training; this adaptive momentum, with no momentum-related hyperparameter required, is motivated by the nonlinear conjugate gradient (NCG) method. Stochastic gradient descent (SGD) with this new adaptive momentum eliminates the need for the momentum hyperparameter calibration, allows using a significantly larger learning rate, accelerates DNN training, and improves the final accuracy and robustness of the trained DNNs. For instance, SGD with this adaptive momentum reduces classification errors for training ResNet110 for CIFAR10 and CIFAR100 from 5.25% to 4.64% and 23.75% to 20.03%, respectively. Furthermore, SGD, with the new adaptive momentum, also benefits adversarial training and, hence, improves the adversarial robustness of the trained DNNs.
动量在基于随机梯度的优化算法中对于加速或改进深度神经网络（DNN）的训练至关重要。在深度学习实践中，动量通常由一个校准良好的常数加权。然而，调整动量的超参数可能是一项重大的计算负担。在本文中，我们提出了一种用于改进DNN训练的新型自适应动量；这种自适应动量无需与动量相关的超参数，其灵感来自非线性共轭梯度（NCG）方法。具有这种新自适应动量的随机梯度下降（SGD）消除了对动量超参数校准的需求，允许使用显著更大的学习率，加速DNN训练，并提高训练后DNN的最终准确性和鲁棒性。例如，具有这种自适应动量的SGD将用于CIFAR10和CIFAR100训练ResNet110的分类错误分别从5.25%降至4.64%和从23.75%降至20.03%。此外，具有新自适应动量的SGD对对抗训练也有益，因此提高了训练后DNN的对抗鲁棒性。
