School of Mechanical and Electrical Engineering, Soochow University, Suzhou 215006, China.
Comput Intell Neurosci. 2021 Nov 10;2021:9882068. doi: 10.1155/2021/9882068. eCollection 2021.
An adaptive clamping method (SGD-MS) based on the radius of curvature is designed to alleviate the local optimal oscillation problem in deep neural network, which combines the radius of curvature of the objective function and the gradient descent of the optimizer. The radius of curvature is considered as the threshold to separate the momentum term or the future gradient moving average term adaptively. In addition, on this basis, we propose an accelerated version (SGD-MA), which further improves the convergence speed by using the method of aggregated momentum. Experimental results on several datasets show that the proposed methods effectively alleviate the local optimal oscillation problem and greatly improve the convergence speed and accuracy. A novel parameter updating algorithm is also provided in this paper for deep neural network.
设计了一种基于曲率半径的自适应夹紧方法（SGD-MS），以缓解深度神经网络中局部最优振荡问题，该方法结合了目标函数的曲率半径和优化器的梯度下降。曲率半径被视为自适应分离动量项或未来梯度移动平均项的阈值。此外，在此基础上，我们提出了一个加速版本（SGD-MA），通过使用聚合动量的方法进一步提高了收敛速度。在几个数据集上的实验结果表明，所提出的方法有效地缓解了局部最优振荡问题，大大提高了收敛速度和准确性。本文还为深度神经网络提供了一种新的参数更新算法。
