Bhaya Amit, Kaszkurewicz Eugenius
Department of Electrical Engineering, Federal University of Rio de Janeiro, PEE/COPPE/UFRJ, PO Box 68504, Rio de Janeiro, RJ 21945-970, Brazil.
Neural Netw. 2004 Jan;17(1):65-71. doi: 10.1016/S0893-6080(03)00170-9.
It is pointed out that the so called momentum method, much used in the neural network literature as an acceleration of the backpropagation method, is a stationary version of the conjugate gradient method. Connections with the continuous optimization method known as heavy ball with friction are also made. In both cases, adaptive (dynamic) choices of the so called learning rate and momentum parameters are obtained using a control Liapunov function analysis of the system.
有人指出，在神经网络文献中作为反向传播方法的一种加速手段而被大量使用的所谓动量法，是共轭梯度法的一种稳态形式。文中还建立了与被称为带摩擦重球的连续优化方法的联系。在这两种情况下，通过对系统进行控制李雅普诺夫函数分析，得到了对所谓学习率和动量参数的自适应（动态）选择。
