Zweiri Yahya H, Seneviratne Lakmal D, Althoefer Kaspar
Department of Mechanical Engineering, King's College London, Strand, WC2R 2LS, UK. yahya.zweiri.kcl.ac.uk
Neural Netw. 2005 Dec;18(10):1341-7. doi: 10.1016/j.neunet.2005.04.007. Epub 2005 Aug 30.
Efficient learning by the backpropagation (BP) algorithm is required for many practical applications. The BP algorithm calculates the weight changes of artificial neural networks, and a common approach is to use a two-term algorithm consisting of a learning rate (LR) and a momentum factor (MF). The major drawbacks of the two-term BP learning algorithm are the problems of local minima and slow convergence speeds, which limit the scope for real-time applications. Recently the addition of an extra term, called a proportional factor (PF), to the two-term BP algorithm was proposed. The third increases the speed of the BP algorithm. However, the PF term also reduces the convergence of the BP algorithm, and criteria for evaluating convergence are required to facilitate the application of the three terms BP algorithm. This paper analyzes the convergence of the new three-term backpropagation algorithm. If the learning parameters of the three-term BP algorithm satisfy the conditions given in this paper, then it is guaranteed that the system is stable and will converge to a local minimum. It is proved that if at least one of the eigenvalues of matrix F (compose of the Hessian of the cost function and the system Jacobian of the error vector at each iteration) is negative, then the system becomes unstable. Also the paper shows that all the local minima of the three-term BP algorithm cost function are stable. The relationship between the learning parameters are established in this paper such that the stability conditions are met.
许多实际应用都需要通过反向传播（BP）算法进行高效学习。BP算法计算人工神经网络的权重变化，一种常见的方法是使用由学习率（LR）和动量因子（MF）组成的双项算法。双项BP学习算法的主要缺点是存在局部极小值问题和收敛速度慢的问题，这限制了实时应用的范围。最近，有人提出在双项BP算法中添加一个额外的项，称为比例因子（PF）。第三项提高了BP算法的速度。然而，PF项也降低了BP算法的收敛性，因此需要评估收敛性的标准来促进三项BP算法的应用。本文分析了新的三项反向传播算法的收敛性。如果三项BP算法的学习参数满足本文给出的条件，那么可以保证系统是稳定的，并且会收敛到一个局部极小值。证明了如果矩阵F（由每次迭代时的代价函数的海森矩阵和误差向量的系统雅可比矩阵组成）的至少一个特征值为负，那么系统就会变得不稳定。本文还表明三项BP算法代价函数的所有局部极小值都是稳定的。本文建立了学习参数之间的关系，以满足稳定性条件。
