Sudharsanan S I, Sundareshan M K
IBM Corporation, Boca Raton, FL 33429.
Int J Neural Syst. 1994 Sep;5(3):165-80. doi: 10.1142/s0129065794000190.
Complexity of implementation has been a major difficulty in the development of gradient descent learning algorithms for dynamical neural networks with feedback and recurrent connections. Some insights from the stability properties of the equilibrium points of the network, which suggest an appropriate tailoring of the sigmoidal nonlinear functions, can however be utilized in obtaining simplified learning rules, as demonstrated in this paper. An analytical proof of convergence of the learning scheme under specific conditions is given and some upper bounds on the adaptation parameters for an efficient implementation of the training procedure are developed. The performance features of the learning algorithm are illustrated by applying it to two problems of importance, viz., design of associative memories and nonlinear input-output mapping. For the first application, a systematic procedure is given for training a network to store multiple memory vectors as its stable equilibrium points, whereas for the second application, specific training rules are developed for a three-layer network architecture comprising a dynamical hidden layer for the identification of nonlinear input-output maps. A comparison with the performance of a standard backpropagation network provides an illustration of the capabilities of the present network architecture and the learning algorithm.
对于具有反馈和递归连接的动态神经网络而言，实现的复杂性一直是梯度下降学习算法开发中的主要难题。然而，正如本文所展示的，从网络平衡点的稳定性特性中获得的一些见解，表明对 sigmoidal 非线性函数进行适当调整，可用于获得简化的学习规则。给出了特定条件下学习方案收敛性的解析证明，并为有效实施训练过程制定了适应参数的一些上限。通过将学习算法应用于两个重要问题，即关联记忆设计和非线性输入 - 输出映射，来说明学习算法的性能特征。对于第一个应用，给出了一个系统的程序来训练网络，以将多个记忆向量存储为其稳定平衡点；而对于第二个应用，为包含用于识别非线性输入 - 输出映射的动态隐藏层的三层网络架构开发了特定的训练规则。与标准反向传播网络的性能比较，说明了当前网络架构和学习算法的能力。
