Hahnloser R
Institute for Theoretical Physics, ETHZ, Zürich, Switzerland.
Network. 1998 Aug;9(3):363-80.
The aim of this article is to investigate a mechanical description of learning. A framework for local and simple learning algorithms based on interpreting a neural network as a set of configuration constraints is proposed. For any architectural design and learning task, unsupervised and supervised algorithms can be derived, optionally using unconstrained and hidden neurons. Unlike algorithms based on the gradient in weight space, the proposed tangential correlation (TC) algorithms move along the gradient in state space. This results in optimal scaling properties and simple expressions for the weight updates. The number of synapses is much larger than the number of neurons. A constraint for neural states does not impose a unique constraint for synaptic weights. Which weights to assign credit to can be selected from a parametrization of all weight changes equivalently satisfying the state constraints. At the heart of the parametrization are minimal weight changes. Two supervised algorithms (differing by their parametrizations) operating on a three-layer perceptron are compared with standard backpropagation. The successful training of fixed points of recurrent networks is demonstrated. The unsupervised learning of oscillations with variable frequencies is performed on standard and more sophisticated recurrent networks. The results presented here can be useful both for the analysis and for the synthesis of learning algorithms.
本文旨在研究学习的一种力学描述。提出了一种基于将神经网络解释为一组配置约束的局部和简单学习算法框架。对于任何架构设计和学习任务，都可以推导出无监督和有监督算法，可选择使用无约束和隐藏神经元。与基于权重空间梯度的算法不同，所提出的切向相关（TC）算法沿着状态空间中的梯度移动。这导致了最优的缩放特性和权重更新的简单表达式。突触的数量远多于神经元的数量。对神经状态的约束并不会对突触权重施加唯一的约束。可以从等效满足状态约束的所有权重变化的参数化中选择赋予信用的权重。参数化的核心是最小权重变化。将在三层感知器上运行的两种有监督算法（因其参数化不同）与标准反向传播进行比较。展示了循环网络固定点的成功训练。在标准和更复杂的循环网络上进行了可变频率振荡的无监督学习。这里给出的结果对于学习算法的分析和合成都可能有用。
