Razavi Saman, Tolson Bryan A
Department of Civil and Environmental Engineering, University of Waterloo, Waterloo, ON N2L 3G1, Canada.
IEEE Trans Neural Netw. 2011 Oct;22(10):1588-98. doi: 10.1109/TNN.2011.2163169. Epub 2011 Aug 22.
Feedforward neural network is one of the most commonly used function approximation techniques and has been applied to a wide variety of problems arising from various disciplines. However, neural networks are black-box models having multiple challenges/difficulties associated with training and generalization. This paper initially looks into the internal behavior of neural networks and develops a detailed interpretation of the neural network functional geometry. Based on this geometrical interpretation, a new set of variables describing neural networks is proposed as a more effective and geometrically interpretable alternative to the traditional set of network weights and biases. Then, this paper develops a new formulation for neural networks with respect to the newly defined variables; this reformulated neural network (ReNN) is equivalent to the common feedforward neural network but has a less complex error response surface. To demonstrate the learning ability of ReNN, in this paper, two training methods involving a derivative-based (a variation of backpropagation) and a derivative-free optimization algorithms are employed. Moreover, a new measure of regularization on the basis of the developed geometrical interpretation is proposed to evaluate and improve the generalization ability of neural networks. The value of the proposed geometrical interpretation, the ReNN approach, and the new regularization measure are demonstrated across multiple test problems. Results show that ReNN can be trained more effectively and efficiently compared to the common neural networks and the proposed regularization measure is an effective indicator of how a network would perform in terms of generalization.
前馈神经网络是最常用的函数逼近技术之一，已被应用于各个学科中出现的各种各样的问题。然而，神经网络是黑箱模型，在训练和泛化方面存在多重挑战/困难。本文首先研究神经网络的内部行为，并对神经网络功能几何进行详细解释。基于这种几何解释，提出了一组描述神经网络的新变量，作为传统网络权重和偏差集的一种更有效且具有几何可解释性的替代方案。然后，本文针对新定义的变量开发了一种神经网络的新公式；这种重新公式化的神经网络（ReNN）等同于普通前馈神经网络，但具有较不复杂的误差响应面。为了证明ReNN的学习能力，本文采用了两种训练方法，一种基于导数（反向传播的一种变体），另一种是无导数优化算法。此外，基于所发展的几何解释提出了一种新的正则化度量，以评估和提高神经网络的泛化能力。在多个测试问题中展示了所提出的几何解释、ReNN方法和新正则化度量的价值。结果表明，与普通神经网络相比，ReNN可以更有效且高效地进行训练，并且所提出的正则化度量是一个网络在泛化方面表现如何的有效指标。
