IEEE Trans Neural Netw Learn Syst. 2020 May;31(5):1763-1770. doi: 10.1109/TNNLS.2019.2920964. Epub 2019 Jul 15.
In this brief, heterogeneity and noise in big data are shown to increase the generalization error for a traditional learning regime utilized for deep neural networks (deep NNs). To reduce this error, while overcoming the issue of vanishing gradients, a direct error-driven learning (EDL) scheme is proposed. First, to reduce the impact of heterogeneity and data noise, the concept of a neighborhood is introduced. Using this neighborhood, an approximation of generalization error is obtained and an overall error, comprised of learning and the approximate generalization errors, is defined. A novel NN weight-tuning law is obtained through a layer-wise performance measure enabling the direct use of overall error for learning. Additional constraints are introduced into the layer-wise performance measure to guide and improve the learning process in the presence of noisy dimensions. The proposed direct EDL scheme effectively addresses the issue of heterogeneity and noise while mitigating vanishing gradients and noisy dimensions. A comprehensive simulation study is presented where the proposed approach is shown to mitigate the vanishing gradient problem while improving generalization by 6%.
在本研究中，我们发现大数据中的异质性和噪声会增加传统学习模式下深度神经网络（DNN）的泛化误差。为了降低这种误差，同时克服梯度消失的问题，我们提出了一种直接的误差驱动学习（EDL）方案。首先，为了降低异质性和数据噪声的影响，引入了邻域的概念。利用这个邻域，我们得到了泛化误差的近似值，并定义了一个包含学习和近似泛化误差的总误差。通过一种基于层的性能度量方法，得到了一个新的神经网络权重调整法则，使得总误差可以直接用于学习。在层的性能度量中引入了附加的约束条件，以在存在噪声维度的情况下指导和改进学习过程。所提出的直接 EDL 方案有效地解决了异质性和噪声问题，同时缓解了梯度消失和噪声维度的问题。我们进行了全面的仿真研究，结果表明，该方法不仅可以缓解梯度消失问题，还可以将泛化误差提高 6%。
