Ho Kevin, Leung Chi-Sing, Sum John
Department of Computer Science and Communication Engineering, Providence University, Taichung 43301, Taiwan.
IEEE Trans Neural Netw. 2011 Feb;22(2):317-23. doi: 10.1109/TNN.2010.2095881. Epub 2010 Dec 23.
Injecting weight noise during training has been a simple strategy to improve the fault tolerance of multilayer perceptrons (MLPs) for almost two decades, and several online training algorithms have been proposed in this regard. However, there are some misconceptions about the objective functions being minimized by these algorithms. Some existing results misinterpret that the prediction error of a trained MLP affected by weight noise is equivalent to the objective function of a weight noise injection algorithm. In this brief, we would like to clarify these misconceptions. Two weight noise injection scenarios will be considered: one is based on additive weight noise injection and the other is based on multiplicative weight noise injection. To avoid the misconceptions, we use their mean updating equations to analyze the objective functions. For injecting additive weight noise during training, we show that the true objective function is identical to the prediction error of a faulty MLP whose weights are affected by additive weight noise. It consists of the conventional mean square error and a smoothing regularizer. For injecting multiplicative weight noise during training, we show that the objective function is different from the prediction error of a faulty MLP whose weights are affected by multiplicative weight noise. With our results, some existing misconceptions regarding MLP training with weight noise injection can now be resolved.
在训练过程中注入权重噪声一直是一种提高多层感知器（MLP）容错能力的简单策略，近二十年来，人们针对这方面提出了几种在线训练算法。然而，对于这些算法所最小化的目标函数存在一些误解。一些现有结果错误地认为，受权重噪声影响的训练后的MLP的预测误差等同于权重噪声注入算法的目标函数。在本简报中，我们希望澄清这些误解。我们将考虑两种权重噪声注入场景：一种基于加性权重噪声注入，另一种基于乘性权重噪声注入。为避免误解，我们使用它们的均值更新方程来分析目标函数。对于在训练过程中注入加性权重噪声的情况，我们表明真实的目标函数与权重受加性权重噪声影响的有故障的MLP的预测误差相同。它由传统的均方误差和平滑正则化项组成。对于在训练过程中注入乘性权重噪声的情况，我们表明目标函数与权重受乘性权重噪声影响的有故障的MLP的预测误差不同。基于我们的结果，现在可以解决一些关于使用权重噪声注入进行MLP训练的现有误解。
