Graduate School of Information Science and Technology, Hokkaido University, Kita 14 Nishi 9 Kita-ku, Sapporo, Japan.
Neural Netw. 2018 May;101:68-78. doi: 10.1016/j.neunet.2018.01.016. Epub 2018 Feb 13.
A method that uses an adaptive learning rate is presented for training neural networks. Unlike most conventional updating methods in which the learning rate gradually decreases during training, the proposed method increases or decreases the learning rate adaptively so that the training loss (the sum of cross-entropy losses for all training samples) decreases as much as possible. It thus provides a wider search range for solutions and thus a lower test error rate. The experiments with some well-known datasets to train a multilayer perceptron show that the proposed method is effective for obtaining a better test accuracy under certain conditions.
本文提出了一种使用自适应学习率的方法来训练神经网络。与大多数传统的更新方法不同，这些方法在训练过程中逐渐降低学习率，本文提出的方法则自适应地增加或减少学习率，以使训练损失（所有训练样本的交叉熵损失之和）尽可能减小。因此，它为解决方案提供了更广泛的搜索范围，从而降低了测试错误率。通过使用一些著名的数据集训练多层感知机的实验表明，在某些条件下，该方法可以有效地获得更好的测试准确性。
