Data61, Commonwealth Scientific and Industrial Research Organization (CSIRO), Canberra ACT 2601, Australia; The Australian National University, Canberra ACT 0200, Australia.
University of Canberra, Bruce ACT 2617, Australia; Inception Institute of AI, Abu Dhabi, United Arab Emirates.
Neural Netw. 2019 Feb;110:82-90. doi: 10.1016/j.neunet.2018.09.009. Epub 2018 Oct 16.
The big breakthrough on the ImageNet challenge in 2012 was partially due to the 'Dropout' technique used to avoid overfitting. Here, we introduce a new approach called 'Spectral Dropout' to improve the generalization ability of deep neural networks. We cast the proposed approach in the form of regular Convolutional Neural Network (CNN) weight layers using a decorrelation transform with fixed basis functions. Our spectral dropout method prevents overfitting by eliminating weak and 'noisy' Fourier domain coefficients of the neural network activations, leading to remarkably better results than the current regularization methods. Furthermore, the proposed is very efficient due to the fixed basis functions used for spectral transformation. In particular, compared to Dropout and Drop-Connect, our method significantly speeds up the network convergence rate during the training process (roughly ×2), with considerably higher neuron pruning rates (an increase of ∼30%). We demonstrate that the spectral dropout can also be used in conjunction with other regularization approaches resulting in additional performance gains.
2012 年的 ImageNet 挑战赛上的重大突破部分归功于用于避免过拟合的“Dropout”技术。在这里，我们引入了一种称为“谱随机失活”的新方法来提高深度神经网络的泛化能力。我们通过使用具有固定基函数的去相关变换，将所提出的方法表示为常规卷积神经网络（CNN）的权重层。我们的谱随机失活方法通过消除神经网络激活的弱和“嘈杂”的傅里叶域系数来防止过拟合，从而导致比当前正则化方法更好的结果。此外，由于使用固定的基函数进行谱变换，因此所提出的方法非常有效。特别地，与 Dropout 和 Drop-Connect 相比，我们的方法在训练过程中显著提高了网络的收敛速度（大约快 2 倍），同时具有更高的神经元修剪率（增加约 30％）。我们证明，谱随机失活也可以与其他正则化方法结合使用，从而获得额外的性能提升。
