CEMS, School of Computing, University of Kent, Canterbury, United Kingdom.
Neural Netw. 2024 Jan;169:572-583. doi: 10.1016/j.neunet.2023.10.051. Epub 2023 Nov 4.
Supervised learning in deep neural networks is commonly performed using error backpropagation. However, the sequential propagation of errors during the backward pass limits its scalability and applicability to low-powered neuromorphic hardware. Therefore, there is growing interest in finding local alternatives to backpropagation. Recently proposed methods based on forward-mode automatic differentiation suffer from high variance in large deep neural networks, which affects convergence. In this paper, we propose the Forward Direct Feedback Alignment algorithm that combines Activity-Perturbed Forward Gradients with Direct Feedback Alignment and momentum. We provide both theoretical proofs and empirical evidence that our proposed method achieves lower variance than forward gradient techniques. In this way, our approach enables faster convergence and better performance when compared to other local alternatives to backpropagation and opens a new perspective for the development of online learning algorithms compatible with neuromorphic systems.
深度学习中的监督学习通常使用误差反向传播来实现。然而，反向传播过程中误差的顺序传播限制了其可扩展性和适用于低功耗神经形态硬件的能力。因此，人们越来越关注寻找反向传播的局部替代方法。最近提出的基于前向模式自动微分的方法在大型深度神经网络中存在较高的方差，这会影响收敛性。在本文中，我们提出了一种结合活动干扰前向梯度、直接反馈对齐和动量的前向直接反馈对齐算法。我们提供了理论证明和实验证据，表明我们提出的方法比前向梯度技术具有更低的方差。通过这种方式，与反向传播的其他局部替代方法相比，我们的方法能够实现更快的收敛和更好的性能，并为开发与神经形态系统兼容的在线学习算法开辟了新的视角。
