García Fernández Jesús, Ahmad Nasir, van Gerven Marcel
Department of Machine Learning and Neural Computing, Donders Institute for Brain, Cognition and Behaviour, Radboud University, 6500HB Nijmegen, The Netherlands.
Entropy (Basel). 2024 Dec 22;26(12):1125. doi: 10.3390/e26121125.
Learning is a fundamental property of intelligent systems, observed across biological organisms and engineered systems. While modern intelligent systems typically rely on gradient descent for learning, the need for exact gradients and complex information flow makes its implementation in biological and neuromorphic systems challenging. This has motivated the exploration of alternative learning mechanisms that can operate locally and do not rely on exact gradients. In this work, we introduce a novel approach that leverages noise in the parameters of the system and global reinforcement signals. Using an Ornstein-Uhlenbeck process with adaptive dynamics, our method balances exploration and exploitation during learning, driven by deviations from error predictions, akin to reward prediction error. Operating in continuous time, Ornstein-Uhlenbeck adaptation (OUA) is proposed as a general mechanism for learning in dynamic, time-evolving environments. We validate our approach across a range of different tasks, including supervised learning and reinforcement learning in feedforward and recurrent systems. Additionally, we demonstrate that it can perform meta-learning, adjusting hyper-parameters autonomously. Our results indicate that OUA provides a promising alternative to traditional gradient-based methods, with potential applications in neuromorphic computing. It also hints at a possible mechanism for noise-driven learning in the brain, where stochastic neurotransmitter release may guide synaptic adjustments.
学习是智能系统的一项基本属性，在生物有机体和工程系统中均有体现。虽然现代智能系统通常依靠梯度下降进行学习，但对精确梯度和复杂信息流的需求使其在生物和神经形态系统中的实现具有挑战性。这激发了人们对可在局部运行且不依赖精确梯度的替代学习机制的探索。在这项工作中，我们引入了一种新颖的方法，该方法利用系统参数中的噪声和全局强化信号。通过具有自适应动力学的奥恩斯坦-乌伦贝克过程，我们的方法在学习过程中平衡探索和利用，由与误差预测的偏差驱动，类似于奖励预测误差。在连续时间内运行，奥恩斯坦-乌伦贝克自适应（OUA）被提议作为在动态、随时间演变的环境中学习的一种通用机制。我们在一系列不同任务中验证了我们的方法，包括前馈和循环系统中的监督学习和强化学习。此外，我们证明它可以进行元学习，自主调整超参数。我们的结果表明，OUA为传统基于梯度的方法提供了一种有前景的替代方案，在神经形态计算中有潜在应用。它还暗示了大脑中噪声驱动学习的一种可能机制，其中随机神经递质释放可能指导突触调整。
