Cold Spring Harbor Laboratory, Simons Center for Quantitative Biology, Cold Spring Harbor, NY 11724, U.S.A.
Neural Comput. 2021 Nov 12;33(12):3179-3203. doi: 10.1162/neco_a_01439.
A fundamental challenge at the interface of machine learning and neuroscience is to uncover computational principles that are shared between artificial and biological neural networks. In deep learning, normalization methods such as batch normalization, weight normalization, and their many variants help to stabilize hidden unit activity and accelerate network training, and these methods have been called one of the most important recent innovations for optimizing deep networks. In the brain, homeostatic plasticity represents a set of mechanisms that also stabilize and normalize network activity to lie within certain ranges, and these mechanisms are critical for maintaining normal brain function. In this article, we discuss parallels between artificial and biological normalization methods at four spatial scales: normalization of a single neuron's activity, normalization of synaptic weights of a neuron, normalization of a layer of neurons, and normalization of a network of neurons. We argue that both types of methods are functionally equivalent-that is, both push activation patterns of hidden units toward a homeostatic state, where all neurons are equally used-and we argue that such representations can improve coding capacity, discrimination, and regularization. As a proof of concept, we develop an algorithm, inspired by a neural normalization technique called synaptic scaling, and show that this algorithm performs competitively against existing normalization methods on several data sets. Overall, we hope this bidirectional connection will inspire neuroscientists and machine learners in three ways: to uncover new normalization algorithms based on established neurobiological principles; to help quantify the trade-offs of different homeostatic plasticity mechanisms used in the brain; and to offer insights about how stability may not hinder, but may actually promote, plasticity.
机器学习和神经科学界面的一个基本挑战是揭示人工神经网络和生物神经网络之间共享的计算原理。在深度学习中，归一化方法（如批量归一化、权重归一化及其许多变体）有助于稳定隐藏单元活动并加速网络训练，这些方法被称为优化深度网络的最重要的创新之一。在大脑中，稳态可塑性代表了一组机制，这些机制也可以稳定和规范网络活动，使其处于一定范围内，这些机制对于维持正常的大脑功能至关重要。在本文中，我们在四个空间尺度上讨论了人工和生物归一化方法之间的相似之处：单个神经元活动的归一化、神经元突触权重的归一化、神经元层的归一化和神经元网络的归一化。我们认为这两种类型的方法在功能上是等效的——也就是说，它们都将隐藏单元的激活模式推向稳态，使所有神经元都得到同等利用——我们认为这种表示可以提高编码能力、区分能力和正则化能力。作为概念验证，我们受一种称为突触缩放的神经归一化技术启发，开发了一种算法，并表明该算法在几个数据集上与现有的归一化方法具有竞争力。总的来说，我们希望这种双向联系能在三个方面启发神经科学家和机器学习研究员：基于已确立的神经生物学原理发现新的归一化算法；帮助量化大脑中使用的不同稳态可塑性机制的权衡取舍；并提供关于稳定性如何可能不会阻碍而是实际上促进可塑性的见解。
