van Meegen Alexander, Sompolinsky Haim
Center for Brain Science, Harvard University, Cambridge, MA, 02138, USA.
Edmond and Lily Safra Center for Brain Sciences, Hebrew University, Jerusalem, 9190401, Israel.
Nat Commun. 2025 Apr 9;16(1):3354. doi: 10.1038/s41467-025-58276-6.
Neural networks posses the crucial ability to generate meaningful representations of task-dependent features. Indeed, with appropriate scaling, supervised learning in neural networks can result in strong, task-dependent feature learning. However, the nature of the emergent representations is still unclear. To understand the effect of learning on representations, we investigate fully-connected, wide neural networks learning classification tasks using the Bayesian framework where learning shapes the posterior distribution of the network weights. Consistent with previous findings, our analysis of the feature learning regime (also known as 'non-lazy' regime) shows that the networks acquire strong, data-dependent features, denoted as coding schemes, where neuronal responses to each input are dominated by its class membership. Surprisingly, the nature of the coding schemes depends crucially on the neuronal nonlinearity. In linear networks, an analog coding scheme of the task emerges; in nonlinear networks, strong spontaneous symmetry breaking leads to either redundant or sparse coding schemes. Our findings highlight how network properties such as scaling of weights and neuronal nonlinearity can profoundly influence the emergent representations.
神经网络具有生成与任务相关特征的有意义表示的关键能力。事实上，通过适当的缩放，神经网络中的监督学习可以导致强大的、与任务相关的特征学习。然而，涌现表示的本质仍然不清楚。为了理解学习对表示的影响，我们使用贝叶斯框架研究全连接的宽神经网络学习分类任务，其中学习塑造了网络权重的后验分布。与之前的发现一致，我们对特征学习机制（也称为“非惰性”机制）的分析表明，网络获得了强大的、依赖数据的特征，称为编码方案，其中对每个输入的神经元反应由其类别成员主导。令人惊讶的是，编码方案的本质关键取决于神经元非线性。在线性网络中，出现任务的模拟编码方案；在非线性网络中，强烈的自发对称性破缺导致冗余或稀疏编码方案。我们的发现突出了诸如权重缩放和神经元非线性等网络属性如何深刻影响涌现表示。
