Dipartimento di Matematica "Guido Castelnuovo", Sapienza Università di Roma, 00185 Roma, Italy.
Dipartimento di Matematica e Fisica "Ennio De Giorgi", Università del Salento, 73100 Lecce, Italy.
Phys Rev Lett. 2020 Jan 17;124(2):028301. doi: 10.1103/PhysRevLett.124.028301.
We consider a three-layer Sejnowski machine and show that features learnt via contrastive divergence have a dual representation as patterns in a dense associative memory of order P=4. The latter is known to be able to Hebbian store an amount of patterns scaling as N^{P-1}, where N denotes the number of constituting binary neurons interacting P wisely. We also prove that, by keeping the dense associative network far from the saturation regime (namely, allowing for a number of patterns scaling only linearly with N, while P>2) such a system is able to perform pattern recognition far below the standard signal-to-noise threshold. In particular, a network with P=4 is able to retrieve information whose intensity is O(1) even in the presence of a noise O(sqrt[N]) in the large N limit. This striking skill stems from a redundancy representation of patterns-which is afforded given the (relatively) low-load information storage-and it contributes to explain the impressive abilities in pattern recognition exhibited by new-generation neural networks. The whole theory is developed rigorously, at the replica symmetric level of approximation, and corroborated by signal-to-noise analysis and Monte Carlo simulations.
我们考虑一个三层的 Sejnowski 机器，并表明通过对比散度学习到的特征具有作为密集联想记忆中模式的对偶表示，其阶数 P=4。后者已知能够通过海伯存储与构成二进制神经元的数量 N^{P-1}成比例的模式数量，其中 N 表示相互作用 P 次的二进制神经元的数量。我们还证明，通过使密集联想网络远离饱和状态（即，允许模式的数量仅与 N 线性缩放，而 P>2），这样的系统能够在远低于标准信噪比阈值的情况下进行模式识别。特别是，一个具有 P=4 的网络能够在大 N 极限下，即使在噪声 O(sqrt[N])的情况下，也能够检索 O(1)强度的信息。这种惊人的技能源于模式的冗余表示，这是由于（相对）低负载信息存储所提供的，它有助于解释新一代神经网络在模式识别方面表现出的令人印象深刻的能力。整个理论是在 replica 对称近似水平上严格发展的，并通过信噪比分析和蒙特卡罗模拟得到证实。
