Department of Computer Science and Engineering, INESC-ID & Instituto Superior Técnico - University of Lisbon, Av. Prof. Dr. Aníbal Cavaco Silva, Porto Salvo, 2744-016, Lisbon, Portugal.
Biol Cybern. 2024 Dec;118(5-6):267-276. doi: 10.1007/s00422-024-00995-y. Epub 2024 Sep 9.
It is generally assumed that the brain uses something akin to sparse distributed representations. These representations, however, are high-dimensional and consequently they affect classification performance of traditional Machine Learning models due to the "curse of dimensionality". In tasks for which there is a vast amount of labeled data, Deep Networks seem to solve this issue with many layers and a non-Hebbian backpropagation algorithm. The brain, however, seems to be able to solve the problem with few layers. In this work, we hypothesize that this happens by using Hebbian learning. Actually, the Hebbian-like learning rule of Restricted Boltzmann Machines learns the input patterns asymmetrically. It exclusively learns the correlation between non-zero values and ignores the zeros, which represent the vast majority of the input dimensionality. By ignoring the zeros the "curse of dimensionality" problem can be avoided. To test our hypothesis, we generated several sparse datasets and compared the performance of a Restricted Boltzmann Machine classifier with some Backprop-trained networks. The experiments using these codes confirm our initial intuition as the Restricted Boltzmann Machine shows a good generalization performance, while the Neural Networks trained with the backpropagation algorithm overfit the training data.
人们普遍认为大脑使用类似于稀疏分布式表示的方法。然而，这些表示是高维的，因此由于“维度灾难”，它们会影响传统机器学习模型的分类性能。在有大量标记数据的任务中，深度网络似乎通过使用许多层和非Hebbian反向传播算法来解决这个问题。然而，大脑似乎可以通过使用少量的层来解决这个问题。在这项工作中，我们假设这是通过Hebbian 学习来实现的。实际上，受限玻尔兹曼机的Hebbian 学习规则不对称地学习输入模式。它专门学习非零值之间的相关性，而忽略了零值，零值代表输入维度的绝大多数。通过忽略零值，可以避免“维度灾难”问题。为了验证我们的假设，我们生成了几个稀疏数据集，并比较了受限玻尔兹曼机分类器与一些反向传播训练网络的性能。使用这些代码进行的实验证实了我们最初的直觉，因为受限玻尔兹曼机表现出良好的泛化性能，而使用反向传播算法训练的神经网络过度拟合训练数据。
