Huang Haiping
PMI Laboratory, School of Physics, Sun Yat-sen University, Guangzhou 510275, People's Republic of China.
Phys Rev E. 2020 Sep;102(3-1):030301. doi: 10.1103/PhysRevE.102.030301.
Unsupervised learning requiring only raw data is not only a fundamental function of the cerebral cortex, but also a foundation for a next generation of artificial neural networks. However, a unified theoretical framework to treat sensory inputs, synapses, and neural activity together is still lacking. The computational obstacle originates from the discrete nature of synapses, and complex interactions among these three essential elements of learning. Here, we propose a variational mean-field theory in which the distribution of synaptic weights is considered. The unsupervised learning can then be decomposed into two intertwined steps: A maximization step is carried out as a gradient ascent of the lower bound on the data log-likelihood, in which the synaptic weight distribution is determined by updating variational parameters, and an expectation step is carried out as a message passing procedure on an equivalent or dual neural network whose parameter is specified by the variational parameters of the weight distribution. Therefore, our framework provides insights on how data (or sensory inputs), synapses, and neural activities interact with each other to achieve the goal of extracting statistical regularities in sensory inputs. This variational framework is verified in restricted Boltzmann machines with planted synaptic weights and handwritten-digits learning.
仅需原始数据的无监督学习不仅是大脑皮层的一项基本功能，也是下一代人工神经网络的基础。然而，目前仍缺乏一个将感觉输入、突触和神经活动统一起来的理论框架。计算障碍源于突触的离散性质以及学习的这三个基本要素之间的复杂相互作用。在此，我们提出一种变分平均场理论，其中考虑了突触权重的分布。然后，无监督学习可分解为两个相互交织的步骤：最大化步骤作为数据对数似然下界的梯度上升来执行，在此步骤中通过更新变分参数来确定突触权重分布；期望步骤作为在一个等效或对偶神经网络上的消息传递过程来执行，该神经网络的参数由权重分布的变分参数指定。因此，我们的框架为数据（或感觉输入）、突触和神经活动如何相互作用以实现提取感觉输入中的统计规律这一目标提供了见解。这种变分框架在具有植入突触权重的受限玻尔兹曼机和手写数字学习中得到了验证。
