Neftci Emre O, Pedroni Bruno U, Joshi Siddharth, Al-Shedivat Maruan, Cauwenberghs Gert
Department of Cognitive Sciences, University of California, Irvine Irvine, CA, USA.
Department of Bioengineering, University of California San Diego, La Jolla, CA, USA.
Front Neurosci. 2016 Jun 29;10:241. doi: 10.3389/fnins.2016.00241. eCollection 2016.
Recent studies have shown that synaptic unreliability is a robust and sufficient mechanism for inducing the stochasticity observed in cortex. Here, we introduce Synaptic Sampling Machines (S2Ms), a class of neural network models that uses synaptic stochasticity as a means to Monte Carlo sampling and unsupervised learning. Similar to the original formulation of Boltzmann machines, these models can be viewed as a stochastic counterpart of Hopfield networks, but where stochasticity is induced by a random mask over the connections. Synaptic stochasticity plays the dual role of an efficient mechanism for sampling, and a regularizer during learning akin to DropConnect. A local synaptic plasticity rule implementing an event-driven form of contrastive divergence enables the learning of generative models in an on-line fashion. S2Ms perform equally well using discrete-timed artificial units (as in Hopfield networks) or continuous-timed leaky integrate and fire neurons. The learned representations are remarkably sparse and robust to reductions in bit precision and synapse pruning: removal of more than 75% of the weakest connections followed by cursory re-learning causes a negligible performance loss on benchmark classification tasks. The spiking neuron-based S2Ms outperform existing spike-based unsupervised learners, while potentially offering substantial advantages in terms of power and complexity, and are thus promising models for on-line learning in brain-inspired hardware.
最近的研究表明，突触不可靠性是诱导在皮层中观察到的随机性的一种强大且充分的机制。在此，我们引入突触采样机器（S2M），这是一类神经网络模型，其将突触随机性用作蒙特卡罗采样和无监督学习的一种手段。与玻尔兹曼机的原始形式类似，这些模型可被视为霍普菲尔德网络的随机对应物，但随机性是由连接上的随机掩码诱导产生的。突触随机性在采样中起到高效机制的双重作用，并且在学习过程中类似于随机失连（DropConnect）起到正则化作用。一种实现事件驱动形式对比散度的局部突触可塑性规则能够以在线方式学习生成模型。S2M使用离散时间人工单元（如在霍普菲尔德网络中）或连续时间泄漏积分发放神经元时表现同样良好。所学习到的表示非常稀疏，并且对于比特精度降低和突触修剪具有鲁棒性：去除超过75%的最弱连接，然后进行粗略的重新学习，在基准分类任务上导致的性能损失可忽略不计。基于脉冲神经元的S2M优于现有的基于脉冲的无监督学习器，同时在功率和复杂性方面可能具有显著优势，因此是用于受脑启发硬件中的在线学习的有前景的模型。
