Department of Applied Science and Technology, Politecnico di Torino, Corso Duca degli Abruzzi 24, I-10129 Torino, Italy.
Human Genetics Foundation-Torino, Via Nizza 52, I-10126 Torino, Italy.
Phys Rev E. 2016 May;93(5):052313. doi: 10.1103/PhysRevE.93.052313. Epub 2016 May 27.
Learning in neural networks poses peculiar challenges when using discretized rather then continuous synaptic states. The choice of discrete synapses is motivated by biological reasoning and experiments, and possibly by hardware implementation considerations as well. In this paper we extend a previous large deviations analysis which unveiled the existence of peculiar dense regions in the space of synaptic states which accounts for the possibility of learning efficiently in networks with binary synapses. We extend the analysis to synapses with multiple states and generally more plausible biological features. The results clearly indicate that the overall qualitative picture is unchanged with respect to the binary case, and very robust to variation of the details of the model. We also provide quantitative results which suggest that the advantages of increasing the synaptic precision (i.e., the number of internal synaptic states) rapidly vanish after the first few bits, and therefore that, for practical applications, only few bits may be needed for near-optimal performance, consistent with recent biological findings. Finally, we demonstrate how the theoretical analysis can be exploited to design efficient algorithmic search strategies.
在使用离散而不是连续的突触状态时，神经网络中的学习带来了特殊的挑战。选择离散突触的动机是出于生物学推理和实验，也可能是出于硬件实现方面的考虑。在本文中，我们扩展了之前的大偏差分析，该分析揭示了突触状态空间中存在特殊密集区域的可能性，这解释了在具有二进制突触的网络中高效学习的可能性。我们将分析扩展到具有多个状态和更合理的生物学特征的突触。结果清楚地表明，与二进制情况相比，整体定性图不变，并且对模型细节的变化非常稳健。我们还提供了定量结果，表明增加突触精度（即内部突触状态的数量）的优势在最初的几个比特后迅速消失，因此，对于实际应用，仅需要几个比特就可以实现接近最佳的性能，这与最近的生物学发现一致。最后，我们展示了如何利用理论分析来设计有效的算法搜索策略。
