Suetake Kazuma, Ikegawa Shin-Ichi, Saiin Ryuji, Sawada Yoshihide
AISIN SOFTWARE, Aichi, Japan.
Tokyo Research Center, AISIN, Tokyo, Japan.
Neural Netw. 2023 Feb;159:208-219. doi: 10.1016/j.neunet.2022.12.008. Epub 2022 Dec 19.
As the scales of neural networks increase, techniques that enable them to run with low computational cost and energy efficiency are required. From such demands, various efficient neural network paradigms, such as spiking neural networks (SNNs) or binary neural networks (BNNs), have been proposed. However, they have sticky drawbacks, such as degraded inference accuracy and latency. To solve these problems, we propose a single-step spiking neural network (SNN), an energy-efficient neural network with low computational cost and high precision. The proposed SNN processes the information between hidden layers by spikes as SNNs. Nevertheless, it has no temporal dimension so that there is no latency within training and inference phases as BNNs. Thus, the proposed SNN has a lower computational cost than SNNs that require time-series processing. However, SNN cannot adopt naïve backpropagation algorithms due to the non-differentiability nature of spikes. We deduce a suitable neuron model by reducing the surrogate gradient for multi-time step SNNs to a single-time step. We experimentally demonstrated that the obtained surrogate gradient allows SNN to be trained appropriately. We also showed that the proposed SNN could achieve comparable accuracy to full-precision networks while being highly energy-efficient.
随着神经网络规模的增加，需要能够使其以低计算成本和高能效运行的技术。基于此类需求，已经提出了各种高效的神经网络范式，如脉冲神经网络（SNN）或二值神经网络（BNN）。然而，它们存在一些棘手的缺点，如推理精度下降和延迟。为了解决这些问题，我们提出了一种单步脉冲神经网络（SNN），这是一种具有低计算成本和高精度的高能效神经网络。所提出的SNN像SNN一样通过脉冲处理隐藏层之间的信息。然而，它没有时间维度，因此在训练和推理阶段没有像BNN那样的延迟。因此，所提出的SNN比需要时间序列处理的SNN具有更低的计算成本。然而，由于脉冲的不可微性质，SNN不能采用朴素的反向传播算法。我们通过将多时间步SNN的替代梯度简化为单时间步来推导合适的神经元模型。我们通过实验证明，所获得的替代梯度允许对SNN进行适当的训练。我们还表明，所提出的SNN在高能效的同时可以实现与全精度网络相当的精度。
