Stan Matei-Ioan, Rhodes Oliver
Department of Computer Science, The University of Manchester, Manchester, UK.
Sci Rep. 2024 Sep 20;14(1):21957. doi: 10.1038/s41598-024-71678-8.
Spiking neural networks (SNNs) take inspiration from the brain to enable energy-efficient computations. Since the advent of Transformers, SNNs have struggled to compete with artificial networks on modern sequential tasks, as they inherit limitations from recurrent neural networks (RNNs), with the added challenge of training with non-differentiable binary spiking activations. However, a recent renewed interest in efficient alternatives to Transformers has given rise to state-of-the-art recurrent architectures named state space models (SSMs). This work systematically investigates, for the first time, the intersection of state-of-the-art SSMs with SNNs for long-range sequence modelling. Results suggest that SSM-based SNNs can outperform the Transformer on all tasks of a well-established long-range sequence modelling benchmark. It is also shown that SSM-based SNNs can outperform current state-of-the-art SNNs with fewer parameters on sequential image classification. Finally, a novel feature mixing layer is introduced, improving SNN accuracy while challenging assumptions about the role of binary activations in SNNs. This work paves the way for deploying powerful SSM-based architectures, such as large language models, to neuromorphic hardware for energy-efficient long-range sequence modelling.
脉冲神经网络（SNN）从大脑中汲取灵感，以实现高效节能的计算。自Transformer问世以来，SNN在现代序列任务中难以与人工网络竞争，因为它们继承了循环神经网络（RNN）的局限性，并且在使用不可微的二进制脉冲激活进行训练时面临额外挑战。然而，最近对Transformer的高效替代方案的重新关注催生了名为状态空间模型（SSM）的先进循环架构。这项工作首次系统地研究了用于长程序列建模的先进SSM与SNN的交集。结果表明，基于SSM的SNN在一个成熟的长程序列建模基准的所有任务上都能超越Transformer。研究还表明，基于SSM的SNN在序列图像分类中可以用更少的参数超越当前最先进的SNN。最后，引入了一种新颖的特征混合层，提高了SNN的准确性，同时对关于二进制激活在SNN中的作用的假设提出了挑战。这项工作为将强大的基于SSM的架构（如大语言模型）部署到神经形态硬件上以进行高效节能的长程序列建模铺平了道路。
