Wang Qingyu, Zhang Duzhen, Cai Xinyuan, Zhang Tielin, Xu Bo
Institute of Automation, Chinese Academy of Sciences, Beijing, China.
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China.
Front Neurosci. 2025 Jan 29;18:1516868. doi: 10.3389/fnins.2024.1516868. eCollection 2024.
Energy-efficient spikformer has been proposed by integrating the biologically plausible spiking neural network (SNN) and artificial transformer, whereby the spiking self-attention (SSA) is used to achieve both higher accuracy and lower computational cost. However, it seems that self-attention is not always necessary, especially in sparse spike-form calculation manners. In this article, we innovatively replace vanilla SSA (using dynamic bases calculating from Query and Key) with spike-form Fourier transform, wavelet transform, and their combinations (using fixed triangular or wavelets bases), based on a key hypothesis that both of them use a set of basis functions for information transformation. Hence, the Fourier-or-Wavelet-based spikformer (FWformer) is proposed and verified in visual classification tasks, including both static image and event-based video datasets. The FWformer can achieve comparable or even higher accuracies (0.4%-1.5%), higher running speed (9%-51% for training and 19%-70% for inference), reduced theoretical energy consumption (20%-25%), and reduced graphic processing unit (GPU) memory usage (4%-26%), compared to the standard spikformer. Our result indicates the continuous refinement of new transformers that are inspired either by biological discovery (spike-form), or information theory (Fourier or Wavelet transform), is promising.
通过将具有生物学合理性的脉冲神经网络（SNN）与人工变压器集成，提出了节能型脉冲former，其中脉冲自注意力（SSA）用于实现更高的准确性和更低的计算成本。然而，自注意力似乎并非总是必要的，特别是在稀疏脉冲形式的计算方式中。在本文中，基于一个关键假设，即它们都使用一组基函数进行信息变换，我们创新性地用脉冲形式的傅里叶变换、小波变换及其组合（使用固定的三角或小波基）取代了普通的SSA（使用从查询和键计算出的动态基）。因此，提出了基于傅里叶或小波的脉冲former（FWformer），并在视觉分类任务中进行了验证，包括静态图像和基于事件的视频数据集。与标准脉冲former相比，FWformer可以实现相当甚至更高的准确率（0.4%-1.5%）、更高的运行速度（训练时提高9%-51%，推理时提高19%-70%）、降低理论能耗（20%-25%）以及减少图形处理单元（GPU）内存使用量（4%-26%）。我们的结果表明，受生物学发现（脉冲形式）或信息理论（傅里叶或小波变换）启发的新型变压器的不断完善是有前景的。
