Zhu Rui-Jie, Zhang Malu, Zhao Qihang, Deng Haoyu, Duan Yule, Deng Liang-Jian
IEEE Trans Neural Netw Learn Syst. 2025 Mar;36(3):5112-5125. doi: 10.1109/TNNLS.2024.3377717. Epub 2025 Feb 28.
Spiking neural networks (SNNs) are attracting widespread interest due to their biological plausibility, energy efficiency, and powerful spatiotemporal information representation ability. Given the critical role of attention mechanisms in enhancing neural network performance, the integration of SNNs and attention mechanisms exhibits tremendous potential to deliver energy-efficient and high-performance computing paradigms. In this article, we present a novel temporal-channel joint attention mechanism for SNNs, referred to as TCJA-SNN. The proposed TCJA-SNN framework can effectively assess the significance of spike sequence from both spatial and temporal dimensions. More specifically, our essential technical contribution lies on: 1) we employ the squeeze operation to compress the spike stream into an average matrix. Then, we leverage two local attention mechanisms based on efficient 1-D convolutions to facilitate comprehensive feature extraction at the temporal and channel levels independently and 2) we introduce the cross-convolutional fusion (CCF) layer as a novel approach to model the interdependencies between the temporal and channel scopes. This layer effectively breaks the independence of these two dimensions and enables the interaction between features. Experimental results demonstrate that the proposed TCJA-SNN outperforms the state-of-the-art (SOTA) on all standard static and neuromorphic datasets, including Fashion-MNIST, CIFAR10, CIFAR100, CIFAR10-DVS, N-Caltech 101, and DVS128 Gesture. Furthermore, we effectively apply the TCJA-SNN framework to image generation tasks by leveraging a variation autoencoder. To the best of our knowledge, this study is the first instance where the SNN-attention mechanism has been employed for high-level classification and low-level generation tasks. Our implementation codes are available at https://github.com/ridgerchu/TCJA.
脉冲神经网络（SNN）因其生物学合理性、能源效率和强大的时空信息表示能力而受到广泛关注。鉴于注意力机制在提升神经网络性能方面的关键作用，SNN与注意力机制的融合展现出了提供节能且高性能计算范式的巨大潜力。在本文中，我们提出了一种用于SNN的新颖的时间-通道联合注意力机制，称为TCJA-SNN。所提出的TCJA-SNN框架能够有效地从空间和时间维度评估脉冲序列的重要性。更具体地说，我们的主要技术贡献在于：1）我们采用挤压操作将脉冲流压缩成一个平均矩阵。然后，我们利用基于高效一维卷积的两种局部注意力机制，分别在时间和通道层面促进全面的特征提取；2）我们引入交叉卷积融合（CCF）层作为一种新颖的方法来建模时间和通道范围之间的相互依赖关系。这一层有效地打破了这两个维度的独立性，并实现了特征之间的交互。实验结果表明，所提出的TCJA-SNN在所有标准静态和神经形态数据集上均优于当前最优方法（SOTA），包括Fashion-MNIST、CIFAR10、CIFAR100、CIFAR10-DVS、N-Caltech 101和DVS128 Gesture。此外，我们通过利用变分自编码器有效地将TCJA-SNN框架应用于图像生成任务。据我们所知，本研究是首次将SNN-注意力机制用于高级分类和低级生成任务。我们的实现代码可在https://github.com/ridgerchu/TCJA获取。
