Lin Xinxu, Liu Mingxuan, Chen Hong
School of Integrated Circuits, Tsinghua University, Beijing, China.
State Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan University, Shanghai, China.
Front Comput Neurosci. 2024 Nov 26;18:1508297. doi: 10.3389/fncom.2024.1508297. eCollection 2024.
Event-based cameras are suitable for human action recognition (HAR) by providing movement perception with highly dynamic range, high temporal resolution, high power efficiency and low latency. Spike Neural Networks (SNNs) are naturally suited to deal with the asynchronous and sparse data from the event cameras due to their spike-based event-driven paradigm, with less power consumption compared to artificial neural networks. In this paper, we propose two end-to-end SNNs, namely Spike-HAR and Spike-HAR++, to introduce spiking transformer into event-based HAR. Spike-HAR includes two novel blocks: a spike attention branch, which enables model to focus on regions with high spike rates, reducing the impact of noise to improve the accuracy, and a parallel spike transformer block with simplified spiking self-attention mechanism, increasing computational efficiency. To better extract crucial information from high-level features, we modify the architecture of the spike attention branch and extend it in Spike-HAR to a higher dimension, proposing Spike-HAR++ to further enhance classification performance. Comprehensive experiments were conducted on four HAR datasets: SL-Animals-DVS, N-LSA64, DVS128 Gesture and DailyAction-DVS, to demonstrate the superior performance of our proposed model. Additionally, the proposed Spike-HAR and Spike-HAR++ require only 0.03 and 0.06 mJ, respectively, to process a sequence of event frames, with model sizes of only 0.7 and 1.8 M. This efficiency positions it as a promising new SNN baseline for the HAR community. Code is available at Spike-HAR++.
基于事件的相机通过提供具有高动态范围、高时间分辨率、高功率效率和低延迟的运动感知，适用于人类动作识别（HAR）。脉冲神经网络（SNN）由于其基于脉冲的事件驱动范式，天然适合处理来自事件相机的异步和稀疏数据，与人工神经网络相比功耗更低。在本文中，我们提出了两种端到端的SNN，即Spike-HAR和Spike-HAR++，将脉冲变换器引入基于事件的HAR中。Spike-HAR包括两个新颖的模块：一个脉冲注意力分支，使模型能够专注于高脉冲率区域，减少噪声影响以提高准确性；以及一个具有简化脉冲自注意力机制的并行脉冲变换器模块，提高计算效率。为了更好地从高级特征中提取关键信息，我们修改了脉冲注意力分支的架构，并在Spike-HAR中将其扩展到更高维度，提出了Spike-HAR++以进一步提高分类性能。我们在四个HAR数据集上进行了综合实验：SL-Animals-DVS、N-LSA64、DVS128 Gesture和DailyAction-DVS，以证明我们提出的模型的优越性能。此外，所提出的Spike-HAR和Spike-HAR++处理一系列事件帧分别仅需0.03和0.06 mJ，模型大小分别仅为0.7和1.8 M。这种效率使其成为HAR社区一个有前途的新SNN基线。代码可在Spike-HAR++获取。
