Sun Yinqian, Zeng Yi, Li Yang
Research Center for Brain-Inspired Intelligence, Institute of Automation, Chinese Academy of Sciences, Beijing, China.
School of Future Technology, University of Chinese Academy of Sciences, Beijing, China.
Front Neurosci. 2022 Aug 25;16:953368. doi: 10.3389/fnins.2022.953368. eCollection 2022.
Brain-inspired spiking neural networks (SNNs) are successfully applied to many pattern recognition domains. The SNNs-based deep structure has achieved considerable results in perceptual tasks, such as image classification and target detection. However, applying deep SNNs in reinforcement learning (RL) tasks is still a problem to be explored. Although there have been previous studies on the combination of SNNs and RL, most focus on robotic control problems with shallow networks or using the ANN-SNN conversion method to implement spiking deep Q networks (SDQN). In this study, we mathematically analyzed the problem of the disappearance of spiking signal features in SDQN and proposed a potential-based layer normalization (pbLN) method to train spiking deep Q networks directly. Experiment shows that compared with state-of-art ANN-SNN conversion method and other SDQN works, the proposed pbLN spiking deep Q networks (PL-SDQN) achieved better performance on Atari game tasks.
受脑启发的脉冲神经网络（SNN）已成功应用于许多模式识别领域。基于SNN的深度结构在诸如图像分类和目标检测等感知任务中取得了显著成果。然而，将深度SNN应用于强化学习（RL）任务仍是一个有待探索的问题。尽管之前已有关于SNN与RL相结合的研究，但大多数研究集中在使用浅层网络的机器人控制问题上，或使用人工神经网络-脉冲神经网络（ANN-SNN）转换方法来实现脉冲深度Q网络（SDQN）。在本研究中，我们从数学角度分析了SDQN中脉冲信号特征消失的问题，并提出了一种基于势的层归一化（pbLN）方法来直接训练脉冲深度Q网络。实验表明，与当前最先进的ANN-SNN转换方法及其他SDQN研究相比，所提出的基于pbLN的脉冲深度Q网络（PL-SDQN）在雅达利游戏任务中表现更佳。
