Newton Thomas Robert, Nicola Wilten
Department of Mathematics and Statistics, University of Calgary, Calgary, Canada.
Hotchkiss Brain Institute, University of Calgary, Calgary, Canada.
PLoS Comput Biol. 2025 Jul 21;21(7):e1013224. doi: 10.1371/journal.pcbi.1013224. eCollection 2025 Jul.
Training spiking recurrent neural networks (SRNNs) presents significant challenges compared to standard recurrent neural networks (RNNs) that model neural firing rates more directly. Here, we investigate the origins of these difficulties by training networks of spiking neurons and their parameter-matched instantaneous rate-based RNNs on supervised learning tasks. We applied FORCE training to leaky integrate-and-fire spiking networks and their matched rate-based counterparts across various dynamical tasks, keeping the FORCE hyperparameters identical. We found that at slow learning rates, spiking and rate networks behaved similarly: FORCE training identified highly correlated weight matrix solutions, and both network types exhibited overlapping hyperparameter regions for successful convergence. Remarkably, these weight solutions were largely interchangeable-weights trained in the spiking network could be transferred to the rate network and vice versa while preserving correct dynamical decoding. However, at fast learning rates, the correlation between learned solutions dropped sharply, and the solutions were no longer fully interchangeable. Despite this, rate networks still functioned well when their weight matrices were replaced with those learned from spiking networks. Additionally, the two network types exhibited distinct behaviours across different sizes: faster learning improved performance in rate networks but had little effect in spiking networks, aside from increasing instability. Through analytic derivation, we further show that slower learning rates in FORCE effectively act as a low-pass filter on the principal components of the neural bases, selectively stabilizing the dominant correlated components across spiking and rate networks. Our results indicate that some of the difficulties in training spiking networks stem from the inherent spike-time variability in spiking systems-variability that is not present in rate networks. These challenges can be mitigated in FORCE training by selecting appropriately slow learning rates. Moreover, our findings suggest that the decoding solutions learned by FORCE for spiking networks approximate a cross-trial firing rate-based decoding.
与更直接地对神经放电率进行建模的标准循环神经网络（RNN）相比，训练脉冲循环神经网络（SRNN）面临着重大挑战。在此，我们通过在监督学习任务上训练脉冲神经元网络及其参数匹配的基于瞬时速率的RNN，来研究这些困难的根源。我们将FORCE训练应用于泄漏积分发放脉冲网络及其匹配的基于速率的对应网络，用于各种动态任务，同时保持FORCE超参数相同。我们发现，在低学习率下，脉冲网络和速率网络的行为相似：FORCE训练识别出高度相关的权重矩阵解，并且两种网络类型在成功收敛时表现出重叠的超参数区域。值得注意的是，这些权重解在很大程度上是可互换的——在脉冲网络中训练的权重可以转移到速率网络，反之亦然，同时保持正确的动态解码。然而，在高学习率下，学习到的解之间的相关性急剧下降，并且这些解不再完全可互换。尽管如此，当速率网络的权重矩阵被替换为从脉冲网络学习到的权重矩阵时，速率网络仍然运行良好。此外，这两种网络类型在不同规模下表现出不同的行为：更快的学习提高了速率网络的性能，但除了增加不稳定性之外，对脉冲网络几乎没有影响。通过解析推导，我们进一步表明，FORCE中较慢的学习率有效地对神经基元的主成分起到了低通滤波器的作用，有选择地稳定了脉冲网络和速率网络中占主导的相关成分。我们的结果表明，训练脉冲网络的一些困难源于脉冲系统中固有的尖峰时间变异性，而这种变异性在速率网络中不存在。通过选择适当的低学习率，这些挑战在FORCE训练中可以得到缓解。此外，我们的研究结果表明，FORCE为脉冲网络学习到的解码解近似于基于跨试验放电率的解码。
