Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France.
Unité Mixte de Physique, CNRS, Thales, Université Paris-Saclay, Palaiseau, France.
Nat Commun. 2021 May 5;12(1):2549. doi: 10.1038/s41467-021-22768-y.
While deep neural networks have surpassed human performance in multiple situations, they are prone to catastrophic forgetting: upon training a new task, they rapidly forget previously learned ones. Neuroscience studies, based on idealized tasks, suggest that in the brain, synapses overcome this issue by adjusting their plasticity depending on their past history. However, such "metaplastic" behaviors do not transfer directly to mitigate catastrophic forgetting in deep neural networks. In this work, we interpret the hidden weights used by binarized neural networks, a low-precision version of deep neural networks, as metaplastic variables, and modify their training technique to alleviate forgetting. Building on this idea, we propose and demonstrate experimentally, in situations of multitask and stream learning, a training technique that reduces catastrophic forgetting without needing previously presented data, nor formal boundaries between datasets and with performance approaching more mainstream techniques with task boundaries. We support our approach with a theoretical analysis on a tractable task. This work bridges computational neuroscience and deep learning, and presents significant assets for future embedded and neuromorphic systems, especially when using novel nanodevices featuring physics analogous to metaplasticity.
虽然深度神经网络在多种情况下已经超越了人类的表现，但它们容易出现灾难性遗忘：在训练新任务时，它们会迅速忘记以前学习过的任务。神经科学研究基于理想化的任务表明，在大脑中，突触通过根据过去的历史调整其可塑性来克服这个问题。然而，这种“类重塑性”行为并不能直接转移到缓解深度神经网络中的灾难性遗忘。在这项工作中，我们将二进制神经网络（深度神经网络的低精度版本）使用的隐藏权重解释为类重塑性变量，并修改它们的训练技术以减轻遗忘。基于这个想法，我们提出并在多任务和流学习的情况下进行了实验验证，提出了一种在不需要以前呈现的数据、也不需要数据集之间的正式界限的情况下减轻灾难性遗忘的训练技术，并且其性能接近具有任务界限的更主流技术。我们在一个可处理的任务上进行了理论分析来支持我们的方法。这项工作连接了计算神经科学和深度学习，并为未来的嵌入式和神经形态系统提供了重要的资产，特别是在使用具有类似于类重塑性的物理特性的新型纳米器件时。
