Laboratory of Sensory and Cognitive Neuroscience, Aix-Marseille University, Marseille, France; Neuroscience Lab, University Hospital Erlangen, Germany; Cognitive Computational Neuroscience Group, University Erlangen-Nürnberg (FAU), Germany.
Chair of Machine Intelligence, University Erlangen-Nürnberg (FAU), Germany.
Neural Netw. 2021 Jul;139:278-293. doi: 10.1016/j.neunet.2021.03.035. Epub 2021 Apr 5.
We introduce the Generalized Discrimination Value (GDV) that measures, in a non-invasive manner, how well different data classes separate in each given layer of an artificial neural network. It turns out that, at the end of the training period, the GDV in each given layer L attains a highly reproducible value, irrespective of the initialization of the network's connection weights. In the case of multi-layer perceptrons trained with error backpropagation, we find that classification of highly complex data sets requires a temporal reduction of class separability, marked by a characteristic 'energy barrier' in the initial part of the GDV(L) curve. Even more surprisingly, for a given data set, the GDV(L) is running through a fixed 'master curve', independently from the total number of network layers. Finally, due to its invariance with respect to dimensionality, the GDV may serve as a useful tool to compare the internal representational dynamics of artificial neural networks with different architectures for neural architecture search or network compression; or even with brain activity in order to decide between different candidate models of brain function.
我们引入了广义判别值（GDV），它以非侵入性的方式衡量不同数据类在人工神经网络的每个给定层中分离的程度。事实证明，在训练期末，GDV 在每个给定层 L 中达到一个高度可重复的值，而与网络连接权重的初始化无关。在使用误差反向传播训练的多层感知机的情况下，我们发现高度复杂数据集的分类需要类可分离性的时间减少，这在 GDV(L)曲线的初始部分标记为特征“能量障碍”。更令人惊讶的是，对于给定的数据集，GDV(L) 正在经历一个固定的“主曲线”，与网络层数的总数无关。最后，由于其对维度的不变性，GDV 可以作为一种有用的工具，用于比较具有不同架构的人工神经网络的内部表示动态，以进行神经架构搜索或网络压缩；甚至可以与大脑活动进行比较，以在不同的候选大脑功能模型之间做出决策。
