Yu Shujian, Wickstrom Kristoffer, Jenssen Robert, Principe Jose
IEEE Trans Neural Netw Learn Syst. 2021 Jan;32(1):435-442. doi: 10.1109/TNNLS.2020.2968509. Epub 2021 Jan 4.
A novel functional estimator for Rényi's α -entropy and its multivariate extension was recently proposed in terms of the normalized eigenspectrum of a Hermitian matrix of the projected data in a reproducing kernel Hilbert space (RKHS). However, the utility and possible applications of these new estimators are rather new and mostly unknown to practitioners. In this brief, we first show that this estimator enables straightforward measurement of information flow in realistic convolutional neural networks (CNNs) without any approximation. Then, we introduce the partial information decomposition (PID) framework and develop three quantities to analyze the synergy and redundancy in convolutional layer representations. Our results validate two fundamental data processing inequalities and reveal more inner properties concerning CNN training.
最近，针对再生核希尔伯特空间（RKHS）中投影数据的埃尔米特矩阵的归一化特征谱，提出了一种用于雷尼α熵及其多元扩展的新型功能估计器。然而，这些新估计器的实用性和可能的应用相当新颖，从业者大多并不了解。在本简报中，我们首先表明，这种估计器能够在现实的卷积神经网络（CNN）中直接测量信息流，而无需任何近似。然后，我们引入部分信息分解（PID）框架，并开发三个量来分析卷积层表示中的协同作用和冗余。我们的结果验证了两个基本的数据处理不等式，并揭示了更多关于CNN训练的内在特性。
