Charvin Hippolyte, Catenacci Volpi Nicola, Polani Daniel
School of Physics, Engineering and Computer Science, University of Hertfordshire, Hatfield AL10 9AB, UK.
Entropy (Basel). 2023 Sep 19;25(9):1355. doi: 10.3390/e25091355.
The information bottleneck (IB) framework formalises the essential requirement for efficient information processing systems to achieve an optimal balance between the complexity of their representation and the amount of information extracted about relevant features. However, since the representation complexity affordable by real-world systems may vary in time, the processing cost of updating the representations should also be taken into account. A crucial question is thus the extent to which adaptive systems can , which target the same relevant features but at a different granularity. We investigate the information-theoretic optimal limits of this process by studying and extending, within the IB framework, the notion of , which describes the ideal situation where no information needs to be discarded for adapting an IB-optimal representation's granularity. Thanks in particular to a new geometric characterisation, we analytically derive the successive refinability of some specific IB problems (for binary variables, for jointly Gaussian variables, and for the relevancy variable being a deterministic function of the source variable), and provide a linear-programming-based tool to numerically investigate, in the discrete case, the successive refinement of the IB. We then soften this notion into a of the loss of information optimality induced by several-stage processing through an existing measure of unique information. Simple numerical experiments suggest that this quantity is typically low, though not entirely negligible. These results could have important implications for (i) the structure and efficiency of incremental learning in biological and artificial agents, (ii) the comparison of IB-optimal observation channels in statistical decision problems, and (iii) the IB theory of deep neural networks.
信息瓶颈（IB）框架将高效信息处理系统的基本要求形式化，以在其表示的复杂性与关于相关特征提取的信息量之间实现最佳平衡。然而，由于现实世界系统能够承受的表示复杂性可能随时间变化，更新表示的处理成本也应予以考虑。因此，一个关键问题是自适应系统能够在何种程度上……，这些系统针对相同的相关特征，但粒度不同。我们通过在IB框架内研究和扩展……的概念来研究此过程的信息论最优极限，该概念描述了在调整IB最优表示的粒度时无需丢弃任何信息的理想情况。特别借助一种新的几何表征，我们解析地推导出一些特定IB问题（针对二元变量、联合高斯变量以及相关性变量是源变量的确定性函数的情况）的逐次可细化性，并提供一种基于线性规划的工具，用于在离散情况下数值研究IB的逐次细化。然后，我们通过现有的唯一信息度量将此概念软化，以衡量多阶段处理导致的信息最优性损失。简单的数值实验表明，这个量通常较低，尽管并非完全可以忽略不计。这些结果可能对（i）生物和人工智能体中增量学习的结构和效率，（ii）统计决策问题中IB最优观测通道的比较，以及（iii）深度神经网络的IB理论具有重要意义。
