Rodríguez Gálvez Borja, Thobaben Ragnar, Skoglund Mikael
Department of Intelligent Systems, Division of Information Science and Engineering (ISE), KTH Royal Institute of Technology, 11428 Stockholm, Sweden.
Entropy (Basel). 2020 Jan 14;22(1):98. doi: 10.3390/e22010098.
The information bottleneck (IB) problem tackles the issue of obtaining relevant compressed representations of some random variable for the task of predicting . It is defined as a constrained optimization problem that maximizes the information the representation has about the task, I ( T ; Y ) , while ensuring that a certain level of compression is achieved (i.e., I ( X ; T ) ≤ r ). For practical reasons, the problem is usually solved by maximizing the IB Lagrangian (i.e., L IB ( T ; β ) = I ( T ; Y ) - β I ( X ; T ) ) for many values of β ∈ [ 0 , 1 ] . Then, the curve of maximal I ( T ; Y ) for a given I ( X ; T ) is drawn and a representation with the desired predictability and compression is selected. It is known when is a deterministic function of , the IB curve cannot be explored and another Lagrangian has been proposed to tackle this problem: the squared IB Lagrangian: L sq - IB ( T ; β sq ) = I ( T ; Y ) - β sq I ( X ; T ) 2 . In this paper, we (i) present a general family of Lagrangians which allow for the exploration of the IB curve in all scenarios; (ii) provide the exact one-to-one mapping between the Lagrange multiplier and the desired compression rate for known IB curve shapes; and (iii) show we can approximately obtain a specific compression level with the convex IB Lagrangian for both known and unknown IB curve shapes. This eliminates the burden of solving the optimization problem for many values of the Lagrange multiplier. That is, we prove that we can solve the original constrained problem with a single optimization.
信息瓶颈（IB）问题解决了在预测任务中获取某个随机变量的相关压缩表示的问题。它被定义为一个约束优化问题，该问题在确保达到一定压缩水平（即(I(X;T) \leq r)）的同时，最大化表示关于任务的信息(I(T;Y))。出于实际原因，通常通过针对许多(\beta \in [0,1])的值最大化IB拉格朗日函数（即(L_{IB}(T;\beta)=I(T;Y)-\beta I(X;T))）来解决该问题。然后，绘制给定(I(X;T))时最大(I(T;Y))的曲线，并选择具有所需可预测性和压缩性的表示。已知当(T)是(X)的确定性函数时，无法探索IB曲线，因此提出了另一种拉格朗日函数来解决此问题：平方IB拉格朗日函数：(L_{sq - IB}(T;\beta_{sq}) = I(T;Y)-\beta_{sq}I(X;T)^2)。在本文中，我们（i）提出了一族通用的拉格朗日函数，它们允许在所有情况下探索IB曲线；（ii）为已知的IB曲线形状提供拉格朗日乘子与所需压缩率之间的确切一一映射；（iii）表明对于已知和未知的IB曲线形状，我们都可以使用凸IB拉格朗日函数近似获得特定的压缩水平。这消除了针对拉格朗日乘子的许多值求解优化问题的负担。也就是说，我们证明了可以通过单次优化来解决原始的约束问题。
