Ngampruetikorn Vudtiwat, Schwab David J
Initiative for the Theoretical Sciences, The Graduate Center, CUNY.
Adv Neural Inf Process Syst. 2022 Dec;35:9784-9796.
Avoiding overfitting is a central challenge in machine learning, yet many large neural networks readily achieve zero training loss. This puzzling contradiction necessitates new approaches to the study of overfitting. Here we quantify overfitting via residual information, defined as the bits in fitted models that encode noise in training data. Information efficient learning algorithms minimize residual information while maximizing the relevant bits, which are predictive of the unknown generative models. We solve this optimization to obtain the information content of optimal algorithms for a linear regression problem and compare it to that of randomized ridge regression. Our results demonstrate the fundamental trade-off between residual and relevant information and characterize the relative information efficiency of randomized regression with respect to optimal algorithms. Finally, using results from random matrix theory, we reveal the information complexity of learning a linear map in high dimensions and unveil information-theoretic analogs of double and multiple descent phenomena.
避免过拟合是机器学习中的核心挑战，然而许多大型神经网络很容易实现零训练损失。这种令人困惑的矛盾需要新的方法来研究过拟合。在这里，我们通过残差信息来量化过拟合，残差信息定义为拟合模型中编码训练数据噪声的比特。信息高效学习算法在最大化相关比特的同时最小化残差信息，这些相关比特可预测未知的生成模型。我们求解此优化问题以获得线性回归问题的最优算法的信息内容，并将其与随机岭回归的信息内容进行比较。我们的结果展示了残差信息与相关信息之间的基本权衡，并刻画了随机回归相对于最优算法的相对信息效率。最后，利用随机矩阵理论的结果，我们揭示了高维中学习线性映射的信息复杂性，并揭示了双重和多重下降现象的信息论类似物。
