Wang Yihan, Liu Shuang, Gao Xiao-Shan
Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, 100190, China; University of Chinese Academy of Sciences, Beijing, 101408, China.
Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, 100190, China; University of Chinese Academy of Sciences, Beijing, 101408, China.
Neural Netw. 2025 Mar;183:106983. doi: 10.1016/j.neunet.2024.106983. Epub 2024 Dec 4.
Stability analysis is an essential aspect of studying the generalization ability of deep learning, as it involves deriving generalization bounds for stochastic gradient descent-based training algorithms. Adversarial training is the most widely used defense against adversarial attacks. However, previous generalization bounds for adversarial training have not included information regarding data distribution. In this paper, we fill this gap by providing generalization bounds for stochastic gradient descent-based adversarial training that incorporate data distribution information. We utilize the concepts of on-average stability and high-order approximate Lipschitz conditions to examine how changes in data distribution and adversarial budget can affect robust generalization gaps. Our derived generalization bounds for both convex and non-convex losses are at least as good as the uniform stability-based counterparts which do not include data distribution information. Furthermore, our findings demonstrate how distribution shifts from data poisoning attacks can impact robust generalization.
稳定性分析是研究深度学习泛化能力的一个重要方面，因为它涉及为基于随机梯度下降的训练算法推导泛化界。对抗训练是抵御对抗攻击最广泛使用的防御方法。然而，先前对抗训练的泛化界并未包含有关数据分布的信息。在本文中，我们通过为基于随机梯度下降的对抗训练提供包含数据分布信息的泛化界来填补这一空白。我们利用平均稳定性和高阶近似利普希茨条件的概念来研究数据分布和对抗预算的变化如何影响鲁棒泛化差距。我们为凸损失和非凸损失推导的泛化界至少与不包含数据分布信息的基于均匀稳定性的对应界一样好。此外，我们的研究结果表明数据中毒攻击导致的分布偏移如何影响鲁棒泛化。
