Deng Wei, Lin Guang, Liang Faming
Department of Mathematics, Purdue University, West Lafayette, IN, USA.
Departments of Mathematics & School of Mechanical Engineering, Purdue University, West Lafayette, IN, USA.
Adv Neural Inf Process Syst. 2020 Dec;34:15725-15736.
We propose an adaptively weighted stochastic gradient Langevin dynamics algorithm (SGLD), so-called contour stochastic gradient Langevin dynamics (CSGLD), for Bayesian learning in big data statistics. The proposed algorithm is essentially a , which automatically the target distribution such that the simulation for a multi-modal distribution can be greatly facilitated. Theoretically, we prove a stability condition and establish the asymptotic convergence of the self-adapting parameter to a , regardless of the non-convexity of the original energy function; we also present an error analysis for the weighted averaging estimators. Empirically, the CSGLD algorithm is tested on multiple benchmark datasets including CIFAR10 and CIFAR100. The numerical results indicate its superiority over the existing state-of-the-art algorithms in training deep neural networks.
我们提出了一种自适应加权随机梯度朗之万动力学算法（SGLD），即所谓的轮廓随机梯度朗之万动力学（CSGLD），用于大数据统计中的贝叶斯学习。所提出的算法本质上是一种，它能自动目标分布，从而极大地促进对多模态分布的模拟。从理论上讲，我们证明了一个稳定性条件，并建立了自适应参数到一个的渐近收敛性，而不考虑原始能量函数的非凸性；我们还对加权平均估计器进行了误差分析。在经验上，CSGLD算法在包括CIFAR10和CIFAR100在内的多个基准数据集上进行了测试。数值结果表明，在训练深度神经网络方面，它优于现有的最先进算法。
