Andrieu C, de Freitas N, Doucet A
Cambridge University Engineering Department, Cambridge CB2 1PZ, England.
Neural Comput. 2001 Oct;13(10):2359-407. doi: 10.1162/089976601750541831.
We propose a hierarchical full Bayesian model for radial basis networks. This model treats the model dimension (number of neurons), model parameters, regularization parameters, and noise parameters as unknown random variables. We develop a reversible-jump Markov chain Monte Carlo (MCMC) method to perform the Bayesian computation. We find that the results obtained using this method are not only better than the ones reported previously, but also appear to be robust with respect to the prior specification. In addition, we propose a novel and computationally efficient reversible-jump MCMC simulated annealing algorithm to optimize neural networks. This algorithm enables us to maximize the joint posterior distribution of the network parameters and the number of basis function. It performs a global search in the joint space of the parameters and number of parameters, thereby surmounting the problem of local minima to a large extent. We show that by calibrating the full hierarchical Bayesian prior, we can obtain the classical Akaike information criterion, Bayesian information criterion, and minimum description length model selection criteria within a penalized likelihood framework. Finally, we present a geometric convergence theorem for the algorithm with homogeneous transition kernel and a convergence theorem for the reversible-jump MCMC simulated annealing method.
我们提出了一种用于径向基网络的分层全贝叶斯模型。该模型将模型维度（神经元数量）、模型参数、正则化参数和噪声参数视为未知随机变量。我们开发了一种可逆跳跃马尔可夫链蒙特卡罗（MCMC）方法来进行贝叶斯计算。我们发现使用该方法获得的结果不仅优于先前报道的结果，而且相对于先验规范似乎也很稳健。此外，我们提出了一种新颖且计算高效的可逆跳跃MCMC模拟退火算法来优化神经网络。该算法使我们能够最大化网络参数和基函数数量的联合后验分布。它在参数和参数数量的联合空间中进行全局搜索，从而在很大程度上克服了局部最小值问题。我们表明，通过校准全分层贝叶斯先验，我们可以在惩罚似然框架内获得经典的赤池信息准则、贝叶斯信息准则和最小描述长度模型选择准则。最后，我们给出了具有齐次转移核的算法的几何收敛定理以及可逆跳跃MCMC模拟退火方法的收敛定理。
