Bengio Y
Département d'informatique et recherche opérationnelle, Université de Montréal, Montréal, Québec, Canada, H3C 3J7.
Neural Comput. 2000 Aug;12(8):1889-900. doi: 10.1162/089976600300015187.
Many machine learning algorithms can be formulated as the minimization of a training criterion that involves a hyperparameter. This hyperparameter is usually chosen by trial and error with a model selection criterion. In this article we present a methodology to optimize several hyperparameters, based on the computation of the gradient of a model selection criterion with respect to the hyperparameters. In the case of a quadratic training criterion, the gradient of the selection criterion with respect to the hyperparameters is efficiently computed by backpropagating through a Cholesky decomposition. In the more general case, we show that the implicit function theorem can be used to derive a formula for the hyperparameter gradient involving second derivatives of the training criterion.
许多机器学习算法都可以表述为对一个涉及超参数的训练准则进行最小化。这个超参数通常通过基于模型选择准则的反复试验来选择。在本文中，我们提出了一种基于模型选择准则相对于超参数的梯度计算来优化多个超参数的方法。在二次训练准则的情况下，通过对乔列斯基分解进行反向传播，可以有效地计算选择准则相对于超参数的梯度。在更一般的情况下，我们表明隐函数定理可用于推导一个涉及训练准则二阶导数的超参数梯度公式。
