Noh Yung-Kyun, Sugiyama Masashi, Liu Song, Plessis Marthinus C du, Park Frank Chongwoo, Lee Daniel D
Seoul National University, Seoul 08826, Korea
RIKEN, Tokyo 103-0027, Japan, and University of Tokyo, Chiba 277-8561, Japan
Neural Comput. 2018 Jul;30(7):1930-1960. doi: 10.1162/neco_a_01092. Epub 2018 Jun 14.
Nearest-neighbor estimators for the Kullback-Leiber (KL) divergence that are asymptotically unbiased have recently been proposed and demonstrated in a number of applications. However, with a small number of samples, nonparametric methods typically suffer from large estimation bias due to the nonlocality of information derived from nearest-neighbor statistics. In this letter, we show that this estimation bias can be mitigated by modifying the metric function, and we propose a novel method for learning a locally optimal Mahalanobis distance function from parametric generative models of the underlying density distributions. Using both simulations and experiments on a variety of data sets, we demonstrate that this interplay between approximate generative models and nonparametric techniques can significantly improve the accuracy of nearest-neighbor-based estimation of the KL divergence.
最近有人提出了用于估计Kullback-Leiber（KL）散度的渐近无偏最近邻估计器，并在许多应用中得到了验证。然而，对于少量样本，由于从最近邻统计中得出的信息具有非局部性，非参数方法通常会遭受较大的估计偏差。在本信函中，我们表明可以通过修改度量函数来减轻这种估计偏差，并且我们提出了一种从基础密度分布的参数生成模型中学习局部最优马氏距离函数的新方法。通过在各种数据集上进行模拟和实验，我们证明了近似生成模型与非参数技术之间的这种相互作用可以显著提高基于最近邻的KL散度估计的准确性。
