Darscheid Paul, Guthke Anneli, Ehret Uwe
Institute of Water Resources and River Basin Management, Karlsruhe Institute of Technology-KIT, 76131 Karlsruhe, Germany.
Institute for Modelling Hydraulic and Environmental Systems (IWS), University of Stuttgart, 70569 Stuttgart, Germany.
Entropy (Basel). 2018 Aug 13;20(8):601. doi: 10.3390/e20080601.
When constructing discrete (binned) distributions from samples of a data set, applications exist where it is desirable to assure that all bins of the sample distribution have nonzero probability. For example, if the sample distribution is part of a predictive model for which we require returning a response for the entire codomain, or if we use Kullback-Leibler divergence to measure the (dis-)agreement of the sample distribution and the original distribution of the variable, which, in the described case, is inconveniently infinite. Several sample-based distribution estimators exist which assure nonzero bin probability, such as adding one counter to each zero-probability bin of the sample histogram, adding a small probability to the sample pdf, smoothing methods such as Kernel-density smoothing, or Bayesian approaches based on the Dirichlet and Multinomial distribution. Here, we suggest and test an approach based on the Clopper-Pearson method, which makes use of the binominal distribution. Based on the sample distribution, confidence intervals for bin-occupation probability are calculated. The mean of each confidence interval is a strictly positive estimator of the true bin-occupation probability and is convergent with increasing sample size. For small samples, it converges towards a uniform distribution, i.e., the method effectively applies a maximum entropy approach. We apply this nonzero method and four alternative sample-based distribution estimators to a range of typical distributions (uniform, Dirac, normal, multimodal, and irregular) and measure the effect with Kullback-Leibler divergence. While the performance of each method strongly depends on the distribution type it is applied to, on average, and especially for small sample sizes, the nonzero, the simple "add one counter", and the Bayesian Dirichlet-multinomial model show very similar behavior and perform best. We conclude that, when estimating distributions without an a priori idea of their shape, applying one of these methods is favorable.
当根据数据集的样本构建离散（分箱）分布时，存在一些应用场景，需要确保样本分布的所有箱都具有非零概率。例如，如果样本分布是预测模型的一部分，我们需要对整个值域返回响应；或者如果我们使用Kullback-Leibler散度来衡量样本分布与变量原始分布的（不）一致性，在上述情况下，这种不一致性是无穷大的，会带来不便。存在几种基于样本的分布估计器可确保箱概率非零，例如给样本直方图的每个零概率箱添加一个计数器，给样本概率密度函数添加一个小概率，采用核密度平滑等平滑方法，或基于狄利克雷分布和多项分布的贝叶斯方法。在此，我们提出并测试一种基于克洛普 - 皮尔逊方法的途径，该方法利用二项分布。基于样本分布，计算箱占用概率的置信区间。每个置信区间的均值是真实箱占用概率的严格正估计器，并且随着样本量的增加而收敛。对于小样本，它收敛于均匀分布，即该方法有效地应用了最大熵方法。我们将这种非零方法和四种基于样本的替代分布估计器应用于一系列典型分布（均匀分布、狄拉克分布、正态分布、多峰分布和不规则分布），并使用Kullback-Leibler散度来衡量效果。虽然每种方法的性能强烈依赖于所应用的分布类型，但总体而言，特别是对于小样本量，非零方法、简单的“添加一个计数器”方法以及贝叶斯狄利克雷 - 多项模型表现出非常相似的行为且性能最佳。我们得出结论，在没有关于分布形状的先验概念的情况下估计分布时，应用这些方法之一是有利的。
