IEEE Trans Pattern Anal Mach Intell. 2016 Dec;38(12):2487-2500. doi: 10.1109/TPAMI.2016.2533382. Epub 2016 Feb 23.
We propose minimum entropy rate simplification (MERS), an information-theoretic, parameterization-independent framework for simplifying generative models of stochastic processes. Applications include improving model quality for sampling tasks by concentrating the probability mass on the most characteristic and accurately described behaviors while de-emphasizing the tails, and obtaining clean models from corrupted data (nonparametric denoising). This is the opposite of the smoothing step commonly applied to classification models. Drawing on rate-distortion theory, MERS seeks the minimum entropy-rate process under a constraint on the dissimilarity between the original and simplified processes. We particularly investigate the Kullback-Leibler divergence rate as a dissimilarity measure, where, compatible with our assumption that the starting model is disturbed or inaccurate, the simplification rather than the starting model is used for the reference distribution of the divergence. This leads to analytic solutions for stationary and ergodic Gaussian processes and Markov chains. The same formulas are also valid for maximum-entropy smoothing under the same divergence constraint. In experiments, MERS successfully simplifies and denoises models from audio, text, speech, and meteorology.
我们提出了最小熵率简化（MERS）方法，这是一种基于信息论的、与参数无关的随机过程生成模型简化框架。其应用包括通过将概率质量集中在最具特征且描述准确的行为上，同时降低尾部的重要性，从而提高采样任务的模型质量，以及从损坏的数据中获得干净的模型（非参数去噪）。这与通常应用于分类模型的平滑步骤相反。MERS 借鉴了率失真理论，在原始过程和简化过程之间的相似度约束下，寻找最小熵率过程。我们特别研究了 Kullback-Leibler 散度率作为相似度度量，其中，与我们的假设一致，即起始模型受到干扰或不准确，简化而不是起始模型用于散度的参考分布。这为平稳和遍历的高斯过程和马尔可夫链提供了分析解。对于相同的散度约束下的最大熵平滑，也有相同的公式。在实验中，MERS 成功地简化和去噪了来自音频、文本、语音和气象学的数据模型。
