Zalman Oshri Dana, Fine Shai
School of Computer Science, Reichman University, Herzliya 4610101, Israel.
Data Science Institute, Reichman University, Herzliya 4610101, Israel.
Entropy (Basel). 2023 Oct 20;25(10):1468. doi: 10.3390/e25101468.
Variational inference provides a way to approximate probability densities through optimization. It does so by optimizing an upper or a lower bound of the likelihood of the observed data (the evidence). The classic variational inference approach suggests maximizing the Evidence Lower Bound (ELBO). Recent studies proposed to optimize the variational Rényi bound (VR) and the χ upper bound. However, these estimates, which are based on the Monte Carlo (MC) approximation, either underestimate the bound or exhibit a high variance. In this work, we introduce a new upper bound, termed the Variational Rényi Log Upper bound (VRLU), which is based on the existing VR bound. In contrast to the existing VR bound, the MC approximation of the VRLU bound maintains the upper bound property. Furthermore, we devise a (sandwiched) upper-lower bound variational inference method, termed the Variational Rényi Sandwich (VRS), to jointly optimize the upper and lower bounds. We present a set of experiments, designed to evaluate the new VRLU bound and to compare the VRS method with the classic Variational Autoencoder (VAE) and the VR methods. Next, we apply the VRS approximation to the Multiple-Source Adaptation problem (MSA). MSA is a real-world scenario where data are collected from multiple sources that differ from one another by their probability distribution over the input space. The main aim is to combine fairly accurate predictive models from these sources and create an accurate model for new, mixed target domains. However, many domain adaptation methods assume prior knowledge of the data distribution in the source domains. In this work, we apply the suggested VRS density estimate to the Multiple-Source Adaptation problem (MSA) and show, both theoretically and empirically, that it provides tighter error bounds and improved performance, compared to leading MSA methods.
变分推断提供了一种通过优化来近似概率密度的方法。它通过优化观测数据（证据）似然性的上界或下界来实现这一点。经典的变分推断方法建议最大化证据下界（ELBO）。最近的研究提出优化变分雷尼界（VR）和χ上界。然而，这些基于蒙特卡罗（MC）近似的估计要么低估了界，要么表现出高方差。在这项工作中，我们引入了一个新的上界，称为变分雷尼对数上界（VRLU），它基于现有的VR界。与现有的VR界不同，VRLU界的MC近似保持了上界性质。此外，我们设计了一种（夹逼）上下界变分推断方法，称为变分雷尼夹逼（VRS），以联合优化上下界。我们展示了一组实验，旨在评估新的VRLU界，并将VRS方法与经典变分自编码器（VAE）和VR方法进行比较。接下来，我们将VRS近似应用于多源适应问题（MSA）。MSA是一种现实世界场景，其中数据是从多个源收集的，这些源在输入空间上的概率分布彼此不同。主要目标是将来自这些源的相当准确的预测模型组合起来，并为新的混合目标域创建一个准确的模型。然而，许多域适应方法假设源域中数据分布的先验知识。在这项工作中，我们将建议的VRS密度估计应用于多源适应问题（MSA），并在理论和实证上表明，与领先的MSA方法相比，它提供了更紧的误差界和更好的性能。
