Xie Jianwen, Lu Yang, Gao Ruiqi, Zhu Song-Chun, Wu Ying Nian
IEEE Trans Pattern Anal Mach Intell. 2020 Jan;42(1):27-45. doi: 10.1109/TPAMI.2018.2879081. Epub 2018 Nov 1.
This paper studies the cooperative training of two generative models for image modeling and synthesis. Both models are parametrized by convolutional neural networks (ConvNets). The first model is a deep energy-based model, whose energy function is defined by a bottom-up ConvNet, which maps the observed image to the energy. We call it the descriptor network. The second model is a generator network, which is a non-linear version of factor analysis. It is defined by a top-down ConvNet, which maps the latent factors to the observed image. The maximum likelihood learning algorithms of both models involve MCMC sampling such as Langevin dynamics. We observe that the two learning algorithms can be seamlessly interwoven into a cooperative learning algorithm that can train both models simultaneously. Specifically, within each iteration of the cooperative learning algorithm, the generator model generates initial synthesized examples to initialize a finite-step MCMC that samples and trains the energy-based descriptor model. After that, the generator model learns from how the MCMC changes its synthesized examples. That is, the descriptor model teaches the generator model by MCMC, so that the generator model accumulates the MCMC transitions and reproduces them by direct ancestral sampling. We call this scheme MCMC teaching. We show that the cooperative algorithm can learn highly realistic generative models.
本文研究用于图像建模与合成的两个生成模型的协同训练。两个模型均由卷积神经网络（ConvNets）进行参数化。第一个模型是基于深度能量的模型，其能量函数由一个自下而上的ConvNet定义，该ConvNet将观测图像映射到能量。我们称其为描述符网络。第二个模型是生成器网络，它是因子分析的非线性版本。它由一个自上而下的ConvNet定义，该ConvNet将潜在因子映射到观测图像。两个模型的最大似然学习算法都涉及诸如朗之万动力学的MCMC采样。我们观察到这两种学习算法可以无缝交织成一种能够同时训练两个模型的协同学习算法。具体而言，在协同学习算法的每次迭代中，生成器模型生成初始合成示例以初始化一个有限步的MCMC，该MCMC对基于能量的描述符模型进行采样和训练。之后，生成器模型从MCMC如何改变其合成示例中学习。也就是说，描述符模型通过MCMC教导生成器模型，以便生成器模型积累MCMC转换并通过直接祖先采样进行再现。我们将此方案称为MCMC教导。我们表明该协同算法能够学习高度逼真的生成模型。
