IEEE Trans Pattern Anal Mach Intell. 2022 Nov;44(11):7474-7489. doi: 10.1109/TPAMI.2021.3115428. Epub 2022 Oct 4.
Learning a good image prior is a long-term goal for image restoration and manipulation. While existing methods like deep image prior (DIP) capture low-level image statistics, there are still gaps toward an image prior that captures rich image semantics including color, spatial coherence, textures, and high-level concepts. This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images. As shown in Fig. 1, the deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, patch, resolution, of various degraded images. It also enables diverse image manipulation including random jittering, image morphing, and category transfer. Such highly flexible restoration and manipulation are made possible through relaxing the assumption of existing GAN inversion methods, which tend to fix the generator. Notably, we allow the generator to be fine-tuned on-the-fly in a progressive manner regularized by feature distance obtained by the discriminator in GAN. We show that these easy-to-implement and practical changes help preserve the reconstruction to remain in the manifold of nature images, and thus lead to more precise and faithful reconstruction for real images. Code is available at https://github.com/XingangPan/deep-generative-prior.
学习一个好的图像先验是图像恢复和处理的长期目标。虽然现有的方法，如深度图像先验（DIP）捕捉到了低层次的图像统计信息，但仍存在差距，即捕捉到包括颜色、空间一致性、纹理和高级概念在内的丰富图像语义的图像先验。这项工作提出了一种有效利用在大规模自然图像上训练的生成对抗网络（GAN）捕获的图像先验的方法。如图 1 所示，深度生成先验（DGP）为恢复缺失的语义提供了令人信服的结果，例如，各种退化图像的颜色、斑块、分辨率等。它还可以实现各种图像操作，包括随机抖动、图像变形和类别转换。通过放宽现有 GAN 反转方法的假设，即倾向于固定生成器，从而实现了这种高度灵活的恢复和操作。值得注意的是，我们允许生成器通过在 GAN 中由鉴别器获得的特征距离正则化的方式，实时进行微调。我们表明，这些易于实现和实用的更改有助于保持重建在自然图像流形内，从而导致对真实图像进行更精确和忠实的重建。代码可在 https://github.com/XingangPan/deep-generative-prior 获得。
