Huang Siyu, An Jie, Wei Donglai, Lin Zudi, Luo Jiebo, Pfister Hanspeter
IEEE Trans Pattern Anal Mach Intell. 2023 Oct;45(10):11707-11719. doi: 10.1109/TPAMI.2023.3287774. Epub 2023 Sep 5.
Unpaired image-to-image translation (UNIT) aims to map images between two visual domains without paired training data. However, given a UNIT model trained on certain domains, it is difficult for current methods to incorporate new domains because they often need to train the full model on both existing and new domains. To address this problem, we propose a new domain-scalable UNIT method, termed as latent space anchoring, which can be efficiently extended to new visual domains and does not need to fine-tune encoders and decoders of existing domains. Our method anchors images of different domains to the same latent space of frozen GANs by learning lightweight encoder and regressor models to reconstruct single-domain images. In the inference phase, the learned encoders and decoders of different domains can be arbitrarily combined to translate images between any two domains without fine-tuning. Experiments on various datasets show that the proposed method achieves superior performance on both standard and domain-scalable UNIT tasks in comparison with the state-of-the-art methods.
无配对图像到图像翻译（UNIT）旨在在没有配对训练数据的情况下在两个视觉域之间映射图像。然而，对于在某些域上训练的UNIT模型，当前方法难以纳入新的域，因为它们通常需要在现有域和新域上对整个模型进行训练。为了解决这个问题，我们提出了一种新的域可扩展UNIT方法，称为潜在空间锚定，它可以有效地扩展到新的视觉域，并且不需要对现有域的编码器和解码器进行微调。我们的方法通过学习轻量级编码器和回归器模型来重建单域图像，将不同域的图像锚定到冻结GAN的相同潜在空间。在推理阶段，不同域的学习到的编码器和解码器可以任意组合，以在任意两个域之间翻译图像而无需微调。在各种数据集上的实验表明，与现有方法相比，所提出的方法在标准和域可扩展UNIT任务上均取得了优异的性能。
