Tang Hao, Shao Ling, Torr Philip H S, Sebe Nicu
IEEE Trans Pattern Anal Mach Intell. 2023 Jan;45(1):768-784. doi: 10.1109/TPAMI.2022.3155989. Epub 2022 Dec 5.
In this paper, we address the task of semantic-guided image generation. One challenge common to most existing image-level generation methods is the difficulty in generating small objects and detailed local textures. To address this, in this work we consider generating images using local context. As such, we design a local class-specific generative network using semantic maps as guidance, which separately constructs and learns subgenerators for different classes, enabling it to capture finer details. To learn more discriminative class-specific feature representations for the local generation, we also propose a novel classification module. To combine the advantages of both global image-level and local class-specific generation, a joint generation network is designed with an attention fusion module and a dual-discriminator structure embedded. Lastly, we propose a novel semantic-aware upsampling method, which has a larger receptive field and can take far-away pixels that are semantically related for feature upsampling, enabling it to better preserve semantic consistency for instances with the same semantic labels. Extensive experiments on two image generation tasks show the superior performance of the proposed method. State-of-the-art results are established by large margins on both tasks and on nine challenging public benchmarks. The source code and trained models are available at https://github.com/Ha0Tang/LGGAN.
在本文中，我们探讨语义引导图像生成任务。大多数现有图像级生成方法面临的一个共同挑战是生成小物体和详细局部纹理的困难。为了解决这个问题，在这项工作中我们考虑使用局部上下文来生成图像。因此，我们设计了一个以语义图为指导的局部特定类生成网络，该网络为不同类分别构建并学习子生成器，使其能够捕捉更精细的细节。为了学习用于局部生成的更具判别力的特定类特征表示，我们还提出了一种新颖的分类模块。为了结合全局图像级和局部特定类生成的优点，设计了一个联合生成网络，其中嵌入了注意力融合模块和双判别器结构。最后，我们提出了一种新颖的语义感知上采样方法，它具有更大的感受野，并且可以采用语义相关的远处像素进行特征上采样，从而能够更好地保持具有相同语义标签的实例的语义一致性。在两个图像生成任务上的大量实验表明了所提出方法的卓越性能。在这两个任务以及九个具有挑战性的公共基准上，该方法均大幅领先于当前最优结果。源代码和训练模型可在https://github.com/Ha0Tang/LGGAN获取。
