IEEE Trans Image Process. 2022;31:1911-1923. doi: 10.1109/TIP.2022.3149237. Epub 2022 Feb 16.
To provide semantic image style transfer results which are consistent with human perception, transferring styles of semantic regions of the style image to their corresponding semantic regions of the content image is necessary. However, when the object categories between the content and style images are not the same, it is difficult to match semantic regions between two images for semantic image style transfer. To solve the semantic matching problem and guide the semantic image style transfer based on matched regions, we propose a novel semantic context-aware image style transfer method by performing semantic context matching followed by a hierarchical local-to-global network architecture. The semantic context matching aims to obtain the corresponding regions between the content and style images by using context correlations of different object categories. Based on the matching results, we retrieve semantic context pairs where each pair is composed of two semantically matched regions from the content and style images. To achieve semantic context-aware style transfer, a hierarchical local-to-global network architecture, which contains two sub-networks including the local context network and the global context network, is proposed. The former focuses on style transfer for each semantic context pair from the style image to the content image, and generates a local style transfer image storing the detailed style feature representations for corresponding semantic regions. The latter aims to derive the stylized image by considering the content, the style, and the intermediate local style transfer images, so that inconsistency between different corresponding semantic regions can be addressed and solved. The experimental results show that the stylized results using our method are more consistent with human perception compared with the state-of-the-art methods.
为了提供与人类感知一致的语义图像风格迁移结果，有必要将风格图像的语义区域的风格转移到内容图像的相应语义区域。然而，当内容图像和风格图像的对象类别不同时，很难匹配两幅图像之间的语义区域以进行语义图像风格迁移。为了解决语义匹配问题并指导基于匹配区域的语义图像风格迁移，我们提出了一种新颖的语义上下文感知图像风格迁移方法，通过执行语义上下文匹配，然后使用分层局部到全局网络架构。语义上下文匹配旨在通过不同对象类别的上下文相关性来获得内容图像和风格图像之间的对应区域。基于匹配结果，我们检索语义上下文对，其中每个对由内容和风格图像中两个语义匹配的区域组成。为了实现语义上下文感知风格迁移，提出了一种分层局部到全局网络架构，该架构包含两个子网，包括局部上下文网络和全局上下文网络。前者专注于从风格图像到内容图像的每个语义上下文对的风格迁移，并生成存储对应语义区域的详细风格特征表示的局部风格迁移图像。后者旨在通过考虑内容、风格和中间局部风格迁移图像来导出样式化图像，从而解决和解决不同对应语义区域之间的不一致性。实验结果表明，与最先进的方法相比，我们的方法生成的样式化结果更符合人类感知。
