Wang Ning, Niu Muyao, Wang Zhihui, Hu Kun, Liu Bin, Wang Zhiyong, Li Haojie
IEEE Trans Image Process. 2023;32:6142-6154. doi: 10.1109/TIP.2023.3326682. Epub 2023 Nov 8.
Automatic sketch colorization is a challenging task that aims to generate a color image from a sketch, primarily due to its inherently ill-posed nature. While many approaches have shown promising results, two significant challenges remain: limited color patterns and a wide range of artifacts such as color bleeding and semantic inconsistencies among relevant regions. These issues stem from the operation of traditional convolutional structures, which capture structural features in a pixel-wise manner, resulting in inadequate utilization of regional information within the sketch. Therefore, we propose the Region-Assisted Sketch Coloring (RASC) method, which introduces an intermediate representation called the 'Region Map' to explicitly characterize the regional information of the sketch. This Region Map is derived from the input sketch and is effectively formulated by our RASC architecture, enhancing the perception of region-wise features beyond the original pixel-wise features. Specifically, we start by employing the sketch encoder to extract hierarchical feature maps from the input sketches. Subsequently, we introduce a coarse-to-fine decoder comprising a series of Region-based Modulation (RM) blocks. This decoder modulates features that combine the modulation results of its previous block and the sketch features of the corresponding encoder block with our Region Formulation module. Each module explicitly formulates the sketch features in a region-wise manner. This accurately captures both the inner-region local style and inter-region global context dependency, resulting in various color patterns and fewer synthesis artifacts. Our experimental results show that our proposed method surpasses state-of-the-art methods in both synthetic and real sketch datasets.
自动草图上色是一项具有挑战性的任务,旨在从草图生成彩色图像,这主要是由于其本质上是不适定的。虽然许多方法都取得了不错的成果,但仍存在两个重大挑战:颜色模式有限以及存在大量伪像,如颜色渗色和相关区域之间的语义不一致。这些问题源于传统卷积结构的操作,该结构以像素方式捕捉结构特征,导致草图中区域信息利用不足。因此,我们提出了区域辅助草图上色(RASC)方法,该方法引入了一种名为“区域图”的中间表示,以明确表征草图的区域信息。此区域图源自输入草图,并由我们的RASC架构有效构建,增强了对超越原始像素特征的区域特征的感知。具体而言,我们首先使用草图编码器从输入草图中提取分层特征图。随后,我们引入了一个由一系列基于区域的调制(RM)块组成的从粗到精的解码器。该解码器通过我们的区域构建模块对结合了其前一个块的调制结果和相应编码器块的草图特征的特征进行调制。每个模块以区域方式明确构建草图特征。这准确地捕捉了区域内的局部风格和区域间的全局上下文依赖性,从而产生各种颜色模式并减少合成伪像。我们的实验结果表明,我们提出的方法在合成草图数据集和真实草图数据集上均优于现有方法。