Su Wanchao, Ye Hui, Chen Shu-Yu, Gao Lin, Fu Hongbo
IEEE Trans Vis Comput Graph. 2023 Oct;29(10):4074-4088. doi: 10.1109/TVCG.2022.3178734. Epub 2023 Sep 1.
The research topic of sketch-to-portrait generation has witnessed a boost of progress with deep learning techniques. The recently proposed StyleGAN architectures achieve state-of-the-art generation ability but the original StyleGAN is not friendly for sketch-based creation due to its unconditional generation nature. To address this issue, we propose a direct conditioning strategy to better preserve the spatial information under the StyleGAN framework. Specifically, we introduce Spatially Conditioned StyleGAN (SC-StyleGAN for short), which explicitly injects spatial constraints to the original StyleGAN generation process. We explore two input modalities, sketches and semantic maps, which together allow users to express desired generation results more precisely and easily. Based on SC-StyleGAN, we present DrawingInStyles, a novel drawing interface for non-professional users to easily produce high-quality, photo-realistic face images with precise control, either from scratch or editing existing ones. Qualitative and quantitative evaluations show the superior generation ability of our method to existing and alternative solutions. The usability and expressiveness of our system are confirmed by a user study.
随着深度学习技术的发展，草图到肖像生成的研究主题取得了显著进展。最近提出的StyleGAN架构实现了先进的生成能力，但由于其无条件生成的性质，原始的StyleGAN对基于草图的创作并不友好。为了解决这个问题，我们提出了一种直接条件策略，以在StyleGAN框架下更好地保留空间信息。具体来说，我们引入了空间条件StyleGAN（简称为SC-StyleGAN），它将空间约束明确地注入到原始的StyleGAN生成过程中。我们探索了两种输入模式，即草图和语义映射，这使得用户能够更精确、更轻松地表达所需的生成结果。基于SC-StyleGAN，我们展示了DrawingInStyles，这是一个新颖的绘图界面，供非专业用户轻松地精确控制生成高质量、逼真的面部图像，既可以从头开始绘制，也可以编辑现有图像。定性和定量评估表明，我们的方法比现有及替代解决方案具有更优越的生成能力。用户研究证实了我们系统的可用性和表现力。
