IEEE Trans Cybern. 2021 Sep;51(9):4350-4362. doi: 10.1109/TCYB.2020.2972944. Epub 2021 Sep 15.
Face photo-sketch synthesis aims at generating a facial sketch/photo conditioned on a given photo/sketch. It covers wide applications including digital entertainment and law enforcement. Precisely depicting face photos/sketches remains challenging due to the restrictions on structural realism and textural consistency. While existing methods achieve compelling results, they mostly yield blurred effects and great deformation over various facial components, leading to the unrealistic feeling of synthesized images. To tackle this challenge, in this article, we propose using facial composition information to help the synthesis of face sketch/photo. Especially, we propose a novel composition-aided generative adversarial network (CA-GAN) for face photo-sketch synthesis. In CA-GAN, we utilize paired inputs, including a face photo/sketch and the corresponding pixelwise face labels for generating a sketch/photo. Next, to focus training on hard-generated components and delicate facial structures, we propose a compositional reconstruction loss. In addition, we employ a perceptual loss function to encourage the synthesized image and real image to be perceptually similar. Finally, we use stacked CA-GANs (SCA-GANs) to further rectify defects and add compelling details. The experimental results show that our method is capable of generating both visually comfortable and identity-preserving face sketches/photos over a wide range of challenging data. In addition, our method significantly decreases the best previous Fréchet inception distance (FID) from 36.2 to 26.2 for sketch synthesis, and from 60.9 to 30.5 for photo synthesis. Besides, we demonstrate that the proposed method is of considerable generalization ability.
人脸照片-素描合成旨在根据给定的照片/素描生成人脸素描/照片。它涵盖了广泛的应用，包括数字娱乐和执法。由于结构现实主义和纹理一致性的限制，精确描绘人脸照片/素描仍然具有挑战性。虽然现有方法取得了引人注目的结果，但它们大多会在各种面部成分上产生模糊效果和较大的变形，导致合成图像的不真实感。为了解决这个挑战，在本文中，我们提出使用面部组成信息来帮助人脸素描/照片的合成。特别是，我们提出了一种新颖的基于组成的生成对抗网络（CA-GAN）用于人脸照片-素描合成。在 CA-GAN 中，我们利用配对输入，包括人脸照片/素描和相应的像素级人脸标签，来生成素描/照片。接下来，为了将训练重点放在难以生成的成分和精细的面部结构上，我们提出了一种组合重建损失。此外，我们采用感知损失函数来鼓励合成图像和真实图像在感知上相似。最后，我们使用堆叠 CA-GAN（SCA-GAN）来进一步纠正缺陷并添加引人注目的细节。实验结果表明，我们的方法能够在广泛的具有挑战性的数据上生成既视觉舒适又保持身份的人脸素描/照片。此外，我们的方法将之前最好的 Fréchet inception distance（FID）从草图合成的 36.2 显著降低到 26.2，从照片合成的 60.9 降低到 30.5。此外，我们证明了所提出的方法具有相当强的泛化能力。
