IEEE Trans Vis Comput Graph. 2023 Apr;29(4):2203-2210. doi: 10.1109/TVCG.2021.3126659. Epub 2023 Feb 28.
Caricature is a type of artistic style of human faces that attracts considerable attention in the entertainment industry. So far a few 3D caricature generation methods exist and all of them require some caricature information (e.g., a caricature sketch or 2D caricature) as input. This kind of input, however, is difficult to provide by non-professional users. In this paper, we propose an end-to-end deep neural network model that generates high-quality 3D caricatures directly from a normal 2D face photo. The most challenging issue for our system is that the source domain of face photos (characterized by normal 2D faces) is significantly different from the target domain of 3D caricatures (characterized by 3D exaggerated face shapes and textures). To address this challenge, we: (1) build a large dataset of 5,343 3D caricature meshes and use it to establish a PCA model in the 3D caricature shape space; (2) reconstruct a normal full 3D head from the input face photo and use its PCA representation in the 3D caricature shape space to establish correspondences between the input photo and 3D caricature shape; and (3) propose a novel character loss and a novel caricature loss based on previous psychological studies on caricatures. Experiments including a novel two-level user study show that our system can generate high-quality 3D caricatures directly from normal face photos.
漫画是一种吸引人关注的人类面部艺术风格。目前已经存在一些 3D 漫画生成方法，它们都需要一些漫画信息（例如，漫画草图或 2D 漫画）作为输入。然而，这种输入对于非专业用户来说很难提供。在本文中，我们提出了一种端到端的深度神经网络模型，可以直接从正常的 2D 人脸照片生成高质量的 3D 漫画。我们系统面临的最具挑战性的问题是，人脸照片的源域（以正常的 2D 人脸为特征）与 3D 漫画的目标域（以 3D 夸张的人脸形状和纹理为特征）有很大的不同。为了解决这个挑战，我们：（1）构建了一个包含 5343 个 3D 漫画网格的大型数据集，并使用它在 3D 漫画形状空间中建立一个 PCA 模型；（2）从输入人脸照片重建一个正常的全 3D 头部，并使用其在 3D 漫画形状空间中的 PCA 表示来建立输入照片与 3D 漫画形状之间的对应关系；（3）根据之前关于漫画的心理学研究，提出了一种新的特征损失和一种新的漫画损失。包括一项新的两级用户研究在内的实验表明，我们的系统可以直接从正常的人脸照片生成高质量的 3D 漫画。
