IEEE Trans Pattern Anal Mach Intell. 2020 Sep;42(9):2148-2164. doi: 10.1109/TPAMI.2019.2914039. Epub 2019 May 2.
In popular TV programs (such as CSI), a very low-resolution face image of a person, who is not even looking at the camera in many cases, is digitally super-resolved to a degree that suddenly the person's identity is made visible and recognizable. Of course, we suspect that this is merely a cinematographic special effect and such a magical transformation of a single image is not technically possible. Or, is it? In this paper, we push the boundaries of super-resolving (hallucinating to be more accurate) a tiny, non-frontal face image to understand how much of this is possible by leveraging the availability of large datasets and deep networks. To this end, we introduce a novel Transformative Adversarial Neural Network (TANN) to jointly frontalize very-low resolution (i.e., 16 × 16 pixels) out-of-plane rotated face images (including profile views) and aggressively super-resolve them (8×), regardless of their original poses and without using any 3D information. TANN is composed of two components: a transformative upsampling network which embodies encoding, spatial transformation and deconvolutional layers, and a discriminative network that enforces the generated high-resolution frontal faces to lie on the same manifold as real frontal face images. We evaluate our method on a large set of synthesized non-frontal face images to assess its reconstruction performance. Extensive experiments demonstrate that TANN generates both qualitatively and quantitatively superior results achieving over 4 dB improvement over the state-of-the-art.
在热门的电视节目（如 CSI）中，一个非常低分辨率的人脸图像，甚至在许多情况下，这个人都没有直视镜头，通过数字技术可以将其超分辨率处理到足以辨认出这个人的身份的程度。当然，我们怀疑这只是电影特效，这种单一图像的神奇转换在技术上是不可能实现的。或者，真的能实现吗？在本文中，我们通过利用大型数据集和深度网络，将极小的、非正面的人脸图像的超分辨率处理（更准确地说是幻觉）推向极限，以了解在多大程度上可以实现这一点。为此，我们引入了一种新颖的变换对抗神经网络（TANN），该网络可以共同将非常低分辨率（即 16×16 像素）的离面旋转人脸图像（包括侧视图）正面化，并对其进行积极的超分辨率处理（8×），而无需考虑其原始姿势，也无需使用任何 3D 信息。TANN 由两个组件组成：一个变换上采样网络，它体现了编码、空间变换和解卷积层，另一个判别网络，它强制生成的高分辨率正面人脸与真实正面人脸图像位于同一流形上。我们在大量合成的非正面人脸图像上评估了我们的方法，以评估其重建性能。广泛的实验表明，TANN 生成的结果在质量和数量上都优于现有方法，超过 4dB 的提升。
