Wang Longguang, Guo Yulan, Liu Li, Lin Zaiping, Deng Xinpu, An Wei
IEEE Trans Image Process. 2020 Jan 23. doi: 10.1109/TIP.2020.2967596.
Video super-resolution (SR) aims at generating a sequence of high-resolution (HR) frames with plausible and temporally consistent details from their low-resolution (LR) counterparts. The key challenge for video SR lies in the effective exploitation of temporal dependency between consecutive frames. Existing deep learning based methods commonly estimate optical flows between LR frames to provide temporal dependency. However, the resolution conflict between LR optical flows and HR outputs hinders the recovery of fine details. In this paper, we propose an end-to-end video SR network to super-resolve both optical flows and images. Optical flow SR from LR frames provides accurate temporal dependency and ultimately improves video SR performance. Specifically, we first propose an optical flow reconstruction network (OFRnet) to infer HR optical flows in a coarse-to-fine manner. Then, motion compensation is performed using HR optical flows to encode temporal dependency. Finally, compensated LR inputs are fed to a super-resolution network (SRnet) to generate SR results. Extensive experiments have been conducted to demonstrate the effectiveness of HR optical flows for SR performance improvement. Comparative results on the Vid4 and DAVIS-10 datasets show that our network achieves the state-of-the-art performance.
视频超分辨率（SR）旨在从低分辨率（LR）对应帧生成一系列具有合理且时间上一致细节的高分辨率（HR）帧。视频SR的关键挑战在于有效利用连续帧之间的时间依赖性。现有的基于深度学习的方法通常估计LR帧之间的光流以提供时间依赖性。然而，LR光流和HR输出之间的分辨率冲突阻碍了精细细节的恢复。在本文中，我们提出了一种端到端的视频SR网络，用于对光流和图像进行超分辨率处理。来自LR帧的光流SR提供了准确的时间依赖性，并最终提高了视频SR性能。具体而言，我们首先提出了一种光流重建网络（OFRnet），以粗到细的方式推断HR光流。然后，使用HR光流进行运动补偿以编码时间依赖性。最后，将补偿后的LR输入馈送到超分辨率网络（SRnet）以生成SR结果。已经进行了广泛的实验来证明HR光流对SR性能提升的有效性。在Vid4和DAVIS-10数据集上的比较结果表明，我们的网络实现了当前的最佳性能。
