Shen Wang, Bao Wenbo, Zhai Guangtao, Chen Li, Min Xiongkuo, Gao Zhiyong
IEEE Trans Image Process. 2021;30:277-292. doi: 10.1109/TIP.2020.3033617. Epub 2020 Nov 20.
Video frame interpolation aims to improve users' watching experiences by generating high-frame-rate videos from low-frame-rate ones. Existing approaches typically focus on synthesizing intermediate frames using high-quality reference images. However, the captured reference frames may suffer from inevitable spatial degradations such as motion blur, sensor noise, etc. Few studies have approached the joint video enhancement problem, namely synthesizing high-frame-rate and high-quality results from low-frame-rate degraded inputs. In this paper, we propose a unified optimization framework for video frame interpolation with spatial degradations. Specifically, we develop a frame interpolation module with a pyramid structure to cyclically synthesize high-quality intermediate frames. The pyramid module features adjustable spatial receptive field and temporal scope, thus contributing to controllable computational complexity and restoration ability. Besides, we propose an inter-pyramid recurrent module to connect sequential models to exploit the temporal relationship. The pyramid module integrates the recurrent module, thus can iteratively synthesize temporally smooth results. And the pyramid modules share weights across iterations, thus it does not expand the model's parameter size. Our model can be generalized to several applications such as up-converting the frame rate of videos with motion blur, reducing compression artifacts, and jointly super-resolving low-resolution videos. Extensive experimental results demonstrate that our method performs favorably against state-of-the-art methods on various video frame interpolation and enhancement tasks.
视频帧插值旨在通过从低帧率视频生成高帧率视频来改善用户的观看体验。现有方法通常专注于使用高质量参考图像合成中间帧。然而，捕获的参考帧可能会受到不可避免的空间退化，如运动模糊、传感器噪声等。很少有研究探讨联合视频增强问题，即从低帧率退化输入合成高帧率和高质量的结果。在本文中，我们提出了一个用于处理具有空间退化的视频帧插值的统一优化框架。具体来说，我们开发了一个具有金字塔结构的帧插值模块，用于循环合成高质量的中间帧。金字塔模块具有可调整的空间感受野和时间范围，从而有助于控制计算复杂度和恢复能力。此外，我们提出了一个金字塔间循环模块来连接顺序模型，以利用时间关系。金字塔模块集成了循环模块，因此可以迭代地合成时间上平滑的结果。并且金字塔模块在迭代过程中共享权重，因此不会扩大模型的参数规模。我们的模型可以推广到多个应用，如对具有运动模糊的视频进行帧率上转换、减少压缩伪像以及联合超分辨率低分辨率视频。大量实验结果表明，我们的方法在各种视频帧插值和增强任务上优于现有方法。
