Kim Dahun, Woo Sanghyun, Lee Joon-Young, Kweon In So
IEEE Trans Pattern Anal Mach Intell. 2020 May;42(5):1038-1052. doi: 10.1109/TPAMI.2019.2958083. Epub 2019 Dec 11.
Video inpainting aims to fill in spatio-temporal holes in videos with plausible content. Despite tremendous progress on deep learning-based inpainting of a single image, it is still challenging to extend these methods to video domain due to the additional time dimension. In this paper, we propose a recurrent temporal aggregation framework for fast deep video inpainting. In particular, we construct an encoder-decoder model, where the encoder takes multiple reference frames which can provide visible pixels revealed from the scene dynamics. These hints are aggregated and fed into the decoder. We apply a recurrent feedback in an auto-regressive manner to enforce temporal consistency in the video results. We propose two architectural designs based on this framework. Our first model is a blind video decaptioning network (BVDNet) that is designed to automatically remove and inpaint text overlays in videos without any mask information. Our BVDNet wins the first place in the ECCV Chalearn 2018 LAP Inpainting Competition Track 2: Video Decaptioning. Second, we propose a network for more general video inpainting (VINet) to deal with more arbitrary and larger holes. Video results demonstrate the advantage of our framework compared to state-of-the-art methods both qualitatively and quantitatively. The codes are available at https://github.com/mcahny/Deep-Video-Inpainting, and https://github.com/shwoo93/video_decaptioning.
视频修复旨在用合理的内容填充视频中的时空空洞。尽管基于深度学习的单图像修复取得了巨大进展，但由于额外的时间维度，将这些方法扩展到视频领域仍然具有挑战性。在本文中，我们提出了一种用于快速深度视频修复的循环时间聚合框架。具体来说，我们构建了一个编码器 - 解码器模型，其中编码器采用多个参考帧，这些参考帧可以提供从场景动态中揭示的可见像素。这些线索被聚合并输入到解码器中。我们以自回归的方式应用循环反馈来增强视频结果中的时间一致性。基于这个框架，我们提出了两种架构设计。我们的第一个模型是一个盲视频字幕去除网络（BVDNet），它被设计用于在没有任何掩码信息的情况下自动去除和修复视频中的文本覆盖。我们的BVDNet在ECCV 2018 Chalearn LAP修复竞赛赛道2：视频字幕去除中获得第一名。其次，我们提出了一个用于更通用视频修复的网络（VINet）来处理更任意和更大的空洞。视频结果在定性和定量方面都证明了我们的框架相对于现有方法的优势。代码可在https://github.com/mcahny/Deep-Video-Inpainting和https://github.com/shwoo93/video_decaptioning获取。
