Li Xiaoyu, Zhang Bo, Liao Jing, Sander Pedro V
IEEE Trans Vis Comput Graph. 2022 Aug;28(8):2938-2952. doi: 10.1109/TVCG.2021.3049419. Epub 2022 Jun 30.
We propose a novel framework to produce cartoon videos by fetching the color information from two input keyframes while following the animated motion guided by a user sketch. The key idea of the proposed approach is to estimate the dense cross-domain correspondence between the sketch and cartoon video frames, and employ a blending module with occlusion estimation to synthesize the middle frame guided by the sketch. After that, the input frames and the synthetic frame equipped with established correspondence are fed into an arbitrary-time frame interpolation pipeline to generate and refine additional inbetween frames. Finally, a module to preserve temporal consistency is employed. Compared to common frame interpolation methods, our approach can address frames with relatively large motion and also has the flexibility to enable users to control the generated video sequences by editing the sketch guidance. By explicitly considering the correspondence between frames and the sketch, we can achieve higher quality results than other image synthesis methods. Our results show that our system generalizes well to different movie frames, achieving better results than existing solutions.
我们提出了一种新颖的框架，通过从两个输入关键帧中获取颜色信息，同时遵循用户草图引导的动画运动来生成卡通视频。所提出方法的关键思想是估计草图与卡通视频帧之间的密集跨域对应关系，并采用带有遮挡估计的混合模块，以草图为引导合成中间帧。之后，将输入帧和具有已建立对应关系的合成帧输入到任意时间帧插值管道中，以生成并细化额外的中间帧。最后，使用一个模块来保持时间一致性。与常见的帧插值方法相比，我们的方法可以处理运动相对较大的帧，并具有灵活性，能够让用户通过编辑草图引导来控制生成的视频序列。通过明确考虑帧与草图之间的对应关系，我们可以获得比其他图像合成方法更高质量的结果。我们的结果表明，我们的系统能够很好地推广到不同的电影帧，比现有解决方案取得了更好的效果。
