Wang Yingqian, Yang Jungang, Wang Longguang, Ying Xinyi, Wu Tianhao, An Wei, Guo Yulan
IEEE Trans Image Process. 2021;30:1057-1071. doi: 10.1109/TIP.2020.3042059. Epub 2020 Dec 11.
Light field (LF) cameras can record scenes from multiple perspectives, and thus introduce beneficial angular information for image super-resolution (SR). However, it is challenging to incorporate angular information due to disparities among LF images. In this paper, we propose a deformable convolution network (i.e., LF-DFnet) to handle the disparity problem for LF image SR. Specifically, we design an angular deformable alignment module (ADAM) for feature-level alignment. Based on ADAM, we further propose a collect-and-distribute approach to perform bidirectional alignment between the center-view feature and each side-view feature. Using our approach, angular information can be well incorporated and encoded into features of each view, which benefits the SR reconstruction of all LF images. Moreover, we develop a baseline-adjustable LF dataset to evaluate SR performance under different disparity variations. Experiments on both public and our self-developed datasets have demonstrated the superiority of our method. Our LF-DFnet can generate high-resolution images with more faithful details and achieve state-of-the-art reconstruction accuracy. Besides, our LF-DFnet is more robust to disparity variations, which has not been well addressed in literature.
光场（LF）相机可以从多个视角记录场景，从而为图像超分辨率（SR）引入有益的角度信息。然而，由于LF图像之间的差异，整合角度信息具有挑战性。在本文中，我们提出了一种可变形卷积网络（即LF-DFnet）来处理LF图像SR的差异问题。具体来说，我们设计了一个角度可变形对齐模块（ADAM）用于特征级对齐。基于ADAM，我们进一步提出了一种收集-分配方法，以在中心视图特征和每个侧视图特征之间进行双向对齐。使用我们的方法，角度信息可以很好地整合并编码到每个视图的特征中，这有利于所有LF图像的SR重建。此外，我们开发了一个基线可调的LF数据集，以评估不同差异变化下的SR性能。在公共数据集和我们自行开发的数据集上进行的实验都证明了我们方法的优越性。我们的LF-DFnet可以生成具有更逼真细节的高分辨率图像，并实现了当前最优的重建精度。此外，我们的LF-DFnet对差异变化更具鲁棒性，这在文献中尚未得到很好的解决。
