Jiang Qi, Gao Shaohua, Gao Yao, Yang Kailun, Yi Zhonghua, Shi Hao, Sun Lei, Wang Kaiwei
IEEE Trans Image Process. 2024;33:4568-4583. doi: 10.1109/TIP.2024.3441370. Epub 2024 Aug 23.
High-quality panoramic images with a Field of View (FoV) of 360° are essential for contemporary panoramic computer vision tasks. However, conventional imaging systems come with sophisticated lens designs and heavy optical components. This disqualifies their usage in many mobile and wearable applications where thin and portable, minimalist imaging systems are desired. In this paper, we propose a Panoramic Computational Imaging Engine (PCIE) to achieve minimalist and high-quality panoramic imaging. With less than three spherical lenses, a Minimalist Panoramic Imaging Prototype (MPIP) is constructed based on the design of the Panoramic Annular Lens (PAL), but with low-quality imaging results due to aberrations and small image plane size. We propose two pipelines, i.e. Aberration Correction (AC) and Super-Resolution and Aberration Correction (SR&AC), to solve the image quality problems of MPIP, with imaging sensors of small and large pixel size, respectively. To leverage the prior information of the optical system, we propose a Point Spread Function (PSF) representation method to produce a PSF map as an additional modality. A PSF-aware Aberration-image Recovery Transformer (PART) is designed as a universal network for the two pipelines, in which the self-attention calculation and feature extraction are guided by the PSF map. We train PART on synthetic image pairs from simulation and put forward the PALHQ dataset to fill the gap of real-world high-quality PAL images for low-level vision. A comprehensive variety of experiments on synthetic and real-world benchmarks demonstrates the impressive imaging results of PCIE and the effectiveness of the PSF representation. We further deliver heuristic experimental findings for minimalist and high-quality panoramic imaging, in terms of the choices of prototype and pipeline, network architecture, training strategies, and dataset construction. Our dataset and code will be available at https://github.com/zju-jiangqi/PCIE-PART.
对于当代全景计算机视觉任务而言，具有360°视场（FoV）的高质量全景图像至关重要。然而，传统成像系统配备了复杂的镜头设计和笨重的光学组件。这使其无法用于许多需要轻薄便携的简约成像系统的移动和可穿戴应用中。在本文中，我们提出了一种全景计算成像引擎（PCIE），以实现简约且高质量的全景成像。基于全景环形透镜（PAL）的设计，使用少于三个球面透镜构建了一个简约全景成像原型（MPIP），但由于像差和小图像平面尺寸，成像结果质量较低。我们提出了两条处理流程，即像差校正（AC）和超分辨率与像差校正（SR&AC），分别用于解决小像素尺寸和大像素尺寸成像传感器的MPIP图像质量问题。为了利用光学系统的先验信息，我们提出了一种点扩散函数（PSF）表示方法，以生成PSF图作为一种附加模态。设计了一种PSF感知像差图像恢复Transformer（PART）作为这两条处理流程的通用网络，其中自注意力计算和特征提取由PSF图引导。我们在模拟生成的合成图像对上训练PART，并提出了PALHQ数据集，以填补用于低级视觉的真实世界高质量PAL图像的空白。在合成和真实世界基准上进行的各种综合实验证明了PCIE令人印象深刻的成像结果以及PSF表示的有效性。我们还在原型和处理流程的选择、网络架构、训练策略和数据集构建方面，为简约且高质量的全景成像提供了启发式实验结果。我们的数据集和代码将在https://github.com/zju-jiangqi/PCIE-PART上提供。
