Jiang Zhiying, Zhang Zengxi, Liu Jinyuan, Fan Xin, Liu Risheng
IEEE Trans Image Process. 2024;33:4288-4302. doi: 10.1109/TIP.2024.3430532. Epub 2024 Jul 30.
Image stitching is a critical task in panorama perception that involves combining images captured from different viewing positions to reconstruct a wider field-of-view (FOV) image. Existing visible image stitching methods suffer from performance drops under severe conditions since environmental factors can easily impair visible images. In contrast, infrared images possess greater penetrating ability and are less affected by environmental factors. Therefore, we propose an infrared and visible image-based multispectral image stitching method to achieve all-weather, broad FOV scene perception. Specifically, based on two pairs of infrared and visible images, we employ the salient structural information from the infrared images and the textual details from the visible images to infer the correspondences within different modality-specific features. For this purpose, a multiscale progressive mechanism coupled with quadrature correlation is exploited to improve regression in different modalities. Exploiting the complementary properties, accurate and credible homography can be obtained by integrating the deformation parameters of the two modalities to compensate for the missing modality-specific information. A global-aware guided reconstruction module is established to generate an informative and broad scene, wherein the attentive features of different viewpoints are introduced to fuse the source images with a more seamless and comprehensive appearance. We construct a high-quality infrared and visible stitching dataset for evaluation, including real-world and synthetic sets. The qualitative and quantitative results demonstrate that the proposed method outperforms the intuitive cascaded fusion-stitching procedure, achieving more robust and credible panorama generation. Code and dataset are available at https://github.com/Jzy2017/MSGA.
图像拼接是全景感知中的一项关键任务，它涉及将从不同视角捕获的图像进行组合，以重建更宽视野（FOV）的图像。现有的可见光图像拼接方法在恶劣条件下会出现性能下降的情况，因为环境因素很容易损害可见光图像。相比之下，红外图像具有更强的穿透能力，且受环境因素的影响较小。因此，我们提出一种基于红外和可见光图像的多光谱图像拼接方法，以实现全天候、宽视野场景感知。具体而言，基于两对红外和可见光图像，我们利用红外图像中的显著结构信息和可见光图像中的文本细节来推断不同模态特定特征之间的对应关系。为此，利用一种结合了正交相关性的多尺度渐进机制来改善不同模态下的回归。利用这些互补特性，通过整合两种模态的变形参数以补偿缺失的模态特定信息，可获得准确且可靠的单应性。建立了一个全局感知引导的重建模块来生成信息丰富且视野宽广的场景，其中引入了不同视角的注意力特征，以使源图像融合得更加无缝和全面。我们构建了一个高质量的红外和可见光拼接数据集用于评估，包括真实世界和合成数据集。定性和定量结果表明，所提出的方法优于直观的级联融合拼接过程，实现了更稳健、可靠的全景生成。代码和数据集可在https://github.com/Jzy2017/MSGA获取。
