Yang Danqing, Wang Xiaorui, Zhu Naibo, Li Shuang, Hou Na
School of Optoelectronic Engineering, Xidian University, Xi'an 710071, China.
Research Institute of System Engineering, PLA Academy of Military Science, Beijing 100091, China.
Sensors (Basel). 2023 Jul 12;23(14):6322. doi: 10.3390/s23146322.
The challenging issues in infrared and visible image fusion (IVIF) are extracting and fusing as much useful information as possible contained in the source images, namely, the rich textures in visible images and the significant contrast in infrared images. Existing fusion methods cannot address this problem well due to the handcrafted fusion operations and the extraction of features only from a single scale. In this work, we solve the problems of insufficient information extraction and fusion from another perspective to overcome the difficulties in lacking textures and unhighlighted targets in fused images. We propose a multi-scale feature extraction (MFE) and joint attention fusion (JAF) based end-to-end method using a generative adversarial network (MJ-GAN) framework for the aim of IVIF. The MFE modules are embedded in the two-stream structure-based generator in a densely connected manner to comprehensively extract multi-grained deep features from the source image pairs and reuse them during reconstruction. Moreover, an improved self-attention structure is introduced into the MFEs to enhance the pertinence among multi-grained features. The merging procedure for salient and important features is conducted via the JAF network in a feature recalibration manner, which also produces the fused image in a reasonable manner. Eventually, we can reconstruct a primary fused image with the major infrared radiometric information and a small amount of visible texture information via a single decoder network. The dual discriminator with strong discriminative power can add more texture and contrast information to the final fused image. Extensive experiments on four publicly available datasets show that the proposed method ultimately achieves phenomenal performance in both visual quality and quantitative assessment compared with nine leading algorithms.
红外与可见光图像融合（IVIF）中的挑战性问题在于提取并融合源图像中尽可能多的有用信息，即可见光图像中的丰富纹理和红外图像中的显著对比度。由于手工融合操作以及仅从单一尺度提取特征，现有的融合方法无法很好地解决这个问题。在这项工作中，我们从另一个角度解决信息提取和融合不足的问题，以克服融合图像中缺乏纹理和目标不突出的困难。我们提出了一种基于多尺度特征提取（MFE）和联合注意力融合（JAF）的端到端方法，使用生成对抗网络（MJ-GAN）框架来实现IVIF。MFE模块以密集连接的方式嵌入到基于双流结构的生成器中，以全面提取源图像对中的多粒度深度特征，并在重建过程中重复使用它们。此外，在MFE中引入了一种改进的自注意力结构，以增强多粒度特征之间的相关性。显著和重要特征的合并过程通过JAF网络以特征重新校准的方式进行，这也以合理的方式生成融合图像。最终，我们可以通过单个解码器网络重建一个包含主要红外辐射信息和少量可见纹理信息的初步融合图像。具有强大判别能力的双判别器可以为最终融合图像添加更多纹理和对比度信息。在四个公开可用数据集上进行的大量实验表明，与九种领先算法相比，所提出的方法最终在视觉质量和定量评估方面都取得了显著的性能。
