Chen Lihui, Vivone Gemine, Qin Jiayi, Chanussot Jocelyn, Yang Xiaomin
IEEE Trans Neural Netw Learn Syst. 2024 Nov;35(11):16733-16747. doi: 10.1109/TNNLS.2023.3297319. Epub 2024 Oct 29.
Convolutional neural networks (CNNs) have recently achieved outstanding performance for hyperspectral (HS) and multispectral (MS) image fusion. However, CNNs cannot explore the long-range dependence for HS and MS image fusion because of their local receptive fields. To overcome this limitation, a transformer is proposed to leverage the long-range dependence from the network inputs. Because of the ability of long-range modeling, the transformer overcomes the sole CNN on many tasks, whereas its use for HS and MS image fusion is still unexplored. In this article, we propose a spectral-spatial transformer (SST) to show the potentiality of transformers for HS and MS image fusion. We devise first two branches to extract spectral and spatial features in the HS and MS images by SST blocks, which can explore the spectral and spatial long-range dependence, respectively. Afterward, spectral and spatial features are fused feeding the result back to spectral and spatial branches for information interaction. Finally, the high-resolution (HR) HS image is reconstructed by dense links from all the fused features to make full use of them. The experimental analysis demonstrates the high performance of the proposed approach compared with some state-of-the-art (SOTA) methods.
卷积神经网络（CNN）最近在高光谱（HS）和多光谱（MS）图像融合方面取得了出色的性能。然而，由于其局部感受野，CNN无法探索HS和MS图像融合中的长距离依赖性。为了克服这一限制，提出了一种变换器来利用网络输入中的长距离依赖性。由于具有长距离建模能力，变换器在许多任务上优于单一的CNN，但其在HS和MS图像融合中的应用仍未得到探索。在本文中，我们提出了一种光谱-空间变换器（SST），以展示变换器在HS和MS图像融合中的潜力。我们首先设计了两个分支，通过SST块分别提取HS和MS图像中的光谱和空间特征，这些块可以分别探索光谱和空间的长距离依赖性。然后，将光谱和空间特征进行融合，并将结果反馈到光谱和空间分支进行信息交互。最后，通过所有融合特征的密集连接重建高分辨率（HR）HS图像，以充分利用这些特征。实验分析表明，与一些最新的（SOTA）方法相比，该方法具有高性能。
