Fan Wensheng, Liu Fan, Li Jingzhi
College of Data Science, Taiyuan University of Technology, Jinzhong, Shanxi, China.
PeerJ Comput Sci. 2023 Jul 28;9:e1488. doi: 10.7717/peerj-cs.1488. eCollection 2023.
Pan-sharpening is a fundamental and crucial task in the remote sensing image processing field, which generates a high-resolution multi-spectral image by fusing a low-resolution multi-spectral image and a high-resolution panchromatic image. Recently, deep learning techniques have shown competitive results in pan-sharpening. However, diverse features in the multi-spectral and panchromatic images are not fully extracted and exploited in existing deep learning methods, which leads to information loss in the pan-sharpening process. To solve this problem, a novel pan-sharpening method based on multi-resolution transformer and two-stage feature fusion is proposed in this article. Specifically, a transformer-based multi-resolution feature extractor is designed to extract diverse image features. Then, to fully exploit features with different content and characteristics, a two-stage feature fusion strategy is adopted. In the first stage, a multi-resolution fusion module is proposed to fuse multi-spectral and panchromatic features at each scale. In the second stage, a shallow-deep fusion module is proposed to fuse shallow and deep features for detail generation. Experiments over QuickBird and WorldView-3 datasets demonstrate that the proposed method outperforms current state-of-the-art approaches visually and quantitatively with fewer parameters. Moreover, the ablation study and feature map analysis also prove the effectiveness of the transformer-based multi-resolution feature extractor and the two-stage fusion scheme.
全色锐化是遥感图像处理领域一项基础且关键的任务，它通过融合低分辨率多光谱图像和高分辨率全色图像来生成高分辨率多光谱图像。近年来，深度学习技术在全色锐化方面展现出了具有竞争力的成果。然而，现有深度学习方法并未充分提取和利用多光谱图像与全色图像中的多样特征，这导致了全色锐化过程中的信息损失。为解决这一问题，本文提出了一种基于多分辨率变换器和两阶段特征融合的新型全色锐化方法。具体而言，设计了一种基于变换器的多分辨率特征提取器来提取多样的图像特征。然后，为充分利用具有不同内容和特征的特征，采用了两阶段特征融合策略。在第一阶段，提出了一个多分辨率融合模块，用于在每个尺度上融合多光谱和全色特征。在第二阶段，提出了一个浅-深融合模块，用于融合浅层和深层特征以生成细节。在QuickBird和WorldView-3数据集上进行的实验表明，所提出的方法在视觉和定量方面均优于当前的最先进方法，且参数更少。此外，消融研究和特征图分析也证明了基于变换器的多分辨率特征提取器和两阶段融合方案的有效性。
