An Efficient Dual Spatial–Spectral Fusion Network

Abstract

The spectral fidelity and the spatial richness enhancement are two primary objectives for the deep learning pan-sharpening algorithms. Consequently, the two-stream fusion architecture is utilized to focus on the individual features of panchromatic (PAN) and multi-spectral (MS) images. However, most existing two-stream networks have a high degree of extracted feature redundancy and large fusion workload. Based on these, we propose a dual spatial-spectral fusion network (DSSN) to implement the divided-objective fusion, with one stream fusing spatial information and the other stream fusing spectral information. We propose a bifurcation spatial fusion stream consisted of backbone and gradient branch. The gradient feature extracted as prior knowledge to direct the backbone to enhance the spatial information richness. Moreover, the multi-scale residual block (MRB) is designed to guarantee the sufficiency of extracted features. In MRBs, the densely connected small size and asymmetric convolution kernels are used to replace the large size ones to reduce the network complexity. In the spectral fusion stream, one-dimensional convolution is adopted to efficiently fuse the spectral features of PAN and MS in the channel dimensions. Finally, the fused spatial and spectral information optimally integrate with a certain proportion to be the final result. The ablation studies confirm that each stream of DSSN is indispensable in the spectral preservation and spatial detail recovery. Three different types of optical data are used to assess practicability and universality of DSSN. The experimental results show that the fusion quality of DSSN exceeds the commonly used pan-sharpening methods, with fast training convergence speed.