P2Sharpen: A progressive pansharpening network with deep spectral transformation

Abstract

Most existing deep learning-based methods for pansharpening task solely rely on the supervision of pseudo-ground-truth multi-spectral images, which exhibits two limitations in producing high-quality images. On the one hand, it is uncontrollable to regulate the full-resolution performance due to the fact that their whole training process only remain at the scale of reduced resolution. On the other hand, they ignore the accurate spatial information reference of high-resolution panchromatic images for supervision, resulting in insufficient spatial structure details. To address these challenges, we propose a progressive pansharpening network with deep spectral transformation, termed as P2Sharpen, where we balance the performance in different resolutions and make full use of the observed satellite data to improve the quality of fused results. First, we design a spectral transformation network (STNet) to cross the modality difference between multi-spectral data and panchromatic data, which establishes an accurate mapping function from MS to PAN images. Second, we propose a progressive pansharpening network (P2Net), in which the optimization of pansharpening at reduced and full resolutions is considered in a two-stage manner, balancing the performance at two scales effectively. In addition, we introduce the trained STNet to construct the consistency constraint between the sharpened result and PAN image at both reduced-resolution stage and full-resolution stage, which further improves the ability of P2Net for preserving spatial textures. Extensive experiments demonstrate that our method shows excellent performance over the state-of-the-arts on the sharpening quality and the spectral response consistency in both reduced and full resolutions. Moreover, the proposed method can be applied to generate the high-resolution normalized difference vegetation index with promising accuracy.