A Unified Pansharpening Method With Structure Tensor Driven Spatial Consistency and Deep Plug-and-Play Priors

Abstract

Pansharpening is to generate a high resolution multispectral (HRMS) image by preserving the spectral information from a low resolution multispectral (LRMS) image and the spatial content from a panchromatic (PAN) image. This article proposes a unified pansharpening method with structure tensor driven spatial consistency and deep plug and play priors. First, the spectral fidelity constraint between HRMS and LRMS is imposed for preserving spectral information. Second, the structure tensor is applied to characterize the spatial geometric information of HRMS and PAN images, thus the structure tensor driven spatial consistency prior between HRMS and PAN is particularly exploited for preserving spatial content. Moreover, by generalizing the convolution neural network (CNN) fusion method into a unified variational framework, a novel CNN-based deep plug and play prior between the HRMS and CNN-based fused MS images is also proposed to generate more image characteristics for further preserving spectral information and spatial content. Besides, the proposed model is solved by the alternating direction method of multipliers (ADMM) algorithm. Finally, extensive experiments on both reduced and full resolution by comparing with various representative approaches exhibit the excellent performance of the proposed method.