Deep Adaptive Pansharpening via Uncertainty-Aware Image Fusion

Abstract

Pansharpening is a procedure that fuses high-resolution panchromatic (PAN) images and low-resolution multispectral (LMS) images to derive high-resolution multispectral (HMS) images. Despite its rapid development, most existing pansharpening techniques integrate the information of PAN and LMS invariantly in the spatial dimension, ignoring the uneven spatial dependence of restoring HMS with the aid of PAN information and resulting in ineffective fusion results. In this work, we propose an Uncertainty-aware Adaptive Pansharpening Network (UAPN) that integrates PAN information spatial-variantly to restore LMS information with an uncertainty mechanism. Specifically, we first estimate the epistemic and aleatoric uncertainties together, which model the spatial-variant distributions of restoring the LMS image to the HMS image. Then, we introduce Uncertainty-conditioned Adaptive Convolution (UAC) to adaptively integrate LMS and PAN information, where its parameters are spatially variable by conditioning on the uncertainty estimations. Furthermore, we propose a multi-stage uncertainty-driven loss function to explicitly force the network to concentrate on restoring challenging areas of the LMS image. Extensive experimental results demonstrate the superiority of our UAPN with fewer parameters and flops, outperforming other state-of-the-art methods both qualitatively and quantitatively on multiple satellite datasets. The code is available at https://github.com/keviner1/UAPN..