Pan-sharpening via deep metric learning

Abstract

Neighbors Embedding based pansharpening methods have received increasing interests in recent years. However, image patches do not strictly follow the similar structure in the shallow MultiSpectral (MS) and PANchromatic (PAN) image spaces, consequently leading to a bias to the pansharpening. In this paper, a new deep metric learning method is proposed to learn a refined geometric multi-manifold neighbor embedding, by exploring the hierarchical features of patches via multiple nonlinear deep neural networks. First of all, down-sampled PAN images from different satellites are divided into a large number of training image patches and are then grouped coarsely according to their shallow geometric structures. Afterwards, several Stacked Sparse AutoEncoders (SSAE) with similar structures are separately constructed and trained by these grouped patches. In the fusion, image patches of the source PAN image pass through the networks to extract features for formulating a deep distance metric and thus deriving their geometric labels. Then, patches with the same geometric labels are grouped to form geometric manifolds. Finally, the assumption that MS patches and PAN patches form the same geometric manifolds in two distinct spaces, is cast on geometric groups to formulate geometric multi-manifold embedding for estimating high resolution MS image patches. Some experiments are taken on datasets acquired by different satellites. The experimental results demonstrate that our proposed method can obtain better fusion results than its counterparts in terms of visual results and quantitative evaluations.