Siamese Networks Based Deep Fusion Framework for Multi-Source Satellite Imagery

Hannan Adeel,Syed Sohaib Ali,M Mohsin Riaz,Javaria Tahir

doi:10.1109/access.2022.3143847

Hannan Adeel, Syed Sohaib Ali + Show 2 more

Open Access

https://doi.org/10.1109/access.2022.3143847

Copy DOI

Abstract

A critical aim of pansharpening is to fuse coherent spatial and spectral features from panchromatic and multispectral images respectively. This study proposes deep siamese network based pansharpening model as a two-stage framework in a multiscale setting. In the first stage, a siamese network learns a common feature space between panchromatic and multispectral bands. The second stage follows by fusing the output feature maps of the siamese network. The parameters of these two stages are shared across scales in order to add spatial information consistently (across scales). The spectral information is preserved by adding appropriate skip connections from input multispectral image. Multi-level network parameters sharing mechanism in pyramidal reconstruction of pansharpened image, better preserves spatial and spectral details simultaneously. Experimental work carried out using deep siamese network in multi-scale setting (to obtain inter-band similarity among different sensor data) outperforms several latest pansharpening methods.

Highlights

Remote sensing imaging systems mainly provide two types of images; one enriched in spectral information while other with high spatial information
Existing methods can be specified into four main categories namely: Component substitution (CS) methods, multi resolution analysis (MRA) based on image decomposition, variational optimization (VO), and deep learning (DL) based on convolutional neural network (CNN) and residual blocks
The second stage is followed by fusing the output feature maps of the siamese network

Summary

INTRODUCTION

Remote sensing imaging systems mainly provide two types of images; one enriched in spectral information while other with high spatial information. Lai et al [14], proposed an encoder-decoder setting for feature extraction of MS and PAN images (from coarsest to finest level) These features and primitive information are used to reconstruct high resolution MS images (in a multi-scale setting). Fast shrinkage optimization and total generalized variation based model are used to improve PAN image spectral details. The scheme uses two discriminators to preserve spectral and spatial details of MS and PAN images in resultant HR-MS image. The scheme uses channel attention mechanism along with feature extraction and residual learning in a multi-scale fashion to improve network convergence time. The siamese networks bank learns inter-band similarity among different MS and PAN bands to maximize obtainable details for fusion These maps are passed to global feature fusion block which adopts local-global fusion strategy to estimate the pansharpened image at a particular level. Section-4 concludes the paper along with some future directions

PROPOSED SCHEME

DEEP FUSION NETWORK

LOCAL FEATURES MAP ESTIMATION

GLOBAL FEATURES MAP ESTIMATION

CONCLUSION