Spatial Data Augmentation: Improving the Generalization of Neural Networks for Pansharpening

Abstract

Deep learning (DL) methods have achieved impressive performance for pansharpening in recent years. However, because of poor generalization, most DL methods achieve unsatisfactory performance for data acquired by sensors not considered during the training phase and decreased performance for samples at full resolution. To solve this issue, we propose a data augmentation framework for pansharpening neural networks. Specifically, we introduce first a random spatial degradation based on anisotropic Gaussian-shaped modulation transfer functions (MTFs) to increase the generalization with respect to different spatial models and sensors. Then, considering that various sensors have different ground sampling distances (GSDs), we randomly rescale the GSD of the training samples to improve the generalization with respect to spatial resolution. Thanks to this module, the generalization to tests from different sensors and samples at full resolution can easily be achieved. Experimental results demonstrate the effectiveness of the proposed approach with better performance when data for training are decoupled with the ones for testing and comparable performance when training and testing are coupled (i.e., data acquired by the same sensor are considered in the two phases). Besides, performance at full resolution for pansharpening neural networks is improved by the proposed approach. The proposed approach has been integrated into existing pansharpening neural networks showing satisfactory performance for widely used sensors, including, GaoFen-1, QuickBird, WorldView-2, WorldView-3, IKONOS, Spot-7, GeoEye, and PHR1A.