Joint spectral-spatial hyperspectral classification based on transfer learning (SSTL) from red-green-blue (RGB) images

Abstract

ABSTRACT Deep learning (DL) approaches are demonstrated excellent performance on hyperspectral images (HSIs) classification tasks. Nevertheless, the imbalance between the scant number of available training samples, and the data-driven requirements of DL becomes the major limitation. In this work, we propose a deep convolutional neural network (CNN) with two-branch architecture based on multiple transfer learning (TL) strategies. Specifically, we designate one-dimensional (1D) CNN and two-dimensional (2D) CNN to extract the spectral and spatial features of the HSIs, respectively. The feature vectors from two branches are then concatenated and fed to fully connected layers to compose the collaborative spectral-spatial features. We implement the classification tasks via training a soft-max layer on the joint spatial-spectral features. To accelerate the process of training models and save calculation consumption, we apply transfer learning (TL) strategies to the spatial branch. The innovation lies in transferring VGG-16 (Visual Geometry Group Network) model pretrained on RGB (Red-Green-Blue) images datasets (ImageNet) to HSIs target domain. To investigate the effectiveness of the transfer learning between heterogeneous physical imaging camera, we design diverse regulating transferring strategies to explore the best performance. The classification results testing on the public HSIs datasets demonstrate our joint spectral-spatial based on transfer learning (SSTL) model performs excellently compared with other state-of-the-art methods.