Novel deep learning data structure for multispectral remote sensing images

Abstract

Standard deep-learning (DL) architectures do not optimize the use of the spatial and spectral information in the multi-spectral images but often consider only one of the two components. Two-stream DL architectures split and process them separately. However, the fusing of the output of the two streams is a challenging task. 3D-CNN processes spatial and spectral information together at the cost of a large number of parameters. To overcome these limitations, we propose a novel DL data structure that re-organizes the spectral and spatial information in remote-sensing (RS) images and process them together. Representing a RS image I as a data cube, we handle the spatial and spectral information by reducing the spectral bands from N to M, where M can drop out to one. The spectral information is projected in the spatial dimensions and re-organized in 2-dimensional B blocks. The proposed approach analyzes the spectral information of each block by using 2-dimensional convolutional kernels of appropriate size and stride. The output represents the relationship between the spectral bands of the input image and preserves the spatial relationship between its neighboring pixels. The spatial relationships are analyzed by processing the output of the previous layer with standard 2D-CNNs. Experiments by using images acquired by Sentinel-2 and Landsat-8 data and the labels of the LUCAS database released in 2018 provide promising results.