Densely Based Multi-Scale and Multi-Modal Fully Convolutional Networks for High-Resolution Remote-Sensing Image Semantic Segmentation

Abstract

Automatic and accurate semantic segmentation from high-resolution remote-sensing images plays an important role in the field of aerial images analysis. The task of dense semantic segmentation requires that semantic labels be assigned to each pixel in the image. Recently, convolutional neural networks (CNNs) have proven to be powerful tools for image classification, and they have been adopted in the remote-sensing community. But many limitations still exist when modern CNN architectures are directly applied to remote-sensing images, such as gradient explosion when the depth of the network increases, over-fitting with limited labeled remote-sensing data, and special differences between remote-sensing images and natural images. In this paper, we present a novel architecture that combines the thought of dense connection and fully convolutional networks, referred as DFCN, to automatically provide fine-grained semantic segmentation maps. In addition, we improve DFCN with multi-scale filters to widen the network and to increase the richness and diversity of extracted information, making the network more powerful and expressive than the naive convolution layer. Furthermore, we investigate a multi-modal network that incorporates digital surface models (DSMs) into a DFCN structure, and then we propose dual-path densely convolutional networks where the encoder consists of two paths that, respectively, extract features from spectral data and DSMs data and then fuse them. Finally, through conducting comprehensive experimental evaluations on two remote sensing benchmark datasets, we test our proposed models and compare them with other deep networks. The results demonstrate the effectiveness of proposed approaches; they can achieve competitive performance compared with the current state-of-the-art methods.