Deep gated attention networks for large-scale street-level scene segmentation

Abstract

Street-level scene segmentation aims to label each pixel of street-view images into specific semantic categories. It has been attracting growing interest due to various real-world applications, especially in the area of autonomous driving. However, this pixel-wise labeling task is very challenging under the complex street-level scenes and large-scale object categories. Motivated by the scene layout of street-view images, in this work we propose a novel Spatial Gated Attention (SGA) module, which automatically highlights the attentive regions for pixel-wise labeling, resulting in effective street-level scene segmentation. The proposed module takes as input the multi-scale feature maps based on a Fully Convolutional Network (FCN) backbone, and produces the corresponding attention mask for each feature map. The learned attention masks can neatly highlight the regions of interest while suppress background clutter. Furthermore, we propose an efficient multi-scale feature interaction mechanism which is able to adaptively aggregate the hierarchical features. Based on the proposed mechanism, the features of different levels are adaptively re-weighted according to the local spatial structure and the surrounding contextual information. Consequently, the proposed modules are able to boost standard FCN architectures and result in an enhanced pixel-wise segmentation for street-level scene images. Extensive experiments on three public available street-level benchmarks demonstrate that the proposed Gated Attention Network (GANet) approach achieves consistently superior performance and outperforms the very recent state-of-the-art methods.