Scene Parsing Research Articles

Pointly-supervised scene parsing with uncertainty mixture

Pointly-supervised learning is an important topic for scene parsing, as dense annotation is extremely expensive and hard to scale. The state-of-the-art method harvests pseudo labels by applying thresholds upon softmax outputs (logits). There are two issues with this practice: (1) Softmax output does not necessarily reflect the confidence of the network output. (2) There is no principled way to decide on the optimal threshold. Tuning thresholds can be time-consuming for deep neural networks. Our method, by contrast, builds upon uncertainty measures instead of logits and is free of threshold tuning. We motivate the method with a large-scale analysis of the distribution of uncertainty measures, using strong models and challenging databases. This analysis leads to the discovery of a statistical phenomenon called uncertainty mixture. Specifically speaking, for each independent category, the distribution of uncertainty measures for unlabeled points is a mixture of two components (certain v.s. uncertain samples). The phenomenon of uncertainty mixture is surprisingly ubiquitous in real-world datasets like PascalContext and ADE20k. Inspired by this discovery, we propose to decompose the distribution of uncertainty measures with a Gamma mixture model, leading to a principled method to harvest reliable pseudo labels. Beyond that, we assume the uncertainty measures for labeled points are always drawn from the certain component. This amounts to a regularized Gamma mixture model. We provide a thorough theoretical analysis of this model, showing that it can be solved with an EM-style algorithm with convergence guarantee. Our method is also empirically successful. On PascalContext and ADE20k, we achieve clear margins over the baseline, notably with no threshold tuning in the pseudo label generation procedure. On the absolute scale, since our method collaborates well with strong baselines, we reach new state-of-the-art performance on both datasets.

Incorporating part-whole hierarchies into fully convolutional network for scene parsing

In this paper, a new approach to scene parsing is proposed which integrates part-whole hierarchies relationship in the last feature map to assign a semantic class label to each pixel. Recently, deep learning-based approaches have had a great impact on scene parsing. However, these methods could not preserve the spatial information about the high-level (or mid-level) features. Hence, Hinton, one of the fathers of deep learning, introduced the capsule concept to encode pose information such as orientation. All of the capsules which have a similar pose matrix value are grouped to form a parent capsule. However, their work has two challenges: 1) the extensive time required to perform dynamic routing agreement to obtain the routing coefficient and 2) the variation of the appearance and the spatial hierarchies between part capsules and their corresponding parent are not encoded. In this study, to consider these challenges, the general Hough transform (GHT) and tensor normal distribution are utilized to propose a novel capsule concept. In this case, each capsule has k offset vectors for each semantic class. The offset vectors are oriented from the capsule to the k other capsules which have an effective role in assigning that capsule to a specific semantic class. The problem formulation is proposed such that evaluating the approach on large datasets is feasable. Also, a new score function is designed to accumulate the vote’s strengths for capsule class estimation. To do so, we use tensor normal distribution in which the covariance matrix is defined as the Kronecker product of the capsule feature covariance and the between-capsule covariance. The proposed approach, for the first time, encodes the relations between part capsules to vote to a whole capsule through the between-capsule covariance matrix. To evaluate our proposed approach, it is applied to SiftFlow, NYUD-v2 and PASCAL VOC 2012 datasets. The results show that our approach achieves superior performance.

Semantic Scene Labeling via Deep Nested Level Set

Semantic scene labeling plays a very important role in intelligent transportation tasks, such as autonomous driving and advanced driver assistance. Recently, thanks to the advances of deep learning, significant improvements have been achieved for this pixel-wise labeling task. Although effective, current methods lack of explicitly modeling the boundary of objects, resulting in inaccurate labeling results. Meanwhile, traditional level set based methods perform better to capture the evolution of boundaries. However, they are sensitive to the model initialization. To address these issues, in this work we propose a novel deep learning framework, named deep nested level set (DNLS) for boundary-aware semantic scene labeling. Different from previous works, our proposed framework explicitly takes deep learned features and object boundary information into account. More specifically, our proposed framework first predicts semantic probability maps and boundary locations of objects using a bifurcated fully convolutional network (BFCN). Then, these probability maps are seamlessly integrated into a nested level set function for accurate scene labeling. As a result, our approach can automatically initialize the nested level set function, and the whole framework can be trained in an end-to-end manner, providing a new solution for accurate semantic scene parsing. Extensive experiments on public CamVid and Cityscapes datasets demonstrate that our proposed framework produces high-quality predictions with clear object boundaries and spatial consistency.

A generalized approach based on convolutional neural networks for large area cropland mapping at very high resolution

Timely and accurate delineation of the cropland extent over large area is crucial for operational agriculture monitoring and is also beneficial to address food security issues. Existing global datasets associated with cropland are limited by insufficient spatial resolution to properly represent areas with small parcel size distributions, and their less-than-ideal accuracies hamper application at regional and local scales. Diverse very high spatial resolution (VHSR) satellite systems are now available, offering sub-meter to five-meter resolution (e.g. Gaofen-1, Gaofen-2, and ZiYuan-3), and hence enabling explicit extraction of cropland areas from heterogeneous and fragmented landscapes. This study presented a generalized methodology for operational cropland mapping at very high resolution using a deep convolutional neural network to automatically learn the robust and discriminative features. Specifically, we slightly modified the pyramid scene parsing network (PSPNet) and combined deep long-range features with shadow local features to provide predictions with high level of detail. We demonstrated the modified PSPNet (MPSPNet) over four province-wide study areas (Heilongjiang, Hebei, Zhejiang and Guangdong) with diverse agrosystems across China from north to south using multi-source very high spatial resolution satellite images (mainly Gaofen-1 supplemented with Gaofen-2 and ZiYuan-3), with the overall accuracies ranging from 89.99% to 92.31%. Moreover, we compared MPSPNet with other CNN models and investigated the its behavior by visualizing the learned features on different layers, indicating that combining low and high level features for final classification was an efficient and accurate strategy for cropland mapping because the former capture edge information related to object boundaries and the latter could learn long-range spatial dependencies that helped recognize croplands. The temporal transfer and spatial transfer assessments from the respects of qualitative and quantitative corroborated the robust generalizability of the proposed method. And the contrast to the traditional object-based classification method also demonstrated the advantages and strong generalization capabilities of MPSPNet in extracting cropland using VHSR remote sensed images. We compared our results with the current cropland maps generated from FROM-GLC10, which further verified the effectiveness of the proposed approach for large-scale cropland mapping at very high resolution.

Non-parametric spatially constrained local prior for scene parsing on real-world data

Scene parsing aims to recognize the object category of every pixel in scene images, and it plays a central role in image content understanding and computer vision applications. However, accurate scene parsing from unconstrained real-world data is still a challenging task. In this paper, we present the non-parametric Spatially Constrained Local Prior (SCLP) for scene parsing on realistic data. For a given query image, the non-parametric SCLP is learnt by first retrieving a subset of most similar training images to the query image and then collecting prior information about object co-occurrence statistics between spatial image blocks and between adjacent superpixels from the retrieved subset. The SCLP is powerful in capturing both long- and short-range context about inter-object correlations in the query image and can be effectively integrated with traditional visual features to refine the classification results. Our experiments on the SIFT Flow and PASCAL-Context benchmark datasets show that the non-parametric SCLP used in conjunction with superpixel-level visual features achieves one of the top performance compared with state-of-the-art approaches.

Built-in Depth-Semantic Coupled Encoding for Scene Parsing, Vehicle Detection, and Road Segmentation

Recent representative scene parsing methods based on Convolutional Neural Networks (CNN) have greatly improved spatial resolution of pixel-wise labelling by exploiting multi-scale features and refined boundaries. However, the vast majority of previous works only utilize the color or textural information of images, without considering the depth information, which is beneficial for semantic reasoning. In this paper, we take advantages of the mutual benefit and strong correlation between depth information and semantic information in scene parsing by introducing the Built-in Depth-Semantic Coupled Encoding (BDSCE) module, which adaptively fuses RGB and depth features, and selectively highlights the depth-discriminative features. The proposed BDSCE module is compatible with existing CNN based methods, and can greatly improve scene parsing performance, particularly in those categories that have clear depth distinction and might be misclassified with RGB-only features. Furthermore, we also extend our proposed module to other urban scene semantic reasoning tasks such as vehicle detection and road segmentation, which are implemented by effectively learning and exploiting the encoded depth-semantics and transferring the learned representations with fine-tuning. The extensive experiments on the popular datasets Cityscapes and KITTI demonstrate that our method performs quite well and can significantly improve the state-of-the-art methods.

Indoor Semantic Mapping with Efficient Convolutional Neural Networks for Resource-constrained SLAM System

Scene parsing and robotic manipulation using visual sensing are of significant importance for mobile and robotics related applications. And increasing efforts are being made to associate semantic concepts to geometric entities in robot’s surroundings. However, they are not well suitable for resource-constrained and mobile platforms. Toward an efficient semantic mapping solution, we propose a specifically light-weight segmentation network for building a dense 3D semantic map in real-time. The propose architecture is based on depth-wise convolution and channel shuffle to reduce computation cost. An improved Atrous Spatial Pyramid Pooling (ASPP) and encoder-decoder structures are presented and multi-scale features are merged for better prediction. Experiments are compared with other methods to prove the effectiveness and superiority.

Incremental Instance-Oriented 3D Semantic Mapping via RGB-D Cameras for Unknown Indoor Scene

Scene parsing plays a crucial role when accomplishing human-robot interaction tasks. As the “eye” of the robot, RGB-D camera is one of the most important components for collecting multiview images to construct instance-oriented 3D environment semantic maps, especially in unknown indoor scenes. Although there are plenty of studies developing accurate object-level mapping systems with different types of cameras, these methods either process the instance segmentation problem in completed mapping or suffer from a critical real-time issue due to heavy computation processing required. In this paper, we propose a novel method to incrementally build instance-oriented 3D semantic maps directly from images acquired by the RGB-D camera. To ensure an efficient reconstruction of 3D objects with semantic and instance IDs, the input RGB images are operated by a real-time deep-learned object detector. To obtain accurate point cloud cluster, we adopt the Gaussian mixture model as an optimizer after processing 2D to 3D projection. Next, we present a data association strategy to update class probabilities across the frames. Finally, a map integration strategy fuses information about their 3D shapes, locations, and instance IDs in a faster way. We evaluate our system on different indoor scenes including offices, bedrooms, and living rooms from the SceneNN dataset, and the results show that our method not only builds the instance-oriented semantic map efficiently but also enhances the accuracy of the individual instance in the scene.

Boosting Real-Time Driving Scene Parsing With Shared Semantics

Real-time scene parsing is a fundamental feature for autonomous driving vehicles with multiple cameras. In this letter we demonstrate that sharing semantics between cameras with different perspectives and overlapped views can boost the parsing performance when compared with traditional methods, which individually process the frames from each camera. Our framework is based on a deep neural network for semantic segmentation but with two kinds of additional modules for sharing and fusing semantics. On the one hand, a semantics sharing module is designed to establish the pixel-wise mapping between the input images. Features as well as semantics are shared by the map to reduce duplicated workload which leads to more efficient computation. On the other hand, feature fusion modules are designed to combine different modal of semantic features, which leverage the information from both inputs for better accuracy. To evaluate the effectiveness of the proposed framework, we have applied our network to a dual-camera vision system for driving scene parsing. Experimental results show that our network outperforms the baseline method on the parsing accuracy with comparable computations.

A Survey on Algorithm Research of Scene Parsing Based on Deep Learning

A Survey on Algorithm Research of Scene Parsing Based on Deep Learning

Feature fusion network based on attention mechanism for 3D semantic segmentation of point clouds

3D scene parsing has always been a hot topic and point clouds are efficient data format to represent scenes. The semantic segmentation of point clouds is critical to the 3D scene, which is a challenging problem due to the unordered structure of point clouds. The max-pooling operation is typically used to obtain the order invariant features, while the point-wise features are destroyed after the max-pooling operation. In this paper, we propose a feature fusion network that fuses point-wise features and local features by attention mechanism to compensate for the loss caused by max-pooling operation. By incorporating point-wise features into local features, the point-wise variation is preserved to obtain a refined segmentation accuracy, and the attention mechanism is used to measure the importance of the point-wise features and local features for each 3D point. Extensive experiments show that our method achieves better performances than other prestigious methods.

A novel approach to use semantic segmentation based deep learning networks to classify multi-temporal SAR data

The use of SAR data for land cover mapping provides many advantages over land cover classification achieved using optical remote sensing data. However, the classification of SAR data has always been a challenging task. In this study, the feasibility of the use of semantic segmentation based deep learning networks to classify temporal SAR data has been demonstrated. It has been achieved by applying six deep learning architectures viz. Pyramid Scene Parsing, UNET, DeepLabv3+, Path Aggregation Network, Encoder-Decoder Network and Feature Pyramid Network over temporally acquired SAR datasets for three different frequencies (triannual, quarterly and bimonthly). Outputs of all six architectures have been assessed using frequency weighted IoU. It was observed that Pyramid Scene Parsing architecture when applied on bimonthly temporal SAR provides the best results.

Semantic combined network for zero‐shot scene parsing

Recently, image-based scene parsing has attracted increasing attention due to its wide application. However, conventional models can only be valid on images with the same domain of the training set and are typically trained using discrete and meaningless labels. Inspired by the traditional zero-shot learning methods which employ auxiliary side information to bridge the source and target domains, the authors propose a novel framework called semantic combined network (SCN), which aims at learning a scene parsing model only from the images of the seen classes while targeting on the unseen ones. In addition, with the assistance of semantic embeddings of classes, the proposed SCN can further improve the performances of traditional fully supervised scene parsing methods. Extensive experiments are conducted on the data set Cityscapes, and the results show that the proposed SCN can perform well on both zero-shot scene parsing (ZSSP) and generalised ZSSP settings based on several state-of-the-art scenes parsing architectures. Furthermore, the authors test the proposed model under the traditional fully supervised setting and the results show that the proposed SCN can also significantly improve the performances of the original network models.

A Simple and Light-Weight Attention Module for Convolutional Neural Networks

Many aspects of deep neural networks, such as depth, width, or cardinality, have been studied to strengthen the representational power. In this work, we study the effect of attention in convolutional neural networks and present our idea in a simple self-contained module, called Bottleneck Attention Module (BAM). Given an intermediate feature map, BAM efficiently produces the attention map along two factorized axes, channel and spatial, with negligible overheads. BAM is placed at bottlenecks of various models where the downsampling of feature maps occurs, and is jointly trained in an end-to-end manner. Ablation studies and extensive experiments are conducted in CIFAR-100/ImageNet classification, VOC2007/MS-COCO detection, super resolution and scene parsing with various architectures including mobile-oriented networks. BAM shows consistent improvements over all experiments, demonstrating the wide applicability of BAM. The code and models are available at https://github.com/Jongchan/attentionmodule.

Preserving details in semantics-aware context for scene parsing

Great success of scene parsing (also known as, semantic segmentation) has been achieved with the pipeline of fully convolutional networks (FCNs). Nevertheless, there are a lot of segmentation failures caused by large similarities between local appearances. To alleviate the problem, most of existing methods attempt to improve the global view of FCNs by introducing different contextual modules. Though the reconstructed high resolution output of these methods is of rich semantics, it cannot faithfully recover the fine image details owing to lack of desired precise low-level information. To overcome the problem, we propose to improve the spatial decoding process through embedding possibly lost low-level information in a principled way. To this end, we make the following three contributions. First, we propose a semantics conformity module to make low-level features variations agnostic. Second, we introduce semantics into the conformed low level features through guidance from semantically aware features. Finally, we institute the availability of various possible contextual features at feature fusion to enrich context information. The proposed approach demonstrates competitive performance on challenging PASCAL VOC 2012, Cityscapes, and ADE20K benchmarks in comparison to the state-of-the-art methods.

ORDNet: Capturing Omni-Range Dependencies for Scene Parsing.

Learning to capture dependencies between spatial positions is essential to many visual tasks, especially the dense labeling problems like scene parsing. Existing methods can effectively capture long-range dependencies with self-attention mechanism while short ones by local convolution. However, there is still much gap between long-range and short-range dependencies, which largely reduces the models' flexibility in application to diverse spatial scales and relationships in complicated natural scene images. To fill such a gap, we develop a Middle-Range (MR) branch to capture middle-range dependencies by restricting self-attention into local patches. Also, we observe that the spatial regions which have large correlations with others can be emphasized to exploit long-range dependencies more accurately, and thus propose a Reweighed Long-Range (RLR) branch. Based on the proposed MR and RLR branches, we build an Omni-Range Dependencies Network (ORDNet) which can effectively capture short-, middle- and long-range dependencies. Our ORDNet is able to extract more comprehensive context information and well adapt to complex spatial variance in scene images. Extensive experiments show that our proposed ORDNet outperforms previous state-of-the-art methods on three scene parsing benchmarks including PASCAL Context, COCO Stuff and ADE20K, demonstrating the superiority of capturing omni-range dependencies in deep models for scene parsing task.

STC-GAN: Spatio-Temporally Coupled Generative Adversarial Networks for Predictive Scene Parsing.

Predictive scene parsing is a task of assigning pixellevel semantic labels to a future frame of a video. It has many applications in vision-based artificial intelligent systems, e.g., autonomous driving and robot navigation. Although previous work has shown its promising performance in semantic segmentation of images and videos, it is still quite challenging to anticipate future scene parsing with limited annotated training data. In this paper, we propose a novel model called STC-GAN, Spatio-Temporally Coupled Generative Adversarial Networks for predictive scene parsing, which employ both convolutional neural networks and convolutional long short-term memory (LSTM) in the encoderdecoder architecture. By virtue of STC-GAN, both spatial layout and semantic context can be captured by the spatial encoder effectively, while motion dynamics are extracted by the temporal encoder accurately. Furthermore, a coupled architecture is presented for establishing joint adversarial training where the weights are shared and features are transformed in an adaptive fashion between the future frame generation model and predictive scene parsing model. Consequently, the proposed STCGAN is able to learn valuable features from unlabeled video data. We evaluate our proposed STC-GAN on two public datasets, i.e., Cityscapes and CamVid. Experimental results demonstrate that our method outperforms the state-of-the-art.

Deep Multiphase Level Set for Scene Parsing.

Recently, Fully Convolutional Network (FCN) seems to be the go-to architecture for image segmentation, including semantic scene parsing. However, it is difficult for a generic FCN to predict semantic labels around the object boundaries, thus FCN-based methods usually produce parsing results with inaccurate boundaries. Meanwhile, many works have demonstrate that level set based active contours are superior to the boundary estimation in sub-pixel accuracy. However, they are quite sensitive to initial settings. To address these limitations, in this paper we propose a novel Deep Multiphase Level Set (DMLS) method for semantic scene parsing, which efficiently incorporates multiphase level sets into deep neural networks. The proposed method consists of three modules, i.e., recurrent FCNs, adaptive multiphase level set, and deeply supervised learning. More specifically, recurrent FCNs learn multi-level representations of input images with different contexts. Adaptive multiphase level set drives the discriminative contour for each semantic class, which makes use of the advantages of both global and local information. In each time-step of the recurrent FCNs, deeply supervised learning is incorporated for model training. Extensive experiments on three public benchmarks have shown that our proposed method achieves new state-of-the-art performances. The source codes will be released at https://github.com/Pchank/DMLS-for-SSP.

RAPNet: Residual Atrous Pyramid Network for Importance-Aware Street Scene Parsing.

Street Scene Parsing (SSP) is a fundamental and important step for autonomous driving and traffic scene understanding. Recently, Fully Convolutional Network (FCN) based methods have delivered expressive performances with the help of large-scale dense-labeling datasets. However, in urban traffic environments, not all the labels contribute equally for making the control decision. Certain labels such as pedestrian, car, bicyclist, road lane or sidewalk would be more important in comparison with labels for vegetation, sky or building. Based on this fact, in this paper we propose a novel deep learning framework, named Residual Atrous Pyramid Network (RAPNet), for importance-aware SSP. More specifically, to incorporate the importance of various object classes, we propose an Importance-Aware Feature Selection (IAFS) mechanism which automatically selects the important features for label predictions. The IAFS can operate in each convolutional block, and the semantic features with different importance are captured in different channels so that they are automatically assigned with corresponding weights. To enhance the labeling coherence, we also propose a Residual Atrous Spatial Pyramid (RASP) module to sequentially aggregate global-to-local context information in a residual refinement manner. Extensive experiments on two public benchmarks have shown that our approach achieves new state-of-the-art performances, and can consistently obtain more accurate results on the semantic classes with high importance levels.

Fully Convolutional Pyramidal Networks for Semantic Segmentation

Semantic segmentation networks focus on the scene parsing of an unrestricted open scene. The typical segmentation architectures are stacks consisting of convolutional layers, which are used to extract semantic features. The feature map dimension is sharply changed at sampling units for most of networks, which ensure effective propagation of the gradient in deep nets. In this article, we proposed a state-of-the-art network model named Fully Convolutional Pyramidal Networks (FC-PRNet), which employs pyramidal residual structure to change the feature map dimension at all convolutional layers. This design is an effective way of improving generalization ability and optimizing parameters, and FC-PRNet could achieve excellent capability of semantic extraction. We used urban scene benchmark CamVid and KITTI dataset to test our network, the experimental results show that FC-PRNet achieves better results without any pre-training or post-treatment module. Moreover, due to smart construction of pyramidal residual structures, FC-PRNet has less parameters than other existing networks trained on these datasets.