Stanford Background Dataset Research Articles

Semantic Segmentation by Multi-Scale Feature Extraction Based on Grouped Dilated Convolution Module

Existing studies have shown that effective extraction of multi-scale information is a crucial factor directly related to the increase in performance of semantic segmentation. Accordingly, various methods for extracting multi-scale information have been developed. However, these methods face problems in that they require additional calculations and vast computing resources. To address these problems, this study proposes a grouped dilated convolution module that combines existing grouped convolutions and atrous spatial pyramid pooling techniques. The proposed method can learn multi-scale features more simply and effectively than existing methods. Because each convolution group has different dilations in the proposed model, they have receptive fields of different sizes and can learn features corresponding to these receptive fields. As a result, multi-scale context can be easily extracted. Moreover, optimal hyper-parameters are obtained from an in-depth analysis, and excellent segmentation performance is derived. To evaluate the proposed method, open databases of the Cambridge Driving Labeled Video Database (CamVid) and the Stanford Background Dataset (SBD) are utilized. The experimental results indicate that the proposed method shows a mean intersection over union of 73.15% based on the CamVid dataset and 72.81% based on the SBD, thereby exhibiting excellent performance compared to other state-of-the-art methods.

ESSN: Enhanced Semantic Segmentation Network by Residual Concatenation of Feature Maps

Semantic segmentation performs pixel-level classification of multiple classes in the input image. Previous studies on semantic segmentation have used various methods such as multi-scale image, encoder-decoder, attention, spatial pyramid pooling, conditional random field, and generative models. However, the contexts of various sizes and types in diverse environments make their performance limited in robustly detecting and classifying objects. To address this problem, we propose an enhanced semantic segmentation network (ESSN) robust to various objects, contexts, and environments. The ESSN can extract multi-scale information well by concatenating the residual feature maps with various receptive fields extracted from sequential convolution blocks, and it can improve the performance of semantic segmentation without additional modules such as loss or attention during the training process. We performed the experiments with two open databases, the Stanford background dataset (SBD) and Cambridge-driving labeled video database (CamVid). Experimental results demonstrated the pixel acc. of 92.74%, class acc. of 79.66%, and mIoU of 71.67% with CamVid, and pixel acc. of 87.46%, class acc. of 81.51%, and mIoU of 71.56% with SBD, which are higher than those of the existing state-of-the-art methods. In addition, the average processing time were 31.12 ms and 92.46 ms on the desktop computer and Jetson TX2 embedded system, respectively, which confirmed that ESSN is applicable to both the desktop computer and Jetson TX2 embedded system which is widely used in autonomous vehicles.

MC-SSM: Nonparametric Semantic Image Segmentation With the ICM Algorithm

In the last few years, there has been considerable interest in scene parsing. This task consists of assigning a predefined class label to each pixel (or pre-segmented region) in an image. To best address the complexity challenge of this task, first, we propose a new geometric retrieval strategy to select nearest neighbors from a database containing fully segmented and annotated images. Then, we introduce a novel and simple energy-minimization model. The proposed cost function of this model combines efficiently different global nonparametric semantic likelihood energy terms. These terms are computed from the (pre-)segmented regions of the (query) image and their structural properties (location, texture, color, context, and shape). Different from the traditional approaches, we use a simple and local optimization procedure derived from the iterative conditional modes algorithm to optimize our energy-based model. Experimental results on two challenging datasets: 1) microsoft research Cambridge dataset and 2) Stanford background dataset demonstrate the feasibility and the success of the proposed approach. Compared to existing annotation methods that require training classifiers for each object and learning many parameters, our method is easy to implement, has a few parameters, and combines different criteria.

EF-Index: Determining number of clusters (K) to estimate number of segments (S) in an image

Estimation of number of segments in an image attracts a formidable interest among the research community. The number of segments in an image is estimated by calculating the number of clusters present in the pixels of that image. The present work offers an unsupervised method, named “Electrostatic Force Index (EF-Index)”, to estimate the number of clusters inherent in an image, reporting of which is very rare in literature. The proposed approach is inspired by Coulomb's law of electrostatics. The EF-Index explores the mutual influence of an arbitrary pixel on another, by considering them similar to point charges. Our proposed cluster indexing method, viz. EF-Index is capable of determining the number of clusters present in an image. It has a strong resemblance to the way the electrostatic force is operative between a pair of static point charges in a closed system as per Coulomb's principle. In order to justify the effectiveness of the proposed approach, we have compared EF-Index of a given image with DB-Index, I-Index, CVNN-Index, DOE-AND-SCA and Sym-Index of the same image. Experimental results show that EF-Index is same as other state-of-the-art indices, whereas EF-Index does not require any clustering algorithm. To establish the applicability of the EF-Index, the same is applied for image segmentation considering Berkeley Segmentation Dataset and Stanford Background Dataset. We observe the results obtained conform to the ground truth and results achieved by applying existing well-established segmentation techniques on the same datasets. The efficacy of the proposed approach is further substantiated in terms of its reduced computational overhead in comparison to the state-of-the-art segmentation algorithms.

Roadside vegetation segmentation with Adaptive Texton Clustering Model

Automatic roadside vegetation segmentation is important for various real-world applications and one main challenge is to design algorithms that are capable of representing discriminative characteristics of vegetation while maintaining robustness against environmental effects. This paper presents an Adaptive Texton Clustering Model (ATCM) that combines pixel-level supervised prediction and cluster-level unsupervised texton occurrence frequencies into superpixel-level majority voting for adaptive roadside vegetation segmentation. The ATCM learns generic characteristics of vegetation from training data using class-specific neural networks with color and texture features, and adaptively incorporates local properties of vegetation in every test image using texton based adaptive K-means clustering. The adaptive clustering groups test pixels into local clusters, accumulates texton frequencies in every cluster and calculates cluster-level class probabilities. The pixel- and cluster-level probabilities are integrated via superpixel-level voting to determine the category of every superpixel. We evaluate the ATCM on three real-world datasets, including the Queensland Department of Transport and Main Roads, the Croatia, and the Stanford background datasets, showing very competitive performance to state-of-the-art approaches.

Piecewise Flat Embedding for Image Segmentation.

We introduce a new multi-dimensional nonlinear embedding-Piecewise Flat Embedding (PFE)-for image segmentation. Based on the theory of sparse signal recovery, piecewise flat embedding with diverse channels attempts to recover a piecewise constant image representation with sparse region boundaries and sparse cluster value scattering. The resultant piecewise flat embedding exhibits interesting properties such as suppressing slowly varying signals, and offers an image representation with higher region identifiability which is desirable for image segmentation or high-level semantic analysis tasks. We formulate our embedding as a variant of the Laplacian Eigenmap embedding with an $L_{1,p} (0<p\leq 1)$L1,p(0<p≤1) regularization term to promote sparse solutions. First, we devise a two-stage numerical algorithm based on Bregman iterations to compute $L_{1,1}$L1,1-regularized piecewise flat embeddings. We further generalize this algorithm through iterative reweighting to solve the general $L_{1,p}$L1,p-regularized problem. To demonstrate its efficacy, we integrate PFE into two existing image segmentation frameworks, segmentation based on clustering and hierarchical segmentation based on contour detection. Experiments on four major benchmark datasets, BSDS500, MSRC, Stanford Background Dataset, and PASCAL Context, show that segmentation algorithms incorporating our embedding achieve significantly improved results.

Semantic Image Segmentation Method with Multiple Adjacency Trees and Multiscale Features

Semantic image segmentation is the basis of image understanding, which is one of the most important human cognitive activities. Cognitive studies have shown that human neocortical information transmission depends on cognitive processing at multiple scales, and contextual information aids the human cognitive system in solving perceptual inference tasks. Inspired by multiscale cognitive mechanisms and contextual effects, in this paper, we propose a semantic image segmentation method addressing multiscale features and contextual information. To integrate multiscale features, after over-segmenting an image into small-scale segments, we employ a segment-based classifier and a CRF (conditional random field) model to generate large-scale regions. We then use the features of regions to train a region-based classifier. To capture context, we propose a multiple adjacency tree model where each tree represents one type of region relevance and can be generated by the adjacency graph corresponding to that relevance metric. Using the multiple tree model instead of a general graph model, we can perform exact inference with some simple assumptions and capture multiple types of regional context dependency. Experiments on the MSRC-21 and Stanford background datasets show advantages of our method over a segment-based CRF model using single-scale features. The results demonstrate the importance of multiscale features and contextual information.

Video parsing via spatiotemporally analysis with images

Effective parsing of video through the spatial and temporal domains is vital to many computer vision problems because it is helpful to automatically label objects in video instead of manual fashion, which is tedious. Some literatures propose to parse the semantic information on individual 2D images or individual video frames, however, these approaches only take use of the spatial information, ignore the temporal continuity information and fail to consider the relevance of frames. On the other hand, some approaches which only consider the spatial information attempt to propagate labels in the temporal domain for parsing the semantic information of the whole video, yet the non-injective and non-surjective natures can cause the black hole effect. In this paper, inspirited by some annotated image datasets (e.g., Stanford Background Dataset, LabelMe, and SIFT-FLOW), we propose to transfer or propagate such labels from images to videos. The proposed approach consists of three main stages: I) the posterior category probability density function (PDF) is learned by an algorithm which combines frame relevance and label propagation from images. II) the prior contextual constraint PDF on the map of pixel categories through whole video is learned by the Markov Random Fields (MRF). III) finally, based on both learned PDFs, the final parsing results are yielded up to the maximum a posterior (MAP) process which is computed via a very efficient graph-cut based integer optimization algorithm. The experiments show that the black hole effect can be effectively handled by the proposed approach.

Labeling Spain With Stanford

We present an end-to-end framework for outdoor scene region decomposition, learned on a small set of randomly selected images that generalizes well to multiple data sets containing images from around the world. We discuss the different aspects of the framework especially a generalized variational inference method with better approximations to the true marginals of a graphical model. Experimentally, we explain why the framework is robust and performs competitively on many diverse scene data sets, including several unseen scene types. We have obtained high pixel-level accuracies ( ≈ 80%) in three of the four data sets, which include a benchmark data set known as the Stanford background data set. Our model obtained over 70% accuracy on the fourth data set, which contained a number of indoor and close-up images that are significantly different from our training examples.

Efficient semantic image segmentation with multi-class ranking prior

Semantic image segmentation is of fundamental importance in a wide variety of computer vision tasks, such as scene understanding, robot navigation and image retrieval, which aims to simultaneously decompose an image into semantically consistent regions. Most of existing works addressed it as structured prediction problem by combining contextual information with low-level cues based on conditional random fields (CRFs), which are often learned by heuristic search based on maximum likelihood estimation. In this paper, we use maximum margin based structural support vector machine (S-SVM) model to combine multiple levels of cues to attenuate the ambiguity of appearance similarity and propose a novel multi-class ranking based global constraint to confine the object classes to be considered when labeling regions within an image. Compared with existing global cues, our method is more balanced between expressive power for heterogeneous regions and the efficiency of searching exponential space of possible label combinations. We then introduce inter-class co-occurrence statistics as pairwise constraints and combine them with the prediction from local and global cues based on S-SVMs framework. This enables the joint inference of labeling within an image for better consistency. We evaluate our algorithm on two challenging datasets which are widely used for semantic segmentation evaluation: MSRC-21 dataset and Stanford Background dataset and experimental results show that we obtain high competitive performance compared with state-of-the-art methods, despite that our model is much simpler and efficient.

Integrating low-level and semantic features for object consistent segmentation

The aim of semantic segmentation is to assign each pixel a semantic label. Numerous methods for semantic segmentation have been proposed in recent years and most of them chose pixel or superpixel as the processing primitives. However, as the information contained in a pixel or a superpixel is not discriminative enough, the outputs of these algorithms are usually not object consistent. To tackle this problem, we introduce the concept of object-like regions as a new and higher level processing primitive. We first experimentally showed that using groundtruth segments as processing primitives can boost semantic segmentation accuracy, and then proposed a novel method to produce regions that resemble the groundtruth regions, which we named them as ‘object-like regions’. We achieve this by integrating state of the art low-level segmentation algorithms with typical semantic segmentation algorithms through a novel semantic feature feedback mechanism. We present experimental results on the publicly available image understanding dataset MSRC21 and stanford background dataset, showing that the new method can achieve relatively good semantic segmentation results with far fewer processing primitives.

Recursive Neural Networks Based on PSO for Image Parsing

This paper presents an image parsing algorithm which is based on Particle Swarm Optimization (PSO) and Recursive Neural Networks (RNNs). State-of-the-art method such as traditional RNN-based parsing strategy uses L-BFGS over the complete data for learning the parameters. However, this could cause problems due to the nondifferentiable objective function. In order to solve this problem, the PSO algorithm has been employed to tune the weights of RNN for minimizing the objective. Experimental results obtained on the Stanford background dataset show that our PSO-based training algorithm outperforms traditional RNN, Pixel CRF, region-based energy, simultaneous MRF, and superpixel MRF.

Learning Hierarchical Features for Scene Labeling

Scene labeling consists of labeling each pixel in an image with the category of the object it belongs to. We propose a method that uses a multiscale convolutional network trained from raw pixels to extract dense feature vectors that encode regions of multiple sizes centered on each pixel. The method alleviates the need for engineered features, and produces a powerful representation that captures texture, shape, and contextual information. We report results using multiple postprocessing methods to produce the final labeling. Among those, we propose a technique to automatically retrieve, from a pool of segmentation components, an optimal set of components that best explain the scene; these components are arbitrary, for example, they can be taken from a segmentation tree or from any family of oversegmentations. The system yields record accuracies on the SIFT Flow dataset (33 classes) and the Barcelona dataset (170 classes) and near-record accuracy on Stanford background dataset (eight classes), while being an order of magnitude faster than competing approaches, producing a $(320\times 240)$ image labeling in less than a second, including feature extraction.