Semantic Segmentation Framework Research Articles

Semi-Supervised Learning for Semantic Segmentation of Emphysema With Partial Annotations.

Segmentation and quantification of each subtype of emphysema is helpful to monitor chronic obstructive pulmonary disease. Due to the nature of emphysema (diffuse pulmonary disease), it is very difficult for experts to allocate semantic labels to every pixel in the CT images. In practice, partially annotating is a better choice for the radiologists to reduce their workloads. In this paper, we propose a new end-to-end trainable semi-supervised framework for semantic segmentation of emphysema with partial annotations, in which a segmentation network is trained from both annotated and unannotated areas. In addition, we present a new loss function, referred to as Fisher loss, to enhance the discriminative power of the model and successfully integrate it into our proposed framework. Our experimental results show that the proposed methods have superior performance over the baseline supervised approach (trained with only annotated areas) and outperform the state-of-the-art methods for emphysema segmentation.

Estimation of crack width based on shape‐sensitive kernels and semantic segmentation

Cracks that develop in railway infrastructural components such as tunnel linings and track systems are not easy to detect on high-speed rail routes, since inspection time is limited during the daytime and visibility is very poor at night. Meanwhile, cracks to structures such as those above mentioned are strictly monitored and treated to prevent possible malfunction or accident. In this regard, a track measurement vehicle is normally deployed to image track components and measure geometric information. The main goal of the present study was to detect cracks in images and to simultaneously measure the maximum crack width by means of newly introduced deep learning technology. For this, a shape-sensitive kernel, that is, crack-like kernel, within a semantic segmentation framework and a modified deep layer model were proposed. In addition to the conventional statistical models such as accuracy and intersection over union, the predicted results of the proposed models were verified by considering the boxplot and root mean square errors of the estimated crack widths. According to the results, the proposed shape-sensitive kernel function was able to predict crack width more precisely by one or two pixels than the conventional semantic segmentation model. Future work will concentrate on the integration of the crack detection model with deterioration prediction of track geometry in order to enable systematic decision making for the predictive maintenance of railway systems.

Spatial information inference net: Road extraction using road-specific contextual information

Deep neural networks perform well in road extraction from very high-resolution satellite imagery. A network with certain reasoning ability will give more satisfactory road network extraction results. In this study, we designed a spatial information inference structure, which enables multidirectional message passing between pixels when it is integrated to a typical semantic segmentation framework. Since the spatial information could be propagated and reinforced via inter layer propagation, the proposed road extraction network can learn both the local visual characteristics of the road and the global spatial structure information (such as the continuity and trend of the road). As a result, this method can effectively solve occlusions and preserve the continuity of the extracted road. The validation experiments using three large datasets of very high-resolution (VHR) satellite imagery show that the proposed method can improve road extraction accuracy and provide an output that is more in line with human expectations.

Vision-Based Autonomous Crack Detection of Concrete Structures Using a Fully Convolutional Encoder-Decoder Network.

The visual inspection of massive civil infrastructure is a common trend for maintaining its reliability and structural health. However, this procedure, which uses human inspectors, requires long inspection times and relies on the subjective and empirical knowledge of the inspectors. To address these limitations, a machine vision-based autonomous crack detection method is proposed using a deep convolutional neural network (DCNN) technique. It consists of a fully convolutional neural network (FCN) with an encoder and decoder framework for semantic segmentation, which performs pixel-wise classification to accurately detect cracks. The main idea is to capture the global context of a scene and determine whether cracks are in the image while also providing a reduced and essential picture of the crack locations. The visual geometry group network (VGGNet), a variant of the DCCN, is employed as a backbone in the proposed FCN for end-to-end training. The efficacy of the proposed FCN method is tested on a publicly available benchmark dataset of concrete crack images. The experimental results indicate that the proposed method is highly effective for concrete crack classification, obtaining scores of approximately 92% for both the recall and F1 average.

PASS: Panoramic Annular Semantic Segmentation

Pixel-wise semantic segmentation is capable of unifying most of driving scene perception tasks, and has enabled striking progress in the context of navigation assistance, where an entire surrounding sensing is vital. However, current mainstream semantic segmenters are predominantly benchmarked against datasets featuring narrow Field of View (FoV), and a large part of vision-based intelligent vehicles use only a forward-facing camera. In this paper, we propose a Panoramic Annular Semantic Segmentation (PASS) framework to perceive the whole surrounding based on a compact panoramic annular lens system and an online panorama unfolding process. To facilitate the training of PASS models, we leverage conventional FoV imaging datasets, bypassing the efforts entailed to create fully dense panoramic annotations. To consistently exploit the rich contextual cues in the unfolded panorama, we adapt our real-time ERF-PSPNet to predict semantically meaningful feature maps in different segments, and fuse them to fulfill panoramic scene parsing. The innovation lies in the network adaptation to enable smooth and seamless segmentation, combined with an extended set of heterogeneous data augmentations to attain robustness in panoramic imagery. A comprehensive variety of experiments demonstrates the effectiveness for real-world surrounding perception in a single PASS, while the adaptation proposal is exceptionally positive for state-of-the-art efficient networks.

Weaklier Supervised Semantic Segmentation With Only One Image Level Annotation per Category.

Image semantic segmentation tasks and methods based on weakly supervised conditions have been proposed and achieve better and better performance in recent years. However, the purpose of these tasks is mainly to simplify the labeling work. In this paper, we establish a new and more challenging task condition: weaklier supervision with one image level annotation per category, which only provides prior knowledge that humans need to recognize new objects, and aims to achieve pixel-level object semantic understanding. In order to solve this problem, a three-stage semantic segmentation framework is put forward, which realizes image level, pixel level, and object common features learning from coarse to fine grade, and finally obtains semantic segmentation results with accurate and complete object regions. Researches on PASCAL VOC 2012 dataset demonstrates the effectiveness of the proposed method, which makes an obvious improvement compared to baselines. Based on fewer supervised information, the method also provides satisfactory performance compared to weakly supervised learning-based methods with complete image-level annotations.

Coarse-to-Fine Semantic Segmentation From Image-Level Labels.

Deep neural network-based semantic segmentation generally requires large-scale cost extensive annotations for training to obtain better performance. To avoid pixel-wise segmentation annotations that are needed for most methods, recently some researchers attempted to use object-level labels (e.g., bounding boxes) or image-level labels (e.g., image categories). In this paper, we propose a novel recursive coarse-to-fine semantic segmentation framework based on only image-level category labels. For each image, an initial coarse mask is first generated by a convolutional neural network-based unsupervised foreground segmentation model and then is enhanced by a graph model. The enhanced coarse mask is fed to a fully convolutional neural network to be recursively refined. Unlike the existing image-level label-based semantic segmentation methods, which require labeling of all categories for images that contain multiple types of objects, our framework only needs one label for each image and can handle images that contain multi-category objects. Only trained on ImageNet, our framework achieves comparable performance on the PASCAL VOC dataset with other image-level label-based state-of-the-art methods of semantic segmentation. Furthermore, our framework can be easily extended to foreground object segmentation task and achieves comparable performance with the state-of-the-art supervised methods on the Internet object dataset.

Self-Supervised Model Adaptation for Multimodal Semantic Segmentation

Learning to reliably perceive and understand the scene is an integral enabler for robots to operate in the real-world. This problem is inherently challenging due to the multitude of object types as well as appearance changes caused by varying illumination and weather conditions. Leveraging complementary modalities can enable learning of semantically richer representations that are resilient to such perturbations. Despite the tremendous progress in recent years, most multimodal convolutional neural network approaches directly concatenate feature maps from individual modality streams rendering the model incapable of focusing only on relevant complementary information for fusion. To address this limitation, we propose a mutimodal semantic segmentation framework that dynamically adapts the fusion of modality-specific features while being sensitive to the object category, spatial location and scene context in a self-supervised manner. Specifically, we propose an architecture consisting of two modality-specific encoder streams that fuse intermediate encoder representations into a single decoder using our proposed self-supervised model adaptation fusion mechanism which optimally combines complementary features. As intermediate representations are not aligned across modalities, we introduce an attention scheme for better correlation. In addition, we propose a computationally efficient unimodal segmentation architecture termed AdapNet++ that incorporates a new encoder with multiscale residual units and an efficient atrous spatial pyramid pooling that has a larger effective receptive field with more than 10x fewer parameters, complemented with a strong decoder with a multi-resolution supervision scheme that recovers high-resolution details. Comprehensive empirical evaluations on several benchmarks demonstrate that both our unimodal and multimodal architectures achieve state-of-the-art performance.

An adaptable approach to automated visual detection of plant organs with applications in grapevine breeding

Periodic and multi-scale field phenotyping of seedlings, mapping populations or genetic repositories are one of the most challenging demands of today's (grapevine) breeding programs and research. The challenges are given by large field sizes with several hundreds of plants that need to be phenotyped nowadays by laborious, manual and subjective classification methods. This work presents an adaptable and transferable approach to the automated detection, localisation and counting of plant organs in RGB images given different growth stages of the plants and different procedures of image acquisition. Adaptability and transferability are achieved by employing a deep learning semantic segmentation framework allowing an end-to-end learning of all system parameters of the complete workflow and thereby replacing the programming of different system variants for different scenarios. Adaptability and transferability are demonstrated within an extensive evaluation of six experimental series using images taken at the Institute for Grapevine Breeding Geilweilerhof yielding localisation accuracies of up to 87.3% IoU and classification accuracies up to 88.6% F1 score.

Voxel-based 3D Point Cloud Semantic Segmentation: Unsupervised Geometric and Relationship Featuring vs Deep Learning Methods

Automation in point cloud data processing is central in knowledge discovery within decision-making systems. The definition of relevant features is often key for segmentation and classification, with automated workflows presenting the main challenges. In this paper, we propose a voxel-based feature engineering that better characterize point clusters and provide strong support to supervised or unsupervised classification. We provide different feature generalization levels to permit interoperable frameworks. First, we recommend a shape-based feature set (SF1) that only leverages the raw X, Y, Z attributes of any point cloud. Afterwards, we derive relationship and topology between voxel entities to obtain a three-dimensional (3D) structural connectivity feature set (SF2). Finally, we provide a knowledge-based decision tree to permit infrastructure-related classification. We study SF1/SF2 synergy on a new semantic segmentation framework for the constitution of a higher semantic representation of point clouds in relevant clusters. Finally, we benchmark the approach against novel and best-performing deep-learning methods while using the full S3DIS dataset. We highlight good performances, easy-integration, and high F1-score (> 85%) for planar-dominant classes that are comparable to state-of-the-art deep learning.

Multi-class multimodal semantic segmentation with an improved 3D fully convolutional networks

Semantic segmentation is an important but challenging task in the field of medical image analysis. Automatic labeling for different anatomical structures can be useful for disease diagnosis, treatment planning and development/degeneration evaluation. However, due to the large shape and appearance variance among different subjects, accurate and reliable semantic segmentation is difficult for automatic labeling and time-consuming for manual labeling. In this paper, we propose a multi-class semantic segmentation algorithm based on the 3D fully convolutional networks, exploring the multimodal image blocks. The use of multimodal image blocks naturally allows an effective data augmentation. Besides, we also investigated the use of ensemble learning and conditional random fields (CRFs) as the post-processing step, which obtains the spatially consistent segmentation results. Our proposed method was evaluated on two public benchmarks: BRATS2017 and MNI-HiSUB25. For both datasets, the experimental results demonstrated that our method was able to establish an effective semantic segmentation framework with multi-modality data and achieve better performance compared to the conventional fully convolutional network.

Joint CRF and Locality-Consistent Dictionary Learning for Semantic Segmentation

Semantic image segmentation can be accomplished by assigning a proper object category label to each meaningful region of an image. Beyond the original bottom-up models, the use of top-down categorization information has been applied to semantic segmentation to improve performance. An excellent example of such a top-down scheme is to integrate a Conditional Random Field (CRF) model with sparse dictionary learning. However, the existing solutions merely consider the discrimination of dictionaries to obtain better sparse codes, without considering the inherent data locality characteristics. In this paper, we explore such characteristics and propose a novel semantic segmentation framework based on an innovative CRF model with locality-consistent dictionary learning. In particular, we propose two new locality-consistent dictionary learning strategies by capturing the local consistencies in the feature space and the label space. In addition, we develop a joint dictionary and a CRF model parameter learning algorithm to seamlessly integrate the proposed locality-consistent dictionary learning strategies into the CRF model. Extensive experiments are conducted with two popular databases of different traits (i.e., Graz-02 and PASCAL-CONTEXT). The simulation results confirm the efficiency of the proposed scheme, especially when training data are limited.

A Novel Object-Based Deep Learning Framework for Semantic Segmentation of Very High-Resolution Remote Sensing Data: Comparison with Convolutional and Fully Convolutional Networks

Deep learning architectures have received much attention in recent years demonstrating state-of-the-art performance in several segmentation, classification and other computer vision tasks. Most of these deep networks are based on either convolutional or fully convolutional architectures. In this paper, we propose a novel object-based deep-learning framework for semantic segmentation in very high-resolution satellite data. In particular, we exploit object-based priors integrated into a fully convolutional neural network by incorporating an anisotropic diffusion data preprocessing step and an additional loss term during the training process. Under this constrained framework, the goal is to enforce pixels that belong to the same object to be classified at the same semantic category. We compared thoroughly the novel object-based framework with the currently dominating convolutional and fully convolutional deep networks. In particular, numerous experiments were conducted on the publicly available ISPRS WGII/4 benchmark datasets, namely Vaihingen and Potsdam, for validation and inter-comparison based on a variety of metrics. Quantitatively, experimental results indicate that, overall, the proposed object-based framework slightly outperformed the current state-of-the-art fully convolutional networks by more than 1% in terms of overall accuracy, while intersection over union results are improved for all semantic categories. Qualitatively, man-made classes with more strict geometry such as buildings were the ones that benefit most from our method, especially along object boundaries, highlighting the great potential of the developed approach.

Synthesizing Supervision for Learning Deep Saliency Network without Human Annotation.

Recently, the research field of salient object detection is undergoing a rapid and remarkable development along with the wide usage of deep neural networks. Being trained with a large number of images annotated with strong pixel-level ground-truth masks, the deep salient object detectors have achieved the state-of-the-art performance. However, it is expensive and time-consuming to provide the pixel-level ground-truth masks for each training image. To address this problem, this paper proposes one of the earliest frameworks to learn deep salient object detectors without requiring any human annotation. The supervisory signals used in our learning framework are generated through a novel supervision synthesis scheme, in which the key insights are "knowledge source transition" and "supervision by fusion". Specifically, in the proposed learning framework, both the external knowledge source and the internal knowledge source are explored dynamically to provide informative cues for synthesizing supervision required in our approach, while a two-stream fusion mechanism is also established to implement the supervision synthesis process. Comprehensive experiments on four benchmark datasets demonstrate that the deep salient object detector trained by our newly proposed learning framework often works well without requiring any human annotated masks, which even approaches to its upper-bound obtained under the fully supervised learning fashion (within only 3 percent performance gap). Besides, we also apply the salient object detector learnt with our annotation-free learning framework to assist the weakly supervised semantic segmentation task, which demonstrates that our approach can also alleviate the heavy supplementary supervision required in the existing weakly supervised semantic segmentation framework.

Meta-Seg: A Generalized Meta-Learning Framework for Multi-Class Few-Shot Semantic Segmentation

Semantic segmentation performs pixel-wise classification for given images, which can be widely used in autonomous driving, robotics, medical diagnostics and etc. The recent advanced approaches have witnessed rapid progress in semantic segmentation. However, these supervised learning based methods rely heavily on large-scale datasets to acquire strong generalizing ability, such that they are coupled with some constraints. Firstly, human annotation of pixel-level segmentation masks is laborious and time-consuming, which causes relatively expensive training data and make it hard to deal with urgent tasks in dynamic environment. Secondly, the outstanding performance of the above data-hungry methods will decrease with few available training examples. In order to overcome the limitations of the supervised learning semantic segmentation methods, this paper proposes a generalized meta-learning framework, named Meta-Seg. It consists of a meta-learner and a base-learner. Specifically, the meta-learner learns a good initialization and a parameter update strategy from a distribution of few-shot semantic segmentation tasks. The base-learner can be any semantic segmentation models theoretically and can implement fast adaptation (that is updating parameters with few iterations) under the guidance of the meta-learner. In this work, the successful semantic segmentation model FCN8s is integrated into Meta-Seg. Experiments on the famous few-shot semantic segmentation dataset PASCAL $5^{i}$ prove Meta-Seg is a promising framework for few-shot semantic segmentation. Besides, this method can provide with reference for the relevant researches of meta-learning semantic segmentation.

Semantic Segmentation of Transmission Lines and Their Accessories Based on UAV-Taken Images

While unmanned aerial vehicles (UAVs) facilitate transmission line inspection to some extent, they also generate numerous raw images rather than defect analysis results. Because analyzing the UAV-taken images by a human approach is an arduous work, an automatic method is needed to improve the analysis efficiency. In this paper, a framework is proposed to perform the image semantic segmentation of transmission lines and their accessories to generate the final defect detection results. A segment connection algorithm based on matrix operations is proposed to rapidly connect the segment features of objects. In line accessory detection, a background filter and an artificial contour segment feature generator are constructed to improve the detection performance. In addition, a distance threshold parameter automatic tuning mechanism is presented. Images provided by China Southern Power Grid Company taken by the UAVs are employed to validate the effectiveness of the proposed image semantic segmentation framework.

An end-to-end differential network learning method for semantic segmentation

Deep convolution neural network has become the primary framework for semantic image segmentation in recent years, and most existing methods using deep learning have achieved a great improvement on the performance compared with traditional methods. Although most methods using fully convolutional networks are concerned about the segmentation of small objects or small/fine parts of objects, the small object segmentation is still a challenging problem. To the best of our knowledge, the main reason is that several pooling or convolution operations with two or more stride size cause the features of small objects to vanish in later layers, even if taking different kinds of multi-scale measures. In the paper, we design a novel differential network which addresses the small object segmentation. Specifically, our networks include two pipelines: the first pipeline is served as the primary segmentation network using existing methods, and the second one is a refine network that we propose. The score maps of two networks are merged by calculating the sum of corresponding channels in their last layers. We first learn the primary segmentation network to get a coarse segmentation model, and then train the two networks jointly in an end-to-end fashion. Experiments show that our method can deal with small objects effectively. The segmentation performance of our method on PASCAL VOC 2012 dataset is superior to the state-of-the-art methods using only the primary segmentation model without applying a differential network.

A novel framework for semantic segmentation with generative adversarial network

Semantic segmentation plays an important role in a series of high-level computer vision applications. In the state-of-the-art semantic segmentation methods based on fully convolutional neural networks, all label variables are predicted independently from each other, and the restricted field-of-views of the convolutional filters are difficult to capture the long-range information. In this paper, a novel post-processing method based on GAN (Generative Adversarial Network) is explored to reinforce spatial contiguity in the output label maps. With the help of fully connected layers in the discriminator, the GAN can capture the long-range information, and provide an auxiliary higher-order potential loss to the segmentation model, thus the segmentation model has the ability of correcting higher order inconsistencies. Furthermore, the optimization scheme in Wasserstein GAN (WGAN) is adopted to the training process of our model to get better performance and stability. Extensive experiments on public benchmarking database demonstrate the effectiveness of the proposed method.

Leveraging semantic segmentation with learning-based confidence measure

The standard paradigm of semantic segmentation is first training a multi-class classifier to estimate the probability distributions of the pixels’ possible labels and then employing a graphical model to encode contextual information to obtain a globally optimal solution. However, evaluating the quality of the trained classifier is a rarely investigated problem so far and its consequential influence on post-processing is also not yet analyzed. In this paper, we focus on the two problems and propose to estimate the confidence of the classifier by additionally learning a binary classifier via ensemble learning and convolutional neural network (CNN) respectively. Compared with the hand-crafted confidence measures, our learning-based methods are proved to be more effective. Moreover, the estimated confidences are employed to modulate the probability distributions to leverage the post-processing step and high-quality label maps can be established finally. We evaluate our methods on two public available datasets Sift Flow and Pascal-Context, and the results demonstrate that learning to predict confidences is an effective and promising strategy for improving semantic segmentation frameworks.

Scene Understanding Based on High-Order Potentials and Generative Adversarial Networks

Scene understanding is to predict a class label at each pixel of an image. In this study, we propose a semantic segmentation framework based on classic generative adversarial nets (GAN) to train a fully convolutional semantic segmentation model along with an adversarial network. To improve the consistency of the segmented image, the high-order potentials, instead of unary or pairwise potentials, are adopted. We realize the high-order potentials by substituting adversarial network for CRF model, which can continuously improve the consistency and details of the segmented semantic image until it cannot discriminate the segmented result from the ground truth. A number of experiments are conducted on PASCAL VOC 2012 and Cityscapes datasets, and the quantitative and qualitative assessments have shown the effectiveness of our proposed approach.