Semi-Supervised Semantic Segmentation Research Articles

Mapping Temporary Slums From Satellite Imagery Using a Semi-Supervised Approach

One billion people worldwide are estimated to be living in slums, and documenting and analyzing these regions is a challenging task. As compared to regular slums; the small, scattered and temporary nature of temporary slums makes data collection and labeling tedious and time-consuming. To tackle this challenging problem of temporary slums detection, we present a semi-supervised deep learning segmentation-based approach; with the strategy to detect initial seed images in the zero-labeled data settings. A small set of seed samples (32 in our case) are automatically discovered by analyzing the temporal changes, which are manually labeled to train a segmentation and representation learning module. The segmentation module gathers high dimensional image representations, and the representation learning module transforms image representations into embedding vectors. After that, a scoring module uses the embedding vectors to sample images from a large pool of unlabeled images and generates pseudo-labels for the sampled images. These sampled images with their pseudo-labels are added to the training set to update the segmentation and representation learning modules iteratively. To analyze the effectiveness of our technique, we construct a large geographically marked dataset of temporary slums. This dataset constitutes more than 200 potential temporary slum locations (2.28 square kilometers) found by sieving sixty-eight thousand images from 12 metropolitan cities of Pakistan covering 8000 square kilometers. Furthermore, our proposed method outperforms several competitive semi-supervised semantic segmentation baselines on a similar setting. The code and the dataset will be made publicly available.

Semi-Supervised Semantic Segmentation of Remote Sensing Images With Iterative Contrastive Network

With the development of deep learning, semantic segmentation of remote sensing images has made great progress. However, segmentation algorithms based on deep learning usually require a huge number of labeled images for model training. For remote sensing images, pixel-level annotation usually consumes expensive resources. To alleviate this problem, this letter proposes a semi-supervised segmentation method of remote sensing images based on an iterative contrastive network. This method combines few labeled images and more unlabeled images to significantly improve the model performance. First, contrastive networks continuously learn more potential information by using better pseudo labels. Then, the iterative training method keeps the differences between models to better improve the segmentation performance. The semi-supervised experiments on different remote sensing datasets prove that this method has a better performance than the related methods. Code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/VCISwang/ICNet</uri> .

Semisupervised Semantic Segmentation of Remote Sensing Images With Consistency Self-Training

Semisupervised semantic segmentation is an effective way to reduce the expensive manual annotation cost and take advantage of the unlabeled data for remote sensing (RS) image interpretation. Recent related research has mainly adopted two strategies: self-training and consistency regularization. Self-training tries to acquire accurate pseudo-labels to explicitly expand the train set. However, the existing methods cannot accurately identify false pseudo-labels, suffering from their negative impact on model optimization. The consistency regularization constrains the model by producing consistent predictions robust to the perturbations introduced in the sample or feature domain but requires a sufficient number of training data. Therefore, we propose a strategy for the semisupervised semantic segmentation of the RS images. The proposed model in the generative adversarial network (GAN) framework is optimized by consistency self-training, learning the distributions of both labeled and unlabeled data. The discriminator is optimized by accurate pixel-level training labels instead of the image-level ones, thereby assessing the confidence for the prediction of each pixel, which is then used to reweight the loss of the unlabeled data in self-training. The generator is optimized with the consistency constraint with respect to all random perturbations on the unlabeled data, which increases the sample diversity and prompts the model to learn the underlying distribution of the unlabeled data. Experimental results on the the large-scale and densely annotated Instance Segmentation in Aerial Images Dataset (iSAID) datasets and the International Society for Photogrammetry and Remote Sensing (ISPRS) datasets show that our framework outperforms several state-of-the-art semisupervised semantic segmentation methods.

Semi-supervised semantic segmentation of prostate and organs-at-risk on 3D pelvic CT images

The recent development of deep learning approaches has revoluted medical data processing, including semantic segmentation, by dramatically improving performance. Automated segmentation can assist radiotherapy treatment planning by saving manual contouring efforts and reducing intra-observer and inter-observer variations. However, training effective deep learning models usually Requires a large amount of high-quality labeled data, often costly to collect. We developed a novel semi-supervised adversarial deep learning approach for 3D pelvic CT image semantic segmentation. Unlike supervised deep learning methods, the new approach can utilize both annotated and un-annotated data for training. It generates un-annotated synthetic data by a data augmentation scheme using generative adversarial networks (GANs). We applied the new approach to segmenting multiple organs in male pelvic CT images. CT images without annotations and GAN-synthesized un-annotated images were used in semi-supervised learning. Experimental results, evaluated by three metrics (Dice similarity coefficient, average Hausdorff distance, and average surface Hausdorff distance), showed that the new method achieved comparable performance with substantially fewer annotated images or better performance with the same amount of annotated data, outperforming the existing state-of-the-art methods.

Deep Learning on Construction Sites: A Case Study of Sparse Data Learning Techniques for Rebar Segmentation.

Recent advances in deep learning models for image interpretation finally made it possible to automate construction site monitoring processes that rely on remote sensing. However, the major drawback of these models is their dependency on large datasets of training images labeled at pixel level, which must be produced manually by skilled personnel. To reduce the need for training data, this study evaluates weakly and semi-supervised semantic segmentation models for construction site imagery to efficiently automate monitoring tasks. As a case study, we compare fully, weakly and semi-supervised methods for the detection of rebar covers, which are useful for quality control. In the experiments, recent models, i.e., IRNet, DeepLabv3+ and the cross-consistency training model are compared for their ability to segment rebar covers from construction site imagery with minimal manual input. The results show that weakly and semi-supervised models can indeed rival with the performance of fully supervised models with the majority of the target objects being properly found. This study provides construction site stakeholders with detailed information on how to leverage deep learning for efficient construction site monitoring and weigh preprocessing, training, and testing efforts against each other in order to decide between fully, weakly and semi-supervised training.

Adversarial network integrating dual attention and sparse representation for semi-supervised semantic segmentation

Semantic segmentation is the important task of assigning a semantic label to each pixel. However, semantic segmentation based on the deep neural network usually requires massive annotations consumption to acquire better performance. To avoid the problem, some algorithms based on weakly-supervised and semi-supervised conditions have been proposed and achieved gradually improving performance in recent years. In this paper, we propose a novel semi-supervised adversarial network to alleviate the shortage of labeled data, which only requires a few labeled images to get competitive performance. The model is composed of two parts: the segmentation network and the discriminator network. The first part aims to semantically generate a segmented result that has the same size as the input color image. The discriminator network is designed in a fully convolutional manner to distinguish the predicted probability maps depending on the ground truth distribution. In particular, the probability maps are regarded as focal attention maps, which are fed back to the segmentation network to make the model converge faster, and the process can induce the model to focus on pixels that are hard to segment. To enhance the representation ability of image features, sparse representation and dual attention are adopted in the segmentation network. The sparse representation module aims to emphasize the object edges and locations by learning the convolutional sparse representation of the input color images, and the dual attention module can exploit the semantic interdependencies in two different dimensions. Moreover, the semi-supervised mechanism is introduced to the network, in which the adaptive parameter T that controls the sensitivity of the self-taught phase is proposed, and the training dataset is split into two parts for fully-supervised learning and semi-supervised learning. Specifically, the first part is unlabeled data, which is applied to provide supervised signals for semi-supervised training. The labeled data drawn from the other part is utilized for fully-supervised learning. Our semi-supervised adversarial framework can improve the learning ability and achieve higher performance, also, provide a novel approach to tackle the semantic segmentation task. Finally, comprehensive experiments on the PASCAL VOC 2012 and Cityscapes datasets are conducted to verify the effectiveness of the proposed model, which achieves the expected performance.

Stable self-attention adversarial learning for semi-supervised semantic image segmentation

The application of adversarial learning for semi-supervised semantic image segmentation based on convolutional neural networks can effectively reduce the number of manually generated labels required in the training process. However, the convolution operator of the generator in the generative adversarial network (GAN) has a local receptive field, so that the long-range dependencies between different image regions can only be modeled after passing through multiple convolutional layers. The present work addresses this issue by introducing a self-attention mechanism in the generator of the GAN to effectively account for relationships between widely separated spatial regions of the input image with supervision based on pixel-level ground truth data. In addition, the adjustment of the discriminator has been demonstrated to affect the stability of GAN training performance. This is addressed by applying spectral normalization to the GAN discriminator during the training process. Our method has better performance than existing full/semi-supervised semantic image segmentation techniques.

Semi-supervised semantic segmentation network for surface crack detection

The detection of surface crack is essential to ensure the safety and the serviceability of civil infrastructure. The automatic method is highly efficient and the test results are objective, which makes it gradually replace conventical manual inspection. Recently, semantic segmentation algorithms based on deep learning have shown excellent performance in crack detection tasks. However, the commonly used fully supervised segmentation method requires manual annotation of large amounts of data, which is time-consuming. In order to solve this problem, we propose a semi-supervised semantic segmentation network for crack detection. The proposed method consists of student model and teacher model. The two models have the same network structure and use the EfficientUNet to extract multi-scale crack feature information, reducing the loss of image information. The student model updates weights through the gradient descent of loss function, and the teacher model uses the exponential moving average weights of the student model. During training, the robustness of the model is improved by adding noise to the input data. When using only 60% of the annotated data, our method achieves an F1 score of 0.6540 on the concrete crack dataset and 0.8321 on the Crack500 dataset. The results show that our method can greatly reduce the workload of annotation while maintaining high accuracy.

Semi-supervised semantic segmentation in Earth Observation: the MiniFrance suite, dataset analysis and multi-task network study

The development of semi-supervised learning techniques is essential to enhance the generalization capacities of machine learning algorithms. Indeed, raw image data are abundant while labels are scarce, therefore it is crucial to leverage unlabeled inputs to build better models. The availability of large databases have been key for the development of learning algorithms with high level performance. Despite the major role of machine learning in Earth Observation to derive products such as land cover maps, datasets in the field are still limited, either because of modest surface coverage, lack of variety of scenes or restricted classes to identify. We introduce a novel large-scale dataset for semi-supervised semantic segmentation in Earth Observation, the MiniFrance suite. MiniFrance has several unprecedented properties: it is large-scale, containing over 2000 very high resolution aerial images, accounting for more than 200 billions samples (pixels); it is varied, covering 16 conurbations in France, with various climates, different landscapes, and urban as well as countryside scenes; and it is challenging, considering land use classes with high-level semantics. Nevertheless, the most distinctive quality of MiniFrance is being the only dataset in the field especially designed for semi-supervised learning: it contains labeled and unlabeled images in its training partition, which reproduces a life-like scenario. Along with this dataset, we present tools for data representativeness analysis in terms of appearance similarity and a thorough study of MiniFrance data, demonstrating that it is suitable for learning and generalizes well in a semi-supervised setting. Finally, we present semi-supervised deep architectures based on multi-task learning and the first experiments on MiniFrance. These results will serve as baselines for future work on semi-supervised learning over the MiniFrance dataset. The Minifrance suite and related semi-supervised networks will be publicly available to promote semi-supervised works in Earth Observation.

Semi-supervised semantic segmentation using an improved generative adversarial network

This paper proposes a better semi-supervised semantic segmentation network using an improved generative adversarial network. It is important for the discriminator on the pixel level to know whether it correctly distinguishes the predicted probability map. However, currently there is no correlation between the actual credibility and the confidence map generated by the pixel-level discriminator. We study this problem and a new network is proposed, which includes one generator and two discriminators. One of the discriminators can output more reliable confidence maps on the pixel level and the other is trained to generate the probability on the image level, which is used as the dynamic threshold in the semi-supervised module instead of being set manually. In addition, the trusted region shared by the two discriminators is used to provide the semi-supervised reference. Through experiments on the PASCAL VOC 2012 and Cityscapes datasets, the proposed network brings better gains, proving the effectiveness of the network.

Semi-supervised generative adversarial networks for the segmentation of the left ventricle in pediatric MRI

Segmentation of the left ventricle in magnetic resonance imaging (MRI) is important for assessing cardiac function. We present DT-GAN, a generative adversarial network (GAN) segmentation approach for the identification of the left ventricle in pediatric MRI. Segmentation of the left ventricle requires a large amount of annotated data; generating such data can be time-consuming and subject to observer variability. Additionally, it can be difficult to accomplish in a clinical setting. During the training of our GAN, we therefore introduce a semi-supervised semantic segmentation to reduce the number of images required for training, while maintaining a good segmentation accuracy. The GAN generator produces a segmentation label map and its discriminator outputs a confidence map, which gives the probability of a pixel coming from the label or from the generator. Moreover, we propose a new formulation of the GAN loss function based on distance transform and pixel-wise cross-entropy. This new loss function provides a better segmentation of boundary pixels, by favoring the correct classification of those pixels rather than focusing on pixels that are farther away from the boundary between anatomical structures. Our proposed method achieves a mean Hausdorff distance of 2.16 mm ± 0.42 mm (2.28 mm ± 0.21 mm for U-Net) and a Dice score of 0.88 ± 0.08 (0.91 ± 0.12 for U-Net) for the endocardium segmentation, using 50% of the annotated data. For the epicardium segmentation, we achieve a mean Hausdorff distance of 2.23 mm ± 0.35 mm (2.34 mm ± 0.39 mm for U-Net) and a Dice score of 0.93 mm ± 0.04 mm (0.89 ± 0.09 for U-Net). For the myocardium segmentation, we achieve a mean Hausdorff distance of 2.98 mm ± 0.43 mm (3.04 mm ± 0.27 mm for U-Net) and a Dice score of 0.79 mm ± 0.10 mm (0.74 ± 0.04 for U-Net). This new model could be very useful for the automatic analysis of cardiac MRI and for conducting large-scale studies based on MRI readings, with a limited amount of training data.

Weakly Supervised and Semi-Supervised Semantic Segmentation for Optic Disc of Fundus Image

Weakly supervised and semi-supervised semantic segmentation has been widely used in the field of computer vision. Since it does not require groundtruth or it only needs a small number of groundtruths for training. Recently, some works use pseudo groundtruths which are generated by a classified network to train the model, however, this method is not suitable for medical image segmentation. To tackle this challenging problem, we use the GrabCut method to generate the pseudo groundtruths in this paper, and then we train the network based on a modified U-net model with the generated pseudo groundtruths, finally we utilize a small amount of groundtruths to fine tune the model. Extensive experiments on the challenging RIM-ONE and DRISHTI-GS benchmarks strongly demonstrate the effectiveness of our algorithm. We obtain state-of-art results on RIM-ONE and DRISHTI-GS databases.

Saliency Guided Self-Attention Network for Weakly and Semi-Supervised Semantic Segmentation

Weakly supervised semantic segmentation (WSSS) using only image-level labels can greatly reduce the annotation cost and therefore has attracted considerable research interest. However, its performance is still inferior to the fully supervised counterparts. To mitigate the performance gap, we propose a saliency guided self-attention network (SGAN) to address the WSSS problem. The introduced self-attention mechanism is able to capture rich and extensive contextual information but may mis-spread attentions to unexpected regions. In order to enable this mechanism to work effectively under weak supervision, we integrate class-agnostic saliency priors into the self-attention mechanism and utilize class-specific attention cues as an additional supervision for SGAN. Our SGAN is able to produce dense and accurate localization cues so that the segmentation performance is boosted. Moreover, by simply replacing the additional supervisions with partially labeled ground-truth, SGAN works effectively for semi-supervised semantic segmentation as well. Experiments on the PASCAL VOC 2012 and COCO datasets show that our approach outperforms all other state-of-the-art methods in both weakly and semi-supervised settings.

Semi-Supervised Semantic Segmentation Using Adversarial Learning for Pavement Crack Detection

Regular inspection of pavement conditions is important to guarantee the safety of transportation. However, current approaches are time-consuming and subjective, which requires the technician to annotate each training image exactly pixel by pixel. To ease the workload of the inspector and lower the cost of acquiring the high-quality training dataset, a semi-supervised method for the pavement crack detection is proposed. Firstly, unlabeled pavement images can be used for the model training in our proposed algorithm, our model can generate a supervisory signal for unlabeled pavement images, which makes up for the deficiency of image annotation. Secondly, an adversarial learning method and a full convolution discriminator are adopted, which can learn to distinguish the ground truth from segmentation predictions. To improve the accuracy of pavement crack detection, the adversarial loss is coupled with the cross-entropy loss in discriminator. Thus, the quality of the training model is no longer dependent on the quantity of the labeled dataset and the accuracy of the labeled. Compared with existing methods that can only employ labeled images, our method utilizes unlabeled images to improve the pavement crack detection accuracy. Moreover, our model is validated on the CFD dataset and AigleRN dataset, the experimental results show that the proposed algorithm is effective. Compared with existing methods, not only can our method detect different types of cracks, but also be particularly effective when only a few labeled are available: when using 118 crack images with a resolution of 480 × 320, using only 50% of the labeled data, the detection accuracy of our model can reach 95.91%.

Digging Into Pseudo Label: A Low-Budget Approach for Semi-Supervised Semantic Segmentation

The capability to understand visual scenes with limited labeled data has been widely concerned in the field of computer vision. Although semi-supervised learning for image classification has been extensively studied in some cases, semantic segmentation with limited data has only recently gained attention. In this work, we follow the standard semi-supervised segmentation pipeline for image classification and propose a two-branch network that can encode strong and pseudo label spaces respectively, extracting reliable supervision information from pseudo-labels to assist in training network with strong labels. Our method outperforms previous semi-supervised methods with limited annotation cost. On standard benchmark PASCAL VOC 2012 for semi-supervised semantic segmentation, the proposed approach gains fresh state-of-the-art performance.

BAS$^{4}$Net: Boundary-Aware Semi-Supervised Semantic Segmentation Network for Very High Resolution Remote Sensing Images

Semantic segmentation is a fundamental task in remote sensing image understanding. Recently, Deep Convolutional Neural Networks (DCNNs) have considerably improved the performance of the semantic segmentation of natural scenes. However, it is still challenging for very high-resolution remote sensing images. Due to the large and complex scenes as well as the influence of illumination and imaging angle, it is particularly difficult for the existing methods to accurately obtain the category of pixels at object boundaries—the so-called boundary blur. We propose a framework called Boundary-Aware Semi-Supervised Semantic Segmentation Network (BAS4Net), which obtains more accurate segmentation results without additional annotation workload, especially at the object boundaries. The Channel-weighted Multi-scale Feature (CMF) module balances semantic and spatial information, and the Boundary Attention Module (BAM) weights the features with rich semantic boundary information to alleviate the boundary blur. Additionally, to decrease the amount of difficult and tedious manual labeling of remote sensing images, a discriminator network infers pseudolabels from unlabeled images to assist semisupervised learning, and further improves the performance of the segmentation network. To validate the effectiveness of the proposed framework, extensive experiments have been performed on both the ISPRS Vaihingen dataset, and the novel remote sensing dataset AIR-SEG with more categories, and complex boundaries. The results demonstrate a significant improvement of accuracy especially on boundaries and for small objects.

Multiscale and Adversarial Learning-Based Semi-Supervised Semantic Segmentation Approach for Crack Detection in Concrete Structures

Typically, the operational lifetime of underground concrete structures is several decades. At present, many such structures are approaching their original life expectancy. In this stage, the essential functionality of the structures may be considerably degraded, leading to various safety hazards such as collapse roof and tunnel flooding. In general, to overcome such problems, the maintenance of underground structures has been conducted through manual subjective inspections so far. However, recently, several objective inspection technologies have been actively developed by fusing artificial intelligence and imaging techniques recently. In particular, deep learning algorithms have been developed to detect concrete cracks, based on a large amount of data for supervised learning, including numerous labeled images. Such data acquisition requires considerable time and effort. To reduce these costs, in this study, multiscale and adversarial learning techniques were applied to realize crack detection. A total of 1,200 labeled data and 3,000 unlabeled data were used to implement and verify the proposed method. The multiscale segmentation neural network, discriminator neural network, and adversarial learning technique were used to realize accurate crack detection, enhance the learning performance, and ensure the efficiency of training data, respectively. The resulting algorithm had a pixel accuracy, mean intersection over union, frequency weighted intersection over union, and F1 score of 98.176%, 88.936%, 96.525%, and 88.789%, respectively. The proposed technique can be used to examine the conditions to ensure the safe maintenance of aging structures.

Semi-Supervised Semantic Segmentation With High- and Low-Level Consistency.

The ability to understand visual information from limited labeled data is an important aspect of machine learning. While image-level classification has been extensively studied in a semi-supervised setting, dense pixel-level classification with limited data has only drawn attention recently. In this work, we propose an approach for semi-supervised semantic segmentation that learns from limited pixel-wise annotated samples while exploiting additional annotation-free images. The proposed approach relies on adversarial training with a feature matching loss to learn from unlabeled images. It uses two network branches that link semi-supervised classification with semi-supervised segmentation including self-training. The dual-branch approach reduces both the low-level and the high-level artifacts typical when training with few labels. The approach attains significant improvement over existing methods, especially when trained with very few labeled samples. On several standard benchmarks-PASCAL VOC 2012, PASCAL-Context, and Cityscapes-the approach achieves new state-of-the-art in semi-supervised learning.

Transferable Semi-Supervised Semantic Segmentation

The performance of deep learning based semantic segmentation models heavily depends on sufficient data with careful annotations. However, even the largest public datasets only provide samples with pixel-level annotations for rather limited semantic categories. Such data scarcity critically limits scalability and applicability of semantic segmentation models in real applications. In this paper, we propose a novel transferable semi-supervised semantic segmentation model that can transfer the learned segmentation knowledge from a few strong categories with pixel-level annotations to unseen weak categories with only image-level annotations, significantly broadening the applicable territory of deep segmentation models. In particular, the proposed model consists of two complementary and learnable components: a Label transfer Network (L-Net) and a Prediction transfer Network (P-Net). The L-Net learns to transfer the segmentation knowledge from strong categories to the images in the weak categories and produces coarse pixel-level semantic maps, by effectively exploiting the similar appearance shared across categories. Meanwhile, the P-Net tailors the transferred knowledge through a carefully designed adversarial learning strategy and produces refined segmentation results with better details. Integrating the L-Net and P-Net achieves 96.5% and 89.4% performance of the fully-supervised baseline using 50% and 0% categories with pixel-level annotations respectively on PASCAL VOC 2012. With such a novel transfer mechanism, our proposed model is easily generalizable to a variety of new categories, only requiring image-level annotations, and offers appealing scalability in real applications.

Deep convolutional neural networks for semi-supervised learning from synthetic aperture sonar (SAS) images

Advancements in deep neural networks for computer-vision tasks have the potential to improve automatic target recognition (ATR) in synthetic aperture sonar (SAS) imagery. Many of the recent improvements in computer vision have been made possible by densely labeled datasets such as ImageNet. In contrast, SAS datasets typically contain far fewer labeled samples than unlabeled samples—often by several orders of magnitude. Yet unlabeled SAS data contain information useful for both generative and discriminative tasks. Here results are shown from semi-supervised ladder networks for learning to classify and localize in SAS images from very few labels. We perform end-to-end training concurrently with unlabeled and labeled samples and find that the unsupervised-learning task improves classification accuracy. Ladder networks are employed to adapt fully convolutional networks used for pixelwise prediction based on supervised training to semi-supervised semantic segmentation and target localization by pixel-level classification of whole SAS images. Using this approach, we find improved segmentation and better generalization in new SAS environments compared to purely supervised learning. We hypothesize that utilizing large unsupervised data in conjunction with the supervised classification task helps the network generalize by learning more invariant hierarchical features. [Work supported by the Office of Naval Research.]