Pixel-level Labels Research Articles

Weakly Supervised Building Semantic Segmentation Based on Spot-Seeds and Refinement Process.

Automatic building semantic segmentation is the most critical and relevant task in several geospatial applications. Methods based on convolutional neural networks (CNNs) are mainly used in current building segmentation. The requirement of huge pixel-level labels is a significant obstacle to achieve the semantic segmentation of building by CNNs. In this paper, we propose a novel weakly supervised framework for building segmentation, which generates high-quality pixel-level annotations and optimizes the segmentation network. A superpixel segmentation algorithm can predict a boundary map for training images. Then, Superpixels-CRF built on the superpixel regions is guided by spot seeds to propagate information from spot seeds to unlabeled regions, resulting in high-quality pixel-level annotations. Using these high-quality pixel-level annotations, we can train a more robust segmentation network and predict segmentation maps. To iteratively optimize the segmentation network, the predicted segmentation maps are refined, and the segmentation network are retrained. Comparative experiments demonstrate that the proposed segmentation framework achieves a marked improvement in the building’s segmentation quality while reducing human labeling efforts.

Semantic segmentation of building extraction in very high resolution imagery via optimal segmentation guided by deep seeds

The automatic segmentation of buildings from satellite images is extremely challenging due to the complex shapes of buildings. Deep convolutional neural networks (DCNNs) have recently enabled accurate pixel-level labeling tasks, which ensures precise boundaries. However, the large number of pixel-level labels required for DCNN-based semantic segmentation of buildings represents a considerable obstacle to the process. We propose a framework that relies on deep seeds and optimal segmentation to extract buildings from very high resolution imagery to solve the aforementioned issue. Input images are cropped rather than resized to a low-resolution image and passed directly to DCNN for segmentation. An image resize transformation will lead to a significant loss of image detail, especially for images with high resolution. For both image patches and resized images, a classification network is utilized to locate deep seeds within building and nonbuilding classes. Then, a boundary map can be predicted by passing the resized image through a convolutional oriented boundary network. A hierarchical segmentation tree is built to determine the optimal segmentation by seeking the optimal tree cut. The final segmentation is achieved by developing a graphic model representing a set of nodes that distribute information from deep seeds to unmarked regions. According to the experiments conducted on the ISPRS two-dimensional semantic labeling contest (Potsdam) and the WHU building datasets, the proposed framework is able to significantly enhance the building segmentation accuracy. We have achieved such improvements through a cost-effective computing method without training a segmentation network.

A coarse-to-fine weakly supervised learning method for green plastic cover segmentation using high-resolution remote sensing images

Green plastic cover (GPC) is a kind of green plastic fine mesh primarily used for covering construction sites and mitigating large amounts of dust during construction. Accurate GPC detection is vital for monitoring urban environment and understanding urban development. Convolutional neural network (CNN)-based segmentation methods are widely used for detecting object extents, while they rely on high-quality pixel-level labels with high acquisition cost. In this regard, weakly supervised learning can achieve pixel-level segmentation using only image-level labels, by first generating the class activation map (CAM) to obtain initial pixel-level labels and then applying the CNN-based segmentation methods to detect object extents. However, these initial labels are usually incomplete and noisy, caused by the local high response property of CAM. Moreover, the CNN-based segmentation methods often lead to blurry object boundaries due to the gradual down-sampling of feature maps, and meanwhile suffer from the class imbalance problem in real scenarios. Given these problems, we introduce weakly supervised learning into GPC detection to lower the label acquisition cost. Furthermore, to improve the completeness and correctness of initial labels and mitigate the blurry boundary problem, we propose a coarse-to-fine weakly supervised segmentation method (called CFWS), consisting of three steps: 1) object-based label extraction; 2) noisy label correction; and 3) boundary-aware semantic segmentation. Moreover, to alleviate the class imbalance problem, we propose a classification-then-segmentation strategy and integrate it into the CFWS to detect GPC. We test the CFWS on two datasets from Google Earth and Gaofen-2 high-resolution images, respectively. The results show that the CFWS obtains more complete GPCs and effectively retains boundaries on both datasets compared to existing state-of-the-art methods. In real scenarios, the classification-then-segmentation strategy significantly reduces a large number of false alarms generated by direct segmentation. These findings confirm that the CFWS holds great potentials for large-scale GPC detection and urban environmental monitoring. The source code will be available at https://github.com/lauraset/Coarse-to-fine-weakly-supervised-GPC-segmentation.

Weakly supervised segmentation with cross-modality equivariant constraints.

Weakly supervised learning has emerged as an appealing alternative to alleviate the need for large labeled datasets in semantic segmentation. Most current approaches exploit class activation maps (CAMs), which can be generated from image-level annotations. Nevertheless, resulting maps have been demonstrated to be highly discriminant, failing to serve as optimal proxy pixel-level labels. We present a novel learning strategy that leverages self-supervision in a multi-modal image scenario to significantly enhance original CAMs. In particular, the proposed method is based on two observations. First, the learning of fully-supervised segmentation networks implicitly imposes equivariance by means of data augmentation, whereas this implicit constraint disappears on CAMs generated with image tags. And second, the commonalities between image modalities can be employed as an efficient self-supervisory signal, correcting the inconsistency shown by CAMs obtained across multiple modalities. To effectively train our model, we integrate a novel loss function that includes a within-modality and a cross-modality equivariant term to explicitly impose these constraints during training. In addition, we add a KL-divergence on the class prediction distributions to facilitate the information exchange between modalities which, combined with the equivariant regularizers further improves the performance of our model. Exhaustive experiments on the popular multi-modal BraTS and prostate DECATHLON segmentation challenge datasets demonstrate that our approach outperforms relevant recent literature under the same learning conditions.

Mining Cross-Domain Structure Affinity for Refined Building Segmentation in Weakly Supervised Constraints

Building segmentation for remote sensing images usually requires pixel-level labels which is difficult to collect when the images are in low resolution and quality. Recently, weakly supervised semantic segmentation methods have achieved promising performance, which only rely on image-level labels for each image. However, buildings in remote sensing images tend to present regular structures. The lack of supervision information may result in the ambiguous boundaries. In this paper, we propose a new weakly supervised network for refined building segmentation by mining the cross-domain structure affinity (CDSA) from multi-source remote sensing images. CDSA integrates the ideas of weak supervision and domain adaptation, where a pixel-level labeled source domain and an image-level labeled target domain are required. The target of CDSA is to learn a powerful segmentation network on the target domain with the guidance of source domain data. CDSA mainly consists of two branches, the structure affinity module (SAM) and the spatial structure adaptation (SSA). In brief, SAM is developed to learn the structure affinity of the buildings from source domain, and SSA infuses the structure affinity to the target domain via a domain adaptation approach. Moreover, we design an end-to-end network structure to simultaneously optimize the SAM and SSA. In this case, SAM can receive pseudosupervised information from SSA, and in turn provide a more accurate affinity matrix for SSA. In the experiments, our model can achieve an IoU score at 57.87% and 79.57% for the WHU and Vaihingen data sets. We compare CDSA with several state-of-the-art weakly supervised and domain adaptation methods, and the results indicate that our method presents advantages on two public data sets.

Troubleshooting image segmentation models with human-in-the-loop

Image segmentation lays the foundation for many high-stakes vision applications such as autonomous driving and medical image analysis. It is, therefore, of great importance to not only improve the accuracy of segmentation models on well-established benchmarks, but also enhance their robustness in the real world so as to avoid sparse but fatal failures. In this paper, instead of chasing state-of-the-art performance on existing benchmarks, we turn our attention to a new challenging problem: how to efficiently expose failures of “top-performing” segmentation models in the real world and how to leverage such counterexamples to rectify the models. To achieve this with minimal human labelling effort, we first automatically sample a small set of images that are likely to falsify the target model from a large corpus of web images via the maximum discrepancy competition principle. We then propose a weakly labelling strategy to further reduce the number of false positives, before time-consuming pixel-level labelling by humans. Finally, we fine-tune the model to harness the identified failures, and repeat the whole process, resulting in an efficient and progressive framework for troubleshooting segmentation models. We demonstrate the feasibility of our framework using the semantic segmentation task in PASCAL VOC, and find that the fine-tuned model exhibits significantly improved generalization when applied to real-world images with greater content diversity. The code is available at https://github.com/VITA-Group/Troubleshooting_Image_Segmentation.

NFANet: A Novel Method for Weakly Supervised Water Extraction From High-Resolution Remote-Sensing Imagery

The use of deep learning for water extraction requires precise pixel-level labels. However, it is very difficult to label high-resolution remote-sensing images at the pixel level. Therefore, we study how to utilize point labels to extract water bodies and propose a novel method called the neighbor feature aggregation network (NFANet). Compared with pixel-level labels, point labels are much easier to obtain, but they will lose much information. In this article, we take advantage of the similarity between the adjacent pixels of a local water body, and propose a neighbor sampler to resample remote-sensing images. Then, the sampled images are sent to the network for feature aggregation. In addition, we use an improved recursive training algorithm to further improve the extraction accuracy, making the water boundary more natural. Furthermore, our method utilizes neighboring features instead of global or local features to learn more representative features. The experimental results show that the proposed NFANet method not only outperforms other studied weakly supervised approaches, but also obtains similar results as the state-of-the-art ones.

Learning From Pixel-Level Label Noise: A New Perspective for Semi-Supervised Semantic Segmentation

This paper addresses semi-supervised semantic segmentation by exploiting a small set of images with pixel-level annotations (strong supervisions) and a large set of images with only image-level annotations (weak supervisions). Most existing approaches aim to generate accurate pixel-level labels from weak supervisions. However, we observe that those generated labels still inevitably contain noisy labels. Motivated by this observation, we present a novel perspective and formulate this task as a problem of learning with pixel-level label noise. Existing noisy label methods, nevertheless, mainly aim at image-level tasks, which can not capture the relationship between neighboring labels in one image. Therefore, we propose a graph based label noise detection and correction framework to deal with pixel-level noisy labels. In particular, for the generated pixel-level noisy labels from weak supervisions by Class Activation Map (CAM), we train a clean segmentation model with strong supervisions to detect the clean labels from these noisy labels according to the cross-entropy loss. Then, we adopt a superpixel-based graph to represent the relations of spatial adjacency and semantic similarity between pixels in one image. Finally we correct the noisy labels using a Graph Attention Network (GAT) supervised by detected clean labels. We comprehensively conduct experiments on PASCAL VOC 2012, PASCAL-Context and MS-COCO datasets. The experimental results show that our proposed semi supervised method achieves the state-of-the-art performances and even outperforms the fully-supervised models on PASCAL VOC 2012 and MS-COCO datasets in some cases.

Guided Filter Network for Semantic Image Segmentation.

The existing publicly available datasets with pixel-level labels contain limited categories, and it is difficult to generalize to the real world containing thousands of categories. In this paper, we propose an approach to generate object masks with detailed pixel-level structures/boundaries automatically to enable semantic image segmentation of thousands of targets in the real world without manually labelling. A Guided Filter Network (GFN) is first developed to learn the segmentation knowledge from an existed dataset, and such GFN then transfers the learned segmentation knowledge to generate initial coarse object masks for the target images. These coarse object masks are treated as pseudo labels to self-optimize the GFN iteratively in the target images. Our experiments on six image sets have demonstrated that our proposed approach can generate object masks with detailed pixel-level structures/boundaries, whose quality is comparable to the manually-labelled ones. Our proposed approach also achieves better performance on semantic image segmentation than most existing weakly-supervised, semi-supervised, and domain adaptation approaches under the same experimental conditions.

Deep Bilateral Filtering Network for Point-Supervised Semantic Segmentation in Remote Sensing Images.

Semantic segmentation methods based on deep neural networks have achieved great success in recent years. However, training such deep neural networks relies heavily on a large number of images with accurate pixel-level labels, which requires a huge amount of human effort, especially for large-scale remote sensing images. In this paper, we propose a point-based weakly supervised learning framework called the deep bilateral filtering network (DBFNet) for the semantic segmentation of remote sensing images. Compared with pixel-level labels, point annotations are usually sparse and cannot reveal the complete structure of the objects; they also lack boundary information, thus resulting in incomplete prediction within the object and the loss of object boundaries. To address these problems, we incorporate the bilateral filtering technique into deeply learned representations in two respects. First, since a target object contains smooth regions that always belong to the same category, we perform deep bilateral filtering (DBF) to filter the deep features by a nonlinear combination of nearby feature values, which encourages the nearby and similar features to become closer, thus achieving a consistent prediction in the smooth region. In addition, the DBF can distinguish the boundary by enlarging the distance between the features on different sides of the edge, thus preserving the boundary information well. Experimental results on two widely used datasets, the ISPRS 2-D semantic labeling Potsdam and Vaihingen datasets, demonstrate that our proposed DBFNet can achieve a highly competitive performance compared with state-of-the-art fully-supervised methods. Code is available at https://github.com/Luffy03/DBFNet.

Cross-Modality Contrastive Learning for Hyperspectral Image Classification

Deep learning has attracted much attention in the field of hyperspectral image classification recently, due to its powerful representation and generalization abilities. Most of current deep learning models are trained in a supervised manner, which require large amounts of labeled samples to achieve state-of-the-art performance. Unfortunately, pixel-level labeling in hyperspectral imageries is difficult, time-consuming, and human-dependent. To address this issue, we propose an unsupervised feature learning model using multi-modal data, hyperspectral and LiDAR in particular. It takes advantage of the relationship between hyperspectral and LiDAR data to extract features, without using any label information. After that, we design a dual fine-tuning strategy to transfer the extracted features for hyperspectral image classification with small numbers of training samples. Such strategy is able to explore not only the semantic information but also the intrinsic structure information of training samples. In order to test the performance of our proposed model, we conduct comprehensive experiments on three hyperspectral and LiDAR datasets. Experimental results show that our proposed model can achieve better performance than several state-of-the-art deep learning models.

Learning-Free Hyperspectral Anomaly Detection With Unpredictive Frequency Residual Priors

Hyperspectral anomaly detection aims to fast and credibly find nontrivial candidate targets without prior knowledge, which has become an increasingly pressing need as imagery swath and resolution are growing rapidly. Relevant state-of-the-art learning-based anomaly detection approaches have benefited from data-driven hierarchical feature embeddings that typically model the geometric distribution of spectral vectors. However, most of these techniques are incompatible with resource-constrained applications: 1) huge computational costs caused by feedforward convolution operation cannot be supported with limited computation resources and storage (e.g., in-orbit processing); 2) detection accuracy relies on large-scale training datasets, which results in labor-expensive imagery collection, pixel-level labeling, and time-consuming learning procedure. To address these issues, we advocate a learning-free, frequency domain anomaly detection method combined with predictive coding, an intriguing heuristic prior from human vision system. Technically, 1) inherently efficient frequency transformation could be implemented with existing image compression modules (e.g., JPEG or JPEG2000 codec), which improves the utilization of computational resources; 2) predictive coding mechanism is exploited for suppressing frequently occurring information represented in the low-entropy frequency domain, such that the “unpredictive” subject (i.e., anomaly spectrums) can “pop out” with naive residuals. Such heuristic prior incorporated into the computational model can reduce dependence on the large-scale training set. Experiments on real-world hyperspectral datasets confirm the efficacy of our model. Besides, low computation cost of the proposed (fast frequency transformation and analytical solutions) anomaly detector facilitates rather straightforward sliding window verification in high-resolution imagery.

Patch-Based Training of Fully Convolutional Network for Hyperspectral Image Classification With Sparse Point Labels

Fully convolutional network (FCN), which has excellent capability for capturing spatial context, was introduced to improve the performance of hyperspectral image classification (HSIC). However, training FCN usually requires a huge amount of pixel-level labels, which is difficult to obtain for HSIC in practical applications. How to train an FCN effectively with the supervision of limited sparse point labels has attracted the attention. The patch-free training pattern with sparse point labels was proved to be effective for HSIC task. Then, as a general training mode for remote sensing image semantic segmentation, is patch-based training of FCN also effective for HSIC with sparse point labels? To answer this question, a patch-based training framework with a novel fully convolutional network is proposed for HSIC in this study. First, cropped hyperspectral image (HSI) patches with sparse labels are input for training. Second, considering the limitation of supervision with sparse points for training, a lightweight network on the basis of an encoder–decoder structure with shallow channels is specially designed for HSIC with the aid of residual connections in the encoder and the integration of multiple attention modules to fully exploit the spectral-spatial information of HSI. Third, conditional random field-loss is adopted as a prior complement to the point supervision for further excavation of spatial context information. The performance of the proposed method is quantitatively evaluated on three HSI datasets and achieves state-of-the-art performance in comparison with other representative methods, demonstrating the effectiveness of the patch-based training framework for HSIC.

Discriminative Multiple-Instance Hyperspectral Subpixel Target Characterization

Subpixel target detection in hyperspectral imagery is challenging since subpixel targets are smaller in size than the resolution of a single pixel and accurate pixel-level labels on subpixel targets are often unavailable. In particular, this article addresses the problem of learning a prime prototype target signature from imprecisely labeled highly mixed hyperspectral data. Two algorithms, multiple-instance subpixel adaptive cosine estimator (MI-SPACE) and multiple-instance subpixel spectral matched filter (MI-SPSMF), based on multiple-instance learning framework are presented. The proposed methods aim to learn a discriminative prime target signature by maximizing the posterior detection statistics of subpixel hyperspectral targets for the correspondingly proposed subpixel adaptive cosine estimator (SPACE) and subpixel spectral matched filter (SPSMF) detectors, which are also developed in this article. Experimental results demonstrate the effectiveness of the proposed methods on both simulated and real-field hyperspectral subpixel target detection tasks.

Sea-Ice Mapping of RADARSAT-2 Imagery by Integrating Spatial Contexture With Textural Features

Mapping different types of sea ice that form, grow, and melt in polar oceans is essential for shipping navigation, climate change modeling, and local community safety. Currently, ice charts are manually generated by analysts at the Canadian Ice Service (CIS) based on dual-polarized RADARSAT-2/RADARSAT Constellation Mission (RCM) imagery on a daily basis. Inspired by the demand for a computer-based mapping system, we have developed an automatic sea ice classification method that integrates spatial contexture (unsupervised segmentation) with textural features (supervised pixel-level labeling). First, the full-scene image is oversegmented, and the segments are merged into homogeneous regions across the entire scene. Second, pixel-based classifiers (support vector machine, random forest) are compared for their ability to label the generated homogeneous regions. Finally, the segmentation and labeling are combined using a proposed energy function. The proposed method was tested on 18 dual-polarization RADARSAT-2 scenes acquired over the Beaufort Sea. This dataset contains water, young ice, first-year ice, and multi-year ice covering melt, summer, and freeze-up seasons. The proposed method obtains an average classification accuracy of 86.33% based on the leave-one-out validation. The experimental results show that the proposed method achieves promising classification results in both quantity and quality measurements compared to benchmark methods. The robustness against incidence angle variance indicates that the proposed method is well-qualified for operational sea ice mapping.

A Semisupervised Convolution Neural Network for Partial Unlabeled Remote-Sensing Image Segmentation

Semantic segmentation methods for remote-sensing images based on the deep learning framework have achieved significant performance improvements. However, most of the existing work is based on fully supervised methods, which rely on a large number of manually annotated pixel-level labels. However, for remote-sensing images, labeling the ground-truth takes time and effort. To solve the problem in existing methods of overly relying on manual labeling, in this study, we propose a semisupervised convolution neural network based on contrastive loss for partial unlabeled remote-sensing image segmentation. In the design of the contrastive loss function, to capture the semantic relationship of pixels and improve the separability between different categories, we propose pixel-level and region-level contrastive loss. The pixel-level contrastive loss is designed to learn the correlation between different images, while region-level contrastive loss is designed to improve the quality of generated pseudo-labels. In addition, we designed a propagated self-training method that further guarantees the quality of the pseudo-labels and improves the richness of the labeled data. Experiments on POTSDAM and Vaihingen datasets demonstrate that the proposed method achieves the highest Mean Intersection over Union (mIOU) and significantly outperforms previous methods.

Multi-view parallel vertebra segmentation and identification on computed tomography (CT) images

Vertebra segmentation and identification is the crucial step for automatic spine analysis. Manual or semi-automatic segmentation and identification is a cumbersome approach used conventionally. This paper proposes an automatic method for accurate pixel-level labeling of vertebrae on CT images. The algorithm consists of two main steps: in the first step, a pixel-link convolutional neural network is trained to generate a binary mask for the vertebral column; and in the second step, a multi-label dilated residual network identifies the labels for each vertebra. The proposed model is evaluated on the VerSe-dataset which contains 374 CT scans. This includes scans with a variety of field-of-views and healthy/disease cases acquired from multiple scanners. The model is trained and evaluated on 2D coronal and sagittal slices extracted from the CT volume. Average dice scores of 0.89 and 0.90 were achieved on two test sets released as public and hidden test sets for VerSe-dataset. The mean pixel accuracy of the predicted segmentation maps for vertebra regions are 0.72–0.86 and 0.68–0.85 for test set 1 and test 2, respectively.

End-to-end multi-task learning for simultaneous optic disc and cup segmentation and glaucoma classification in eye fundus images

The automated analysis of eye fundus images is crucial towards facilitating the screening and early diagnosis of glaucoma. Nowadays, there are two common alternatives for the diagnosis of this disease using deep neural networks. One is the segmentation of the optic disc and cup followed by the morphological analysis of these structures. The other is to directly address the diagnosis as an image classification task. The segmentation approach presents the advantage of using pixel-level labels with precise morphological information for training. However, while this detailed training feedback is not available for the classification approach, in this case the image-level labels may allow the discovery of additional non-morphological cues that are also relevant for the diagnosis.In this work, we propose a novel multi-task approach for the simultaneous classification of glaucoma and segmentation of the optic disc and cup. This approach can improve the overall performance by taking advantage of both pixel-level and image-level labels during the network training. Additionally, the segmentation maps that are predicted together with the diagnosis allow the extraction of relevant biomarkers such as the cup-to-disc ratio. The proposed methodology presents two relevant technical novelties. First, a network architecture for simultaneous segmentation and classification that increases the number of shared parameters between both tasks. Second, a multi-adaptive optimization strategy that ensures that both tasks contribute similarly to the parameter updates during training, avoiding the use of loss weighting hyperparameters.To validate our proposal, an exhaustive experimentation was performed on the public REFUGE and DRISHTI-GS datasets. The results show that our proposal outperforms comparable multi-task baselines and is highly competitive with existing state-of-the-art approaches. Additionally, the provided ablation study shows that both the network architecture and the optimization approach are independently advantageous for multi-task learning.

Lesion Segmentation Framework Based on Convolutional Neural Networks with Dual Attention Mechanism

Computational intelligence has been widely used in medical information processing. The deep learning methods, especially, have many successful applications in medical image analysis. In this paper, we proposed an end-to-end medical lesion segmentation framework based on convolutional neural networks with a dual attention mechanism, which integrates both fully and weakly supervised segmentation. The weakly supervised segmentation module achieves accurate lesion segmentation by using bounding-box labels of lesion areas, which solves the problem of the high cost of pixel-level labels with lesions in the medical images. In addition, a dual attention mechanism is introduced to enhance the network’s ability for visual feature learning. The dual attention mechanism (channel and spatial attention) can help the network pay attention to feature extraction from important regions. Compared with the current mainstream method of weakly supervised segmentation using pseudo labels, it can greatly reduce the gaps between ground-truth labels and pseudo labels. The final experimental results show that our proposed framework achieved more competitive performances on oral lesion dataset, and our framework further extended to dermatological lesion segmentation.

A Chan-Vese model based on the Markov chain for unsupervised medical image segmentation

The accurate segmentation of medical images is crucial to medical care and research; however, many efficient supervised image segmentation methods require sufficient pixel level labels. Such requirement is difficult to meet in practice and even impossible in some cases, e.g., rare Pathoma images. Inspired by traditional unsupervised methods, we propose a novel Chan-Vese model based on the Markov chain for unsupervised medical image segmentation. It combines local information brought by superpixels with the global difference between the target tissue and the background. Based on the Chan-Vese model, we utilize weight maps generated by the Markov chain to model and solve the segmentation problem iteratively using the min-cut algorithm at the superpixel level. Our method exploits abundant boundary and local region information in segmentation and thus can handle images with intensity inhomogeneity and object sparsity. In our method, users gain the power of fine-tuning parameters to achieve satisfactory results for each segmentation. By contrast, the result from deep learning based methods is rigid. The performance of our method is assessed by using four Computerized Tomography (CT) datasets. Experimental results show that the proposed method outperforms traditional unsupervised segmentation techniques.