Semi-Supervised Semantic Segmentation Research Articles

Efficient and accurate semi-supervised semantic segmentation for industrial surface defects

Deep learning-based defect detection methods have gained widespread application in industrial quality inspection. However, limitations such as insufficient sample sizes, low data utilization, and issues with accuracy and speed persist. This paper proposes a semi-supervised semantic segmentation framework that addresses these challenges through perturbation invariance at both the image and feature space. The framework employs diverse perturbation cross-pseudo-supervision to reduce dependency on extensive labeled datasets. Our lightweight method incorporates edge pixel-level semantic information and shallow feature fusion to enhance real-time performance and improve the accuracy of defect edge detection and small target segmentation in industrial inspection. Experimental results demonstrate that the proposed method outperforms the current state-of-the-art (SOTA) semi-supervised semantic segmentation methods across various industrial scenarios. Specifically, our method achieves a mean Intersection over Union (mIoU) 3.11% higher than the SOTA method on our dataset and 4.39% higher on the public KolektorSDD dataset. Additionally, our semantic segmentation network matches the speed of the fastest network, U-net, while achieving a mIoU 2.99% higher than DeepLabv3Plus.

Structural tensor and frequency guided semi-supervised segmentation for medical images.

The method of semi-supervised semantic segmentation entails training with a limited number of labeled samples alongside many unlabeled samples, aiming to reduce dependence on pixel-level annotations. Most semi-supervised semantic segmentation methods primarily focus on sample augmentation in spatial dimensions to reduce the shortage of labeled samples. These methods tend to ignore the structural information of objects. In addition, frequency-domain information also supplies another perspective to evaluate information from images, which includes different properties compared to the spatialdomain. In this study, we attempt to answer these two questions: (1) is it helpful to provide structural information of objects in semi-supervised semantic segmentation tasks for medical images? (2) is it more effective to evaluate the segmentation performance in the frequency domain compared to the spatial domain for semi-supervised medical image segmentation? Therefore, we seek to introduce structural and frequency information to improve the performance of semi-supervised semantic segmentation for medicalimages. We present a novel structural tensor loss (STL) to guide feature learning on the spatial domain for semi-supervised semantic segmentation. Specifically, STL utilizes the structural information encoded in the tensors to enforce the consistency of objects across spatial regions, thereby promoting more robust and accurate feature extraction. Additionally, we proposed a frequency-domain alignment loss (FAL) to enable the neural networks to learn frequency-domain information across different augmented samples. It leverages the inherent patterns present in frequency-domain representations to guide the network in capturing and aligning features across diverse augmentation variations, thereby enhancing the model's robustness for the inputting variations. We conduct our experiments on three benchmark datasets, which include MRI (ACDC) for cardiac, CT (Synapse) for abdomen organs, and ultrasound image (BUSI) for breast lesion segmentation. The experimental results demonstrate that our method outperforms state-of-the-art semi-supervised approaches regarding the Dice similaritycoefficient. We find the proposed approach could improve the final performance of the semi-supervised medical image segmentation task. It will help reduce the need for medical image labels. Our code will are available at https://github.com/apple1986/STLFAL.

Enhancing the utilization of uncertain pixels in semi-supervised semantic segmentation

In semi-supervised semantic segmentation, determining the correct label for uncertain pixels is crucial yet challenging. The recently proposed virtual category (VC) learning achieves excellent performance by creating a potential category (PC) set to exploit the valuable part of the uncertain pixels effectively. However, the potential category set in the existing method is not accurate enough at the beginning of iterations, which could lead to less accurate segmentation results. To better utilize uncertain pixels, we propose to improve the strategy of constructing the potential category set. Specifically, we adjust the threshold based on the model’s optimization status and split the pseudo-labels into low/high-confidence areas to create a more appropriate PC set. At the same time, due to uncertainty in boundary pixels, it is challenging to achieve precise object segmentation, and there is currently no method available to optimize these areas in semi-supervised segmentation tasks. Therefore, we propose a entropy-based method to identify boundary areas and design a novel boundary refinement network to process labeled and unlabeled data separately to optimize segmentation. The experimental results demonstrate that our proposed method excels in accurately segmenting the boundary areas of the targets. Furthermore, it achieves state-of-the-art semi-supervised segmentation on the Pascal VOC and Cityscapes datasets.

Category-sensitive semi-supervised semantic segmentation framework for land-use/land-cover mapping with optical remote sensing images

High-quality land-use/land-cover mapping with optical remote sensing images yet presents significant work. Even though fully convolutional semantic segmentation models have recently contributed to popular solutions, the lack of annotation data may lead to severe degradations in their inference performance. Besides, the category confusion in high-resolution representations will further exacerbate the adverse effects. In this paper, we propose a category-sensitive semi-supervised semantic segmentation framework to address these weaknesses by employing massive unlabeled data. With the perturbations from adopted hybrid data augmentation structures, we first focus on the output space and execute regularization constraints to learn category-specific discriminative features. It is formulated with a consistency self-training procedure where a dynamic class-balanced threshold selection scheme is proposed to provide high-confident pseudo supervisions for each category. In addition, we introduce pixel-wise contrastive learning on the common embedding space from both labeled and unlabeled data domains to further facilitate the semantic dependencies among category features, in which the reliable labels are leveraged as guidance for pixel sample selection. We verify the proposed framework on two benchmark land-use/land-cover datasets, and the experimental results demonstrate its competitive performance to other state-of-the-art semi-supervised methods.

A method for seismic fault identification based on self-training with high-stability pseudo-labels

The imaging principle of seismic image is different from that of natural image. There are many problems on seismic images, such as limited resolution, complex reflection feature and strong uncertainty, which leads to significant difference in the research emphases of computer vision on seismic image. In response to the above issues, the semi-supervised semantic segmentation method based on self-training with high-stability pseudo-labels is proposed for seismic fault identification. Specifically, the self-training strategy Fault-Seg-ST is proposed to decouple Teacher–Student model predictions by adding random noise to unlabeled images. The selective retraining strategy Fault-Seg-SST is proposed to evaluate the reliability of pseudo-labels based on the stability of model predictions. The contrastive loss function is proposed to learn pixel-level feature representation with intra class affinity and inter class separability. The Fault-Seg-SST has achieved the optimal segmentation performance on both DeepLabV2 (Dice 86%, mIoU 77%) and UNet with ResNet 18 (Dice 83%, mIoU 76%). Experimental results show that the fault identification method based on self-training with high-stability pseudo-labels demonstrates the superiority of selective retraining strategy, and the fault identification model trained with Fault-Seg-SST strategy can achieve the fine-grained fault segmentation.

Multi-Dimensional Manifolds Consistency Regularization for semi-supervised remote sensing semantic segmentation

Semi-supervised semantic segmentation in remote sensing is critical for urban planning, environmental monitoring and disaster response. The high cost and time required for high-quality data annotation limits its wider application. Traditional semi-supervised deep learning methods, which operate in a single dimension, limit model robustness and generalization. Our study addresses this issue by proposing an effective semi-supervised learning method. This method improves model robustness and generalization in remote sensing semantic segmentation. We introduce the Multi-Dimensional Manifolds Consistency Regularization (MDMCR) approach. It applies multi-dimensional perturbations to input images and features, expanding the sample library and improving learning efficiency. Our method has been rigorously tested on various datasets. With only 1/8 of the data labeled, it achieved mean Intersection over Union (mIoU) scores of 74.48% on ISPRS Vaihingen and 78.80% on Potsdam. With only 5% labeled data, it reached 49.93% mIoU on DeepGlobe Roads and 57.90% on Massachusetts Roads. These results show the superiority of our method over existing techniques.

PRCL: Probabilistic Representation Contrastive Learning for Semi-Supervised Semantic Segmentation

PRCL: Probabilistic Representation Contrastive Learning for Semi-Supervised Semantic Segmentation

Decouple and weight semi-supervised semantic segmentation of remote sensing images

Semantic understanding of high-resolution remote sensing (RS) images is of great value in Earth observation, however, it heavily depends on numerous pixel-wise manually-labeled data, which is laborious and thereby limits its practical application. Semi-supervised semantic segmentation (SSS) of RS images would be a promising solution, which uses both limited labeled data and dominant unlabeled data to train segmentation models, significantly mitigating the annotation burden. The current mainstream methods of remote sensing semi-supervised semantic segmentation (RS-SSS) utilize the hard or soft pseudo-labels of unlabeled data for model training and achieve excellent performance. Nevertheless, their performance is bottlenecked because of two inherent problems: irreversible wrong pseudo-labels and long-tailed distribution among classes. Aiming at them, we propose a decoupled weighting learning (DWL) framework for RS-SSS in this study, which consists of two novel modules, decoupled learning and ranking weighting, corresponding to the above two problems, respectively. During training, the decoupled learning module separates the predictions of the labeled and unlabeled data to decrease the negative impact of the self-training of the wrongly pseudo-labeled unlabeled data on the supervised training of the labeled data. Furthermore, the ranking weighting module tries to adaptively weight each pseudo-label of the unlabeled data according to its relative confidence ranking in its pseudo-class to alleviate model bias to majority classes as a result of the long-tailed distribution. To verify the effectiveness of the proposed DWL framework, extensive experiments are conducted on three widely-used RS semantic segmentation datasets in the semi-supervised setting. The experimental results demonstrate the superiority of our method to some state-of-the-art SSS methods. Our code will be available at https://github.com/zhu-xlab/RS-DWL.

FishSegSSL: A Semi-Supervised Semantic Segmentation Framework for Fish-Eye Images.

The application of large field-of-view (FoV) cameras equipped with fish-eye lenses brings notable advantages to various real-world computer vision applications, including autonomous driving. While deep learning has proven successful in conventional computer vision applications using regular perspective images, its potential in fish-eye camera contexts remains largely unexplored due to limited datasets for fully supervised learning. Semi-supervised learning comes as a potential solution to manage this challenge. In this study, we explore and benchmark two popular semi-supervised methods from the perspective image domain for fish-eye image segmentation. We further introduce FishSegSSL, a novel fish-eye image segmentation framework featuring three semi-supervised components: pseudo-label filtering, dynamic confidence thresholding, and robust strong augmentation. Evaluation on the WoodScape dataset, collected from vehicle-mounted fish-eye cameras, demonstrates that our proposed method enhances the model's performance by up to 10.49% over fully supervised methods using the same amount of labeled data. Our method also improves the existing image segmentation methods by 2.34%. To the best of our knowledge, this is the first work on semi-supervised semantic segmentation on fish-eye images. Additionally, we conduct a comprehensive ablation study and sensitivity analysis to showcase the efficacy of each proposed method in this research.

Anti-Adversarially Manipulated Attributions for Weakly Supervised Semantic Segmentation and Object Localization.

Obtaining accurate pixel-level localization from class labels is a crucial process in weakly supervised semantic segmentation and object localization. Attribution maps from a trained classifier are widely used to provide pixel-level localization, but their focus tends to be restricted to a small discriminative region of the target object. An AdvCAM is an attribution map of an image that is manipulated to increase the classification score produced by a classifier before the final softmax or sigmoid layer. This manipulation is realized in an anti-adversarial manner, so that the original image is perturbed along pixel gradients in directions opposite to those used in an adversarial attack. This process enhances non-discriminative yet class-relevant features, which make an insufficient contribution to previous attribution maps, so that the resulting AdvCAM identifies more regions of the target object. In addition, we introduce a new regularization procedure that inhibits the incorrect attribution of regions unrelated to the target object and the excessive concentration of attributions on a small region of the target object. Our method achieves a new state-of-the-art performance in weakly and semi-supervised semantic segmentation, on both the PASCAL VOC 2012 and MS COCO 2014 datasets. In weakly supervised object localization, it achieves a new state-of-the-art performance on the CUB-200-2011 and ImageNet-1K datasets.

Deep learning for head and neck semi-supervised semantic segmentation

Objective. Radiation therapy (RT) represents a prevalent therapeutic modality for head and neck (H&N) cancer. A crucial phase in RT planning involves the precise delineation of organs-at-risks (OARs), employing computed tomography (CT) scans. Nevertheless, the manual delineation of OARs is a labor-intensive process, necessitating individual scrutiny of each CT image slice, not to mention that a standard CT scan comprises hundreds of such slices. Furthermore, there is a significant domain shift between different institutions’ H&N data, which makes traditional semi-supervised learning strategies susceptible to confirmation bias. Therefore, effectively using unlabeled datasets to support annotated datasets for model training has become a critical issue for preventing domain shift and confirmation bias. Approach. In this work, we proposed an innovative cross-domain orthogon-based-perspective consistency (CD-OPC) strategy within a two-branch collaborative training framework, which compels the two sub-networks to acquire valuable features from unrelated perspectives. More specifically, a novel generative pretext task cross-domain prediction (CDP) was designed for learning inherent properties of CT images. Then this prior knowledge was utilized to promote the independent learning of distinct features by the two sub-networks from identical inputs, thereby enhancing the perceptual capabilities of the sub-networks through orthogon-based pseudo-labeling knowledge transfer. Main results. Our CD-OPC model was trained on H&N datasets from nine different institutions, and validated on the four local intuitions’ H&N datasets. Among all datasets CD-OPC achieved more advanced performance than other semi-supervised semantic segmentation algorithms. Significance. The CD-OPC method successfully mitigates domain shift and prevents network collapse. In addition, it enhances the network’s perceptual abilities, and generates more reliable predictions, thereby further addressing the confirmation bias issue.

Subdivided Mask Dispersion Framework for semi-supervised semantic segmentation

Learning the relationship between weak and strong perturbations has been considered a major part of semi-supervised semantic segmentation. We observed two problems with a publicly used perturbation method, which randomly generates a mask with a single large bounding box. The large single bounding box that entirely covers the important object components in an image, hindering the model from capturing partial object information. Furthermore, training the model with a single large bounding box as an image-level perturbation causes the model to be biased towards the shape of the large squared box, rather than the deformable object component shapes. In this paper, we propose Subdivided Mask Dispersion Framework (SMDF) to solve these problems. Our framework disperses the large squared box into small multi-scale boxes, capturing the crucial multi-scaled object components in the image. SMDF achieves state-of-the-art performance on five data partitions of PASCAL dataset and three partitions of Extended SBD dataset. Our extensive ablation studies show the effectiveness of dispersed small multi-scale bounding boxes in semi-supervised semantic segmentation.

Information Transfer in Semi-Supervised Semantic Segmentation

Information Transfer in Semi-Supervised Semantic Segmentation

NUNI-Waste: novel semi-supervised semantic segmentation waste classification with non-uniform data augmentation

Waste categorization and recycling are critical approaches for converting waste into valuable and functional materials, thereby significantly aiding in land preservation, reducing pollution, and optimizing resource usages. However, real-world classification and identification of recyclable waste face substantial hurdles due to the intricate and unpredictable nature of wastes, as well as the limited availability of comprehensive waste datasets. These factors limit efficacy of the existing research work in the domain of waste management. In this paper, we utilize semantic segmentation at individual pixel level and introduce a semi-supervised metod for authentic waste classification scenarios, leveraging the Zerowaste dataset. We devise a non-standard data augmentation strategy that mimics the ever-changing conditions of real-world waste environments. Additionally, we introduce an adaptive weighted loss function and dynamically adjust the ratio of positive to negative samples through a masking method, ensuring the model learns from relevant samples. Lastly, to maintain consistency between predictions made on data-augmented images and the original counterparts, we remove input perturbations. Our method proves to be effective, as verified by an array of standard experiments and ablation studies, achieved an accuracy improvement of 3.74% over the baseline Zerowaste method.

Enhancing Semi-Supervised Semantic Segmentation of Remote Sensing Images via Feature Perturbation-Based Consistency Regularization Methods

In the field of remote sensing technology, the semantic segmentation of remote sensing images carries substantial importance. The creation of high-quality models for this task calls for an extensive collection of image data. However, the manual annotation of these images can be both time-consuming and labor-intensive. This has catalyzed the advent of semi-supervised semantic segmentation methodologies. Yet, the complexities inherent within the foreground categories of these remote sensing images present challenges in preserving prediction consistency. Moreover, remote sensing images possess more complex features, and different categories are confused within the feature space, making optimization based on the feature space challenging. To enhance model consistency and to optimize feature-based class categorization, this paper introduces a novel semi-supervised semantic segmentation framework based on Mean Teacher (MT). Unlike the conventional Mean Teacher that only introduces perturbations at the image level, we incorporate perturbations at the feature level. Simultaneously, to maintain consistency after feature perturbation, we employ contrastive learning for feature-level learning. In response to the complex feature space of remote sensing images, we utilize entropy threshold to assist contrastive learning, selecting feature key-values more precisely, thereby enhancing the accuracy of segmentation. Extensive experimental results on the ISPRS Potsdam dataset and the challenging iSAID dataset substantiate the superior performance of our proposed methodology.

Progressive Learning with Cross-Window Consistency for Semi-Supervised Semantic Segmentation.

Semi-supervised semantic segmentation focuses on the exploration of a small amount of labeled data and a large amount of unlabeled data, which is more in line with the demands of real-world image understanding applications. However, it is still hindered by the inability to fully and effectively leverage unlabeled images. In this paper, we reveal that cross-window consistency (CWC) is helpful in comprehensively extracting auxiliary supervision from unlabeled data. Additionally, we propose a novel CWC-driven progressive learning framework to optimize the deep network by mining weak-to-strong constraints from massive unlabeled data. More specifically, this paper presents a biased cross-window consistency (BCC) loss with an importance factor, which helps the deep network explicitly constrain confidence maps from overlapping regions in different windows to maintain semantic consistency with larger contexts. In addition, we propose a dynamic pseudo-label memory bank (DPM) to provide high-consistency and high-reliability pseudo-labels to further optimize the network. Extensive experiments on three representative datasets of urban views, medical scenarios, and satellite scenes with consistent performance gain demonstrate the superiority of our framework. Our code is released at https://jack-bo1220.github.io/project/CWC.html.

Class-Wise Adaptive Strategy for Semi Supervised Semantic Segmentation

Class-Wise Adaptive Strategy for Semi Supervised Semantic Segmentation

Innovative Research on Illustration Design Integrating Color Science and Image Processing Technology

Abstract How to create intelligently and efficiently has become a hot research topic in the field of illustration design. In this paper, starting from the acceptance intention to satisfy users’ favorites and needs, we propose an intelligent analysis method of style based on a deep clustering model and use the categorized style attributes as a reference to disperse designers’ creative thinking. In addition, a semi-supervised semantic segmentation method is introduced to construct an automatic coloring model for sketches, and the automatic coloring function is realized by incorporating the color label recognition process, which further improves the efficiency of designers’ work. The results of clustering analysis of the STL dataset show that the ACC (78.39%), NMI (67.78%), and ARI (62.93%) metrics of the VIT+K-Means model have the largest results compared with other feature extraction methods. Results are superior to other feature extraction methods. Not only that, adding a pseudo-label retraining process for this model further improves the results of the three metrics by 9.13%, 9.97%, and 10.78%, and the visual analysis experiments of clustering clusters also verified the performance enhancement effect. In the comparative analysis of illustration coloring models, the AdvSSL scheme with semi-supervised strategy achieves the best performance in the semantic segmentation task, with an improvement of 4.31% and 2.29% over the SSAN scheme and the S4GCN scheme, respectively. The experts’ coloring evaluation shows that the improved sketch coloring model has the highest results in all four dimensions compared to the traditional Tag2Pix model.

Enhancing the Semi-Supervised Semantic Segmentation With Prototype-Based Supervision for Remote Sensing Images

Enhancing the Semi-Supervised Semantic Segmentation With Prototype-Based Supervision for Remote Sensing Images

Confidence-Weighted Dual-Teacher Networks With Biased Contrastive Learning for Semi-Supervised Semantic Segmentation in Remote Sensing Images

Confidence-Weighted Dual-Teacher Networks With Biased Contrastive Learning for Semi-Supervised Semantic Segmentation in Remote Sensing Images