Semantic Segmentation Network Research Articles

Unsupervised domain adaptive building semantic segmentation network by edge-enhanced contrastive learning

Unsupervised domain adaptation (UDA) is a weakly supervised learning technique that classifies images in the target domain when the source domain has labeled samples, and the target domain has unlabeled samples. Due to the complexity of imaging conditions and the content of remote sensing images, the use of UDA to accurately extract artificial features such as buildings from high-spatial-resolution (HSR) imagery is still challenging. In this study, we propose a new UDA method for building extraction, the contrastive domain adaptation network (CDANet), by utilizing adversarial learning and contrastive learning techniques. CDANet consists of a single multitask generator and dual discriminators. The generator employs a region and edge dual-branch structure that strengthens its edge extraction ability and is beneficial for the extraction of small and densely distributed buildings. The dual discriminators receive the region and edge prediction outputs and achieve multilevel adversarial learning. During adversarial training processing, CDANet aligns the cross-domain of similar pixel features in the embedding space by constructing the regional pixelwise contrastive loss. A self-training (ST) strategy based on pseudolabel generation is further utilized to address the target intradomain discrepancy. Comprehensive experiments are conducted to validate CDANet on three publicly accessible datasets, namely the WHU, Austin, and Massachusetts. Ablation experiments show that the generator network structure, contrastive loss and ST strategy all improve the building extraction accuracy. Method comparisons validate that CDANet achieves superior performance to several state-of-the-art methods, including AdaptSegNet, AdvEnt, IntraDA, FDANet and ADRS, in terms of F1 score and mIoU.

A Novel Global-Local Feature Aggregation Framework for Semantic Segmentation of Large-Format High-Resolution Remote Sensing Images

In high-resolution remote sensing images, there are areas with weak textures such as large building roofs, which occupy a large number of pixels in the image. These areas pose a challenge for traditional semantic segmentation networks to obtain ideal results. Common strategies like downsampling, patch cropping, and cascade models often sacrifice fine details or global context, resulting in limited accuracy. To address these issues, a novel semantic segmentation framework has been designed specifically for large-format high-resolution remote sensing images by aggregating global and local features in this paper. The framework consists of two branches: one branch deals with low-resolution downsampled images to capture global features, while the other branch focuses on cropped patches to extract high-resolution local details. Also, this paper introduces a feature aggregation module based on the Transformer structure, which effectively aggregates global and local information. Additionally, to save GPU memory usage, a novel three-step training method has been developed. Extensive experiments on two public datasets demonstrate the effectiveness of the proposed approach, with an IoU of 90.83% on the AIDS dataset and 90.30% on the WBDS dataset, surpassing state-of-the-art methods such as DANet, DeepLab v3+, U-Net, ViT, TransUNet, CMTFNet, and UANet.

Rethinking encoder-decoder architecture using vision transformer for colorectal polyp and surgical instruments segmentation

Accurate polyp segmentation from colonoscopy images is important for the immediate diagnosis and effective treatment of colon cancer. While significant progress has been made in the polyps segmentation task, there are various challenges that need to be addressed. Polyps can vary greatly in size and shape, and often has no clear boundary between surrounding tissues. Furthermore, surgical instrument segmentation can aid surgeons with the precise positioning and orientation of the instruments, helping them to plan the next steps in the robot-assisted surgery. The proposed colorectal polyp segmentation Transformer (CPS-Former) uses innovative attention blocks in a network that encodes-decodes features like classic Semantic Segmentation Network (SegNet). However, it has special self-attention modules with small convolutional kernels that efficiently extract information from different feature-channels. Moreover, it is equipped with an effective positional embedding to capture information from a large area of context for long distance interactions. Additionally, a fusion block is embedded for scaling-attention that combines the outputs from the encoder-decoder blocks to enhance the semantic features and reduce the non-semantic ones. Transformer encoder blocks also modified by adding a local feedforward layer and skips connections, and adjust the channel sizes to reduce the model trainable parameters. We evaluate our colorectal polyp segmentation network (CPS-Former) on four colorectal polyp public datasets and one surgical instrument segmentation dataset, which show its superiority over other state-of-the-art polyp segmentation models. Our implementation source code and network weights are available at GitHub: https://github.com/ahmedeqbal/CPS-Former.

FP-Deeplab: a segmentation model for fabric defect detection

Abstract Deep learning for defect detection has become prevalent to improve fabric production and efficiency and quality. However, fabric defect detection faces challenges such as low recognition ratio, suboptimal classification performance, poor detection inefficiency, and high model complexity. To address these issues, an end-to-end semantic segmentation network is proposed employing an efficient encoder-decoder structure, denoted as Feature Pyramid-Deeplab (FP-Deeplab). The improvements involves enhancing the backbone network by improving the MobileNetV3 network for superior performance, a novel Atrous Spatial Pyramid Pooling with Dilated Strip Pooling (ASPP-DSP) module which combines strip pooling, dilated convolution and ASPP, to ensure an expanded receptive field and the capability to gather distant contextual information. Additionally, a Feature Pyramid module (FP module) is proposed to integrate multiscale features at various stages more efficiently. The incorporating of depth-wise separable convolution in FP-Deeplab enables significant parameter and computational cost reduction, catering to online detection requirements. Experimental results certify the superiority of FP-Deeplab over classical and recent segmentation models. Comparative analysis demonstrates higher segmentation accuracy and reduced parameter quantity. Specifically, compared to the benchmark Deeplabv3+ model with MobileV2 as the backbone, FP-Deeplab achieves a notable increase in segmentation accuracy (F1 score and MIoU) by 4.26% and 5.81%, respectively. Moreover, the model parameters (params) are only one-fifth of the original model, indicating the efficiency and effectiveness of our proposed approach.

The Effect of Annotation Quality on Wear Semantic Segmentation by CNN.

In this work, we investigate the impact of annotation quality and domain expertise on the performance of Convolutional Neural Networks (CNNs) for semantic segmentation of wear on titanium nitride (TiN) and titanium carbonitride (TiCN) coated end mills. Using an innovative measurement system and customized CNN architecture, we found that domain expertise significantly affects model performance. Annotator 1 achieved maximum mIoU scores of 0.8153 for abnormal wear and 0.7120 for normal wear on TiN datasets, whereas Annotator 3 with the lowest expertise achieved significantly lower scores. Sensitivity to annotation inconsistencies and model hyperparameters were examined, revealing that models for TiCN datasets showed a higher coefficient of variation (CV) of 16.32% compared to 8.6% for TiN due to the subtle wear characteristics, highlighting the need for optimized annotation policies and high-quality images to improve wear segmentation.

Gated feature aggregate and alignment network for real-time semantic segmentation of street scenes

Gated feature aggregate and alignment network for real-time semantic segmentation of street scenes

Ionograms Trace Extraction Method Based on Multiscale Transformer Network

The echo traces in the ionograms contain key information about the ionosphere. Therefore, the accurate extraction of these traces is crucial for the subsequent work. This paper transforms the original signal processing problem into a semantic segmentation task, combines it with the currently popular deep learning techniques, and proposes a multiscale Transformer network to achieve pixel-level trace extraction. To train the proposed model, we built a dataset by discretizing the original echo data, labeling, and other preprocessing work. A series of advanced semantic segmentation networks are utilized for comparative experiments. The analysis of the results indicates that the proposed network excels in performance, achieving the highest scores on key semantic segmentation evaluation metrics, including mIoU, Kappa, Dice, and AUC-ROC. In addition, this paper also designs a series of ablation experiments to observe the changes in network performance and to evaluate the rationality of the network design. The experimental results demonstrate the effectiveness of the network in the trace extraction task, which plays a positive role in the subsequent electron density reversal work.

NLA-GCL-Net: semantic segmentation of large-scale surveying point clouds based on neighborhood label aggregation (NLA) and global context learning (GCL)

For large-scale 3D point clouds from surveying, the RandLA-Net semantic segmentation network model is unable to learn global context features efficiently during training, leading to suboptimal feature acquisition of globally related features. Furthermore, inconsistent predictions are made in the vicinity of data boundaries due to the oversimplified usage of multi-layer perceptrons (MLPs) and non-linear activation functions following decoding, which has a detrimental impact on both model training and performance. This study provides two solutions to address these issues: a neighborhood label aggregation (NLA) classifier that aggregates neighborhood information for segmentation tasks, and a global context learning (GCL) module that learns global volume-relative information to improve the semantic segmentation network. Overall accuracy (OA) obtained via experimental validation on large-scale SensatUrban and S3DIS datasets is 91.80% and 88.8%, respectively. The proposed model outperforms RandLA-Net by 3.8% and 3.0% in overall mean Intersection over Union (mIOU), respectively, significantly improving model performance and generalization abilities. This work offers new perspectives on how to effectively segment point cloud data.

LiteMSNet: a lightweight semantic segmentation network with multi-scale feature extraction for urban streetscape scenes

LiteMSNet: a lightweight semantic segmentation network with multi-scale feature extraction for urban streetscape scenes

Evaluating single event upsets in deep neural networks for semantic segmentation: An embedded system perspective

As the deployment of artificial intelligence (AI) algorithms at edge devices becomes increasingly prevalent, enhancing the robustness and reliability of autonomous AI-based perception and decision systems is becoming as relevant as precision and performance, especially in applications areas considered safety-critical such as autonomous driving and aerospace. This paper delves into the robustness assessment in embedded Deep Neural Networks (DNNs), particularly focusing on the impact of parameter perturbations produced by single event upsets (SEUs) on convolutional neural networks (CNN) for image semantic segmentation. By scrutinizing the layer-by-layer and bit-by-bit sensitivity of various encoder–decoder models to soft errors, this study thoroughly investigates the vulnerability of segmentation DNNs to SEUs and evaluates the consequences of techniques like model pruning and parameter quantization on the robustness of compressed models aimed at embedded implementations. The findings offer valuable insights into the mechanisms underlying SEU-induced failures that allow for evaluating the robustness of DNNs once trained in advance. Moreover, based on the collected data, we propose a set of practical lightweight error mitigation techniques with no memory or computational cost suitable for resource-constrained deployments. The code used to perform the fault injection (FI) campaign is available at https://github.com/jonGuti13/TensorFI2, while the code to implement proposed techniques is available at https://github.com/jonGuti13/parameterProtection.

DECNet: Dense embedding contrast for unsupervised semantic segmentation

Unsupervised semantic segmentation is important for understanding that each pixel belongs to known categories without annotation. Recent studies have demonstrated promising outcomes by employing a vision transformer backbone pre-trained on an image-level dataset in a self-supervised manner. However, those methods always depend on complex architectures or meticulously designed inputs. Naturally, we are attempting to explore the investment with a straightforward approach. To prevent over-complication, we introduce a simple Dense Embedding Contrast network (DECNet) for unsupervised semantic segmentation in this paper. Specifically, we propose a Nearest Neighbor Similarity strategy (NNS) to establish well-defined positive and negative pairs for dense contrastive learning. Meanwhile, we optimize a contrastive objective named Ortho-InfoNCE to alleviate the false negative problem inherent in contrastive learning for further enhancing dense representations. Finally, extensive experiments conducted on COCO-Stuff and Cityscapes datasets demonstrate that our approach outperforms state-of-the-art methods.

DEDNet: Dual-Encoder DeeplabV3+ Network for Rock Glacier Recognition Based on Multispectral Remote Sensing Image

Understanding the distribution of rock glaciers provides key information for investigating and recognizing the status and changes of the cryosphere environment. Deep learning algorithms and red–green–blue (RGB) bands from high-resolution satellite images have been extensively employed to map rock glaciers. However, the near-infrared (NIR) band offers rich spectral information and sharp edge features that could significantly contribute to semantic segmentation tasks, but it is rarely utilized in constructing rock glacier identification models due to the limitation of three input bands for classical semantic segmentation networks, like DeeplabV3+. In this study, a dual-encoder DeeplabV3+ network (DEDNet) was designed to overcome the flaws of the classical DeeplabV3+ network (CDNet) when identifying rock glaciers using multispectral remote sensing images by extracting spatial and spectral features from RGB and NIR bands, respectively. This network, trained with manually labeled rock glacier samples from the Qilian Mountains, established a model with accuracy, precision, recall, specificity, and mIoU (mean intersection over union) of 0.9131, 0.9130, 0.9270, 0.9195, and 0.8601, respectively. The well-trained model was applied to identify new rock glaciers in a test region, achieving a producer’s accuracy of 93.68% and a user’s accuracy of 94.18%. Furthermore, the model was employed in two study areas in northern Tien Shan (Kazakhstan) and Daxue Shan (Hengduan Shan, China) with high accuracy, which proved that the DEDNet offers an innovative solution to more accurately map rock glaciers on a larger scale due to its robustness across diverse geographic regions.

From image-level to pixel-level labeling: A weakly-supervised learning method for identifying aquaculture ponds using iterative anti-adversarial attacks guided by aquaculture features

Aquaculture mapping is essential for monitoring and managing aquaculture resources. However, accurately geotargeting individual aquaculture ponds from medium-resolution remote sensing imagery remains challenging, and convolutional deep learning methods for identifying aquaculture ponds require labor-intensive pixel-level annotations. This paper presents a novel weakly-supervised learning method to derive pixel-level labels from image-level annotations for aquaculture ponds. Our approach uses iterative anti-adversarial attacks to refine localization results from multi-scale class activation maps (CAMs). The improved method integrates two regularization methods guided by aquaculture features to form a joint loss function for adversarial samples: discriminative water region suppression and non-aquaculture water class suppression. We also propose an aquaculture feature termed CFNDWI to constrain the localization results and generate high-quality pseudo-labels. As a result, the pseudo-labels are used to train semantic segmentation networks for accurately identifying aquaculture ponds. We evaluated the performance of our method using commonly-used backbones on 10 m Sentinel-2 imagery. Our method achieves Intersection over Union (IoU) values of 0.618–0.655 for pseudo-label generation, and IoU values of 0.664–0.708 for semantic segmentation, outperforming state-of-the-art weakly-supervised methods and public datasets. The effectiveness of each module of our method was also testified through ablation experiments. Our method leverages knowledge-driven aquaculture features to guide the data-driven adversarial learning process, addressing the lack of high-quality aquaculture datasets for model training. The code for implementing our method will be accessible at https://github.com/designer1024/WSLM-AQ.

Multi-query and multi-level enhanced network for semantic segmentation

Plain transformer-based methods have achieved promising performance on semantic segmentation recently. These methods adopt a single set of class queries to predict masks of different semantic categories based on multi-level feature maps. We argue that this single-query design cannot fully exploit diverse information of different levels for improved semantic segmentation. To address this issue, we propose a multi-query and multi-level enhanced network for semantic segmentation (named QLSeg). Our QLSeg first performs multi-level feature enhancement on plain transformer to improve feature discriminability. Afterwards, we introduce multi-query decoder to respectively extract feature embeddings and predict mask logits at different levels, where feature embeddings are adaptively merged for classification and mask logits are summed for output masks. In addition, we introduce masked attention-to-mask to focus on local regions with the same class. We perform the experiments on three widely-used semantic segmentation datasets: ADE20K, COCO-Stuff-10K, and PASCAL-Context. Our proposed QLSeg achieves competitive results on all these three datasets.

Monocular Pose Estimation Method for Automatic Citrus Harvesting Using Semantic Segmentation and Rotating Target Detection.

The lack of spatial pose information and the low positioning accuracy of the picking target are the key factors affecting the picking function of citrus-picking robots. In this paper, a new method for automatic citrus fruit harvest is proposed, which uses semantic segmentation and rotating target detection to estimate the pose of a single culture. First, Faster R-CNN is used for grab detection to identify candidate grab frames. At the same time, the semantic segmentation network extracts the contour information of the citrus fruit to be harvested. Then, the capture frame with the highest confidence is selected for each target fruit using the semantic segmentation results, and the rough angle is estimated. The network uses image-processing technology and a camera-imaging model to further segment the mask image of the fruit and its epiphyllous branches and realize the fitting of contour, fruit centroid, and fruit minimum outer rectangular frame and three-dimensional boundary frame. The positional relationship of the citrus fruit to its epiphytic branches was used to estimate the three-dimensional pose of the citrus fruit. The effectiveness of the method was verified through citrus-planting experiments, and then field picking experiments were carried out in the natural environment of orchards. The results showed that the success rate of citrus fruit recognition and positioning was 93.6%, the average attitude estimation angle error was 7.9°, and the success rate of picking was 85.1%. The average picking time is 5.6 s, indicating that the robot can effectively perform intelligent picking operations.

Multi-Resolution Learning and Semantic Edge Enhancement for Super-Resolution Semantic Segmentation of Urban Scene Images.

Super-resolution semantic segmentation (SRSS) is a technique that aims to obtain high-resolution semantic segmentation results based on resolution-reduced input images. SRSS can significantly reduce computational cost and enable efficient, high-resolution semantic segmentation on mobile devices with limited resources. Some of the existing methods require modifications of the original semantic segmentation network structure or add additional and complicated processing modules, which limits the flexibility of actual deployment. Furthermore, the lack of detailed information in the low-resolution input image renders existing methods susceptible to misdetection at the semantic edges. To address the above problems, we propose a simple but effective framework called multi-resolution learning and semantic edge enhancement-based super-resolution semantic segmentation (MS-SRSS) which can be applied to any existing encoder-decoder based semantic segmentation network. Specifically, a multi-resolution learning mechanism (MRL) is proposed that enables the feature encoder of the semantic segmentation network to improve its feature extraction ability. Furthermore, we introduce a semantic edge enhancement loss (SEE) to alleviate the false detection at the semantic edges. We conduct extensive experiments on the three challenging benchmarks, Cityscapes, Pascal Context, and Pascal VOC 2012, to verify the effectiveness of our proposed MS-SRSS method. The experimental results show that, compared with the existing methods, our method can obtain the new state-of-the-art semantic segmentation performance.

Research on the Method of Foreign Object Detection for Railway Tracks Based on Deep Learning.

Addressing the limitations of current railway track foreign object detection techniques, which suffer from inadequate real-time performance and diminished accuracy in detecting small objects, this paper introduces an innovative vision-based perception methodology harnessing the power of deep learning. Central to this approach is the construction of a railway boundary model utilizing a sophisticated track detection method, along with an enhanced UNet semantic segmentation network to achieve autonomous segmentation of diverse track categories. By employing equal interval division and row-by-row traversal, critical track feature points are precisely extracted, and the track linear equation is derived through the least squares method, thus establishing an accurate railway boundary model. We optimized the YOLOv5s detection model in four aspects: incorporating the SE attention mechanism into the Neck network layer to enhance the model's feature extraction capabilities, adding a prediction layer to improve the detection performance for small objects, proposing a linear size scaling method to obtain suitable anchor boxes, and utilizing Inner-IoU to refine the boundary regression loss function, thereby increasing the positioning accuracy of the bounding boxes. We conducted a detection accuracy validation for railway track foreign object intrusion using a self-constructed image dataset. The results indicate that the proposed semantic segmentation model achieved an MIoU of 91.8%, representing a 3.9% improvement over the previous model, effectively segmenting railway tracks. Additionally, the optimized detection model could effectively detect foreign object intrusions on the tracks, reducing missed and false alarms and achieving a 7.4% increase in the mean average precision (IoU = 0.5) compared to the original YOLOv5s model. The model exhibits strong generalization capabilities in scenarios involving small objects. This proposed approach represents an effective exploration of deep learning techniques for railway track foreign object intrusion detection, suitable for use in complex environments to ensure the operational safety of rail lines.

Ultra-fast semantic map perception model for autonomous driving

Autonomous driving relies on real-time perception of environmental semantic maps to make decisions, accurate and real-time perception of environmental semantics is crucial for safe navigation, as well as for efficient construction of high-definition maps in the backend. Our research found that when the output space of the semantic map perception model is an orthogonal projection subspace of the input perception space, it significantly improves the accuracy and real-time performance of map semantic segmentation while reducing learning pressure on deep network models. Based on this, we propose a simple yet efficient semantic map segmentation network that aims to achieve extremely fast speed and precision. Furthermore, we implement a semantic map fusion process using a Bayesian framework that promotes global semantic information accumulation in local semantics over time and space. Our experimental results demonstrate the superiority of our method, achieving state-of-the-art performance in network speed and mIoU metrics. At the same spatial resolution of semantic maps, our method reaches over 350+ frames per second, which is at least 15 times faster than the previous state-of-the-art method. Code released at https://github.com/xzx-ai/MDSegMap.

Semantic segmentation of point clouds of ancient buildings based on weak supervision

Semantic segmentation of point clouds of ancient buildings plays an important role in Historical Building Information Modelling (HBIM). As the annotation task of point cloud of ancient architecture is characterised by strong professionalism and large workload, which greatly restricts the application of point cloud semantic segmentation technology in the field of ancient architecture, therefore, this paper launches a research on the semantic segmentation method of point cloud of ancient architecture based on weak supervision. Aiming at the problem of small differences between classes of ancient architectural components, this paper introduces a self-attention mechanism, which can effectively distinguish similar components in the neighbourhood. Moreover, this paper explores the insufficiency of positional encoding in baseline and constructs a high-precision point cloud semantic segmentation network model for ancient buildings—Semantic Query Network based on Dual Local Attention (SQN-DLA). Using only 0.1% of the annotations in our homemade dataset and the Architectural Cultural Heritage (ArCH) dataset, the mean Intersection over Union (mIoU) reaches 66.02% and 58.03%, respectively, which is an improvement of 3.51% and 3.91%, respectively, compared to the baseline.

Active Bidirectional Self-Training Network for Cross-Domain Segmentation in Remote-Sensing Images

Semantic segmentation with cross-domain adaptation in remote-sensing images (RSIs) is crucial and mitigates the expense of manually labeling target data. However, the performance of existing unsupervised domain adaptation (UDA) methods is still significantly impacted by domain bias, leading to a considerable gap compared to supervised trained models. To address this, our work focuses on semi-supervised domain adaptation, selecting a small subset of target annotations through active learning (AL) that maximize information to improve domain adaptation. Overall, we propose a novel active bidirectional self-training network (ABSNet) for cross-domain semantic segmentation in RSIs. ABSNet consists of two sub-stages: a multi-prototype active region selection (MARS) stage and a source-weighted class-balanced self-training (SCBS) stage. The MARS approach captures the diversity in labeled source data by introducing multi-prototype density estimation based on Gaussian mixture models. We then measure inter-domain similarity to select complementary and representative target samples. Through fine-tuning with the selected active samples, we propose an enhanced self-training strategy SCBS, designed for weighted training on source data, aiming to avoid the negative effects of interfering samples. We conduct extensive experiments on the LoveDA and ISPRS datasets to validate the superiority of our method over existing state-of-the-art domain-adaptive semantic segmentation methods.