Problem Of Semantic Segmentation Research Articles

Edge Detection of Source Body from Magnetic Anomaly Based on ResNet

Utilizing magnetic anomaly data for effective edge detection of source bodies can provide crucial evidence for the delineation of geological units and the division of fault structures. However, the existing edge detection methods of source bodies from magnetic anomalies are influenced by factors such as the source bodies’ burial depth, magnetization direction, and mutual interference of magnetic anomalies, leading to errors in subsequent interpretation tasks. The advanced convolutional neural network possesses robust capabilities for feature representation and deep learning, prompting this paper to introduce an edge detection method for source bodies based on convolutional neural networks. The issue is initially framed as a semantic segmentation problem, and four network architectures aimed at edge detection of a source body from magnetic anomaly are designed and modified based on the U-Net and ResNet. Subsequently, a multitude of high-quality sample data sets are constructed using models with varying locations, scales, quantities, and physical properties to train the network. This paper then details model experiments that escalate from simple to complex, taking into account the combined effects of burial depth and inclined magnetization on edge detection. Compared to conventional edge detection methods, the method proposed in this paper is shown to accurately identify edges of source bodies at various depths with little impact from inclined magnetization and can automatically extract edge information without manual intervention. The method’s efficacy is corroborated through real data tests.

A Comparison of Uncertainty Estimation Methods for Building Footprint Change Detection from Sentinel-2 Imagery

Abstract. This manuscript investigates the effects of uncertainty methods applied to the problem of deep learning-based semantic segmentation of building footprints on moderate-resolution satellite imagery. While the recent efforts of big corporations to add information about building locations and sizes on a global or continental scale are generally valuable, still the overall challenge persists in identifying the spatial-temporal patterns of growing urbanization. In this work, we extend UNet-type architectures to perform binary building footprint classification based on Sentinel-2 imagery resulting in five different models. While previous studies focused on urban areas in the Western world we conduct all training and evaluation in India. All models are trained on Microsoft building footprint products while for evaluation purposes high-quality reference data is manually selected from regions with especially good open-street-map coverage. Quantitative and qualitative experiments are conducted where a significant performance gain is found for a model trained with a mixture of aleatoric and epistemic uncertainty measures. The performance gain is even more pronounced for subsequent quantitative multi-temporal change detection experiments.

A Nighttime Driving-Scene Segmentation Method Based on Light-Enhanced Network

To solve the semantic segmentation problem of night driving-scene images, which often have low brightness, low contrast, and uneven illumination, a nighttime driving-scene segmentation method based on a light-enhanced network was proposed. Firstly, we designed a light enhancement network, which comprises two parts: a color correction module and a parameter predictor. The color correction module mitigates the impact of illumination variations on the segmentation network by adjusting the color information of the image. Meanwhile, the parameter predictor accurately predicts the parameters of the image filter through the analysis of global content, including factors such as brightness, contrast, hue, and exposure level, thereby effectively enhancing the image quality. Subsequently, the output of the light enhancement network is input into the segmentation network to obtain the final segmentation prediction. Experimental results show that the proposed method achieves mean Intersection over Union (mIoU) values of 59.4% on the Dark Zurich-test dataset, outperforming other segmentation algorithms for nighttime driving-scenes.

Cross-Image Pixel Contrasting for Semantic Segmentation.

This work studies the problem of image semantic segmentation. Current approaches focus mainly on mining "local" context, i.e., dependencies between pixels within individual images, by specifically-designed, context aggregation modules (e.g., dilated convolution, neural attention) or structure-aware optimization objectives (e.g., IoU-like loss). However, they ignore "global" context of the training data, i.e., rich semantic relations between pixels across different images. Inspired by recent advance in unsupervised contrastive representation learning, we propose a pixel-wise contrastive algorithm, dubbed as PiCo, for semantic segmentation in the fully supervised learning setting. The core idea is to enforce pixel embeddings belonging to a same semantic class to be more similar than embeddings from different classes. It raises a pixel-wise metric learning paradigm for semantic segmentation, by explicitly exploring the structures of labeled pixels, which were rarely studied before. Our training algorithm is compatible with modern segmentation solutions without extra overhead during testing. We experimentally show that, with famous segmentation models (i.e., DeepLabV3, HRNet, OCRNet, SegFormer, Segmenter, MaskFormer) and backbones (i.e., MobileNet, ResNet, HRNet, MiT, ViT), our algorithm brings consistent performance improvements across diverse datasets (i.e., Cityscapes, ADE20 K, PASCAL-Context, COCO-Stuff, CamVid). We expect that this work will encourage our community to rethink the current de facto training paradigm in semantic segmentation.

PatchNet: Maximize the Exploration of Congeneric Semantics for Weakly Supervised Semantic Segmentation.

With the increase in the number of image data and the lack of corresponding labels, weakly supervised learning has drawn a lot of attention recently in computer vision tasks, especially in the fine-grained semantic segmentation problem. To alleviate human efforts from expensive pixel-by-pixel annotations, our method focuses on weakly supervised semantic segmentation (WSSS) with image-level labels, which are much easier to obtain. As a considerable gap exists between pixel-level segmentation and image-level labels, how to reflect the image-level semantic information on each pixel is an important question. To explore the congeneric semantic regions from the same class to the maximum, we construct the patch-level semantic augmentation network (PatchNet) based on the self-detected patches from different images that contain the same class labels. Patches can frame the objects as much as possible and include as little background as possible. The patch-level semantic augmentation network that is established with patches as the nodes can maximize the mutual learning of similar objects. We regard the embedding vectors of patches as nodes and use a transformer-based complementary learning module to construct weighted edges according to the embedding similarity between different nodes. Moreover, to better supplement semantic information, we propose softcomplementary loss functions matched with the whole network structure. We conduct experiments on the popular PASCAL VOC 2012 and MS COCO 2014 benchmarks, and our model yields the state-of-the-art performance.

Semantic segmentation for large-scale point clouds based on hybrid attention and dynamic fusion

This paper investigates the semantic segmentation problem for large-scale point clouds. Recent segmentation methods usually employ an encoder–decoder architecture. However, these methods may not effectively extract neighboring information in the encoder. Additionally, they typically use nearest neighbor interpolation and skip connections in the decoder, overlooking the semantic gap between encoder and decoder features. To resolve these issues, we propose HADF-Net, which consists of a Hybrid Attention Encoder (HAE), an Edge Dynamic Fusion module (EDF), and a Dynamic Cross-attention Decoder (DCD). HAE leverages the distinctive properties of geometric and semantic relations to aggregate local features at different stages. EDF aims to alleviate information loss during decoder upsampling by dynamically integrating the neighboring information. DCD employs an enhanced fusion mechanism with spatial-wise cross-attention to bridge the semantic gap between encoder and decoder features. Experimental results on 4 datasets demonstrate that our HADF-Net achieves superior performance.

Unet-boosted Classifier – Multi-Task Architecture for Small Datasets Applied to Brain MRI Classification

The problem of training deep neural networks on small samples is especially relevant for medical issues. The paper examines the impact of pixel-wise marking of significant objects in the image, over the true class label, on the quality of the classification. To achieve better classification results on small samples, we propose a multitasking architecture – Unet-boosted classifier (UBC), that is trained simultaneously to solve classification and semantic segmentation problems. As the exploratory dataset, MRI images of patients with benign glioma and glioblastoma taken from the BRaTS 2019 data set are used. One horizontal slice of the MRI image containing a glioma is considered as the input (a total of 380 frames in the training set), and the probability of glioblastoma – as the output. Resnet34 was chosen as the baseline, trained without augmentations with a loss function based on cross-entropy. As an alternative solution, UBC-resnet34 is used – the same resnet34, boosted by a decoder built on the U-Net principle and predicting the pixels with glioma. The smoothed Sorensen-Dice coefficient (DiceLoss) is used as a decoder loss function. Results on the test sample: accuracy for the baseline reached 0.71, for the proposed model – 0.81, and the Dice score – 0.77. Thus, a deep model can be well trained even on a small data set, using the proposed architecture, provided that marking of the affected tissues in the form of a semantic mask is provided.

Improving CNN-based semantic segmentation on structurally similar data using contrastive graph convolutional networks

Structurally similar data exist in most practical semantic segmentation applications. For example, objects can appear identical or positionally similar in many images, such as video frames. Objects with structural similarity in data samples can confuse deep neural networks (DNNs) in semantic segmentation applications. These challenges often lead to lower pixel classification accuracy of natural object segmentation. This study proposes a novel approach (S2-GCN) that enhances CNN-based semantic segmentation for structurally similar data using a contrastive graph convolutional network (GCN). By selecting specific label pairs and developing a customized GCN branch parallel to an encoder-decoder backbone, our method significantly improves accuracy, IoU, and F1-score, by up to 8 %, as demonstrated through an extensive evaluation of five datasets. Our findings show that the proposed method effectively addresses the structural similarity problem of CNN-based semantic segmentation and can be applied to a wide range of practical applications.

Применение свёрточных нейронных сетей в задаче устранения теней с фотографий

<p>The article proposes a method for removing shadows from photographs using deep learning methods. The proposed method consists of several stages: dividing the image into rectangular fragments of 32x32 pixels, localizing shadows on each fragment, restoring the color of shadowed objects, and combining the fragments back into a whole image. Shadow localization is considered as a semantic segmentation problem; to solve it, a neural network of encoder-decoder architecture has been developed and trained. To restore the color of objects in identified shaded areas, another neural network based on the CDNet architecture is used. Examples of image processing using the developed method are given, including images from a drone, and the high quality of restoration of shaded areas is demonstrated.</p>

Alignment and fusion for adaptive domain nighttime semantic segmentation

In the field of autonomous driving technology, both daytime and nighttime scenes are common. However, due to the poor illumination and difficulty in manual annotation of nighttime images, semantic segmentation of nighttime scenes is more challenging compared to daytime scenes. Therefore, achieving significant progress in nighttime semantic segmentation would greatly enhance the effectiveness of the application of autonomous driving scenarios. In this work, we investigate the problem of domain-adaptive nighttime semantic segmentation (DANSS). The problem aims to learn semantic segmentation in nighttime scenes by leveraging a labeled Cityscapes dataset and unlabeled but roughly aligned day-night image pairs. To address this, we propose an Align and Fusion Network (AAFnet), a network for adaptive domain nighttime semantic segmentation. AAFnet utilizes a novel DAFormer as the backbone to separately compute features for dynamic and static objects during the training process. It also employs methods such as small category sampling and image blending to improve learning effectiveness. Additionally, we propose utilizing local image patches to enhance the results during the testing process. Experimental results demonstrate a significant improvement of 10.4% compared to the previous method, DANNet. Extensive experiments conducted on the Dark Zurich dataset and Nightdriving dataset validate the effectiveness of the proposed approach. Our method outperforms previous backbones and achieves top-ranking results.

Real-time semantic segmentation based on BiSeNetV2 for wild road

Abstract State-of-the-art segmentation models have shown great performance in structured road segmentation. However, these models are not suitable for the wild roads, which are highly unstructured. To tackle the problem of real-time semantic segmentation of wild roads, we propose a Multi-Information Concatenate Network based on BiSeNetV2 and construct a segmentation dataset Dalle Molle institute for artificial intelligence feature segmentation (IDSIAFS) based on Dalle Molle institute for artificial intelligence. The proposed model removes structural redundancy and optimizes the semantic branch based on BiSeNetV2. Moreover, the Dual-Path Semantic Inference Layer (TPSIL) reduces computation by designing the channel dimension of the semantic branch feature map and aggregates feature maps of different depths. Finally, the segmentation results are achieved by fusing both shallow detail information and deep semantic information. Experiments on the IDSIAFS dataset demonstrate that our proposed model achieves an 89.5% Intersection over Union. The comparative experiments on Cityscapes and India driving dataset benchmarks show that proposed model achieves good inference accuracy and faster inference speed.

Exploring Temporal Feature Correlation for Efficient and Stable Video Semantic Segmentation

This paper tackles the problem of efficient and stable video semantic segmentation. While stability has been under-explored, prevalent work in efficient video semantic segmentation uses the keyframe paradigm. They efficiently process videos by only recomputing the low-level features and reusing high-level features computed at selected keyframes. In addition, the reused features stabilize the predictions across frames, thereby improving video consistency. However, dynamic scenes in the video can easily lead to misalignments between reused and recomputed features, which hampers performance. Moreover, relying on feature reuse to improve prediction consistency is brittle; an erroneous alignment of the features can easily lead to unstable predictions. Therefore, the keyframe paradigm exhibits a dilemma between stability and performance. We address this efficiency and stability challenge using a novel yet simple Temporal Feature Correlation (TFC) module. It uses the cosine similarity between two frames’ low-level features to inform the semantic label’s consistency across frames. Specifically, we selectively reuse label-consistent features across frames through linear interpolation and update others through sparse multi-scale deformable attention. As a result, we no longer directly reuse features to improve stability and thus effectively solve feature misalignment. This work provides a significant step towards efficient and stable video semantic segmentation. On the VSPW dataset, our method significantly improves the prediction consistency of image-based methods while being as fast and accurate.

Semantic segmentation of thermal defects in belt conveyor idlers using thermal image augmentation and U-Net-based convolutional neural networks

The belt conveyor (BC) is the main means of horizontal transportation of bulk materials at mining sites. The sudden fault in BC modules may cause unexpected stops in production lines. With the increasing number of applications of inspection mobile robots in condition monitoring (CM) of industrial infrastructure in hazardous environments, in this article we introduce an image processing pipeline for automatic segmentation of thermal defects in thermal images captured from BC idlers using a mobile robot. This study follows the fact that CM of idler temperature is an important task for preventing sudden breakdowns in BC system networks. We compared the performance of three different types of U-Net-based convolutional neural network architectures for the identification of thermal anomalies using a small number of hand-labeled thermal images. Experiments on the test data set showed that the attention residual U-Net with binary cross entropy as the loss function handled the semantic segmentation problem better than our previous research and other studied U-Net variations.

Improving Semantic Segmentation via Efficient Self-Training.

Starting from the seminal work of Fully Convolutional Networks (FCN), there has been significant progress on semantic segmentation. However, deep learning models often require large amounts of pixelwise annotations to train accurate and robust models. Given the prohibitively expensive annotation cost of segmentation masks, we introduce a self-training framework in this paper to leverage pseudo labels generated from unlabeled data. In order to handle the data imbalance problem of semantic segmentation, we propose a centroid sampling strategy to uniformly select training samples from every class within each epoch. We also introduce a fast training schedule to alleviate the computational burden. This enables us to explore the usage of large amounts of pseudo labels. Our Centroid Sampling based Self-Training framework (CSST) achieves state-of-the-art results on Cityscapes and CamVid datasets. On PASCAL VOC 2012 test set, our models trained with the original train set even outperform the same models trained on the much bigger augmented train set. This indicates the effectiveness of CSST when there are fewer annotations. We also demonstrate promising few-shot generalization capability from Cityscapes to BDD100K and from Cityscapes to Mapillary datasets.

Location and segmentation of important railway signs based on improved segmentation

Purpose With the rapid development of railway-intelligent video technology, scene understanding is becoming more and more important. Semantic segmentation is a major part of scene understanding. There is an urgent need for an algorithm with high accuracy and real-time to meet the current railway requirements for railway identification. In response to this demand, this paper aims to explore a variety of models, accurately locate and segment important railway signs based on the improved SegNeXt algorithm, supplement the railway safety protection system and improve the intelligent level of railway safety protection. Design/methodology/approach This paper studies the performance of existing models on RailSem19 and explores the defects of each model through performance so as to further explore an algorithm model dedicated to railway semantic segmentation. In this paper, the authors explore the optimal solution of SegNeXt model for railway scenes and achieve the purpose of this paper by improving the encoder and decoder structure. Findings This paper proposes an improved SegNeXt algorithm: first, it explores the performance of various models on railways, studies the problems of semantic segmentation on railways and then analyzes the specific problems. On the basis of retaining the original excellent MSCAN encoder of SegNeXt, multiscale information fusion is used to further extract detailed features such as multihead attention and mask, solving the problem of inaccurate segmentation of current objects by the original SegNeXt algorithm. The improved algorithm is of great significance for the segmentation and recognition of railway signs. Research limitations/implications The model constructed in this paper has advantages in the feature segmentation of distant small objects, but it still has the problem of segmentation fracture for the railway, which is not completely segmented. In addition, in the throat area, due to the complexity of the railway, the segmentation results are not accurate. Social implications The identification and segmentation of railway signs based on the improved SegNeXt algorithm in this paper is of great significance for the understanding of existing railway scenes, which can greatly improve the classification and recognition ability of railway small object features and can greatly improve the degree of railway security. Originality/value This article introduces an enhanced version of the SegNeXt algorithm, which aims to improve the accuracy of semantic segmentation on railways. The study begins by investigating the performance of different models in railway scenarios and identifying the challenges associated with semantic segmentation on this particular domain. To address these challenges, the proposed approach builds upon the strong foundation of the original SegNeXt algorithm, leveraging techniques such as multi-scale information fusion, multi-head attention, and masking to extract finer details and enhance feature representation. By doing so, the improved algorithm effectively resolves the issue of inaccurate object segmentation encountered in the original SegNeXt algorithm. This advancement holds significant importance for the accurate recognition and segmentation of railway signage.

CreativeSeg: Semantic Segmentation of Creative Sketches.

The problem of sketch semantic segmentation is far from being solved. Despite existing methods exhibiting near-saturating performances on simple sketches with high recognisability, they suffer serious setbacks when the target sketches are products of an imaginative process with high degree of creativity. We hypothesise that human creativity, being highly individualistic, induces a significant shift in distribution of sketches, leading to poor model generalisation. Such hypothesis, backed by empirical evidences, opens the door for a solution that explicitly disentangles creativity while learning sketch representations. We materialise this by crafting a learnable creativity estimator that assigns a scalar score of creativity to each sketch. It follows that we introduce CreativeSeg, a learning-to-learn framework that leverages the estimator in order to learn creativity-agnostic representation, and eventually the downstream semantic segmentation task. We empirically verify the superiority of CreativeSeg on the recent "Creative Birds" and "Creative Creatures" creative sketch datasets. Through a human study, we further strengthen the case that the learned creativity score does indeed have a positive correlation with the subjective creativity of human. Codes are available at https://github.com/PRIS-CV/Sketch-CS.

Semantic segmentation of remote sensing imagery for road extraction via joint angle prediction: comparisons to deep learning

Accurate road network information is required to study and analyze the relationship between land usage type and land subsidence, and road extraction from remote sensing images is an important data source for updating road networks. This task has been considered a significant semantic segmentation problem, given the many road extraction methods developed for remote sensing images in recent years. Although impressive results have been achieved by classifying each pixel in the remote sensing image using a semantic segmentation network, traditional semantic segmentation methods often lack clear constraints of road features. Consequently, the geometric features of the results might deviate from actual roads, leading to issues like road fractures, rough edges, inconsistent road widths, and more, which hinder their effectiveness in road updates. This paper proposes a novel road semantic segmentation algorithm for remote sensing images based on the joint road angle prediction. By incorporating the angle prediction module and the angle feature fusion module, constraints are added to the angle features of the road. Through the angle prediction and angle feature fusion, the information contained in the remote sensing images can be better utilized. The experimental results show that the proposed method outperforms existing semantic segmentation methods in both quantitative evaluation and visual effects. Furthermore, the extracted roads were consecutive with distinct edges, making them more suitable for mapping road updates.

The Segment Anything Model (SAM) for accelerating the smart farming revolution

Precision agriculture uses accurate identification and mapping of crop features by automated mechanisms. Using computer vision techniques implemented by supervised deep learning systems to solve many precision agricultural problems necessitates large-scale data collection and prolonged ground truth annotation by humans. The so-called foundation models in Artificial Intelligence (AI) are becoming increasingly significant. Meta AI Research is working on a project called Segment Anything to provide a base model for image segmentation. It can accomplish zero-shot generalisation to strange objects and images without additional training. This study evaluates the performance of the Segment Anything Model (SAM) for the problem of semantic segmentation of objects in the context of precision agriculture.

Semantic Segmentation of High-Resolution Remote Sensing Images with Improved U-Net Based on Transfer Learning

Semantic segmentation of high-resolution remote sensing images has emerged as one of the foci of research in the remote sensing field, which can accurately identify objects on the ground and determine their localization. In contrast, the traditional deep learning-based semantic segmentation, on the other hand, requires a large amount of annotated data, which is unsuitable for high-resolution remote sensing tasks with limited resources. It is therefore important to build a semantic segmentation method for high-resolution remote sensing images. In this paper, it is proposed an improved U-Net model based on transfer learning to solve the semantic segmentation problem of high-resolution remote sensing images. The model is based on the symmetric encoder–decoder structure of U-Net. For the encoder, transfer learning is applied and VGG16 is used as the backbone of the feature extraction network, and in the decoder, after upsampling using bilinear interpolation, it is performed multiscale fusion with the feature maps of the corresponding layers of the encoder in turn and is finally obtained the predicted value of each pixel to achieve precise localization. To verify the efficacy of the proposed network, experiments are performed on the ISPRS Vaihingen dataset. The experiments show that the applied method has achieved high-quality semantic segmentation results on the high-resolution remote sensing dataset, and the MIoU is 1.70%, 2.20%, and 2.33% higher on the training, validation, and test sets, respectively, and the IoU is 4.26%, 6.89%, and 5.44% higher for the automotive category compared to the traditional U-Net.

Evolutionary design of explainable algorithms for biomedical image segmentation

An unresolved issue in contemporary biomedicine is the overwhelming number and diversity of complex images that require annotation, analysis and interpretation. Recent advances in Deep Learning have revolutionized the field of computer vision, creating algorithms that compete with human experts in image segmentation tasks. However, these frameworks require large human-annotated datasets for training and the resulting “black box” models are difficult to interpret. In this study, we introduce Kartezio, a modular Cartesian Genetic Programming-based computational strategy that generates fully transparent and easily interpretable image processing pipelines by iteratively assembling and parameterizing computer vision functions. The pipelines thus generated exhibit comparable precision to state-of-the-art Deep Learning approaches on instance segmentation tasks, while requiring drastically smaller training datasets. This Few-Shot Learning method confers tremendous flexibility, speed, and functionality to this approach. We then deploy Kartezio to solve a series of semantic and instance segmentation problems, and demonstrate its utility across diverse images ranging from multiplexed tissue histopathology images to high resolution microscopy images. While the flexibility, robustness and practical utility of Kartezio make this fully explicable evolutionary designer a potential game-changer in the field of biomedical image processing, Kartezio remains complementary and potentially auxiliary to mainstream Deep Learning approaches.