Problem Of Semantic Segmentation Research Articles

Semantic segmentation for large-scale point clouds based on hybrid attention and dynamic fusion

This paper investigates the semantic segmentation problem for large-scale point clouds. Recent segmentation methods usually employ an encoder–decoder architecture. However, these methods may not effectively extract neighboring information in the encoder. Additionally, they typically use nearest neighbor interpolation and skip connections in the decoder, overlooking the semantic gap between encoder and decoder features. To resolve these issues, we propose HADF-Net, which consists of a Hybrid Attention Encoder (HAE), an Edge Dynamic Fusion module (EDF), and a Dynamic Cross-attention Decoder (DCD). HAE leverages the distinctive properties of geometric and semantic relations to aggregate local features at different stages. EDF aims to alleviate information loss during decoder upsampling by dynamically integrating the neighboring information. DCD employs an enhanced fusion mechanism with spatial-wise cross-attention to bridge the semantic gap between encoder and decoder features. Experimental results on 4 datasets demonstrate that our HADF-Net achieves superior performance.

Применение свёрточных нейронных сетей в задаче устранения теней с фотографий

<p>The article proposes a method for removing shadows from photographs using deep learning methods. The proposed method consists of several stages: dividing the image into rectangular fragments of 32x32 pixels, localizing shadows on each fragment, restoring the color of shadowed objects, and combining the fragments back into a whole image. Shadow localization is considered as a semantic segmentation problem; to solve it, a neural network of encoder-decoder architecture has been developed and trained. To restore the color of objects in identified shaded areas, another neural network based on the CDNet architecture is used. Examples of image processing using the developed method are given, including images from a drone, and the high quality of restoration of shaded areas is demonstrated.</p>

Alignment and fusion for adaptive domain nighttime semantic segmentation

In the field of autonomous driving technology, both daytime and nighttime scenes are common. However, due to the poor illumination and difficulty in manual annotation of nighttime images, semantic segmentation of nighttime scenes is more challenging compared to daytime scenes. Therefore, achieving significant progress in nighttime semantic segmentation would greatly enhance the effectiveness of the application of autonomous driving scenarios. In this work, we investigate the problem of domain-adaptive nighttime semantic segmentation (DANSS). The problem aims to learn semantic segmentation in nighttime scenes by leveraging a labeled Cityscapes dataset and unlabeled but roughly aligned day-night image pairs. To address this, we propose an Align and Fusion Network (AAFnet), a network for adaptive domain nighttime semantic segmentation. AAFnet utilizes a novel DAFormer as the backbone to separately compute features for dynamic and static objects during the training process. It also employs methods such as small category sampling and image blending to improve learning effectiveness. Additionally, we propose utilizing local image patches to enhance the results during the testing process. Experimental results demonstrate a significant improvement of 10.4% compared to the previous method, DANNet. Extensive experiments conducted on the Dark Zurich dataset and Nightdriving dataset validate the effectiveness of the proposed approach. Our method outperforms previous backbones and achieves top-ranking results.

Real-time semantic segmentation based on BiSeNetV2 for wild road

Abstract State-of-the-art segmentation models have shown great performance in structured road segmentation. However, these models are not suitable for the wild roads, which are highly unstructured. To tackle the problem of real-time semantic segmentation of wild roads, we propose a Multi-Information Concatenate Network based on BiSeNetV2 and construct a segmentation dataset Dalle Molle institute for artificial intelligence feature segmentation (IDSIAFS) based on Dalle Molle institute for artificial intelligence. The proposed model removes structural redundancy and optimizes the semantic branch based on BiSeNetV2. Moreover, the Dual-Path Semantic Inference Layer (TPSIL) reduces computation by designing the channel dimension of the semantic branch feature map and aggregates feature maps of different depths. Finally, the segmentation results are achieved by fusing both shallow detail information and deep semantic information. Experiments on the IDSIAFS dataset demonstrate that our proposed model achieves an 89.5% Intersection over Union. The comparative experiments on Cityscapes and India driving dataset benchmarks show that proposed model achieves good inference accuracy and faster inference speed.

Exploring Temporal Feature Correlation for Efficient and Stable Video Semantic Segmentation

This paper tackles the problem of efficient and stable video semantic segmentation. While stability has been under-explored, prevalent work in efficient video semantic segmentation uses the keyframe paradigm. They efficiently process videos by only recomputing the low-level features and reusing high-level features computed at selected keyframes. In addition, the reused features stabilize the predictions across frames, thereby improving video consistency. However, dynamic scenes in the video can easily lead to misalignments between reused and recomputed features, which hampers performance. Moreover, relying on feature reuse to improve prediction consistency is brittle; an erroneous alignment of the features can easily lead to unstable predictions. Therefore, the keyframe paradigm exhibits a dilemma between stability and performance. We address this efficiency and stability challenge using a novel yet simple Temporal Feature Correlation (TFC) module. It uses the cosine similarity between two frames’ low-level features to inform the semantic label’s consistency across frames. Specifically, we selectively reuse label-consistent features across frames through linear interpolation and update others through sparse multi-scale deformable attention. As a result, we no longer directly reuse features to improve stability and thus effectively solve feature misalignment. This work provides a significant step towards efficient and stable video semantic segmentation. On the VSPW dataset, our method significantly improves the prediction consistency of image-based methods while being as fast and accurate.

Semantic segmentation of thermal defects in belt conveyor idlers using thermal image augmentation and U-Net-based convolutional neural networks

The belt conveyor (BC) is the main means of horizontal transportation of bulk materials at mining sites. The sudden fault in BC modules may cause unexpected stops in production lines. With the increasing number of applications of inspection mobile robots in condition monitoring (CM) of industrial infrastructure in hazardous environments, in this article we introduce an image processing pipeline for automatic segmentation of thermal defects in thermal images captured from BC idlers using a mobile robot. This study follows the fact that CM of idler temperature is an important task for preventing sudden breakdowns in BC system networks. We compared the performance of three different types of U-Net-based convolutional neural network architectures for the identification of thermal anomalies using a small number of hand-labeled thermal images. Experiments on the test data set showed that the attention residual U-Net with binary cross entropy as the loss function handled the semantic segmentation problem better than our previous research and other studied U-Net variations.

Improving Semantic Segmentation via Efficient Self-Training.

Starting from the seminal work of Fully Convolutional Networks (FCN), there has been significant progress on semantic segmentation. However, deep learning models often require large amounts of pixelwise annotations to train accurate and robust models. Given the prohibitively expensive annotation cost of segmentation masks, we introduce a self-training framework in this paper to leverage pseudo labels generated from unlabeled data. In order to handle the data imbalance problem of semantic segmentation, we propose a centroid sampling strategy to uniformly select training samples from every class within each epoch. We also introduce a fast training schedule to alleviate the computational burden. This enables us to explore the usage of large amounts of pseudo labels. Our Centroid Sampling based Self-Training framework (CSST) achieves state-of-the-art results on Cityscapes and CamVid datasets. On PASCAL VOC 2012 test set, our models trained with the original train set even outperform the same models trained on the much bigger augmented train set. This indicates the effectiveness of CSST when there are fewer annotations. We also demonstrate promising few-shot generalization capability from Cityscapes to BDD100K and from Cityscapes to Mapillary datasets.

Location and segmentation of important railway signs based on improved segmentation

Purpose With the rapid development of railway-intelligent video technology, scene understanding is becoming more and more important. Semantic segmentation is a major part of scene understanding. There is an urgent need for an algorithm with high accuracy and real-time to meet the current railway requirements for railway identification. In response to this demand, this paper aims to explore a variety of models, accurately locate and segment important railway signs based on the improved SegNeXt algorithm, supplement the railway safety protection system and improve the intelligent level of railway safety protection. Design/methodology/approach This paper studies the performance of existing models on RailSem19 and explores the defects of each model through performance so as to further explore an algorithm model dedicated to railway semantic segmentation. In this paper, the authors explore the optimal solution of SegNeXt model for railway scenes and achieve the purpose of this paper by improving the encoder and decoder structure. Findings This paper proposes an improved SegNeXt algorithm: first, it explores the performance of various models on railways, studies the problems of semantic segmentation on railways and then analyzes the specific problems. On the basis of retaining the original excellent MSCAN encoder of SegNeXt, multiscale information fusion is used to further extract detailed features such as multihead attention and mask, solving the problem of inaccurate segmentation of current objects by the original SegNeXt algorithm. The improved algorithm is of great significance for the segmentation and recognition of railway signs. Research limitations/implications The model constructed in this paper has advantages in the feature segmentation of distant small objects, but it still has the problem of segmentation fracture for the railway, which is not completely segmented. In addition, in the throat area, due to the complexity of the railway, the segmentation results are not accurate. Social implications The identification and segmentation of railway signs based on the improved SegNeXt algorithm in this paper is of great significance for the understanding of existing railway scenes, which can greatly improve the classification and recognition ability of railway small object features and can greatly improve the degree of railway security. Originality/value This article introduces an enhanced version of the SegNeXt algorithm, which aims to improve the accuracy of semantic segmentation on railways. The study begins by investigating the performance of different models in railway scenarios and identifying the challenges associated with semantic segmentation on this particular domain. To address these challenges, the proposed approach builds upon the strong foundation of the original SegNeXt algorithm, leveraging techniques such as multi-scale information fusion, multi-head attention, and masking to extract finer details and enhance feature representation. By doing so, the improved algorithm effectively resolves the issue of inaccurate object segmentation encountered in the original SegNeXt algorithm. This advancement holds significant importance for the accurate recognition and segmentation of railway signage.

CreativeSeg: Semantic Segmentation of Creative Sketches.

The problem of sketch semantic segmentation is far from being solved. Despite existing methods exhibiting near-saturating performances on simple sketches with high recognisability, they suffer serious setbacks when the target sketches are products of an imaginative process with high degree of creativity. We hypothesise that human creativity, being highly individualistic, induces a significant shift in distribution of sketches, leading to poor model generalisation. Such hypothesis, backed by empirical evidences, opens the door for a solution that explicitly disentangles creativity while learning sketch representations. We materialise this by crafting a learnable creativity estimator that assigns a scalar score of creativity to each sketch. It follows that we introduce CreativeSeg, a learning-to-learn framework that leverages the estimator in order to learn creativity-agnostic representation, and eventually the downstream semantic segmentation task. We empirically verify the superiority of CreativeSeg on the recent "Creative Birds" and "Creative Creatures" creative sketch datasets. Through a human study, we further strengthen the case that the learned creativity score does indeed have a positive correlation with the subjective creativity of human. Codes are available at https://github.com/PRIS-CV/Sketch-CS.

Cross-Image Pixel Contrasting for Semantic Segmentation.

This work studies the problem of image semantic segmentation. Current approaches focus mainly on mining "local" context, i.e., dependencies between pixels within individual images, by specifically-designed, context aggregation modules (e.g., dilated convolution, neural attention) or structure-aware optimization objectives (e.g., IoU-like loss). However, they ignore "global" context of the training data, i.e., rich semantic relations between pixels across different images. Inspired by recent advance in unsupervised contrastive representation learning, we propose a pixel-wise contrastive algorithm, dubbed as PiCo, for semantic segmentation in the fully supervised learning setting. The core idea is to enforce pixel embeddings belonging to a same semantic class to be more similar than embeddings from different classes. It raises a pixel-wise metric learning paradigm for semantic segmentation, by explicitly exploring the structures of labeled pixels, which were rarely studied before. Our training algorithm is compatible with modern segmentation solutions without extra overhead during testing. We experimentally show that, with famous segmentation models (i.e., DeepLabV3, HRNet, OCRNet, SegFormer, Segmenter, MaskFormer) and backbones (i.e., MobileNet, ResNet, HRNet, MiT, ViT), our algorithm brings consistent performance improvements across diverse datasets (i.e., Cityscapes, ADE20K, PASCAL-Context, COCO-Stuff, CamVid). We expect that this work will encourage our community to rethink the current de facto training paradigm in semantic segmentation. Our code is available at https://github.com/tfzhou/ContrastiveSeg.

Semantic segmentation of remote sensing imagery for road extraction via joint angle prediction: comparisons to deep learning

Accurate road network information is required to study and analyze the relationship between land usage type and land subsidence, and road extraction from remote sensing images is an important data source for updating road networks. This task has been considered a significant semantic segmentation problem, given the many road extraction methods developed for remote sensing images in recent years. Although impressive results have been achieved by classifying each pixel in the remote sensing image using a semantic segmentation network, traditional semantic segmentation methods often lack clear constraints of road features. Consequently, the geometric features of the results might deviate from actual roads, leading to issues like road fractures, rough edges, inconsistent road widths, and more, which hinder their effectiveness in road updates. This paper proposes a novel road semantic segmentation algorithm for remote sensing images based on the joint road angle prediction. By incorporating the angle prediction module and the angle feature fusion module, constraints are added to the angle features of the road. Through the angle prediction and angle feature fusion, the information contained in the remote sensing images can be better utilized. The experimental results show that the proposed method outperforms existing semantic segmentation methods in both quantitative evaluation and visual effects. Furthermore, the extracted roads were consecutive with distinct edges, making them more suitable for mapping road updates.

The Segment Anything Model (SAM) for accelerating the smart farming revolution

Precision agriculture uses accurate identification and mapping of crop features by automated mechanisms. Using computer vision techniques implemented by supervised deep learning systems to solve many precision agricultural problems necessitates large-scale data collection and prolonged ground truth annotation by humans. The so-called foundation models in Artificial Intelligence (AI) are becoming increasingly significant. Meta AI Research is working on a project called Segment Anything to provide a base model for image segmentation. It can accomplish zero-shot generalisation to strange objects and images without additional training. This study evaluates the performance of the Segment Anything Model (SAM) for the problem of semantic segmentation of objects in the context of precision agriculture.

Semantic Segmentation of High-Resolution Remote Sensing Images with Improved U-Net Based on Transfer Learning

Semantic segmentation of high-resolution remote sensing images has emerged as one of the foci of research in the remote sensing field, which can accurately identify objects on the ground and determine their localization. In contrast, the traditional deep learning-based semantic segmentation, on the other hand, requires a large amount of annotated data, which is unsuitable for high-resolution remote sensing tasks with limited resources. It is therefore important to build a semantic segmentation method for high-resolution remote sensing images. In this paper, it is proposed an improved U-Net model based on transfer learning to solve the semantic segmentation problem of high-resolution remote sensing images. The model is based on the symmetric encoder–decoder structure of U-Net. For the encoder, transfer learning is applied and VGG16 is used as the backbone of the feature extraction network, and in the decoder, after upsampling using bilinear interpolation, it is performed multiscale fusion with the feature maps of the corresponding layers of the encoder in turn and is finally obtained the predicted value of each pixel to achieve precise localization. To verify the efficacy of the proposed network, experiments are performed on the ISPRS Vaihingen dataset. The experiments show that the applied method has achieved high-quality semantic segmentation results on the high-resolution remote sensing dataset, and the MIoU is 1.70%, 2.20%, and 2.33% higher on the training, validation, and test sets, respectively, and the IoU is 4.26%, 6.89%, and 5.44% higher for the automotive category compared to the traditional U-Net.

Evolutionary design of explainable algorithms for biomedical image segmentation

An unresolved issue in contemporary biomedicine is the overwhelming number and diversity of complex images that require annotation, analysis and interpretation. Recent advances in Deep Learning have revolutionized the field of computer vision, creating algorithms that compete with human experts in image segmentation tasks. However, these frameworks require large human-annotated datasets for training and the resulting “black box” models are difficult to interpret. In this study, we introduce Kartezio, a modular Cartesian Genetic Programming-based computational strategy that generates fully transparent and easily interpretable image processing pipelines by iteratively assembling and parameterizing computer vision functions. The pipelines thus generated exhibit comparable precision to state-of-the-art Deep Learning approaches on instance segmentation tasks, while requiring drastically smaller training datasets. This Few-Shot Learning method confers tremendous flexibility, speed, and functionality to this approach. We then deploy Kartezio to solve a series of semantic and instance segmentation problems, and demonstrate its utility across diverse images ranging from multiplexed tissue histopathology images to high resolution microscopy images. While the flexibility, robustness and practical utility of Kartezio make this fully explicable evolutionary designer a potential game-changer in the field of biomedical image processing, Kartezio remains complementary and potentially auxiliary to mainstream Deep Learning approaches.

Cloud-to-Ground and Intra-Cloud Nowcasting Lightning Using a Semantic Segmentation Deep Learning Network

Weather forecasting requires a comprehensive analysis of various types of meteorology data, and with the wide application of deep learning in various fields, deep learning has proved to have powerful feature extraction capabilities. In this paper, from the viewpoint of an image semantic segmentation problem, a deep learning framework based on semantic segmentation is proposed to nowcast Cloud-to-Ground and Intra-Cloud lightning simultaneously within an hour. First, a dataset with spatiotemporal features is constructed using radar echo reflectivity data and lightning observation data. More specifically, each sample in the dataset consists of the past half hour of observations. Then, a Light3DUnet is presented based on 3D U-Net. The three-dimensional structured network can extract spatiotemporal features, and the encoder–decoder structure and the skip connection can handle small targets and recover more details. Due to the sparsity of lightning observations, a weighted cross-loss function was used to evaluate network performance. Finally, Light3DUnet was trained using the dataset to predict Cloud-to-Ground and Intra-Cloud lightning in the next hour. We evaluated the prediction performance of the network using a real-world dataset from middle China. The results show that Light3DUnet has a good ability to nowcast IC and CG lightning. Meanwhile, due to the spatial position coupling of IC and CG on a two-dimensional plane, predictions from summing the probabilistic prediction matrices will be augmented to obtain accurate prediction results for total flashes.

Application of artificial intelligence in ophthalmology for solving the problem of semantic segmentation of fundus images

The paper presents main aspects of the application of artificial intelligence in ophthalmology for the diagnosis and treatment of eye diseases, considering the problem of semantic segmentation of fundus images as an example. The classic approach to semantic segmentation on the basis of textural features is compared to the proposed approach based on neural networks. Basic problems of using the neural network approach in biomedicine are formulated. We propose a new method for selecting an optimal zone of laser exposure for laser coagulation based on two neural networks. The first network is used for detecting anatomical objects in the fundus and the second one is used for selecting the area of macular edema. The region of interest is formed from the edema area while taking into account the location of anatomical objects in it. A comparative analysis of sev-eral architectures of neural networks for solving the problem of selecting the edema area is carried out. The best results in the selection of the edema area are shown by the neural network architecture of Unet++.

Enhancing Semantic Segmentation: Design and Analysis of Improved U-Net Based Deep Convolutional Neural Networks

In this research, we provide a state-of-the-art method for semantic segmentation that makes use of a modified version of the U-Net architecture, which is itself based on deep convolutional neural networks (CNNs). This research delves into the ins and outs of this cutting-edge approach to semantic segmentation in an effort to boost its precision and productivity. To perform semantic segmentation, a crucial operation in computer vision, each pixel in an image must be assigned to one of many predefined item classes. The proposed Improved U-Net architecture makes use of deep CNNs to efficiently capture complex spatial characteristics while preserving associated context. The study illustrates the efficacy of the Improved U-Net in a variety of real-world circumstances through thorough experimentation and assessment. Intricate feature extraction, down-sampling, and up-sampling are all part of the network's design in order to produce high-quality segmentation results. The study demonstrates comparative evaluations against classic U-Net and other state-of-the-art models and emphasizes the significance of hyperparameter fine-tuning. The suggested architecture shows excellent performance in terms of accuracy and generalization, demonstrating its promise for a variety of applications. Finally, the problem of semantic segmentation is addressed in a novel way. The experimental findings validate the relevance of the architecture's design decisions and demonstrate its potential to boost computer vision by enhancing segmentation precision and efficiency.

3D Point Cloud Semantic Segmentation Through Functional Data Analysis

AbstractHere, we propose a method for the semantic segmentation of 3D point clouds based on functional data analysis. For each point of a training set, a number of handcrafted features representing the local geometry around it are calculated at different scales, that is, varying the spatial extension of the local analysis. Calculating the scales at small intervals allows each feature to be accurately approximated using a smooth function and, for the problem of semantic segmentation, to be tackled using functional data analysis. We also present a step-wise method to select the optimal features to include in the model based on the calculation of the distance correlation between each feature and the response variable. The algorithm showed promising results when applied to simulated data. When applied to the semantic segmentation of a point cloud of a forested plot, the results proved better than when using a standard multiscale semantic segmentation method. The comparison with two popular deep learning models showed that our proposal requires smaller training samples sizes and that it can compete with these methods in terms of prediction.

Segmentation of structural defects in polymer composite computed tomography images with deep learning models

We investigate appliance of different deep learning models to the problem of semantic segmentation of structural defects in computed tomography images of fiber-reinforced polymer composite material. Specifically, we try to segment porosities and delaminations in a specimen using U-Net and DeepLabv3 neural networks. We find out that complex models struggle to generalize solutions on small data samples that are generally available to individual research teams, whereas smaller models are the right choice for approaching defect segmentation in CT images. Our experiments are based on our own laboratory data, collected with X-ray microtomography and labeled manually for the semantic segmentation task.

Crop Mapping Based on Sentinel-2 Images Using Semantic Segmentation Model of Attention Mechanism.

Using remote sensing images to identify crop plots and estimate crop planting area is an important part of agricultural remote sensing monitoring. High-resolution remote sensing images can provide rich information regarding texture, tone, shape, and spectrum of ground objects. With the advancement of sensor and information technologies, it is now possible to categorize crops with pinpoint accuracy. This study defines crop mapping as a semantic segmentation problem; therefore, a deep learning method is proposed to identify the distribution of corn and soybean using the differences in the spatial and spectral features of crops. The study area is located in the southwest of the Great Lakes in the United States, where corn and soybean cultivation is concentrated. The proposed attention mechanism deep learning model, A2SegNet, was trained and evaluated using three years of Sentinel-2 data, collected between 2019 and 2021. The experimental results show that this method is able to fully extract the spatial and spectral characteristics of crops, and its classification effect is significantly better than that of the baseline method, and it has better classification performance than other deep learning models. We cross verified the trained model on the test sets of different years through transfer learning in both spatiotemporal and spatial dimensions. Proving the effectiveness of the attention mechanism in the process of knowledge transfer, A2SegNet showed better adaptability.