Problem Of Semantic Segmentation Research Articles

Cloud-to-Ground and Intra-Cloud Nowcasting Lightning Using a Semantic Segmentation Deep Learning Network

Weather forecasting requires a comprehensive analysis of various types of meteorology data, and with the wide application of deep learning in various fields, deep learning has proved to have powerful feature extraction capabilities. In this paper, from the viewpoint of an image semantic segmentation problem, a deep learning framework based on semantic segmentation is proposed to nowcast Cloud-to-Ground and Intra-Cloud lightning simultaneously within an hour. First, a dataset with spatiotemporal features is constructed using radar echo reflectivity data and lightning observation data. More specifically, each sample in the dataset consists of the past half hour of observations. Then, a Light3DUnet is presented based on 3D U-Net. The three-dimensional structured network can extract spatiotemporal features, and the encoder–decoder structure and the skip connection can handle small targets and recover more details. Due to the sparsity of lightning observations, a weighted cross-loss function was used to evaluate network performance. Finally, Light3DUnet was trained using the dataset to predict Cloud-to-Ground and Intra-Cloud lightning in the next hour. We evaluated the prediction performance of the network using a real-world dataset from middle China. The results show that Light3DUnet has a good ability to nowcast IC and CG lightning. Meanwhile, due to the spatial position coupling of IC and CG on a two-dimensional plane, predictions from summing the probabilistic prediction matrices will be augmented to obtain accurate prediction results for total flashes.

Application of artificial intelligence in ophthalmology for solving the problem of semantic segmentation of fundus images

The paper presents main aspects of the application of artificial intelligence in ophthalmology for the diagnosis and treatment of eye diseases, considering the problem of semantic segmentation of fundus images as an example. The classic approach to semantic segmentation on the basis of textural features is compared to the proposed approach based on neural networks. Basic problems of using the neural network approach in biomedicine are formulated. We propose a new method for selecting an optimal zone of laser exposure for laser coagulation based on two neural networks. The first network is used for detecting anatomical objects in the fundus and the second one is used for selecting the area of macular edema. The region of interest is formed from the edema area while taking into account the location of anatomical objects in it. A comparative analysis of sev-eral architectures of neural networks for solving the problem of selecting the edema area is carried out. The best results in the selection of the edema area are shown by the neural network architecture of Unet++.

Saliency as Pseudo-Pixel Supervision for Weakly and Semi-Supervised Semantic Segmentation.

Existing studies on semantic segmentation using image-level weak supervision have several limitations, including sparse object coverage, inaccurate object boundaries, and co-occurring pixels from non-target objects. To overcome these challenges, we propose a novel framework, an improved version of Explicit Pseudo-pixel Supervision (EPS++), which learns from pixel-level feedback by combining two types of weak supervision. Specifically, the image-level label provides the object identity via the localization map, and the saliency map from an off-the-shelf saliency detection model offers rich object boundaries. We devise a joint training strategy to fully utilize the complementary relationship between disparate information. Notably, we suggest an Inconsistent Region Drop (IRD) strategy, which effectively handles errors in saliency maps using fewer hyper-parameters than EPS. Our method can obtain accurate object boundaries and discard co-occurring pixels, significantly improving the quality of pseudo-masks. Experimental results show that EPS++ effectively resolves the key challenges of semantic segmentation using weak supervision, resulting in new state-of-the-art performances on three benchmark datasets in a weakly supervised semantic segmentation setting. Furthermore, we show that the proposed method can be extended to solve the semi-supervised semantic segmentation problem using image-level weak supervision. Surprisingly, the proposed model also achieves new state-of-the-art performances on two popular benchmark datasets.

Enhancing Semantic Segmentation: Design and Analysis of Improved U-Net Based Deep Convolutional Neural Networks

In this research, we provide a state-of-the-art method for semantic segmentation that makes use of a modified version of the U-Net architecture, which is itself based on deep convolutional neural networks (CNNs). This research delves into the ins and outs of this cutting-edge approach to semantic segmentation in an effort to boost its precision and productivity. To perform semantic segmentation, a crucial operation in computer vision, each pixel in an image must be assigned to one of many predefined item classes. The proposed Improved U-Net architecture makes use of deep CNNs to efficiently capture complex spatial characteristics while preserving associated context. The study illustrates the efficacy of the Improved U-Net in a variety of real-world circumstances through thorough experimentation and assessment. Intricate feature extraction, down-sampling, and up-sampling are all part of the network's design in order to produce high-quality segmentation results. The study demonstrates comparative evaluations against classic U-Net and other state-of-the-art models and emphasizes the significance of hyperparameter fine-tuning. The suggested architecture shows excellent performance in terms of accuracy and generalization, demonstrating its promise for a variety of applications. Finally, the problem of semantic segmentation is addressed in a novel way. The experimental findings validate the relevance of the architecture's design decisions and demonstrate its potential to boost computer vision by enhancing segmentation precision and efficiency.

3D Point Cloud Semantic Segmentation Through Functional Data Analysis

Here, we propose a method for the semantic segmentation of 3D point clouds based on functional data analysis. For each point of a training set, a number of handcrafted features representing the local geometry around it are calculated at different scales, that is, varying the spatial extension of the local analysis. Calculating the scales at small intervals allows each feature to be accurately approximated using a smooth function and, for the problem of semantic segmentation, to be tackled using functional data analysis. We also present a step-wise method to select the optimal features to include in the model based on the calculation of the distance correlation between each feature and the response variable. The algorithm showed promising results when applied to simulated data. When applied to the semantic segmentation of a point cloud of a forested plot, the results proved better than when using a standard multiscale semantic segmentation method. The comparison with two popular deep learning models showed that our proposal requires smaller training samples sizes and that it can compete with these methods in terms of prediction.

Segmentation of structural defects in polymer composite computed tomography images with deep learning models

We investigate appliance of different deep learning models to the problem of semantic segmentation of structural defects in computed tomography images of fiber-reinforced polymer composite material. Specifically, we try to segment porosities and delaminations in a specimen using U-Net and DeepLabv3 neural networks. We find out that complex models struggle to generalize solutions on small data samples that are generally available to individual research teams, whereas smaller models are the right choice for approaching defect segmentation in CT images. Our experiments are based on our own laboratory data, collected with X-ray microtomography and labeled manually for the semantic segmentation task.

Crop Mapping Based on Sentinel-2 Images Using Semantic Segmentation Model of Attention Mechanism.

Using remote sensing images to identify crop plots and estimate crop planting area is an important part of agricultural remote sensing monitoring. High-resolution remote sensing images can provide rich information regarding texture, tone, shape, and spectrum of ground objects. With the advancement of sensor and information technologies, it is now possible to categorize crops with pinpoint accuracy. This study defines crop mapping as a semantic segmentation problem; therefore, a deep learning method is proposed to identify the distribution of corn and soybean using the differences in the spatial and spectral features of crops. The study area is located in the southwest of the Great Lakes in the United States, where corn and soybean cultivation is concentrated. The proposed attention mechanism deep learning model, A2SegNet, was trained and evaluated using three years of Sentinel-2 data, collected between 2019 and 2021. The experimental results show that this method is able to fully extract the spatial and spectral characteristics of crops, and its classification effect is significantly better than that of the baseline method, and it has better classification performance than other deep learning models. We cross verified the trained model on the test sets of different years through transfer learning in both spatiotemporal and spatial dimensions. Proving the effectiveness of the attention mechanism in the process of knowledge transfer, A2SegNet showed better adaptability.

Crop classification in high-resolution remote sensing images based on multi-scale feature fusion semantic segmentation model.

The great success of deep learning in the field of computer vision provides a development opportunity for intelligent information extraction of remote sensing images. In the field of agriculture, a large number of deep convolutional neural networks have been applied to crop spatial distribution recognition. In this paper, crop mapping is defined as a semantic segmentation problem, and a multi-scale feature fusion semantic segmentation model MSSNet is proposed for crop recognition, aiming at the key problem that multi-scale neural networks can learn multiple features under different sensitivity fields to improve classification accuracy and fine-grained image classification. Firstly, the network uses multi-branch asymmetric convolution and dilated convolution. Each branch concatenates conventional convolution with convolution nuclei of different sizes with dilated convolution with different expansion coefficients. Then, the features extracted from each branch are spliced to achieve multi-scale feature fusion. Finally, a skip connection is used to combine low-level features from the shallow network with abstract features from the deep network to further enrich the semantic information. In the experiment of crop classification using Sentinel-2 remote sensing image, it was found that the method made full use of spectral and spatial characteristics of crop, achieved good recognition effect. The output crop classification mapping was better in plot segmentation and edge characterization of ground objects. This study can provide a good reference for high-precision crop mapping and field plot extraction, and at the same time, avoid excessive data acquisition and processing.

Handling Open-Set Noise and Novel Target Recognition in Domain Adaptive Semantic Segmentation.

This paper studies a practical domain adaptive (DA) semantic segmentation problem where only pseudo-labeled target data is accessible through a black-box model. Due to the domain gap and label shift between two domains, pseudo-labeled target data contains mixed closed-set and open-set label noises. In this paper, we propose a simplex noise transition matrix (SimT) to model the mixed noise distributions in DA semantic segmentation, and leverage SimT to handle open-set label noise and enable novel target recognition. When handling open-set noises, we formulate the problem as estimation of SimT. By exploiting computational geometry analysis and properties of segmentation, we design four complementary regularizers, i.e., volume regularization, anchor guidance, convex guarantee, and semantic constraint, to approximate the true SimT. Specifically, volume regularization minimizes the volume of simplex formed by rows of the non-square SimT, ensuring outputs of model to fit into the ground truth label distribution. To compensate for the lack of open-set knowledge, anchor guidance, convex guarantee, and semantic constraint are devised to enable the modeling of open-set noise distribution. The estimated SimT is utilized to correct noise issues in pseudo labels and promote the generalization ability of segmentation model on target domain data. In the task of novel target recognition, we first propose closed-to-open label correction (C2OLC) to explicitly derive the supervision signal for open-set classes by exploiting the estimated SimT, and then advance a semantic relation (SR) loss that harnesses the inter-class relation to facilitate the open-set class sample recognition in target domain. Extensive experimental results demonstrate that the proposed SimT can be flexibly plugged into existing DA methods to boost both closed-set and open-set class performance.

Pixel-wise phase unwrapping of fringe projection profilometry based on deep learning

Phase unwrapping is a crucial step in high-precision optical metrology, aimed at retrieving absolute phases from wrapped phases. This paper presents a deep learning-based phase unwrapping method that enhances the conventional three-wavelength heterodyne method and achieves pixel-wise phase unwrapping using a wrapped phase map and two fringe patterns. The proposed method formulates the phase unwrapping task as a semantic segmentation problem that infers an absolute fringe order for each wrapped phase pixel. The verification results demonstrate that the method can accurately measure surfaces with complex topologies. Under low noise conditions, the proposed method achieves performance comparable to the three-wavelength heterodyne method, with similarity rates between the fringe order maps obtained by the two methods of up to 0.99; additionally, the method exhibits superb resistance to severe noise. Moreover, the proposed method is more efficient in terms of fringe pattern efficiency by at least 44.44%.

SMBCNet: A Transformer-Based Approach for Change Detection in Remote Sensing Images through Semantic Segmentation

Remote sensing change detection (RSCD) is crucial for our understanding of the dynamic pattern of the Earth’s surface and human influence. Recently, transformer-based methodologies have advanced from their powerful global modeling capabilities in RSCD tasks. Nevertheless, they remain under excessive parameterization, which continues to be severely constrained by time and computation resources. Here, we present a transformer-based RSCD model called the Segmentation Multi-Branch Change Detection Network (SMBCNet). Our proposed approach combines a hierarchically structured transformer encoder with a cross-scale enhancement module (CEM) to extract global information with lower complexity. To account for the diverse nature of changes, we introduce a plug-and-play multi-branch change fusion module (MCFM) that integrates temporal features. Within this module, we transform the change detection task into a semantic segmentation problem. Moreover, we identify the Temporal Feature Aggregation Module (TFAM) to facilitate integrating features from diverse spatial scales. These results demonstrate that semantic segmentation is an effective solution to change detection (CD) problems in remote sensing images.

Exploratory Inference Learning for Scribble Supervised Semantic Segmentation

Scribble supervised semantic segmentation has achieved great advances in pseudo label exploitation, yet suffers insufficient label exploration for the mass of unannotated regions. In this work, we propose a novel exploratory inference learning (EIL) framework, which facilitates efficient probing on unlabeled pixels and promotes selecting confident candidates for boosting the evolved segmentation. The exploration of unannotated regions is formulated as an iterative decision-making process, where a policy searcher learns to infer in the unknown space and the reward to the exploratory policy is based on a contrastive measurement of candidates. In particular, we devise the contrastive reward with the intra-class attraction and the inter-class repulsion in the feature space w.r.t the pseudo labels. The unlabeled exploration and the labeled exploitation are jointly balanced to improve the segmentation, and framed in a close-looping end-to-end network. Comprehensive evaluations on the benchmark datasets (PASCAL VOC 2012 and PASCAL Context) demonstrate the superiority of our proposed EIL when compared with other state-of-the-art methods for the scribble-supervised semantic segmentation problem.

A semantic segmentation scheme for night driving improved by irregular convolution.

In order to solve the poor performance of real-time semantic segmentation of night road conditions in video images due to insufficient light and motion blur, this study proposes a scheme: a fuzzy information complementation strategy based on generative models and a network that fuses different intermediate layer outputs to complement spatial semantics which also embeds irregular convolutional attention modules for fine extraction of motion target boundaries. First, DeblurGan is used to generate information to fix the lost semantics in the original image; then, the outputs of different intermediate layers are taken out, assigned different weight scaling factors, and fused; finally, the irregular convolutional attention with the best effect is selected. The scheme achieves Global Accuracy of 89.1% Mean and IOU 94.2% on the night driving dataset of this experiment, which exceeds the best performance of DeepLabv3 by 1.3 and 7.2%, and achieves an Accuracy of 83.0% on the small volume label (Moveable). The experimental results demonstrate that the solution can effectively cope with various problems faced by night driving and enhance the model's perception. It also provides a technical reference for the semantic segmentation problem of vehicles driving in the nighttime environment.

Large-scale Unsupervised Semantic Segmentation.

Empowered by large datasets, e.g., ImageNet and MS COCO, unsupervised learning on large-scale data has enabled significant advances for classification tasks. However, whether the large-scale unsupervised semantic segmentation can be achieved remains unknown. There are two major challenges: i) we need a large-scale benchmark for assessing algorithms; ii) we need to develop methods to simultaneously learn category and shape representation in an unsupervised manner. In this work, we propose a new problem of large-scale unsupervised semantic segmentation (LUSS) with a newly created benchmark dataset to help the research progress. Building on the ImageNet dataset, we propose the ImageNet-Sdataset with 1.2 million training images and 50k high-quality semantic segmentation annotations for evaluation. Our benchmark has a high data diversity and a clear task objective. We also present a simple yet effective method that works surprisingly well for LUSS. In addition, we benchmark related un/weakly/fully supervised methods accordingly, identifying the challenges and possible directions of LUSS. The benchmark and source code is publicly available at https://github.com/LUSSeg.

2D Semantic Segmentation: Recent Developments and Future Directions

Semantic segmentation is a critical task in computer vision that aims to assign each pixel in an image a corresponding label on the basis of its semantic content. This task is commonly referred to as dense labeling because it requires pixel-level classification of the image. The research area of semantic segmentation is vast and has achieved critical advances in recent years. Deep learning architectures in particular have shown remarkable performance in generating high-level, hierarchical, and semantic features from images. Among these architectures, convolutional neural networks have been widely used to address semantic segmentation problems. This work aims to review and analyze recent technological developments in image semantic segmentation. It provides an overview of traditional and deep-learning-based approaches and analyzes their structural characteristics, strengths, and limitations. Specifically, it focuses on technical developments in deep-learning-based 2D semantic segmentation methods proposed over the past decade and discusses current challenges in semantic segmentation. The future development direction of semantic segmentation and the potential research areas that need further exploration are also examined.

Cross-scale Graph Interaction Network for Semantic Segmentation of Remote Sensing Images

Semantic segmentation of remote sensing (RS) images plays a vital role in a variety of fields, including urban planning, natural disaster monitoring, and land resource management. Due to the complexity and low resolution of RS images, many approaches have been proposed to handle the related task. However, these previously developed approaches dedicate to contextual interaction but ignore the cross-scale semantic correlation and multi-scale boundary information. Therefore, we propose a Cross-scale Graph Interaction Network (CGIN) to address semantic segmentation problems of RS images, which consists of a semantic branch and a boundary branch. In the semantic branch, we first apply atrous convolution to extract multi-scale semantic features of RS images. Particularly, based on the multi-scale semantic features, a Cross-scale Graph Interaction (CGI) module is introduced, which establishes cross-scale graph structures and performs adaptive graph reasoning to capture the cross-scale semantic correlation of RS objects. In the boundary branch, we propose a Multi-scale Boundary Feature Extraction (MBFE) module that utilizes atrous convolutions with different dilation rates to extract multi-scale boundary features. Finally, to address the problem of sparse boundary pixels in the fusion process of the two branches, we propose a Multi-scale Similarity-guided Aggregation (MSA) module by calculating the similarity of semantic features and boundary features at the corresponding scale, which can emphasize the boundary information in semantic features. Our proposed CGIN outperforms state-of-the-art approaches in numerical experiments conducted on two benchmark remote sensing datasets.

SPDC: A SUPER-POINT AND POINT COMBINING BASED DUAL-SCALE CONTRASTIVE LEARNING NETWORK FOR POINT CLOUD SEMANTIC SEGMENTATION

Abstract. Semantic segmentation of point clouds is one of the fundamental tasks of point cloud processing and is the basis for other downstream tasks. Deep learning has become the main method to solve point cloud processing. Most existing 3D deep learning models require large amounts of point cloud data to drive them, but annotating the data requires significant time and economic costs. To address the problem of semantic segmentation requiring large amounts of annotated data for training, this paper proposes a Super-point-level and Point-level Dual-scale Contrast learning network (SPDC). To solve the problem that contrastive learning is difficult to train and feature extraction is not sufficient, we introduce super-point maps to assist the network in feature extraction. We use a pre-trained super-point generation network to convert the original point cloud into a super-point map. A dynamic data augmentation(DDA) module is designed for the super-point maps for super-point-level contrastive learning. We map the extracted super-point-level features back to the original point-level scale and conduct secondary contrastive learning with the original point features. The whole feature extraction network is parameter sharing and to reduce the number of parameters we used the lightweight network DGCNN (encoder)+Self-attention as the backbone network. And we did a few-shot pre-training of the backbone network to make the network converge easily. Analogous to CutMix, we designed a new method for point cloud data augmentation called PointObjectMix (POM). This method solves the sample imbalance problem while preserving the overall characteristics of the objects in the scene. We conducted experiments on the S3DIS dataset and obtained 63.3% mIoU. We have also done a large number of ablation experiments to verify the effectiveness of the modules in our method. Experimental results show that our method outperforms the best-unsupervised network available.

An overview on Meta-learning approaches for Few-shot Weakly-supervised Segmentation

Semantic segmentation is a difficult task in computer vision that have applications in many scenarios, often as a preprocessing step for a tool. Current solutions are based on Deep Neural Networks, which often require a large amount of data for learning a task. Aiming to alleviate the strenuous data-collecting/annotating labor, research fields have emerged in recent years. One of them is Meta-Learning, which tries to improve the generability of models to learn in a restricted amount of data. In this work, we extend a previous paper conducting a more extensive overview of the still under-explored problem of Few-Shot Weakly-supervised Semantic Segmentation. We refined the previous taxonomy and included the review of additional methods, including Few-Shot Segmentation methods that could be adapted to the weak supervision. The goal is to provide a simple organization of literature and highlight aspects observed in the current moment, and be a starting point to foment research on this problem with applications in areas like medical imaging, remote sensing, video segmentation, and others.

Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficient

Metrics optimized in complex machine learning tasks are often selected in an ad-hoc manner. It is unknown how they align with human expert perception. We explore the correlations between established quantitative segmentation quality metrics and qualitative evaluations by professionally trained human raters. Therefore, we conduct psychophysical experiments for two complex biomedical semantic segmentation problems. We discover that current standard metrics and loss functions correlate only moderately with the segmentation quality assessment of experts. Importantly, this effect is particularly pronounced for clinically relevant structures, such as the enhancing tumor compartment of glioma in brain magnetic resonance and grey matter in ultrasound imaging. It is often unclear how to optimize abstract metrics, such as human expert perception, in convolutional neural network (CNN) training. To cope with this challenge, we propose a novel strategy employing techniques of classical statistics to create complementary compound loss functions to better approximate human expert perception. Across all rating experiments, human experts consistently scored computer-generated segmentations better than the human-curated reference labels. Our results, therefore, strongly question many current practices in medical image segmentation and provide meaningful cues for future research.

Few-shot learning for seismic facies segmentation via prototype learning

The mapping of seismic facies from seismic data is considered a multiclass image semantic segmentation problem. Despite the signification progress made by the deep learning methods in seismic prospecting, the dense prediction problem of seismic facies requires large amounts of annotated seismic facies data, which often are unavailable. These valuable labels are only helpful in one model and field due to geologic heterogeneity. To overcome these challenges, we have developed a few-shot seismic facies segmentation model. Few-shot learning has been designed to learn to perform with very few labels and we design reconstructing masked traces as a pretext task for self-supervised learning to obtain a good feature extractor. By these, this model can use all seismic data from different fields, which is different from image data as the texture-based data. With two different seismic data in turn as a meta-training set and a meta-testing set, our model works well in one- and five-shot settings, which means only one label and five labels, respectively.