Accurate Semantic Segmentation Research Articles

Semantic segmentation of remote sensing images based on dilated convolution and spatial-channel attention mechanism

The rich context information and multiscale ground information in remote sensing images are crucial to improving the semantic segmentation accuracy. Therefore, we propose a remote sensing image semantic segmentation method that integrates multilevel spatial channel attention and multi-scale dilated convolution, effectively addressing the issue of poor segmentation performance of small target objects in remote sensing images. This method builds a multilevel characteristic fusion structure, combining deep-level semantic characteristics with the details of the shallow levels to generate multiscale feature diagrams. Then, we introduce the dilated convolution of the series combination in each layer of the atrous spatial pyramid pooling structure to reduce the loss of small target information. Finally, using convolutional conditional random field to describe the context information on the space and edges to improve the model’s ability to extract details. We prove the effectiveness of the model on the three public datasets. From the quantitative point of view, we mainly evaluate the four indicators of the model’s F1 score, overall accuracy (OA), Intersection over Union (IoU), and Mean Intersection over Union (MIoU). On GID dataset, F1 score, OA, and MIoU reach 87.27, 87.80, and 77.70, respectively, superior to most mainstream semantic segmentation networks.

Deep semantic segmentation of unmanned aerial vehicle remote sensing images based on fully convolutional neural network

In the era of artificial intelligence and big data, semantic segmentation of images plays a vital role in various fields, such as people’s livelihoods and the military. The accuracy of semantic segmentation results directly affects the subsequent data analysis and intelligent applications. Presently, semantic segmentation of unmanned aerial vehicle (UAV) remote-sensing images is a research hotspot. Compared with manual segmentation and object-based segmentation methods, semantic segmentation methods based on deep learning are efficient and highly accurate segmentation methods. The author has seriously studied the implementation principle and process of the classical deep semantic segmentation model—the fully convolutional neural network (FCN), including convolution and pooling in the encoding stage, deconvolution and upsampling, etc., in the decoding stage. The author has applied the three structures (i.e., FCN-32s, FCN-16s, and FCN-8s) to the UAV remote sensing image dataset AeroScapes. And the results show that the accuracy of vegetation recognition is stable at about 94%. The accuracy of road recognition can reach up to more than 88%. The mean pixel accuracy rate of the whole test dataset is above 91%. Applying full convolution neural network to semantic segmentation of UAV remote sensing images can improve the efficiency and accuracy of semantic segmentation significantly.

Boundary–Inner Disentanglement Enhanced Learning for Point Cloud Semantic Segmentation

In a point cloud semantic segmentation task, misclassification usually appears on the semantic boundary. A few studies have taken the boundary into consideration, but they relied on complex modules for explicit boundary prediction, which greatly increased model complexity. It is challenging to improve the segmentation accuracy of points on the boundary without dependence on additional modules. For every boundary point, this paper divides its neighboring points into different collections, and then measures its entanglement with each collection. A comparison of the measurement results before and after utilizing boundary information in the semantic segmentation network showed that the boundary could enhance the disentanglement between the boundary point and its neighboring points in inner areas, thereby greatly improving the overall accuracy. Therefore, to improve the semantic segmentation accuracy of boundary points, a Boundary–Inner Disentanglement Enhanced Learning (BIDEL) framework with no need for additional modules and learning parameters is proposed, which can maximize feature distinction between the boundary point and its neighboring points in inner areas through a newly defined boundary loss function. Experiments with two classic baselines across three challenging datasets demonstrate the benefits of BIDEL for the semantic boundary. As a general framework, BIDEL can be easily adopted in many existing semantic segmentation networks.

Defense against Adversarial Patch Attacks for Aerial Image Semantic Segmentation by Robust Feature Extraction

Deep learning (DL) models have recently been widely used in UAV aerial image semantic segmentation tasks and have achieved excellent performance. However, DL models are vulnerable to adversarial examples, which bring significant security risks to safety-critical systems. Existing research mainly focuses on solving digital attacks for aerial image semantic segmentation, but adversarial patches with physical attack attributes are more threatening than digital attacks. In this article, we systematically evaluate the threat of adversarial patches on the aerial image semantic segmentation task for the first time. To defend against adversarial patch attacks and obtain accurate semantic segmentation results, we construct a novel robust feature extraction network (RFENet). Based on the characteristics of aerial images and adversarial patches, RFENet designs a limited receptive field mechanism (LRFM), a spatial semantic enhancement module (SSEM), a boundary feature perception module (BFPM) and a global correlation encoder module (GCEM), respectively, to solve adversarial patch attacks from the DL model architecture design level. We discover that semantic features, shape features and global features contained in aerial images can significantly enhance the robustness of the DL model against patch attacks. Extensive experiments on three aerial image benchmark datasets demonstrate that the proposed RFENet has strong resistance to adversarial patch attacks compared with the existing state-of-the-art methods.

A multitask model for realtime fish detection and segmentation based on YOLOv5.

The accuracy of fish farming and real-time monitoring are essential to the development of "intelligent" fish farming. Although the existing instance segmentation networks (such as Maskrcnn) can detect and segment the fish, most of them are not effective in real-time monitoring. In order to improve the accuracy of fish image segmentation and promote the accurate and intelligent development of fish farming industry, this article uses YOLOv5 as the backbone network and object detection branch, combined with semantic segmentation head for real-time fish detection and segmentation. The experiments show that the object detection precision can reach 95.4% and the semantic segmentation accuracy can reach 98.5% with the algorithm structure proposed in this article, based on the golden crucian carp dataset, and 116.6 FPS can be achieved on RTX3060. On the publicly available dataset PASCAL VOC 2007, the object detection precision is 73.8%, the semantic segmentation accuracy is 84.3%, and the speed is up to 120 FPS on RTX3060.

Real-Time Semantic Segmentation of Point Clouds Based on an Attention Mechanism and a Sparse Tensor

A 3D point cloud is one of the main data sources for robot environmental cognition and understanding. Due to the limited computation and memory capacities of the robotic platform, existing semantic segmentation models of 3D point clouds cannot meet the requirements of real-time applications. To solve this problem, a lightweight, fully convolutional network based on an attention mechanism and a sparse tensor is proposed to better balance the accuracy and real-time performance of point cloud semantic segmentation. On the basis of the 3D-Unet structure, a global feature-learning module and a multi-scale feature fusion module are designed. The former improves the ability of features to describe important areas by learning the importance of spatial neighborhoods. The latter realizes the fusion of multi-scale semantic information and suppresses useless information through the task correlation learning of multi-scale features. Additionally, to efficiently process the large-scale point clouds acquired in real time, a sparse tensor-based implementation method is introduced. It is able to reduce unnecessary computation according to the sparsity of the 3D point cloud. As demonstrated by the results of experiments conducted with the SemanticKITTI and NuScenes datasets, our model improves the mIoU metric by 6.4% and 5%, respectively, over existing models that can be applied in real time. Our model is a lightweight model that can meet the requirements of real-time applications.

Deep Dual-Resolution Networks for Real-Time and Accurate Semantic Segmentation of Traffic Scenes

Using light-weight architectures or reasoning on low-resolution images, recent methods realize very fast scene parsing, even running at more than 100 FPS on a single GPU. However, there is still a significant gap in performance between these real-time methods and the models based on dilation backbones. To this end, we proposed a family of deep dual-resolution networks (DDRNets) for real-time and accurate semantic segmentation, which consist of deep dual-resolution backbones and enhanced low-resolution contextual information extractors. The two deep branches and multiple bilateral fusions of backbones generate higher quality details compared to existing two-pathway methods. The enhanced contextual information extractor named Deep Aggregation Pyramid Pooling Module (DAPPM) enlarges effective receptive fields and fuses multi-scale context based on low-resolution feature maps with little time cost. Our method achieves a new state-of-the-art trade-off between accuracy and speed on both Cityscapes and CamVid dataset. For the input of full resolution, on a single 2080Ti GPU without hardware acceleration, DDRNet-23-slim yields 77.4% mIoU at 102 FPS on Cityscapes test set and 74.7% mIoU at 230 FPS on CamVid test set. With widely used test augmentation, our method is superior to most state-of-the-art models and requires much less computation. Codes and trained models are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ydhongHIT/DDRNet</uri> .

MAFF-HRNet: Multi-Attention Feature Fusion HRNet for Building Segmentation in Remote Sensing Images

Built-up areas and buildings are two main targets in remote sensing research; consequently, automatic extraction of built-up areas and buildings has attracted extensive attention. This task is usually difficult because of boundary blur, object occlusion, and intra-class inconsistency. In this paper, we propose the multi-attention feature fusion HRNet, MAFF-HRNet, which can retain more detailed features to achieve accurate semantic segmentation. The design of a pyramidal feature attention (PFA) hierarchy enhances the multilevel semantic representation of the model. In addition, we develop a mixed convolutional attention (MCA) block, which increases the capture range of receptive fields and overcomes the problem of intra-class inconsistency. To alleviate interference due to occlusion, a multiscale attention feature aggregation (MAFA) block is also proposed to enhance the restoration of the final prediction map. Our approach was systematically tested on the WHU (Wuhan University) Building Dataset and the Massachusetts Buildings Dataset. Compared with other advanced semantic segmentation models, our model achieved the best IoU results of 91.69% and 68.32%, respectively. To further evaluate the application significance of the proposed model, we migrated a pretrained model based on the World-Cover Dataset training to the Gaofen 16 m dataset for testing. Quantitative and qualitative experiments show that our model can accurately segment buildings and built-up areas from remote sensing images.

Processing of micro-CT images of granodiorite rock samples using convolutional neural networks (CNN), Part II: Semantic segmentation using a 2.5D CNN

X-ray computed tomography (XCT) is routinely used in geosciences for the purpose of rock characterisation. High-quality micro-CT images are successfully used for fracture characterisation, as well as analysis of grains and pores. In contrast, the use of XCT for mineral identification is uncommon and often ineffective. Implementation of micro-CT imaging techniques for mineral identification is affected by the accuracy and precision of the image segmentation results. Conventional segmentation methods such as thresholding, watershed, and active contouring are user-biased and do not provide the robust distinction between various heavy accessory minerals in granite rocks. Heavy ore minerals such as pyrite, chalcopyrite, molybdenite, and ilmenite are readily recognised in grey-scale micro-CT images because of their high attenuation coefficient, but further differentiation between these minerals using only traditional segmentation methods is challenging. Conversely, deep convolutional neural networks (CNNs) are fully self-trained, and they have demonstrated accurate semantic segmentation results for rock images. However, the application of CNN semantic segmentation for igneous rocks is not well documented. In this research, the U-Net 2.5D CNN was deployed to train the neural network on a combination of high-resolution micro-CT and mineral liberation analysis (MLA) images to identify different accessory mineral regions of interest (ROIs). The image segmentation results were assessed using MLA and SEM data, and the accuracy of segmentation was found to be greater than 97%. The methodology developed in this study can be extended to map the mineralogy of granite samples unseen by the CNN to further validate the robustness of the approach.

Multi‐directional feature refinement network for real‐time semantic segmentation in urban street scenes

AbstractEfficient and accurate semantic segmentation is crucial for autonomous driving scene parsing. Capturing detailed information and semantic information efficiently through two‐branch networks has been widely utilised in real‐time semantic segmentation. This study proposes a network named MRFNet based on two‐branch strategy to solve the problem of accuracy and speed of segmentation in urban scenes. Many real‐time networks do not comprehensively consider contextual information from sub‐regions in different directions and at different scales. To handle this problem, a Multi‐directional Feature Refinement Module (MFRM) which has three sub‐paths to capture information at different scales and directions is proposed. And MFRM reduces computation by using strip pooling and dilated convolution operations. In particular, the authors propose a Feature Cross‐guide Aggregation Module to aggregate detailed information and contextual information through the mutual guidance of detailed information and semantic information. This module guides the extraction of feature maps in a more precise direction. Experiments on Cityscapes and CamVid datasets demonstrate the effectiveness of our method by achieving a balance between accuracy and inference speed. Specially, on single 1080Ti GPU, our method yields 78.9% mean intersection over union (mIoU) and 77.4% mIoU at speed of 144.5 frames per second (FPS) and 120.8 FPS on Cityscapes and CamVid datasets respectively.

Development of a compressed FCN architecture for semantic segmentation using Particle Swarm Optimization.

Researchers have adapted the conventional deep learning classification networks to generate Fully Conventional Networks (FCN) for carrying out accurate semantic segmentation. However, such models are expensive both in terms of storage and inference time and not readily employable on edge devices. In this paper, a compressed version of VGG16-based Fully Convolution Network (FCN) has been developed using Particle Swarm Optimization. It has been shown that the developed model can offer tremendous saving in storage space and also faster inference time, and can be implemented on edge devices. The efficacy of the proposed approach has been tested using potato late blight leaf images from publicly available PlantVillage dataset, street scene image dataset and lungs X-Ray dataset and it has been shown that it approaches the accuracies offered by standard FCN even after 851× compression.

Semantic segmentation of surface defects of smooth parts based on deep convolutional neural networks

Machine vision plays an increasingly important role in industrial product quality detection. During processing, scratches, dents and other defects are inevitable on the surface of a smooth part. Although surface defects do not affect the overall performance of the product, their existence is unacceptable when a perfect product is required. The surface defect detection method based on machine vision and deep convolutional neural networks overcomes, to a certain extent, the problem of low detection efficiency, high false detection and missing detection rates in the traditional detection method. In this paper, a multistream semantic segmentation neural network is proposed to identify defects on smooth parts. Taking a seatbelt buckle as an example, the scratch and crush defects on the surface are classified. The network takes DeepLabV3+ as the framework and three types of image stream as the input of the network. In the backbone feature extraction network, the Xception structure is improved to MobilenetV2 and the convolutional block attention module (CBAM) is introduced into the decoding network, which improves the operational efficiency and accuracy. Compared with other classical networks, this network demonstrates good performance in the image dataset of the seatbelt buckle and realises fast and accurate semantic segmentation and classification of surface defects. The evaluation results of the network model have been significantly improved.

SEMANTIC SEGMENTATION OF UAV IMAGES BASED ON U-NET IN URBAN AREA

Abstract. Semantic segmentation of aerial data has been one of the leading researches in the field of photogrammetry, remote sensing, and computer vision in recent years. Many applications, including airborne mapping of urban scenes, object positioning in aerial images, automatic extraction of buildings from remote sensing or high-resolution aerial images, etc., require accurate and efficient segmentation algorithms. According to the high potential of deep learning algorithms in the classification of complex scenes, this paper aims to train a deep learning model to evaluate the semantic segmentation accuracy of UAV-based images in urban areas. The proposed method implements a deep learning framework based on the U-Net encoder-decoder architecture, which extracts and classifies features through layers of convolution, max pooling, activation, and concatenation in an end-to-end process. The obtained results compare with two traditional machine learning models, Random Forest (RF) and Multi-Layer Perceptron (MLP). They rely on two steps that involve extracting features and classifying images. In this study, the experiments are performed on the UAVid2020 semantic segmentation dataset from the ISPRS database. Results show the effectiveness of the proposed deep learning framework, so that the U-Net architecture achieved the best results with 75.15% overall accuracy, compared to RF and MLP algorithms with 52.51% and 54.65% overall accuracy, respectively.

IMPROVING SEMANTIC SEGMENTATION OF HIGH-RESOLUTION REMOTE SENSING IMAGES USING WASSERSTEIN GENERATIVE ADVERSARIAL NETWORK

Abstract. Semantic segmentation of remote sensing images with high spatial resolution has many applications in a wide range of problems in this field. In recent years, the use of advanced techniques based on fully convolutional neural networks have achieved high and impressive accuracies. However, the labels of different classes are estimated independently in this method. In general, the segmentation effect is too coarse to take the relationship between pixels into account. On the other hand, due to the use of convolution filters and limitations of calculations, the field of view information of these filters will be limited in deep layers. In this study, a method based on generative adversarial network (GAN) is proposed to strengthen spatial vicinity in the output segmentation map. The segmentation model receive assistance from the GAN model in the form of a higher order potential loss. Furthermore, for better stability and performance in model training the Wasserstein GAN is used for optimization of the model. We successfully show an increase in semantic segmentation accuracy using the challenging ISPRS Vaihingen benchmark dataset.

Semi-Supervised Remote Sensing Image Semantic Segmentation Method Based on Deep Learning

In this paper, we study the semi-supervised semantic segmentation problem via limited labeled samples and a large number of unlabeled samples. We propose a self-learning semi-supervised approach for the semantic segmentation of high-resolution remote sensing images. Our approach uses two networks (UNet and DeepLabV3) to predict the labels of the same unlabeled sample, and the pseudo labels samples with high prediction consistency are added to the training samples to improve the accuracy of semantic segmentation under the condition of limited labeled samples. Our method expands training data samples by using unlabeled data samples with pseudo labels. In order to verify the effectiveness of the proposed method, some experiments were conducted on the improved ISPRS Vaihingen 2D Semantic Labeling dataset using the method that we proposed. We focus on the extraction of forest and vegetation information and focus on the impact of a large number of unlabeled samples on the precision of semantic segmentation, we combine water, surface, buildings, cars, and background into one category and named others, and we call the changed dataset the improved ISPRS Vaihingen dataset. The experimental results show that the proposed method can effectively improve the semantic segmentation accuracy of high-scoring remote sensing images with limited samples than common deep semi-supervised learning.

ST-Unet: Swin Transformer boosted U-Net with Cross-Layer Feature Enhancement for medical image segmentation

Medical image segmentation is an essential task in clinical diagnosis and case analysis. Most of the existing methods are based on U-shaped convolutional neural networks (CNNs), and one of disadvantages is that the long-term dependencies and global contextual connections cannot be effectively established, which results in inaccuracy segmentation. For fully using low-level features to enhance global features and reduce the semantic gap between encoding and decoding stages, we propose a novel Swin Transformer boosted U-Net (ST-Unet) for medical image processing in this paper, in which Swin Transformer and CNNs are used as encoder and decoder respectively. Then a novel Cross-Layer Feature Enhancement (CLFE) module is proposed to realize cross-layer feature learning, and a Spatial and Channel Squeeze & Excitation module is adopted to highlight the saliency of specific regions. Finally, we learn the features fused by the CLFE module through CNNs to recover low-level features and localize local features for realizing more accurate semantic segmentation. Experiments on widely used public datasets Synapse and ISIC 2018 prove that our proposed ST-Unet can achieve 78.86 of dice and 0.9243 of recall performance, outperforming most current medical image segmentation methods.

SBSS: Stacking-Based Semantic Segmentation Framework for Very High-Resolution Remote Sensing Image

Semantic segmentation of very high-resolution (VHR) remote sensing images is a fundamental task for many applications. However, large variations in the scales of objects in those VHR images pose a challenge for performing accurate semantic segmentation. Existing semantic segmentation networks are able to analyze an input image at up to four resizing scales, but this may be insufficient given the diversity of object scales. Therefore, multiscale (MS) test-time data augmentation is often used in practice to obtain more accurate segmentation results, which makes equal use of the segmentation results obtained at the different resizing scales. However, it was found in this study that different classes of objects had their preferred resizing scale for more accurate semantic segmentation. Based on this behavior, a stacking-based semantic segmentation (SBSS) framework is proposed to improve the segmentation results by learning this behavior, which contains a learnable error correction module (ECM) for segmentation result fusion and an error correction scheme (ECS) for computational complexity control. Two ECS, i.e., ECS-MS and ECS-single-scale (SS), are proposed and investigated in this study. The floating-point operations (Flops) required for ECS-MS and ECS-SS are similar to the commonly used MS test and the SS test, respectively. Extensive experiments on four datasets (i.e., Cityscapes, UAVid, LoveDA, and Potsdam) show that SBSS is an effective and flexible framework. It achieved higher accuracy than MS when using ECS-MS, and similar accuracy as SS with a quarter of the memory footprint when using ECS-SS.

Category-Level Assignment for Cross-Domain Semantic Segmentation in Remote Sensing Images

Deep learning-based semantic segmentation has made great progress in understanding very-high-resolution (VHR) remote sensing images (RSIs). However, large-scale applications are still limited. The main reason is that diverse imaging modes and geographical differences make it difficult to transfer a model trained in the source domain to the target domain. To solve this problem, unsupervised domain adaptation (UDA) for VHR RSIs has received some attention, but the accuracy of cross-domain semantic segmentation still needs to be improved. Currently, one reasonable proposal for improving accuracy is to take a close look at the category-level information. In this paper, we reveal an integer programming mechanism for modeling the category-level relationship between the source and target domains. The mechanism is based on the solution of the assignment problem, and thus, the proposed method is called category-level assignment for UDA (ClA-UDA). In ClA-UDA, a category-level assignment problem with additional constraints is defined for UDA tasks, and the solution is provided. Based on the solution, an assignment-based image-to-image transferring algorithm (AIT) is first proposed to transfer the source-domain images based on the style of the target-domain images. AIT minimizes a weighted discrepancy, and provides an analytical solution for the transfer. Two assignment-based alignment losses are then introduced to align the source and target domains based on the category-level relationship in a concise way. To validate the performance of ClA-UDA, three VHR remote sensing image datasets are employed, and six UDA tasks are designed. Extensive experiments are conducted, and the results demonstrate the superiority of ClA-UDA compared to the existing methods.

Reaction-diffusion model applied to enhancing U-Net accuracy for semantic image segmentation

&lt;p style='text-indent:20px;'&gt;In our current paper we are introducing a new method to enhance semantic image segmentation accuracy of a U-Net neural network model by integrating it with a mathematical model based on reaction-diffusion equations.&lt;/p&gt;&lt;p style='text-indent:20px;'&gt;The methods currently used for semantic image segmentation, including U-Net neural networks, are processing images as blocks of pixels in which the boundaries, the colors and patterns are all mixed together as inputs to the transformations that take place inside the layers of the convolutional neural networks. In our method we are modifying the architecture of a U-Net network and introduce a new data input feed in parallel to the image feed that needs to be segmented. The new input feed is mathematically extracted from the input image and contains the edges (shape) information of the image to be processed. The new input feed it's used during the U-Net decoding phase in order to help shape more precisely the up-scaled output edges, thus leading to improved accuracy performance of the network.&lt;/p&gt;&lt;p style='text-indent:20px;'&gt;Introducing the parallel feed shows an improvement of accuracy metrics up to 4% (if compared to the U-Net model) and has a limited impact on computational resources consumed at training, because we are only adding a small number of new parameters to be calculated.&lt;/p&gt;

Lightweight Attention Network for Very High-Resolution Image Semantic Segmentation

Semantic segmentation is one of the most challenging tasks for very high resolution (VHR) remote sensing applications. Deep convolutional neural networks (CNN) based on the attention mechanism have shown outstanding performance in VHR remote sensing images semantic segmentation. However, existing attention-guided methods require the estimation of a large number of parameters that are affected by the limited number of available labeled samples that results in underperforming segmentation results. In this paper, we propose a multi-scale feature fusion lightweight model (MSFFL) to greatly reduce the number of parameters and improve the accuracy of semantic segmentation. In this model, two parallel enhanced attention modules, i.e., the spatial attention module (SAM) and the channel attention module (CAM) are designed by introducing encoding position information. Then a covariance calculation strategy is adopted to recalibrate the generated attention maps. The integration of enhanced attention modules into the proposed lightweight module results in an efficient lightweight attention network (LiANet). The performance of the proposed LiANet is assessed on two benchmark datasets. Experimental results demonstrate that LiANet can achieve promising performance with a small number of parameters.