Semantic Segmentation Research Articles

Techniques for detecting and masking faults in semantic segmentation applications

Semantic segmentation of images is essential for many applications including autonomous driving and modern DNNs now achieve high accuracy. Automotive systems are safety critical systems and in turn they must comply with safety standards, requiring at least hardware fault detection capability. Small embedded applications also require some level of fault tolerance, while operating with a tight power budget. In this paper, we first present a detailed analysis of the effects of faults using Google’s DeepLabV3+ network processing an industrial data-set. Further to that, two techniques to mitigate hardware faults are proposed. The first one is a symptom-based fault detection algorithm shown to detect >99% of critical faults with zero false positives and a compute overhead of 0.2%. The second one is a simpler technique, using a clipped ReLU activation function, to quickly mask over 99% of the critical faults in the activation values.

Combining dual attention mechanism and efficient feature aggregation for road and vehicle segmentation from UAV imagery

Unmanned aerial vehicles (UAVs) have gained significant popularity in recent years due to their ability to capture high-resolution aerial imagery for various applications, including traffic monitoring, urban planning, and disaster management. Accurate road and vehicle segmentation from UAV imagery plays a crucial role in these applications. In this paper, we propose a novel approach combining dual attention mechanisms and efficient multi-layer feature aggregation to enhance the performance of road and vehicle segmentation from UAV imagery. Our approach integrates a spatial attention mechanism and a channel-wise attention mechanism to enable the model to selectively focus on relevant features for segmentation tasks. In conjunction with these attention mechanisms, we introduce an efficient multi-layer feature aggregation method that synthesizes and integrates multi-scale features at different levels of the network, resulting in a more robust and informative feature representation. Our proposed method is evaluated on the UAVid semantic segmentation dataset, showcasing its exceptional performance in comparison to renowned approaches such as U-Net, DeepLabv3+, and SegNet. The experimental results affirm that our approach surpasses these state-of-the-art methods in terms of segmentation accuracy.

Label-Efficient Semantic Segmentation of LiDAR Point Clouds in Adverse Weather Conditions

Label-Efficient Semantic Segmentation of LiDAR Point Clouds in Adverse Weather Conditions

Effect of dataset distribution on automatic road extraction in very high-resolution orthophoto using DeepLab V3+

Road extraction is one of the stages in the map-making process, which has been done manually, takes a long time, and costs a lot. Deep Learning is used to speed up the road extraction process by performing binary semantic segmentation on the image. We propose DeepLab V3+ to produce road extraction from very high-resolution orthophoto for Indonesia study area, which poses many challenges, such as road obstruction by trees, clouds, building shadows, dense traffic, and similarities to rivers and rice fields. We compared the distribution of datasets to obtain the optimal performance of the DeepLab V3+ model in relation to the dataset. The results showed that dataset ratio of 75:10:15 resulted in mean Intersection Over Union (mIoU) of 0.92 and Dice Loss of 0.042. Visually, the results of road extraction are more accurate when compared to the results obtained from different distributions of the dataset.

Cross-Correlation Driven Aerial Image Segmentation: Leveraging Multi-Scale Features and Edge Information

Abstract: Semantic segmentation of remote sensing images is crucial for interpreting these large, rich in information scenes and our study introduces a new method for segmenting remote sensing imagery (RSI) when faced with limited training data and imbalanced classes. Our approach utilizes a unique potential function that merges information from both super pixel segmentation and edge detection. This combination allows the model to effectively analyze features at various scales and reduce the influence of potential errors in super pixel segmentation. Furthermore, the inclusion of edge details extracted via the Sketch token algorithm refines object boundaries, yielding more accurate segmentation results. This work offers a promising solution for achieving reliable interpretation of RSIs in scenarios with limited training data.

Research on Efficient Asymmetric Attention Module for Real-Time Semantic Segmentation Networks in Urban Scenes

Currently, numerous high-precision models have been proposed for semantic segmentation, but the model parameters are large and the segmentation speed is slow. Real-time semantic segmentation for urban scenes necessitates a balance between accuracy, inference speed, and model size. In this paper, we present an efficient solution to this challenge, efficient asymmetric attention module net (EAAMNet) for the semantic segmentation of urban scenes, which adopts an asymmetric encoder–decoder structure. The encoder part of the network utilizes an efficient asymmetric attention module to form the network backbone. In the decoding part, we propose a lightweight multi-feature fusion decoder that can maintain good segmentation accuracy with a small number of parameters. Our extensive evaluations demonstrate that EAAMNet achieves a favorable equilibrium between segmentation efficiency, model parameters, and segmentation accuracy, rendering it highly suitable for real-time semantic segmentation in urban scenes. Remarkably, EAAMNet attains a 73.31% mIoU at 128 fps on Cityscapes and a 69.32% mIoU at 141 fps on CamVid without any pre-training. Compared to state-of-the-art models, our approach not only matches their model parameters but also enhances accuracy and increases speed.

Target recognition and segmentation in turbid water using data from non-turbid conditions: a unified approach and experimental validation

Semantic segmentation of targets in underwater images within turbid water environments presents significant challenges, hindered by factors such as environmental variability, difficulties in acquiring datasets, imprecise data annotation, and the poor robustness of conventional methods. This paper addresses this issue by proposing a novel joint method using deep learning to effectively perform semantic segmentation tasks in turbid environments, with the practical case of efficiently collecting polymetallic nodules in deep-sea while minimizing damage to the seabed environment. Our approach includes a novel data expansion technique and a modified U-net based model. Drawing on the underwater image formation model, we introduce noise to clear water images to simulate images captured under varying degrees of turbidity, thus providing an alternative to the required data. Furthermore, traditional U-net-based modified models have shown limitations in enhancing performance in such tasks. Based on the primary factors underlying image degradation, we propose a new model which incorporates an improved dual-channel encoder. Our method significantly advances the fine segmentation of underwater images in turbid media, and experimental validation demonstrates its effectiveness and superiority under different turbidity conditions. The study provides new technical means for deep-sea resource development, holding broad application prospects and scientific value.

Concrete spalling detection system based on semantic segmentation using deep architectures

This paper presents a method for detecting the location of spalling and assessing the severity level of the spalling in concrete surfaces. The proposed method is constructed based on deep learning architectures and multi-class semantic segmentation. The proposed method can detect each pixel as a non-spalling, a deep-spalling, or a shallow-spalling. The proposed method consists of three different deep learning architectures with several encoders as backbone networks. Both qualitative and quantitative analyses show that the deep learning architecture with a certain encoder network can detect spalling with different severity levels very well. Additionally, the paper proposes a method to analyze the deep spalling areas of concrete to show their severity levels. The performance analysis shows that this approach provides very convincing results with respect to the actual affected spalling areas. The results convey that this paper achieved a higher level of performance for detecting spalling and assessing the severity of the spalling.

Segmentation of ethnic clothing patterns with fusion of multiple attention mechanisms

AbstractTo address the challenges posed by diverse pattern-background elements, intricate details, and complex textures in the semantic segmentation of ethnic clothing patterns, this research introduces a novel semantic segmentation network model called MST-Unet (Mixed Swin Transformer U-net). The proposed model combines a U-shaped network structure with multiple attention mechanisms. The upper layers of the model employ classical convolutional operations, focusing on local relationships in the initial layers containing high-resolution details. In deeper layers, Swin Transformer modules are utilized, capable of efficient feature extraction with smaller spatial dimensions, maintaining performance while reducing computational burden. An attention gate mechanism is integrated into the decoder, contributing to enhanced performance in ethnic clothing pattern segmentation tasks by allowing the model to better capture crucial image features and achieve precise segmentation results. In visual comparisons of segmentation results, our proposed model demonstrates superior performance. The segmentation results exhibit more complete preservation of edge contours and fewer misclassifications in irrelevant regions within the images. In qualitative and quantitative experiments conducted on the ethnic clothing pattern dataset, our model achieves the highest Dice score for segmentation results in all four subclasses of ethnic clothing patterns. The average Dice score of our model reaches an impressive 89.80%, surpassing other algorithms in the same category. When compared to Deeplab_V3+, ResUnet, SwinUnet, and Unet networks, our model outperforms them by 7.72%, 5.09%, 5.05%, and 0.67%.

Community Quality Evaluation for Socially Sustainable Regeneration: A Study Using Multi-Sourced Geospatial Data and AI-Based Image Semantic Segmentation

The Chinese urban regeneration movement underscores a “people-oriented” paradigm, aimed at addressing urban challenges stemming from rapid prior urbanization, while striving for high-quality and sustainable urban development. At the community level, fostering quality through a socially sustainable perspective (SSP) is a pivotal strategy for people-oriented urban regeneration. Nonetheless, explorations of community quality assessments grounded in an SSP have been notably scarce in recent scholarly discourse. This study pioneers a multidimensional quantitative model (MQM) for gauging community quality, leveraging diverse geospatial data sources from the SSP framework. The MQM introduces an evaluative framework with “Patency, Convenience, Comfort, and Safety” as primary indicators, integrating multi-sourced data encompassing the area of interest (AOI), Point of Interest (POI), Weibo check-ins, and Dianping data. The model’s efficacy is demonstrated through a case study in the Gusu district, Suzhou. Furthermore, semantic analysis of the Gusu district’s street view photos validates the MQM results. Our findings reveal the following: (1) AI-based semantic analysis accurately verifies the validity of MQM-generated community quality measurements, establishing its robust applicability with multi-sourced geospatial data; (2) the community quality distribution in Gusu district is notably correlated with the urban fabric, exhibiting lower quality within the ancient town area and higher quality outside it; and (3) communities of varying quality coexist spatially, with high- and low-quality communities overlapping in the same regions. This research pioneers a systematic, holistic methodology for quantitatively measuring community quality, laying the groundwork for informed urban regeneration policies, planning, and place making. The MQM, fortified by multi-sourced geospatial data and AI-based semantic analysis, offers a rigorous foundation for assessing community quality, thereby guiding socially sustainable regeneration initiatives and decision making at the community scale.

Remote intelligent perception system for multi-object detection

IntroductionDuring the last few years, a heightened interest has been shown in classifying scene images depicting diverse robotic environments. The surge in interest can be attributed to significant improvements in visual sensor technology, which has enhanced image analysis capabilities.MethodsAdvances in vision technology have a major impact on the areas of multiple object detection and scene understanding. These tasks are an integral part of a variety of technologies, including integrating scenes in augmented reality, facilitating robot navigation, enabling autonomous driving systems, and improving applications in tourist information. Despite significant strides in visual interpretation, numerous challenges persist, encompassing semantic understanding, occlusion, orientation, insufficient availability of labeled data, uneven illumination including shadows and lighting, variation in direction, and object size and changing background. To overcome these challenges, we proposed an innovative scene recognition framework, which proved to be highly effective and yielded remarkable results. First, we perform preprocessing using kernel convolution on scene data. Second, we perform semantic segmentation using UNet segmentation. Then, we extract features from these segmented data using discrete wavelet transform (DWT), Sobel and Laplacian, and textual (local binary pattern analysis). To recognize the object, we have used deep belief network and then find the object-to-object relation. Finally, AlexNet is used to assign the relevant labels to the scene based on recognized objects in the image.ResultsThe performance of the proposed system was validated using three standard datasets: PASCALVOC-12, Cityscapes, and Caltech 101. The accuracy attained on the PASCALVOC-12 dataset exceeds 96% while achieving a rate of 95.90% on the Cityscapes dataset.DiscussionFurthermore, the model demonstrates a commendable accuracy of 92.2% on the Caltech 101 dataset. This model showcases noteworthy advancements beyond the capabilities of current models.

OffRoadSynth Open Dataset for Semantic Segmentation using Synthetic-Data-Based Weight Initialization for Autonomous UGV in Off-Road Environments

This article concerns the issue of image semantic segmentation for the machine vision system of an autonomous Unmanned Ground Vehicle (UGV) moving in an off-road environment. Determining the meaning (semantics) of the areas visible in the recorded image provides a complete understanding of the scene surrounding the autonomous vehicle. It is crucial for the correct determination of a passable route. Nowadays, semantic segmentation is generally solved using convolutional neural networks (CNN), which can take an image as input and output the segmented image. However, proper training of the neural network requires the use of large amounts of data, which becomes problematic in the situation of low availability of large, dedicated image data sets that consider various off-road situations - driving on various types of roads, surrounded by diverse vegetation and in various weather and light conditions. This study introduces a synthetic image dataset called “OffRoadSynth” to address the training data scarcity for off-road scenarios. It has been shown that pre-training the neural network on this synthetic dataset improves image segmentation accuracy compared to other methods, such as random network weight initialization or using larger, generic datasets. Results suggest that using a smaller but domain-dedicated set of synthetic images to initialize network weights for training on the target real-world dataset may be an effective approach to improving semantic segmentation results of images, including those from off-road environments.

A comprehensive overview of deep learning techniques for 3D point cloud classification and semantic segmentation

A comprehensive overview of deep learning techniques for 3D point cloud classification and semantic segmentation

Hybrid Shunted Transformer embedding UNet for remote sensing image semantic segmentation

Hybrid Shunted Transformer embedding UNet for remote sensing image semantic segmentation

Multi-modal background-aware for defect semantic segmentation with limited data

Multi-modal background-aware for defect semantic segmentation with limited data

Segmentation of Apparent Multi-Defect Images of Concrete Bridges Based on PID Encoder and Multi-Feature Fusion

To address the issue of insufficient deep contextual information mining in the semantic segmentation task of multiple defects in concrete bridges, due to the diversity in texture, shape, and scale of the defects as well as significant differences in the background, we propose the Concrete Bridge Apparent Multi-Defect Segmentation Network (PID-MHENet) based on a PID encoder and multi-feature fusion. PID-MHENet consists of a PID encoder, skip connection, and decoder. The PID encoder adopts a multi-branch structure, including an integral branch and a proportional branch with a “thick and long” design principle and a differential branch with a “thin and short” design principle. The PID Aggregation Enhancement (PAE) combines the detail information of the proportional branch and the semantic information of the differential branch to enhance the fusion of contextual information and, at the same time, introduces the self-learning parameters, which can effectively extract the information of the boundary details of the lesions, the texture, and the background differences. The Multi-Feature Fusion Enhancement Decoding Block (MFEDB) in the decoding stage enhances the information and globally fuses the different feature maps introduced by the three-channel skip connection, which improves the segmentation accuracy of the network for the background similarity and the micro-defects. The experimental results show that the mean Pixel accuracy (mPa) and mean Intersection over Union (mIoU) values of PID-MHENet on the concrete bridge multi-defect semantic segmentation dataset improved by 5.17% and 5.46%, respectively, compared to the UNet network.

ABANet: Attention boundary‐aware network for image segmentation

AbstractDeep learning techniques have attained substantial progress in various face‐related tasks, such as face recognition, face inpainting, and facial expression recognition. To prevent infection or the spread of the virus, wearing of masks in public places has been mandated following the COVID‐19 epidemic, which has led to face occlusion and posed significant challenges for face recognition systems. Most prominent masked face recognition solutions rely on mask segmentation tasks. Therefore, segmentation can be used to mitigate the negative impacts of wearing a mask and improve recognition accuracy. Mask region segmentation suffers from two main problems: there is no standard type of masks that people wear, they come in different colours and designs, and there is no publicly available masked face dataset with appropriate ground truth for the mask region. In order to address these issues, we propose an encoder–decoder framework that utilizes a boundary‐aware attention network combined with a new hybrid loss to provide a map, patch, and pixel‐level supervision. We also introduce a dataset called MFSD, with 11,601 images and 12,758 masked faces for masked face segmentation. Furthermore, we compare the performance of different cutting‐edge deep learning semantic segmentation models on the presented dataset. Experimental results on the MSFD dataset reveal that the suggested approach outperforms state‐of‐the‐art, algorithms with 97.623% accuracy, 93.814% IoU, and 96.817% F1‐score rate. Our dataset of masked faces with mask region labels and source code will be available online.

An Evaluation Model of Urban Green Space Based on Residents’ Physical Activity

Urban green spaces (UGSs) possess a status in improving public health; thus, it is crucial to emphasize the evaluation of UGSs in terms of residents’ physical activity (PA). This study utilizes the semantic segmentation method and Geographic Information System tools to quantify the key values of UGSs, including aesthetic and attractions, natural world experience, nature conservation, encouraging physical activity, cultural value, and social value, which are set as the evaluation indexes to investigate their impacts on residents’ PA based on the six UGSs in Changsha city, Hunan Province, China. The PA-oriented UGS evaluation model is realized through the index optimal combination weights obtained by the Improved Combination Weighting Method of Game Theory, combining the subjective and objective weights from the Uncertainty Analytic Hierarchy Process method and Entropy Weight Method, respectively. By collecting and analyzing the exercise data of residents, we can accurately assess the level of residents’ PA and frequency within various UGSs. The proposed model herein has a positive significance for evaluating the value of public green space in residents’ PA in Changsha city and provides a reference for the construction of an urban green space evaluation model from multiple perspectives in the future.

Unstructured Road Extraction in UAV Images based on Lightweight Model

There is no unified planning standard for unstructured roads, and the morphological structures of these roads are complex and varied. It is important to maintain a balance between accuracy and speed for unstructured road extraction models. Unstructured road extraction algorithms based on deep learning have problems such as high model complexity, high computational cost, and the inability to adapt to current edge computing devices. Therefore, it is best to use lightweight network models. Considering the need for lightweight models and the characteristics of unstructured roads with different pattern shapes, such as blocks and strips, a TMB (Triple Multi-Block) feature extraction module is proposed, and the overall structure of the TMBNet network is described. The TMB module was compared with SS-nbt, Non-bottleneck-1D, and other modules via experiments. The feasibility and effectiveness of the TMB module design were proven through experiments and visualizations. The comparison experiment, using multiple convolution kernel categories, proved that the TMB module can improve the segmentation accuracy of the network. The comparison with different semantic segmentation networks demonstrates that the TMBNet network has advantages in terms of unstructured road extraction.

Combined multi-level context aggregation and attention mechanism method for photovoltaic panel extraction from high resolution remote sensing images

ABSTRACT In the context of global carbon emission reduction, solar photovoltaic (PV) technology is experiencing rapid development. Using high-resolution remote sensing images to accurately obtain PV information over a large region, including location and size, has the advantages of high statistical efficiency and timely data update for the PV energy management. Due to the intra-class diversity of PV panels and the intricate variability in their deployment environments, existing semantic segmentation methods often have problems such as under-segmentation and mis-segmentation. To alleviate these problems, this paper proposes an improved DeepLabv3+ semantic segmentation network to more accurately extract PV panels from high-resolution remote sensing images. With the aim of alleviating under-segmentation, a multi-level context aggregation module is developed. This module can enhance the model’s ability to learn the characteristics of PV panels and their surrounding environment by aggregating rich contextual information from multi-scale and semantic levels. To alleviate the problem of mis-segmentation, a hybrid attention module is introduced. This module sequentially and adaptively adjusts the weight distribution in both the channel and spatial dimensions, thus enabling the model to focus more on the feature information and spatial positions of PV objects. Experiments conducted on a self-constructed Beijing PV segmentation dataset show that the method in this paper has advantages of completeness and accuracy in extracting PV panels compared to the baseline model and current mainstream semantic segmentation network. In addition, the results of experiments on extracting PV panels in real region show that our model also has good stability and generalization capability.