Neural Network For Semantic Segmentation Research Articles

Comparative Assessment of Different Architectures of Convolutional Neural Network for Semantic Segmentation of Forest Disturbances from Multi-Temporal Satellite Images

Algorithms based on convolutional neural networks are the most efficient for semantic segmentation of images, including segmentation of forest cover disturbances from satellite images. In this study, we consider the applicability of various modifications of the U-net architecture of convolutional neural network for recognizing logged, burnt and windthrow areas in forests from multi-temporal and multi-seasonal Sentinel-2 satellite images. The assessment was carried out on three test sites that differ significantly in the characteristics of forest stands and forest management. The highest accuracy (average F-measure of 0.59) was obtained from the U-net model, while the models that showed the best results during training (Attention U-Net and MobilNetv2 U-Net) did not improve segmentation of independent data. The resulting accuracy estimates are close to those previously published for forests with a substantial proportion of selective logged areas. Characteristics of logged areas and windthrows, namely their area and type are the main factor determining the accuracy of semantic segmentation. Substantial differences were also revealed between the images taken in different seasons of the year, with the maximum segmentation accuracy based on winter pairs of images. According to summertime and different-season pairs of images, the area of forest disturbances is substantially underestimated. Forest species composition has a less significant effect, although for two of the three test sites, the maximum accuracy was observed in dark coniferous forests, and the minimum in deciduous forests. There was no statistically significant effect of slope lighting factor calculated from digital elevation model on segmentation accuracy based for winter pairs of images. The accuracy of segmentation of burnt areas, which was assessed using the example of 14 large forest fires in 2021-2022, is unsatisfactory, which is probably due to the varying degrees of damage to the forest cover in the burnt areas.

Development of an Algorithm for Semantic Segmentation of Earth Remote Sensing Data to Determine Phytoplankton Populations

Introduction. Computer vision is widely used for semantic segmentation of Earth remote sensing (ERS) data. The method allows monitoring ecosystems, including aquatic ones. Algorithms that maintain the quality of semantic segmentation of ERS images are in demand, specifically, to identify areas with phytoplankton, where water blooms— the cause of suffocation — are possible. The objective of the study is to create an algorithm that processes satellite data as input information for the formation and checking of mathematical models of hydrodynamics, which are used to monitor the state of water bodies. Various algorithms for semantic segmentation are described in the literature. New research focuses on enhancing the reliability of recognition — often using neural networks. This approach is modified in the presented work. To develop the direction, a new set of information from open sources and synthetic data are proposed. They are aimed at improving the generalization ability of the model. For the first time, the contour area of the phytoplankton population is compared to the database — and thus the boundary conditions are formed for the implementation of mathematical models and the construction of boundary-adaptive grids.Materials and Methods. The set of remote sensing images was supplemented with the author's augmentation algorithm in Python. Computer vision segmented areas of phytoplankton populations in the images. The U-Net convolutional neural network (CNN) was trained on the basis of NVIDIA Tesla T4 computing accelerators.Results. To automate the detection of phytoplankton distribution areas, a computer vision algorithm based on the U-Net CNN was developed. The model was evaluated by the calculated values of the main quality metrics related to segmentation tasks. The following metric values were obtained: Precision = 0.89, Recall = 0.88, F1 = 0.87, Dice = 0.87, and IoU = 0.79. Graphical visualization of the results of CNN learning on the training and validation sets showed good quality of model learning. This is evidenced by small changes in the loss function at the end of training. The segmentation performed by the model turned out to be close to manual marking, which indicated the high quality of the proposed solution. The area of the segmented region of the phytoplankton population was calculated by the area of one pixel. The result obtained for the original image was 51202.5 (based on information about the number of pixels related to the bloom of blue-green algae). The corresponding result of the modeling was 51312.Discussion and Conclusion. The study expands theoretical and practical knowledge on the use of convolutional neural networks for semantic segmentation of space imagery data. Given the results of the work, it is possible to assess the potential for automating the process of semantic segmentation of remote sensing data to determine the boundaries of phytoplankton populations using artificial intelligence. The use of the proposed computer vision model to obtain contours of water bloom due to phytoplankton will provide for the creation of databases — the basis for environmental monitoring of water resources and predictive modeling of hydrobiological processes.

Semantic Segmentation and Classification of Active and Abandoned Agricultural Fields through Deep Learning in the Southern Peruvian Andes

The monumental scale agricultural infrastructure systems built by Andean peoples during pre-Hispanic times have enabled intensive agriculture in the high-relief, arid/semi-arid landscape of the Southern Peruvian Andes. Large tracts of these labor-intensive systems have been abandoned, however, owing in large measure to a range of demographic, economic, and political crises precipitated by the Spanish invasion of the 16th century CE. This research seeks to better understand the dynamics of agricultural intensification and deintensification in the Andes by inventorying through the semantic segmentation of active and abandoned agricultural fields in satellite imagery across approximately 77,000 km2 of the Southern Peruvian Highlands. While manual digitization of agricultural fields in satellite imagery is time-consuming and labor-intensive, deep learning-based semantic segmentation makes it possible to map and classify en masse Andean agricultural infrastructure. Using high resolution satellite imagery, training and validation data were manually produced in distributed sample areas and were used to transfer-train a convolutional neural network for semantic segmentation. The resulting dataset was compared to manual surveys of the region and results suggest that deep learning can generate larger and more accurate datasets than those generated by hand.

Exploring Factors Affecting the Performance of Neural Network Algorithm for Detecting Clouds, Snow, and Lakes in Sentinel-2 Images

Detecting clouds, snow, and lakes in remote sensing images is vital due to their propensity to obscure underlying surface information and hinder data extraction. In this study, we utilize Sentinel-2 images to implement a two-stage random forest (RF) algorithm for image labeling and delve into the factors influencing neural network performance across six aspects: model architecture, encoder, learning rate adjustment strategy, loss function, input image size, and different band combinations. Our findings indicate the Feature Pyramid Network (FPN) achieved the highest MIoU of 87.14%. The multi-head self-attention mechanism was less effective compared to convolutional methods for feature extraction with small datasets. Incorporating residual connections into convolutional blocks notably enhanced performance. Additionally, employing false-color images (bands 12-3-2) yielded a 4.86% improvement in MIoU compared to true-color images (bands 4-3-2). Notably, variations in model architecture, encoder structure, and input band combination had a substantial impact on performance, with parameter variations resulting in MIoU differences exceeding 5%. These results provide a reference for high-precision segmentation of clouds, snow, and lakes and offer valuable insights for applying deep learning techniques to the high-precision extraction of information from remote sensing images, thereby advancing research in deep neural networks for semantic segmentation.

Artificial Intelligence Measurement of Preoperative Radiographs in Adolescent Idiopathic Scoliosis Based on Multiple-View Semantic Segmentation.

Cross-sectional study. Imaging classification of adolescent idiopathic scoliosis (AIS) is directly related to the surgical strategy, but the artificial classification is complex and depends on doctors' experience. This study investigated deep learning-based automated classification methods (DL group) for AIS and validated the consistency of machine classification and manual classification (M group). A total of 506 cases (81 males and 425 females) and 1812 AIS full spine images in the anteroposterior (AP), lateral (LAT), left bending (LB) and right bending (RB) positions were retrospectively used for training. The mean age was 13.6 ± 1.8. The mean maximum Cobb angle was 46.8 ± 12.0. U-Net semantic segmentation neural network technology and deep learning methods were used to automatically segment and establish the alignment relationship between multiple views of the spine, and to extract spinal features such as the Cobb angle. The type of each test case was automatically calculated according to Lenke's rule. An additional 107 cases of adolescent idiopathic scoliosis imaging were prospectively used for testing. The consistency of the DL group and M group was compared. Automatic vertebral body segmentation and recognition, multi-view alignment of the spine and automatic Cobb angle measurement were implemented. Compare to the M group, the consistency of the DL group was significantly higher in 3 aspects: type of lateral convexity (0.989 vs 0.566), lumbar curvature modifier (0.932 vs 0.738), and sagittal plane modifier (0.987 vs 0.522). Deep learning enables automated Cobb angle measurement and automated Lenke classification of idiopathic scoliosis whole spine radiographs with higher consistency than manual measurement classification.

Evaluating single event upsets in deep neural networks for semantic segmentation: An embedded system perspective

As the deployment of artificial intelligence (AI) algorithms at edge devices becomes increasingly prevalent, enhancing the robustness and reliability of autonomous AI-based perception and decision systems is becoming as relevant as precision and performance, especially in applications areas considered safety-critical such as autonomous driving and aerospace. This paper delves into the robustness assessment in embedded Deep Neural Networks (DNNs), particularly focusing on the impact of parameter perturbations produced by single event upsets (SEUs) on convolutional neural networks (CNN) for image semantic segmentation. By scrutinizing the layer-by-layer and bit-by-bit sensitivity of various encoder–decoder models to soft errors, this study thoroughly investigates the vulnerability of segmentation DNNs to SEUs and evaluates the consequences of techniques like model pruning and parameter quantization on the robustness of compressed models aimed at embedded implementations. The findings offer valuable insights into the mechanisms underlying SEU-induced failures that allow for evaluating the robustness of DNNs once trained in advance. Moreover, based on the collected data, we propose a set of practical lightweight error mitigation techniques with no memory or computational cost suitable for resource-constrained deployments. The code used to perform the fault injection (FI) campaign is available at https://github.com/jonGuti13/TensorFI2, while the code to implement proposed techniques is available at https://github.com/jonGuti13/parameterProtection.

Fine grained automatic left ventricle segmentation via ROI based Tri-Convolutional neural networks.

The left ventricle segmentation (LVS) is crucial to the assessment of cardiac function. Globally, cardiovascular disease accounts for the majority of deaths, posing a significant health threat. In recent years, LVS has gained important attention due to its ability to measure vital parameters such as myocardial mass, end-diastolic volume, and ejection fraction. Medical professionals realize that manually segmenting data to evaluate these processes takes a lot of time, effort when diagnosing heart diseases. Yet, manually segmenting these images is labour-intensive and may reduce diagnostic accuracy. This paper, propose a combination of different deep neural networks for semantic segmentation of the left ventricle based on Tri-Convolutional Networks (Tri-ConvNets) to obtain highly accurate segmentation. CMRI images are initially pre-processed to remove noise artefacts and enhance image quality, then ROI-based extraction is done in three stages to accurately identify the LV. The extracted features are given as input to three different deep learning structures for segmenting the LV in an efficient way. The contour edges are processed in the standard ConvNet, the contour points are processed using Fully ConvNet and finally the noise free images are converted into patches to perform pixel-wise operations in ConvNets. The proposed Tri-ConvNets model achieves the Jaccard indices of 0.9491 ± 0.0188 for the sunny brook dataset and 0.9497 ± 0.0237 for the York dataset, and the dice index of 0.9419 ± 0.0178 for the ACDC dataset and 0.9414 ± 0.0247 for LVSC dataset respectively. The experimental results also reveal that the proposed Tri-ConvNets model is faster and requires minimal resources compared to state-of-the-art models.

An FPGA-Based Lightweight Semantic Segmentation Neural Network With Optimized Ghost Module

An FPGA-Based Lightweight Semantic Segmentation Neural Network With Optimized Ghost Module

FPGA-based tiling scheme and on-chip memory scheduling scheme for multi-branch semantic segmentation neural network accelerator

Abstract According to the characteristics of a multi-branch semantic segmentation neural network with a large amount of intermediate data during the inference process, we reduce the requirement of off-chip memory access and the redundant calculation due to tiling through better blocking scheme, better on-chip memory partitioning scheme, and on-chip memory scheduling scheme, improving the energy efficiency to 77.4 GOPS/W.

Road Anomaly Detection with Unknown Scenes Using DifferNet-Based Automatic Labeling Segmentation

Obstacle avoidance is essential for the effective operation of autonomous mobile robots, enabling them to detect and navigate around obstacles in their environment. While deep learning provides significant benefits for autonomous navigation, it typically requires large, accurately labeled datasets, making the data’s preparation and processing time-consuming and labor-intensive. To address this challenge, this study introduces a transfer learning (TL)-based automatic labeling segmentation (ALS) framework. This framework utilizes a pretrained attention-based network, DifferNet, to efficiently perform semantic segmentation tasks on new, unlabeled datasets. DifferNet leverages prior knowledge from the Cityscapes dataset to identify high-entropy areas as road obstacles by analyzing differences between the input and resynthesized images. The resulting road anomaly map was refined using depth information to produce a robust drivable area and map of road anomalies. Several off-the-shelf RGB-D semantic segmentation neural networks were trained using pseudo-labels generated by the ALS framework, with validation conducted on the GMRPD dataset. Experimental results demonstrated that the proposed ALS framework achieved mean precision, mean recall, and mean intersection over union (IoU) rates of 80.31%, 84.42%, and 71.99%, respectively. The ALS framework, through the use of transfer learning and the DifferNet network, offers an efficient solution for semantic segmentation of new, unlabeled datasets, underscoring its potential for improving obstacle avoidance in autonomous mobile robots.

Building Extraction from LiDAR Point Clouds Based on Revised RandLA-Net

Abstract. 3D building models is crucial for applications in smart cities. Automatic reconstruction of 3D buildings has been investigated based on various data sources. Point clouds from airborne LiDAR scanners can be used to extract buildings data due to its high accuracy and point density. In this paper, we present a methodology to segment buildings and corresponding rooftop structure from point clouds. First, RandLA-Net, which is an efficient and lightweight neural network for semantic segmentation of large-scale point clouds, is revised and adopted for building segmentation. By implementing local feature aggregation of each point, RandLA-Net can effectively preserve geometric details in point clouds. Besides 3D coordinates of point clouds, we incorporated point attributes including pulse intensity and return numbers into the network as additional features. Feature normalizations are applied to the input features. To achieve a better result of the local feature aggregation, hyperparameters of the network are fine-tuned according to the density of points and building size. Based on the classified building point clouds, DBSCAN clustering algorithm is implemented for segmenting individual buildings. Elevation histogram analysis is conducted to determine optimal threshold values for delineating candidate rooftop point clouds of individual buildings. For the buildings with multiple rooftops, multiple elevation threshold values are necessary to extract corresponding rooftops or walls. Then DBSCAN is employed again for segmentation of individual rooftops and denoising of point clouds of each building. Finally, Alpha-shape analysis is applied based on adaptive threshold values to build the envelope of each rooftop. Experiments show that our implementation of building segmentation using RandLA-net achieves higher mean IoU (Intersection over Union) and better classification performance in building segmentation. ISPRS benchmark data was used in our experiment and our methodology produce results with accuracy of 90.79%.

Denoised Non-Local Neural Network for Semantic Segmentation.

The non-local (NL) network has become a widely used technique for semantic segmentation, which computes an attention map to measure the relationships of each pixel pair. However, most of the current popular NL models tend to ignore the phenomenon that the calculated attention map appears to be very noisy, containing interclass and intraclass inconsistencies, which lowers the accuracy and reliability of the NL methods. In this article, we figuratively denote these inconsistencies as attention noises and explore the solutions to denoise them. Specifically, we inventively propose a denoised NL network, which consists of two primary modules, i.e., the global rectifying (GR) block and the local retention (LR) block, to eliminate the interclass and intraclass noises, respectively. First, GR adopts the class-level predictions to capture a binary map to distinguish whether the selected two pixels belong to the same category. Second, LR captures the ignored local dependencies and further uses them to rectify the unwanted hollows in the attention map. The experimental results on two challenging semantic segmentation datasets demonstrate the superior performance of our model. Without any external training data, our proposed denoised NL can achieve the state-of-the-art performance of 83.5% and 46.69% mean of classwise intersection over union (mIoU) on Cityscapes and ADE20K, respectively.

A large-scale point cloud semantic segmentation neural network based on long-range contextual dependencies enhancement

ABSTRACT This letter presents a long-range contextual dependency enhanced network (LCDE-Net) for semantic segmentation of large-scale point cloud, which employs a U-shaped framework. Firstly, point clouds are subsampled with grid sampling and fed into convolutional layers to learn more representative local features of points. Then global and local encoders (GLE) are designed to exploit long-range contextual dependencies and local features simultaneously. The core of GLE consists of two parts: global feature enhancement (GFE) module and feature channel modulation (FCM) module. Secondly, through decoder layers, the encoded features are upsampled through the nearest-neighbour interpolation and aggregated with the intermediate encoded features by skip connections to capture multi-scale discriminative features for semantic segmentation of point cloud. Finally, via Fully Connection layer and Softmax classifier, each point’s label is assigned. Two different benchmark datasets are conducted to evaluate the performance of the proposed method, Experimental results report that the proposed LCDE-Net achieves 78.6% in terms of mean intersection over union (mIoU) on Semantic3D, and 68.2% on S3DIS, which is the highest among the comparison methods. The code of LCDE-Net is available at https://github.com/xrzmyz/LCDE-Net.

Per-class curriculum for Unsupervised Domain Adaptation in semantic segmentation

AbstractAccurate training of deep neural networks for semantic segmentation requires a large number of pixel-level annotations of real images, which are expensive to generate or not even available. In this context, Unsupervised Domain Adaptation (UDA) can transfer knowledge from unlimited synthetic annotations to unlabeled real images of a given domain. UDA methods are composed of an initial training stage with labeled synthetic data followed by a second stage for feature alignment between labeled synthetic and unlabeled real data. In this paper, we propose a novel approach for UDA focusing the initial training stage, which leads to increased performance after adaptation. We introduce a curriculum strategy where each semantic class is learned progressively. Thereby, better features are obtained for the second stage. This curriculum is based on: (1) a class-scoring function to determine the difficulty of each semantic class, (2) a strategy for incremental learning based on scoring and pacing functions that limits the required training time unlike standard curriculum-based training and (3) a training loss to operate at class level. We extensively evaluate our approach as the first stage of several state-of-the-art UDA methods for semantic segmentation. Our results demonstrate significant performance enhancements across all methods: improvements of up to 10% for entropy-based techniques and 8% for adversarial methods. These findings underscore the dependency of UDA on the accuracy of the initial training. The implementation is available at https://github.com/vpulab/PCCL.

The integration of GPS and visual navigation for autonomous navigation of an Ackerman steering mobile robot in cotton fields.

Autonomous navigation in agricultural fields presents a unique challenge due to the unpredictable outdoor environment. Various approaches have been explored to tackle this task, each with its own set of challenges. These include GPS guidance, which faces availability issues and struggles to avoid obstacles, and vision guidance techniques, which are sensitive to changes in light, weeds, and crop growth. This study proposes a novel idea that combining GPS and visual navigation offers an optimal solution for autonomous navigation in agricultural fields. Three solutions for autonomous navigation in cotton fields were developed and evaluated. The first solution utilized a path tracking algorithm, Pure Pursuit, to follow GPS coordinates and guide a mobile robot. It achieved an average lateral deviation of 8.3 cm from the pre-recorded path. The second solution employed a deep learning model, specifically a fully convolutional neural network for semantic segmentation, to detect paths between cotton rows. The mobile rover then navigated using the Dynamic Window Approach (DWA) path planning algorithm, achieving an average lateral deviation of 4.8 cm from the desired path. Finally, the two solutions were integrated for a more practical approach. GPS served as a global planner to map the field, while the deep learning model and DWA acted as a local planner for navigation and real-time decision-making. This integrated solution enabled the robot to navigate between cotton rows with an average lateral distance error of 9.5 cm, offering a more practical method for autonomous navigation in cotton fields.

Autonomous Robotic System to Prune Sweet Pepper Leaves Using Semantic Segmentation with Deep Learning and Articulated Manipulator.

This paper proposes an autonomous robotic system to prune sweet pepper leaves using semantic segmentation with deep learning and an articulated manipulator. This system involves three main tasks: the perception of crop parts, the detection of pruning position, and the control of the articulated manipulator. A semantic segmentation neural network is employed to recognize the different parts of the sweet pepper plant, which is then used to create 3D point clouds for detecting the pruning position and the manipulator pose. Eventually, a manipulator robot is controlled to prune the crop part. This article provides a detailed description of the three tasks involved in building the sweet pepper pruning system and how to integrate them. In the experiments, we used a robot arm to manipulate the pruning leaf actions within a certain height range and a depth camera to obtain 3D point clouds. The control program was developed in different modules using various programming languages running on the ROS (Robot Operating System).

Sweet Pepper Leaf Area Estimation Using Semantic 3D Point Clouds Based on Semantic Segmentation Neural Network

In the field of agriculture, measuring the leaf area is crucial for the management of crops. Various techniques exist for this measurement, ranging from direct to indirect approaches and destructive to non-destructive techniques. The non-destructive approach is favored because it preserves the plant's integrity. Among these, several methods utilize leaf dimensions, such as width and length, to estimate leaf areas based on specific models that consider the unique shapes of leaves. Although this approach does not damage plants, it is labor-intensive, requiring manual measurements of leaf dimensions. In contrast, some indirect non-destructive techniques leveraging convolutional neural networks can predict leaf areas more swiftly and autonomously. In this paper, we propose a new direct method using 3D point clouds constructed by semantic RGB-D (Red Green Blue and Depth) images generated by a semantic segmentation neural network and RGB-D images. The key idea is that the leaf area is quantified by the count of points depicting the leaves. This method demonstrates high accuracy, with an R2 value of 0.98 and a RMSE (Root Mean Square Error) value of 3.05 cm2. Here, the neural network's role is to segregate leaves from other plant parts to accurately measure the leaf area represented by the point clouds, rather than predicting the total leaf area of the plant. This method is direct, precise, and non-invasive to sweet pepper plants, offering easy leaf area calculation. It can be implemented on laptops for manual use or integrated into robots for automated periodic leaf area assessments. This innovative method holds promise for advancing our understanding of plant responses to environmental changes. We verified the method's reliability and superior performance through experiments on individual leaves and whole plants.

Vision based process monitoring in wire arc additive manufacturing (WAAM)

AbstractA stable welding process is crucial to obtain high quality parts in wire arc additive manufacturing. The complexity of the process makes it inherently unstable, which can cause various defects, resulting in poor geometric accuracy and material properties. This demands for in-process monitoring and control mechanisms to industrialize the technology. In this work, process monitoring algorithms based on welding camera image analysis are presented. A neural network for semantic segmentation of the welding wire is used to monitor the working distance as well as the horizontal position of the wire during welding and classic image processing techniques are applied to capture spatter formation. Using these algorithms, the process stability is evaluated in real time and the analysis results enable the direction independent closed-loop-control of the manufacturing process. This significantly improves geometric fidelity as well as mechanical properties of the fabricated part and allows the automated production of parts with complex deposition paths including weld bead crossings, curvatures and overhang structures.

Deep neural network for semantic segmentation of satellite images

Deep neural network for semantic segmentation of satellite images

CSC-Unet: A Novel Convolutional Sparse Coding Strategy Based Neural Network for Semantic Segmentation

CSC-Unet: A Novel Convolutional Sparse Coding Strategy Based Neural Network for Semantic Segmentation