Neural Network For Semantic Segmentation Research Articles

Building Extraction from LiDAR Point Clouds Based on Revised RandLA-Net

Abstract. 3D building models is crucial for applications in smart cities. Automatic reconstruction of 3D buildings has been investigated based on various data sources. Point clouds from airborne LiDAR scanners can be used to extract buildings data due to its high accuracy and point density. In this paper, we present a methodology to segment buildings and corresponding rooftop structure from point clouds. First, RandLA-Net, which is an efficient and lightweight neural network for semantic segmentation of large-scale point clouds, is revised and adopted for building segmentation. By implementing local feature aggregation of each point, RandLA-Net can effectively preserve geometric details in point clouds. Besides 3D coordinates of point clouds, we incorporated point attributes including pulse intensity and return numbers into the network as additional features. Feature normalizations are applied to the input features. To achieve a better result of the local feature aggregation, hyperparameters of the network are fine-tuned according to the density of points and building size. Based on the classified building point clouds, DBSCAN clustering algorithm is implemented for segmenting individual buildings. Elevation histogram analysis is conducted to determine optimal threshold values for delineating candidate rooftop point clouds of individual buildings. For the buildings with multiple rooftops, multiple elevation threshold values are necessary to extract corresponding rooftops or walls. Then DBSCAN is employed again for segmentation of individual rooftops and denoising of point clouds of each building. Finally, Alpha-shape analysis is applied based on adaptive threshold values to build the envelope of each rooftop. Experiments show that our implementation of building segmentation using RandLA-net achieves higher mean IoU (Intersection over Union) and better classification performance in building segmentation. ISPRS benchmark data was used in our experiment and our methodology produce results with accuracy of 90.79%.

Denoised Non-Local Neural Network for Semantic Segmentation.

The non-local (NL) network has become a widely used technique for semantic segmentation, which computes an attention map to measure the relationships of each pixel pair. However, most of the current popular NL models tend to ignore the phenomenon that the calculated attention map appears to be very noisy, containing interclass and intraclass inconsistencies, which lowers the accuracy and reliability of the NL methods. In this article, we figuratively denote these inconsistencies as attention noises and explore the solutions to denoise them. Specifically, we inventively propose a denoised NL network, which consists of two primary modules, i.e., the global rectifying (GR) block and the local retention (LR) block, to eliminate the interclass and intraclass noises, respectively. First, GR adopts the class-level predictions to capture a binary map to distinguish whether the selected two pixels belong to the same category. Second, LR captures the ignored local dependencies and further uses them to rectify the unwanted hollows in the attention map. The experimental results on two challenging semantic segmentation datasets demonstrate the superior performance of our model. Without any external training data, our proposed denoised NL can achieve the state-of-the-art performance of 83.5% and 46.69% mean of classwise intersection over union (mIoU) on Cityscapes and ADE20K, respectively.

A large-scale point cloud semantic segmentation neural network based on long-range contextual dependencies enhancement

ABSTRACT This letter presents a long-range contextual dependency enhanced network (LCDE-Net) for semantic segmentation of large-scale point cloud, which employs a U-shaped framework. Firstly, point clouds are subsampled with grid sampling and fed into convolutional layers to learn more representative local features of points. Then global and local encoders (GLE) are designed to exploit long-range contextual dependencies and local features simultaneously. The core of GLE consists of two parts: global feature enhancement (GFE) module and feature channel modulation (FCM) module. Secondly, through decoder layers, the encoded features are upsampled through the nearest-neighbour interpolation and aggregated with the intermediate encoded features by skip connections to capture multi-scale discriminative features for semantic segmentation of point cloud. Finally, via Fully Connection layer and Softmax classifier, each point’s label is assigned. Two different benchmark datasets are conducted to evaluate the performance of the proposed method, Experimental results report that the proposed LCDE-Net achieves 78.6% in terms of mean intersection over union (mIoU) on Semantic3D, and 68.2% on S3DIS, which is the highest among the comparison methods. The code of LCDE-Net is available at https://github.com/xrzmyz/LCDE-Net.

Per-class curriculum for Unsupervised Domain Adaptation in semantic segmentation

AbstractAccurate training of deep neural networks for semantic segmentation requires a large number of pixel-level annotations of real images, which are expensive to generate or not even available. In this context, Unsupervised Domain Adaptation (UDA) can transfer knowledge from unlimited synthetic annotations to unlabeled real images of a given domain. UDA methods are composed of an initial training stage with labeled synthetic data followed by a second stage for feature alignment between labeled synthetic and unlabeled real data. In this paper, we propose a novel approach for UDA focusing the initial training stage, which leads to increased performance after adaptation. We introduce a curriculum strategy where each semantic class is learned progressively. Thereby, better features are obtained for the second stage. This curriculum is based on: (1) a class-scoring function to determine the difficulty of each semantic class, (2) a strategy for incremental learning based on scoring and pacing functions that limits the required training time unlike standard curriculum-based training and (3) a training loss to operate at class level. We extensively evaluate our approach as the first stage of several state-of-the-art UDA methods for semantic segmentation. Our results demonstrate significant performance enhancements across all methods: improvements of up to 10% for entropy-based techniques and 8% for adversarial methods. These findings underscore the dependency of UDA on the accuracy of the initial training. The implementation is available at https://github.com/vpulab/PCCL.

The integration of GPS and visual navigation for autonomous navigation of an Ackerman steering mobile robot in cotton fields.

Autonomous navigation in agricultural fields presents a unique challenge due to the unpredictable outdoor environment. Various approaches have been explored to tackle this task, each with its own set of challenges. These include GPS guidance, which faces availability issues and struggles to avoid obstacles, and vision guidance techniques, which are sensitive to changes in light, weeds, and crop growth. This study proposes a novel idea that combining GPS and visual navigation offers an optimal solution for autonomous navigation in agricultural fields. Three solutions for autonomous navigation in cotton fields were developed and evaluated. The first solution utilized a path tracking algorithm, Pure Pursuit, to follow GPS coordinates and guide a mobile robot. It achieved an average lateral deviation of 8.3 cm from the pre-recorded path. The second solution employed a deep learning model, specifically a fully convolutional neural network for semantic segmentation, to detect paths between cotton rows. The mobile rover then navigated using the Dynamic Window Approach (DWA) path planning algorithm, achieving an average lateral deviation of 4.8 cm from the desired path. Finally, the two solutions were integrated for a more practical approach. GPS served as a global planner to map the field, while the deep learning model and DWA acted as a local planner for navigation and real-time decision-making. This integrated solution enabled the robot to navigate between cotton rows with an average lateral distance error of 9.5 cm, offering a more practical method for autonomous navigation in cotton fields.

Autonomous Robotic System to Prune Sweet Pepper Leaves Using Semantic Segmentation with Deep Learning and Articulated Manipulator.

This paper proposes an autonomous robotic system to prune sweet pepper leaves using semantic segmentation with deep learning and an articulated manipulator. This system involves three main tasks: the perception of crop parts, the detection of pruning position, and the control of the articulated manipulator. A semantic segmentation neural network is employed to recognize the different parts of the sweet pepper plant, which is then used to create 3D point clouds for detecting the pruning position and the manipulator pose. Eventually, a manipulator robot is controlled to prune the crop part. This article provides a detailed description of the three tasks involved in building the sweet pepper pruning system and how to integrate them. In the experiments, we used a robot arm to manipulate the pruning leaf actions within a certain height range and a depth camera to obtain 3D point clouds. The control program was developed in different modules using various programming languages running on the ROS (Robot Operating System).

Sweet Pepper Leaf Area Estimation Using Semantic 3D Point Clouds Based on Semantic Segmentation Neural Network

In the field of agriculture, measuring the leaf area is crucial for the management of crops. Various techniques exist for this measurement, ranging from direct to indirect approaches and destructive to non-destructive techniques. The non-destructive approach is favored because it preserves the plant's integrity. Among these, several methods utilize leaf dimensions, such as width and length, to estimate leaf areas based on specific models that consider the unique shapes of leaves. Although this approach does not damage plants, it is labor-intensive, requiring manual measurements of leaf dimensions. In contrast, some indirect non-destructive techniques leveraging convolutional neural networks can predict leaf areas more swiftly and autonomously. In this paper, we propose a new direct method using 3D point clouds constructed by semantic RGB-D (Red Green Blue and Depth) images generated by a semantic segmentation neural network and RGB-D images. The key idea is that the leaf area is quantified by the count of points depicting the leaves. This method demonstrates high accuracy, with an R2 value of 0.98 and a RMSE (Root Mean Square Error) value of 3.05 cm2. Here, the neural network's role is to segregate leaves from other plant parts to accurately measure the leaf area represented by the point clouds, rather than predicting the total leaf area of the plant. This method is direct, precise, and non-invasive to sweet pepper plants, offering easy leaf area calculation. It can be implemented on laptops for manual use or integrated into robots for automated periodic leaf area assessments. This innovative method holds promise for advancing our understanding of plant responses to environmental changes. We verified the method's reliability and superior performance through experiments on individual leaves and whole plants.

Vision based process monitoring in wire arc additive manufacturing (WAAM)

AbstractA stable welding process is crucial to obtain high quality parts in wire arc additive manufacturing. The complexity of the process makes it inherently unstable, which can cause various defects, resulting in poor geometric accuracy and material properties. This demands for in-process monitoring and control mechanisms to industrialize the technology. In this work, process monitoring algorithms based on welding camera image analysis are presented. A neural network for semantic segmentation of the welding wire is used to monitor the working distance as well as the horizontal position of the wire during welding and classic image processing techniques are applied to capture spatter formation. Using these algorithms, the process stability is evaluated in real time and the analysis results enable the direction independent closed-loop-control of the manufacturing process. This significantly improves geometric fidelity as well as mechanical properties of the fabricated part and allows the automated production of parts with complex deposition paths including weld bead crossings, curvatures and overhang structures.

CSC-Unet: A Novel Convolutional Sparse Coding Strategy Based Neural Network for Semantic Segmentation

CSC-Unet: A Novel Convolutional Sparse Coding Strategy Based Neural Network for Semantic Segmentation

An FPGA-Based Lightweight Semantic Segmentation Neural Network With Optimized Ghost Module

An FPGA-Based Lightweight Semantic Segmentation Neural Network With Optimized Ghost Module

温室大棚道路图像的语义分割研究

In order to realize the automatic driving of agricultural machinery in greenhouse, this paper uses image acquisition equipment to collect road images in greenhouse and makes data sets, builds SETR model based on Transformer framework and DeepLabv3+ model based on convolution neural network for semantic segmentation of road images in greenhouse, and verifies the semantic segmentation ability of the two models to road images in greenhouse. Several groups of training periods are set as observation points to observe the semantic segmentation effect of the two models on the greenhouse road image, and the test set which has not been trained by the model is used as the prediction object to verify the performance of the two models on the semantic segmentation of greenhouse road image. The SETR model reached 94.64% PA on the greenhouse road data set, and 82.72% mIoU, DeepLabv3+ model reached 90.80% PA and 72.35% mIoU on the greenhouse road data set. Both models have excellent performance in semantic segmentation of greenhouse road images, and the performance of SETR model is slightly better than that of DeepLabv3+ model. The semantic segmentation performance of the two models for greenhouse road images can meet the needs of actual deployment.

Absolute retinal blood flow in healthy eyes and in eyes with retinal vein occlusion

PurposeTo measure non-invasively retinal venous blood flow (RBF) in healthy subjects and patients with retinal venous occlusion (RVO). MethodsThe prototype named AO-LDV (Adaptive Optics Laser Doppler Velocimeter), which combines a new absolute laser Doppler velocimeter with an adaptive optics fundus camera (rtx1, Imagine Eyes®, Orsay, France), was studied for the measurement of absolute RBF as a function of retinal vessel diameters and simultaneous measurement of red blood cell velocity. RBF was measured in healthy subjects (n = 15) and patients with retinal venous occlusion (RVO, n = 6). We also evaluated two softwares for the measurement of retinal vessel diameters: software 1 (automatic vessel detection, profile analysis) and software 2 (based on the use of deep neural networks for semantic segmentation of vessels, using a M2u-Net architecture). ResultsSoftware 2 provided a higher rate of automatic retinal vessel measurement (99.5 % of 12,320 AO images) than software 1 (64.9 %) and wider measurements (75.5 ± 15.7 μm vs 70.9 ± 19.8 μm, p < 0.001). For healthy subjects (n = 15), all the retinal veins in one eye were measured to obtain the total RBF. In healthy subjects, the total RBF was 37.8 ± 6.8 μl/min. There was a significant linear correlation between retinal vessel diameter and maximal velocity (slope = 0.1016; p < 0.001; r2 = 0.8597) and a significant power curve correlation between retinal vessel diameter and blood flow (3.63 × 10−5 × D2.54; p < 0.001; r2 = 0.7287). No significant relationship was found between total RBF and systolic and diastolic blood pressure, ocular perfusion pressure, heart rate, or hematocrit. For RVO patients (n = 6), a significant decrease in RBF was noted in occluded veins (3.51 ± 2.25 μl/min) compared with the contralateral healthy eye (11.07 ± 4.53 μl/min). For occluded vessels, the slope between diameter and velocity was 0.0195 (p < 0.001; r2 = 0.6068) and the relation between diameter and flow was Q = 9.91 × 10−6 × D2.41 (p < 0.01; r2 = 0.2526). ConclusionThis AO-LDV prototype offers new opportunity to study RBF in humans and to evaluate treatment in retinal vein diseases.

An Iterative Closest Point Method for Lidar Odometry with Fused Semantic Features

Lidar sensors play a pivotal role in a multitude of remote sensing domains, finding extensive applications in various sectors, including robotics, unmanned aerial vehicles (UAVs), autonomous driving, and 3D reconstruction, among others. Their significance and versatility in these areas have made them indispensable tools for a wide range of applications. The accuracy of Lidar odometry (LO), which serves as the front end of SLAM, is crucial. In this paper, we propose a novel iterative closest point (ICP) technique that combines semantic features to improve LO precision. First, the semantic segmentation neural network is used to extract the semantic features from each frame of the point cloud. Then, the obtained semantic features assist in extracting the local geometric features of the point cloud. Also, the residual blocks of the ICP algorithm’s least squares are combined with semantic confidence functions to better predict the exact pose. Compared to LOAM, there is an average improvement of 4 cm in accuracy per 100 m. Experimental results illustrate the superiority of the proposed method and indicate that the fusion of semantic features can robustly improve the precision of LO.

Vision-enhanced Peg-in-Hole for automotive body parts using semantic image segmentation and object detection

Artificial Intelligence (AI) is an enabling technology in the context of Industry 4.0. In particular, the automotive sector is among those who can benefit most of the use of AI in conjunction with advanced vision techniques. The scope of this work is to integrate deep learning algorithms in an industrial scenario involving a robotic Peg-in-Hole task. More in detail, we focus on a scenario where a human operator manually positions a carbon fiber automotive part in the workspace of a 7 Degrees of Freedom (DOF) manipulator. To cope with the uncertainty on the relative position between the robot and the workpiece, we adopt a three stage strategy. The first stage concerns the Three-Dimensional (3D) reconstruction of the workpiece using a registration algorithm based on the Iterative Closest Point (ICP) paradigm. Such a procedure is integrated with a semantic image segmentation neural network, which is in charge of removing the background of the scene to improve the registration. The adoption of such network allows to reduce the registration time of about 28.8%. In the second stage, the reconstructed surface is compared with a Computer Aided Design (CAD) model of the workpiece to locate the holes and their axes. In this stage, the adoption of a Convolutional Neural Network (CNN) allows to improve the holes’ position estimation of about 57.3%. The third stage concerns the insertion of the peg by implementing a search phase to handle the remaining estimation errors. Also in this case, the use of the CNN reduces the search phase duration of about 71.3%. Quantitative experiments, including a comparison with a previous approach without both the segmentation network and the CNN, have been conducted in a realistic scenario. The results show the effectiveness of the proposed approach and how the integration of AI techniques improves the success rate from 84.5% to 99.0%.

Retracted: PointLAE: A Point Cloud Semantic Segmentation Neural Network via Multifeature Aggregation for Large-Scale Application

Retracted: PointLAE: A Point Cloud Semantic Segmentation Neural Network via Multifeature Aggregation for Large-Scale Application

Lithology Identification of UAV Oblique Photography Images Based on Semantic Segmentation Neural Network Algorithm

Lithology Identification of UAV Oblique Photography Images Based on Semantic Segmentation Neural Network Algorithm

The Development and Initial Capabilities of ThunderCast, a Deep Learning Model for Thunderstorm Nowcasting in the United States

Abstract This paper presents the Thunderstorm Nowcasting Tool (ThunderCast), a 24-h, year-round model for predicting the location of convection that is likely to initiate or remain a thunderstorm in the next 0–60 min in the continental United States, adapted from existing deep learning convection applications. ThunderCast utilizes a U-Net convolutional neural network for semantic segmentation trained on 320 km × 320 km data patches with four inputs and one target dataset. The inputs are satellite bands from the Geostationary Operational Environmental Satellite-16 (GOES-16) Advanced Baseline Imager (ABI) in the visible, shortwave infrared, and longwave infrared spectra, and the target is Multi-Radar Multi-Sensor (MRMS) radar reflectivity at the −10°C isotherm in the atmosphere. On a pixel-by-pixel basis, ThunderCast has high accuracy, recall, and specificity but is subject to false-positive predictions resulting in low precision. However, the number of false positives decreases when buffering the target values with a 15 km × 15 km centered window, indicating ThunderCast’s predictions are useful within a buffered area. To demonstrate the initial prediction capabilities of ThunderCast, three case studies are presented: a mesoscale convective vortex, sea-breeze convection, and monsoonal convection in the southwestern United States. The case studies illustrate that the ThunderCast model effectively nowcasts the location of newly initiated and ongoing active convection, within the next 60 min, under a variety of geographical and meteorological conditions. Significance Statement In this research, a machine learning model is developed for short-term (0–60 min) forecasting of thunderstorms in the continental United States using geostationary satellite imagery as inputs for predicting active convection based on radar thresholds. Pending additional testing, the model may be able to provide decision-support services for thunderstorm forecasting. The case studies presented here indicate the model is able to nowcast convective initiation with 5–35 min of lead time in areas without radar coverage and anticipate future locations of storms without additional environmental context.

An artificial intelligence algorithm for automated blastocyst morphometric parameters demonstrates a positive association with implantation potential

Blastocyst selection is primarily based on morphological scoring systems and morphokinetic data. These methods involve subjective grading and time-consuming techniques. Artificial intelligence allows for objective and quick blastocyst selection. In this study, 608 blastocysts were selected for transfer using morphokinetics and Gardner criteria. Retrospectively, morphometric parameters of blastocyst size, inner cell mass (ICM) size, ICM-to-blastocyst size ratio, and ICM shape were automatically measured by a semantic segmentation neural network model. The model was trained on 1506 videos with 102 videos for validation with no overlap between the ICM and trophectoderm models. Univariable logistic analysis found blastocyst size and ICM-to-blastocyst size ratio to be significantly associated with implantation potential. Multivariable regression analysis, adjusted for woman age, found blastocyst size to be significantly associated with implantation potential. The odds of implantation increased by 1.74 for embryos with a blastocyst size greater than the mean (147 ± 19.1 μm). The performance of the algorithm was represented by an area under the curve of 0.70 (p < 0.01). In conclusion, this study supports the association of a large blastocyst size with higher implantation potential and suggests that automatically measured blastocyst morphometrics can be used as a precise, consistent, and time-saving tool for improving blastocyst selection.

Corrigendum to CrackDenseLinkNet: a deep convolutional neural network for semantic segmentation of cracks on concrete surface images

Corrigendum to CrackDenseLinkNet: a deep convolutional neural network for semantic segmentation of cracks on concrete surface images

Enhancing identification of digital rock images using super-resolution deep neural network

Identification of pore and minerals from digital rock images plays an important role in characterization of oil and gas reservoirs. Traditional methods usually require high costs of time and labor. With the rapid development of artificial intelligence, deep learning methods have been applied to geosciences. Among them, semantic segmentation neural networks are used to identify rock images, but traditional semantic segmentation networks require high-resolution input images to improve performance and can only generate masks with the same resolutions as the input images, which again requires high-performance equipment and increases computational and equipment costs. In this paper, we propose an approach to enhancing identification of digital rock images with super-resolution neural network (SSRN) which enables low-resolution input images to generate high-resolution representations. We use SSRN to train the 10-classification thin-section images, and compare it with traditional deep learning methods. It indicates that SSRN can effectively improve the performance of the generated high-resolution masks.