Real-Time Environmental Contour Construction Using 3D LiDAR and Image Recognition with Object Removal

The 3D point cloud accumulates laser scans at different locations and times. Since laser scanning captures a snapshot of surrounding environment, moving objects are often observed and contained now and then. The dynamic objects in the point cloud map can degrade the quality of the map and affect the localization accuracy, thus it is critical to remove the dynamic objects from the 3D point cloud map. In this paper, a baseline based on dynamic object removal for 3d point cloud loop detection is proposed. To eliminate the interference of moving objects in the environment. First, the radar point cloud data is preprocessed, namely, the 3D object detection model OpenPCDet is employed to detect dynamic objects in the outdoor scene, such as vehicles, pedestrians, etc. Second, we use the bounding box detected by the model to perform cube filtering on the original data to remove dynamics objects. Finally, the processed data is utilized to extract scene descriptors for loop detection. In the road scene, experimental results demonstrate that our approach yields superior performance against the traditional methods.

Aiming at addressing the defect of the data blindness of a LiDAR point cloud in transparent media such as glass in low illumination environments, a new method is proposed to realize covert target reconnaissance, identification and ranging using the fusion of a shimmering polarized image and a laser LiDAR point cloud, and the corresponding system is constructed. Based on the extraction of pixel coordinates from the 3D LiDAR point cloud, the method adds information on the polarization degree and polarization angle of the micro-light polarization image, as well as on the reflective intensity of each point of the LiDAR. The mapping matrix of the radar point cloud to the pixel coordinates is made to contain depth offset information and show better fitting, thus optimizing the 3D point cloud converted from the micro-light polarization image. On this basis, algorithms such as 3D point cloud fusion and pseudo-color mapping are used to further optimize the matching and fusion procedures for the micro-light polarization image and the radar point cloud, so as to successfully realize the alignment and fusion of the 2D micro-light polarization image and the 3D LiDAR point cloud. The experimental results show that the alignment rate between the 2D micro-light polarization image and the 3D LiDAR point cloud reaches 74.82%, which can effectively detect the target hidden behind the glass under the low illumination condition and fill the blind area of the LiDAR point cloud data acquisition. This study verifies the feasibility and advantages of “polarization + LiDAR” fusion in low-light glass scene reconnaissance, and it provides a new technological means of covert target detection in complex environments.

In this study, we develop a system for efficiently measuring detailed information of trees in a forest environment using a small unmanned aerial vehicle (UAV) equipped with light detection and ranging (lidar). The main purpose of forest measurement is to predict the volume of wood for harvesting and delineating forest boundaries by tree location. Herein, we propose a method for extracting the position, number of trees, and vertical height of trees from a set of three-dimensional (3D) point clouds acquired by a UAV lidar system. The point cloud obtained from a UAV is dense in the tree’s crown, and the trunk 3D points are sparse because the crown of the tree obstructs the laser beam. Therefore, it is difficult to extract single-tree information from 3D point clouds because the characteristics of 3D point clouds differ significantly from those of conventional 3D point clouds using ground-based laser scanners. In this study, we segment the forest point cloud into three regions with different densities of point clouds, i.e., canopy, trunk, and ground, and process each region individually to extract the target information. By comparing a ground laser survey and the proposed method in an actual forest environment, it is discovered that the number of trees in an area measuring 100 m × 100 m is 94.6% of the total number of trees. The root mean square error of the tree position is 0.3 m, whereas that of the vertical height is 2.3 m, indicating that single-tree information can be measured with sufficient accuracy for forest management.

Three-dimensional (3D) imaging provides detailed geometry of real-world objects, unlike 2D image texture. The rudimentary form of 3D imaging is point clouds that are distinctly different from image pixels in terms of structure and processing methods. The 3D computer vision literature primarily retrieves global shape patterns in 3D data for object and face recognition tasks. In contrast, mining local deformation patterns in 3D data that are independent of global shape is a nontrivial task. This paper proposes a computational pipeline for mining baseline local patterns in 3D point clouds and identifies informative segments of point clouds for data selection and interpretation. We investigate the performance of several clustering algorithms in 3D point cloud segmentation and propose a computationally fast multi-stage clustering pipeline with parametric modeling of local patterns in point clouds. The proposed pipeline has achieved an area under the ROC curve of 0.72 in classifying seven emotional expressions (including the neutral expression) using 3D human facial point clouds. Our results demonstrate the baseline efficacy of raw 3D point coordinates in mining local patterns without involving feature engineering or deep learning. Therefore, the proposed pipeline can serve as a baseline for 1) rapid mining of informative local patterns and 2) selecting important segments of 3D point cloud data. The source code is made publicly available to promote future work in this area.

With the further development of the construction of “smart mine,” the establishment of three-dimensional (3D) point cloud models of mines has become very common. However, the truck operation caused the 3D point cloud model of the mining area to contain dust points, and the 3D point cloud model established by the Context Capture modeling software is a hollow structure. The previous point cloud denoising algorithms caused holes in the model. In view of the above problems, this paper proposes the point cloud denoising method based on orthogonal total least squares fitting and two-layer extreme learning machine improved by genetic algorithm (GA-TELM). The steps are to separate dust points and ground points by orthogonal total least squares fitting and use GA-TELM to repair holes. The advantages of the proposed method are listed as follows. First, this method could denoise without generating holes, which solves engineering problems. Second, GA-TELM has a better effect in repairing holes compared with the other methods considered in this paper. Finally, this method starts from actual problems and could be used in mining areas with the same problems. Experimental results demonstrate that it can remove dust spots in the flat area of the mine effectively and ensure the integrity of the model.

An encoder-decoder deep learning method for multi-class object segmentation from 3D tunnel point clouds

In three-dimensional (3D) shape measurement based on fringe projection, various factors can degrade the quality of the point cloud. Existing point cloud filtering methods involve analyzing the geometric relationship between 3D space and point cloud, which poses challenges such as complex calculation and low efficiency. To improve the accuracy and speed of point cloud filtering, this paper proposes a new point cloud filtering method based on image segmentation and the absolute phase for the 3D imaging obtained by fringe projection. Firstly, a two-dimensional (2D) point cloud mapping image is established based on the 3D point cloud obtained from fringe projection. Secondly, threshold segmentation and region growing methods are used to segment the 2D point cloud mapping image, followed by recording and removal of the segmented noise region. Using the relationship between the noise point cloud and the absolute phase noise point in fringe projection, a reference noise-free point is established, and the absolute phase line segment is restored to obtain the absolute phase of the noise-free point. Finally, a new 2D point cloud mapping image is reconstructed in 3D space to obtain a point cloud with noise removed. Experimental results show that the point cloud denoising accuracy calculated by this method can reach up to 99.974%, and the running time is 0.954 s. The proposed method can effectively remove point cloud noise and avoid complex calculations in 3D space. This method can not only remove the noise of the 3D point cloud but also can restore the partly removed noise point cloud into a noise-free 3D point cloud, which can improve the accuracy of the 3D point cloud.

Accurate and high-throughput determination of plant morphological traits is essential for phenotyping studies. Nowadays, there are many approaches to acquire high-quality three-dimensional (3D) point clouds of plants. However, it is difficult to estimate phenotyping parameters accurately of the whole growth stages of maize plants using these 3D point clouds. In this paper, an accurate skeleton extraction approach was proposed to bridge the gap between 3D point cloud and phenotyping traits estimation of maize plants. The algorithm first uses point cloud clustering and color difference denoising to reduce the noise of the input point clouds. Next, the Laplacian contraction algorithm is applied to shrink the points. Then the key points representing the skeleton of the plant are selected through adaptive sampling, and neighboring points are connected to form a plant skeleton composed of semantic organs. Finally, deviation skeleton points to the input point cloud are calibrated by building a step forward local coordinate along the tangent direction of the original points. The proposed approach successfully generates accurately extracted skeleton from 3D point cloud and helps to estimate phenotyping parameters with high precision of maize plants. Experimental verification of the skeleton extraction process, tested using three cultivars and different growth stages maize, demonstrates that the extracted matches the input point cloud well. Compared with 3D digitizing data-derived morphological parameters, the NRMSE of leaf length, leaf inclination angle, leaf top length, leaf azimuthal angle, leaf growth height, and plant height, estimated using the extracted plant skeleton, are 5.27, 8.37, 5.12, 4.42, 1.53, and 0.83%, respectively, which could meet the needs of phenotyping analysis. The time required to process a single maize plant is below 100 s. The proposed approach may play an important role in further maize research and applications, such as genotype-to-phenotype study, geometric reconstruction, functional structural maize modeling, and dynamic growth animation.

Three-dimensional (3D) point clouds are becoming an important part of the geospatial domain. During research on 3D point clouds, deep-learning models have been widely used for the classification and segmentation of 3D point clouds observed by airborne LiDAR. However, most previous studies used discriminative models, whereas few studies used generative models. Specifically, one unsolved problem is the synthesis of large-scale 3D point clouds, such as those observed in outdoor scenes, because of the 3D point clouds' complex geometric structure. In this paper, we propose a generative model for generating large-scale 3D point clouds observed from airborne LiDAR. Generally, because the training process of the famous generative model called generative adversarial network (GAN) is unstable, we combine a variational autoen-coder and GAN to generate a suitable 3D point cloud. We experimentally demonstrate that our framework can generate high-density 3D point clouds by using data from the 2018 IEEE GRSS Data Fusion Contest.

Abstract. With the rapid development of new indoor sensors and acquisition techniques, the amount of indoor three dimensional (3D) point cloud models was significantly increased. However, these massive “blind” point clouds are difficult to satisfy the demand of many location-based indoor applications and GIS analysis. The robust semantic segmentation of 3D point clouds remains a challenge. In this paper, a segmentation with layout estimation network (SLENet)-based 2D–3D semantic transfer method is proposed for robust segmentation of image-based indoor 3D point clouds. Firstly, a SLENet is devised to simultaneously achieve the semantic labels and indoor spatial layout estimation from 2D images. A pixel labeling pool is then constructed to incorporate the visual graphical model to realize the efficient 2D–3D semantic transfer for 3D point clouds, which avoids the time-consuming pixel-wise label transfer and the reprojection error. Finally, a 3D-contextual refinement, which explores the extra-image consistency with 3D constraints is developed to suppress the labeling contradiction caused by multi-superpixel aggregation. The experiments were conducted on an open dataset (NYUDv2 indoor dataset) and a local dataset. In comparison with the state-of-the-art methods in terms of 2D semantic segmentation, SLENet can both learn discriminative enough features for inter-class segmentation while preserving clear boundaries for intra-class segmentation. Based on the excellence of SLENet, the final 3D semantic segmentation tested on the point cloud created from the local image dataset can reach a total accuracy of 89.97%, with the object semantics and indoor structural information both expressed.

Currently, the application of ground surface extraction technology in 3D point clouds has attracted extensive research attention. Such technology is widely used in the environmental perception and local navigation functions of Unmanned Ground Vehicles (UGVs). However, due to the heterogeneous density and unstructured spatial distribution of point clouds, the computational time and space complexity is relatively high in ground detection. In addition, since light detection and ranging (LiDAR) sensing can obtain more than 700,000 points per second, ground point clouds require more memory, compared to non-ground point clouds. Thus, ground point clouds extraction is a vitally important function for UGVs to realize intelligent driving. To extract precise ground information, this paper proposes a 3D Hough transform (3DHT) algorithm for ground detection from 3D LiDAR point clouds. We create a special Hough space within the walking slope range and transform all of the 3D points into this 3D Hough space according to the proposed 3DHT algorithm. Then, the maximal peak is extracted and ground equation parameters are obtained using the inverse Hough transform. To reduce the time required, we apply a Graphics Processing Unit (GPU) parallel computation method to reduce the number of typically exhaustive iterations.

Point cloud (PC) compression is crucial to immersive visual applications such as autonomous vehicles to classify objects on the roads. The Motion Picture Experts Group (MPEG) standardization group has achieved a notable compression efficiency, called video-based PC compression (V-PCC), which consists of an encoder-decoder. The V-PCC encoder takes original 3D PC data and projects them onto multiple 2D planes to generate several 2D feature images. These images are then compressed using the well-established High-Efficiency Video Coding (HEVC) method. The V-PCC decoder uses compressed information and decoding techniques to reconstruct the 3D PC. However, the PCs produced by V-PCC are often sparse, non-uniform, and contain artifacts. In many practical applications, it is necessary to recover complete PCs from partial ones in real time. This article presents a method for enhancing decoded PCs as a post-processing step in the V-PCC with reduced computational time. Our approach involves a 2D upsampling for the V-PCC occupancy image, which increases the density of the PC, and a 2D high-resolution auxiliary information modification algorithm for the 2D-3D conversion of high-resolution 3D PCs, which improves the uniformity and reduces the noise in the PC. The 3D high-resolution PC has been further enhanced using the developed 3D outlier removal and point regeneration algorithm. Our proposed work can significantly simplify the state-of-the-art super resolution methods for PCs and reduce the time complexity of 61–75% while maintaining a high level of quality in PCs.

3D point cloud generation is of great use for 3D scene modeling and understanding. Real-world 3D object point clouds can be properly described by a collection of low-level and high-level structures such as surfaces, geometric primitives, semantic parts,etc. In fact, there exist many different representations of a 3D object point cloud as a set of point groups. Existing frameworks for point cloud genera-ion either do not consider structure in their proposed solutions, or assume and enforce a specific structure/topology,e.g. a collection of manifolds or surfaces, for the generated point cloud of a 3D object. In this work, we pro-pose a novel decoder that generates a structured point cloud without assuming any specific structure or topology on the underlying point set. Our decoder is softly constrained to generate a point cloud following a hierarchical rooted tree structure. We show that given enough capacity and allowing for redundancies, the proposed decoder is very flexible and able to learn any arbitrary grouping of points including any topology on the point set. We evaluate our decoder on the task of point cloud generation for 3D point cloud shape completion. Combined with encoders from existing frameworks, we show that our proposed decoder significantly outperforms state-of-the-art 3D point cloud completion methods on the Shapenet dataset

Using 3D information is expected to handle challenges in 2D face recognition and improve system performance. Extracting pure facial part in face point cloud is usually the first step in a 3D face recognition system, which was mainly operated by manual in most previous studies. In this paper we propose a fully automatic approach for pure face extraction from 3D point cloud. Considering that 3D face point cloud can often be sensed in combination with color information, we use random forest classifiers to classify skin points and non-skin points in 3D point clouds. Usually there will be a few holes in the obtained skin point cloud, which mainly correspond to eyes, mouth, moustache, etc. We propose an approach based on nearest neighbor search method to fulfill the holes. Experiments show that the proposed approach can extract pure faces with different sizes, poses and expressions under various illumination conditions.

Converting three-dimensional (3D) point cloud data to two-dimensional (2D) image based on virtual structured-light system is a popular method in 3D data compression because the image format is easier to process by the exiting storing and transmitting methods. When this method is used in the 3D point cloud compression, the quantization error is introduced during the coordinate mapping. To solve this problem, a virtual structured-light 3D point cloud compression algorithm based on geometric reshaping is presented. And a least-square system parameter optimization method is proposed to further improve the data decoding accuracy. In the proposed method, the 3D spatial coordinates are reshaped to a 2D matrix first, and then the 2D matrix is stored as a “Holoimage” by using the parameter optimized virtual structured-light system. The quantization error introduced by coordinate mapping between X and Y coordinates of the 3D point cloud and the 2D image pixel coordinates is suppressed, so the decoding accuracy is improved. In addition, the geometric information of the 3D point cloud is hidden synchronously when the 3D point cloud is compressed, which is of great significance in the copyright protection of the point cloud data. Experiments verify the effectiveness of the proposed algorithm. The decoding root mean square error (RMSE) of the proposed method is decreased by 79.86% on average compared with the traditional one under the same compression ratio, and the compression ratio of the proposed method is 1.96 times bigger than the traditional one under the similar decoding accuracy.