Three-dimensional Point Cloud Research Articles

TIF-Reg: Point Cloud Registration with Transform-Invariant Features in SE(3).

Three-dimensional point cloud registration (PCReg) has a wide range of applications in computer vision, 3D reconstruction and medical fields. Although numerous advances have been achieved in the field of point cloud registration in recent years, large-scale rigid transformation is a problem that most algorithms still cannot effectively handle. To solve this problem, we propose a point cloud registration method based on learning and transform-invariant features (TIF-Reg). Our algorithm includes four modules, which are the transform-invariant feature extraction module, deep feature embedding module, corresponding point generation module and decoupled singular value decomposition (SVD) module. In the transform-invariant feature extraction module, we design TIF in SE(3) (which means the 3D rigid transformation space) which contains a triangular feature and local density feature for points. It fully exploits the transformation invariance of point clouds, making the algorithm highly robust to rigid transformation. The deep feature embedding module embeds TIF into a high-dimension space using a deep neural network, further improving the expression ability of features. The corresponding point cloud is generated using an attention mechanism in the corresponding point generation module, and the final transformation for registration is calculated in the decoupled SVD module. In an experiment, we first train and evaluate the TIF-Reg method on the ModelNet40 dataset. The results show that our method keeps the root mean squared error (RMSE) of rotation within 0.5 and the RMSE of translation error close to 0 m, even when the rotation is up to [−180, 180] or the translation is up to [−20 m, 20 m]. We also test the generalization of our method on the TUM3D dataset using the model trained on Modelnet40. The results show that our method’s errors are close to the experimental results on Modelnet40, which verifies the good generalization ability of our method. All experiments prove that the proposed method is superior to state-of-the-art PCReg algorithms in terms of accuracy and complexity.

3D Instance Segmentation and Object Detection Framework Based on the Fusion of Lidar Remote Sensing and Optical Image Sensing

Since single sensor and high-density point cloud data processing have certain direct processing limitations in urban traffic scenarios, this paper proposes a 3D instance segmentation and object detection framework for urban transportation scenes based on the fusion of Lidar remote sensing technology and optical image sensing technology. Firstly, multi-source and multi-mode data pre-fusion and alignment of Lidar and camera sensor data are effectively carried out, and then a unique and innovative network of stereo regional proposal selective search-driven DAGNN is constructed. Finally, using the multi-dimensional information interaction, three-dimensional point clouds with multi-features and unique concave-convex geometric characteristics are instance over-segmented and clustered by the hypervoxel storage in the remarkable octree and growing voxels. Finally, the positioning and semantic information of significant 3D object detection in this paper are visualized by multi-dimensional mapping of the boundary box. The experimental results validate the effectiveness of the proposed framework with excellent feedback for small objects, object stacking, and object occlusion. It can be a remediable or alternative plan to a single sensor and provide an essential theoretical and application basis for remote sensing, autonomous driving, environment modeling, autonomous navigation, and path planning under the V2X intelligent network space– ground integration in the future.

АЛГОРИТМ КОМПЛЕКСИРОВАНИЯ НЕСКОЛЬКИХ ИСТОЧНИКОВ ДАННЫХ В ОБЩУЮ КАРТУ ЗАНЯТОСТИ

The paper deals with the problem of constructing a passability model of environment witha large number of dynamic objects based on data from several different sensors. The aim of thework is to improve the algorithm for constructing the occupancy map by adding data from bothexisting algorithms for moving obstacles detection and from millimeter-wave automotive radar.The study solves the problem of combining data on static environment and dynamic objects intoone general passability model for further trajectory planning. The modification of the algorithmpresented in the article is able to combine data from both occupancy maps based on a threedimensional point cloud from any sensor such as lidar or radar, and arrays of bounding boxes ofobjects with known coordinates, sizes, and orientation. Data aggregation occurs at the level ofbuilding occupancy maps and does not impose requirements on the source of information aboutdynamic obstacles. The algorithm is able to refine the data on the position and size of the dynamicobject by speed from the radar, which allows to plan the trajectory taking into account the movementof dynamic objects. The parallel use of the classical approach allows to detect obstacles inthe event of an error in the output of the dynamic obstacle detection algorithm. The developedalgorithm works in real time on the Jetson AGX Xavier module, and is tested in real conditions ona mobile robotic platform in autonomous mode. Promising directions for further research to improvethe presented approach are formulated.

Low Cost Automatic Reconstruction of Tree Structure by AdQSM with Terrestrial Close-Range Photogrammetry

The quantitative structure model (QSM) contains the branch geometry and attributes of the tree. AdQSM is a new, accurate, and detailed tree QSM. In this paper, an automatic modeling method based on AdQSM is developed, and a low-cost technical scheme of tree structure modeling is provided, so that AdQSM can be freely used by more people. First, we used two digital cameras to collect two-dimensional (2D) photos of trees and generated three-dimensional (3D) point clouds of plot and segmented individual tree from the plot point clouds. Then a new QSM-AdQSM was used to construct tree model from point clouds of 44 trees. Finally, to verify the effectiveness of our method, the diameter at breast height (DBH), tree height, and trunk volume were derived from the reconstructed tree model. These parameters extracted from AdQSM were compared with the reference values from forest inventory. For the DBH, the relative bias (rBias), root mean square error (RMSE), and coefficient of variation of root mean square error (rRMSE) were 4.26%, 1.93 cm, and 6.60%. For the tree height, the rBias, RMSE, and rRMSE were—10.86%, 1.67 m, and 12.34%. The determination coefficient (R2) of DBH and tree height estimated by AdQSM and the reference value were 0.94 and 0.86. We used the trunk volume calculated by the allometric equation as a reference value to test the accuracy of AdQSM. The trunk volume was estimated based on AdQSM, and its bias was 0.07066 m3, rBias was 18.73%, RMSE was 0.12369 m3, rRMSE was 32.78%. To better evaluate the accuracy of QSM’s reconstruction of the trunk volume, we compared AdQSM and TreeQSM in the same dataset. The bias of the trunk volume estimated based on TreeQSM was −0.05071 m3, and the rBias was −13.44%, RMSE was 0.13267 m3, rRMSE was 35.16%. At 95% confidence interval level, the concordance correlation coefficient (CCC = 0.77) of the agreement between the estimated tree trunk volume of AdQSM and the reference value was greater than that of TreeQSM (CCC = 0.60). The significance of this research is as follows: (1) The automatic modeling method based on AdQSM is developed, which expands the application scope of AdQSM; (2) provide low-cost photogrammetric point cloud as the input data of AdQSM; (3) explore the potential of AdQSM to reconstruct forest terrestrial photogrammetric point clouds.

Achieve accurate recognition of 3D point cloud images by studying the scattering characteristics of typical targets

Three-dimensional imaging lidar has been widely used in various fields due to its high measurement accuracy, strong directionality, and three-dimensional visualization. At present, the three-dimensional point cloud imaging method of lidar mainly depends on the distance information of the target, but the intensity information of the target can play an auxiliary role in the recognition of the target, and it has also received extensive attention. The combination of intensity information and distance information enables the system to make more accurate decisions and improve the redundancy of the system. When scanning non-cooperative targets, if there is only distance information, the target will not be accurately identified, which may cause errors in decision-making. This paper studies and analyzes the scattering characteristics of typical lidar targets. The intensity of the scattered echo signal of the target is related to the target material, target distance, gray value and detection angle, so the detector's receiving scattering rate is different. For the extraction of the intensity and the distance signal, we propose a double-scale intensity-weighted centroid algorithm to achieve it, which ensures the accuracy of the signal. To this end, we build a database of the scattering characteristics of related targets, combined with the original distance information to realize the accurate recognition of the target in the lidar 3D point cloud image.

Research on splicing method of point cloud with insufficient features based on spatial reference

Suppose the current three-dimensional point cloud splicing method is well applicable to point clouds with insufficient features, it will benefit the noncontact surface quality detection and reverse design of objects with insufficient surface features. However, unfortunately, because the point cloud features are not rich, the current method is not applicable. We propose a point cloud splicing method based on spatial reference objects to solve the problem. Also, we describe the data processing method in detail. We first preprocess the point cloud and segment the spherical reference point cloud. Then, fit the spherical center coordinates of the spherical point cloud. Next, we calculate the relative spatial relationship between the target point cloud and the test point cloud and achieve coarse registration. Finally, we use the iterative closest point algorithm to complete the point cloud pair’s precise registration. From the experimental results, we can find that the proposed method has a splicing root mean square error of <0.03 mm for point clouds with insufficient features. When the point clouds’ initial relative spatial positions are significantly different, the average distance between the registered point clouds is <0.04 mm. In addition, experiments have proved that the arrangement of reference objects, the point cloud’s initial relative position, and the cumulative error have little effect on point cloud splicing accuracy. The proposed method does not rely on point cloud features and can achieve high-precision splicing of point clouds with insufficient features.

Analysis of college martial arts teaching posture based on 3D image reconstruction and wavelet transform

Martial arts is a traditional sports event of the Chinese nation, which carries history and culture. With the development of “Martial Arts on Campus Activities” in recent years, more and more schools have opened general martial arts courses. However, due to the more complex technical movements of martial arts, there are often varying degrees of gaps between the movements and standard movements. Based on this, this research introduces three-dimensional imaging technology on the basis of traditional physical education teaching methods, aiming to explore new martial arts teaching models through image reconstruction and posture analysis. First of all, in order to obtain a three-dimensional point cloud and three-dimensional line, this paper extracts and matches feature points and feature lines on the input image. Secondly, on the basis of obtaining dense matching and straight-line matching, this paper selects the image with the most feature line matching for reprojection. Wavelet transform is used in the process of image compression and coding, including signal decomposition and reconstruction steps. Finally, through the experimental test of martial arts teaching posture images in colleges and universities, it shows that the method of combining three-dimensional image reconstruction and wavelet transform proposed in this paper has good applicability and efficiency, and can provide a scientific reference for college martial arts teaching.

Pairwise Registration Algorithm for Large-Scale Planar Point Cloud Used in Flatness Measurement.

In this paper, an optimized three-dimensional (3D) pairwise point cloud registration algorithm is proposed, which is used for flatness measurement based on a laser profilometer. The objective is to achieve a fast and accurate six-degrees-of-freedom (6-DoF) pose estimation of a large-scale planar point cloud to ensure that the flatness measurement is precise. To that end, the proposed algorithm extracts the boundary of the point cloud to obtain more effective feature descriptors of the keypoints. Then, it eliminates the invalid keypoints by neighborhood evaluation to obtain the initial matching point pairs. Thereafter, clustering combined with the geometric consistency constraints of correspondences is conducted to realize coarse registration. Finally, the iterative closest point (ICP) algorithm is used to complete fine registration based on the boundary point cloud. The experimental results demonstrate that the proposed algorithm is superior to the current algorithms in terms of boundary extraction and registration performance.

Reconstruction of Two-dimensional Image to Three-dimensional Point Cloud Data for Detection of Empty Rot Defects in Tree Growth

In the application of nondestructive detecting of trees, it is a technical problem to use radar waves to detect tree specimens with growth defects, how to segment defect areas after obtaining two-dimensional images, and reverse simulate the detection results with three-dimensional point cloud data. Therefore, the method of extracting boundary information according to color features is studied to extract the boundary curve of empty rot area, and the selection of higher precision extraction algorithm is determined by comparing the boundary extraction results of HSV color space and RGB color space in laboratory According to the extracted void boundary curve, the reverse modeling is carried out, and the mapping from 2D inspection gray image to 3D space is realized, The point cloud data reconstruction needed for 3D modeling of multi-curved surfaces is obtained in reverse. The boundary curve extraction algorithm in this study is used to process the images of nondestructive testing of trees. Through comparative experiments and error analysis, the accurate modeling conclusion from inversion of 2D images to 3D point cloud data reconstruction by radar wave detection is verified, and the Core issue problem of point cloud reconstruction in the ill-conditioned area of tree growth and decay detected by radar wave is solved.

Automatic stem-leaf segmentation of maize shoots using three-dimensional point cloud

The application of 3D point cloud data in maize research is increasingly extensive. Currently, there are many approaches to acquiring three-dimensional (3D) point clouds of maize plants. However, automatic stem-leaf segmentation of maize shoots from 3D point clouds remains challenging, especially for new emerging leaves that are wrapped very closely together during the seedling stage. To address this issue, we propose an automatic segmentation method consisting of three steps: skeleton extraction, coarse segmentation based on the skeleton, and fine segmentation based on stem-leaf classification. The segmentation method was tested on 75 maize seedlings and compared with the manually obtained ground truth. The mean precision, mean recall, mean micro F1 score, and mean overall accuracy of our segmentation algorithm were 0.944, 0.956, 0.950 and 0.953, respectively. Using the segmentation results, two applications were also developed in this study, namely, phenotypic trait extraction and skeleton optimization. Six phenotypic parameters, namely, plant height, crown diameter, stem height and diameter, leaf width, and length, can be accurately and automatically measured. Furthermore, the values of R2 for the six phenotypic traits were all above 0.92. We also propose a skeleton optimization method that can extract the skeletons of the upper leaves completely and clearly. The results indicate that the proposed algorithm can automatically and precisely segment not only the fully expanded leaves but also the new leaves wrapped closely together. The proposed approach can play an important role in further maize research and applications, such as genotype-to-phenotype study, geometric reconstruction, and dynamic growth animation. We released the source code and test data at the web site https://github.com/syau-miao/seg4maize.git.

SepPCNET: Deeping Learning on a 3D Surface Electrostatic Potential Point Cloud for Enhanced Toxicity Classification and Its Application to Suspected Environmental Estrogens.

Deep learning (DL) offers an unprecedented opportunity to revolutionize the landscape of toxicity prediction based on quantitative structure-activity relationship (QSAR) studies in the big data era. However, the structural description in the reported DL-QSAR models is still restricted to the two-dimensional level. Inspired by point clouds, a type of geometric data structure, a novel three-dimensional (3D) molecular surface point cloud with electrostatic potential (SepPC) was proposed to describe chemical structures. Each surface point of a chemical is assigned its 3D coordinate and molecular electrostatic potential. A novel DL architecture SepPCNET was then introduced to directly consume unordered SepPC data for toxicity classification. The SepPCNET model was trained on 1317 chemicals tested in a battery of 18 estrogen receptor-related assays of the ToxCast program. The obtained model recognized the active and inactive chemicals at accuracies of 82.8 and 88.9%, respectively, with a total accuracy of 88.3% on the internal test set and 92.5% on the external test set, which outperformed other up-to-date machine learning models and succeeded in recognizing the difference in the activity of isomers. Additional insights into the toxicity mechanism were also gained by visualizing critical points and extracting data-driven point features of active chemicals.

Refraction error correction of Airborne LiDAR Bathymetry data considering sea surface waves

Airborne Light Detection And Ranging (LiDAR) Bathymetry (ALB) has an unparalleled advantage in integrated sea-land measurements, especially in the acquisition of topographic data in shallow water areas. Sea surface waves are an important factor affecting the quality of ALB data. This paper presents a method that strictly corrects the sea surface wave-induced refraction error for each seabed laser pulse without any complex modeling of the sea surface. In this method, an adaptive neighborhood selection method is first used to calculate the normal sea surface at the moment when a laser pulse enters the water, and then a laser pulse refraction error correction model considering sea surface waves is constructed. The distance condition equation, the angle condition equation and the coplanar condition equation are established based on Snell's law and the geometric relationship among the incident laser pulse, the normal vector of the wavy sea surface and the actual refracted ray in order to calculate the coordinates of each laser point at the seabed after refraction error correction. The experimental results show that sea surface waves have a significant impact on the three-dimensional point cloud coordinates of the underwater topography. Even in calm sea conditions, the plane coordinate displacement error of the seabed point caused by the sea surface waves may reach the meter level, and the depth coordinate displacement error can also exceed 0.2 m. The corrected displacement errors of the planimetric coordinates are significantly greater than the corrected displacement errors of the depth coordinates, which can effectively improve the quality of the ALB data.

High-volume point cloud data simplification based on decomposed graph filtering

Recent studies on three-dimensional (3D) point cloud data (PCD) simplification have played significant roles in computer-aided models for alleviating computational and storage burden. However, existing simplification methods are not suitable for the high-volume PCD with billions of point number, especially in construction industry. In this paper, a decomposed simplification method is developed to handle the high-volume buildings' PCD. Based on the divide-and-conquer philosophy, the new decomposed approach effectively reduces the memory usage. In the proposed approach, PCD is divided into several subsets according to the relationship of natural neighbor, and decomposed graph filtering with adaptive resampling rate is designed. It can be proved that the decomposed simplification results have no variance in the comparison with the optimal simplification of entire PCD. Verification experiments are conducted on different sizes of PCD, which indicate the effectiveness and feasibility of the proposed approach.

3D MSSD: A multilayer spatial structure 3D object detection network for mobile LiDAR point clouds

Point cloud-based object detection is vital and essential for many real-world applications, such as autonomous driving and robot vision. The PointPillars model has achieved the efficient detection of objects in front of a vehicle. However, the algorithm does not consider the spatial structures semantic information stored in the three-dimensional point cloud for a given spatial structure, thus leading to missed or false detections for objects with complex spatial structures or singular structures. We propose an approach based on PointPillars, which considers the spatial structure characteristics of 3D point clouds to enhance the detection accuracy. First, based on the specified range of the z-axis coordinates, the entire point cloud scene is divided into several layers so that the point cloud areas in the same height interval form one layer. Data from several layers are obtained. Second, the point clouds of several layers are processed with Pillar Feature Net to obtain several pseudoimages. Each pseudoimage represents the semantic information from the corresponding level of the point cloud. Third, the obtained pseudoimages from each level are merged with the pseudoimages of the entire scene to obtain a feature map with spatial structure characteristics. We apply a Region Proposal Network, and an object detection operator processes the feature map and obtains the result of object detection. Experiments show that the proposed method has a highly accurate detection effect for objects with complex spatial structures. In addition, the proposed method does not erroneously detect objects with similar semantic information after vertical dimension projection.

High resolution 3D terrestrial LiDAR for cotton plant main stalk and node detection

Dense three-dimensional point clouds provide opportunities to retrieve detailed characteristics of plant organ-level phenotypic traits, which are helpful to better understand plant architecture leading to its improvements via new plant breeding approaches. In this study, a high-resolution terrestrial LiDAR was used to acquire point clouds of plants under field conditions, and a data processing pipeline was developed to detect plant main stalks and nodes, and then to extract two phenotypic traits including node number and main stalk length. The proposed method mainly consisted of three steps: first, extract skeletons from original point clouds using a Laplacian-based contraction algorithm; second, identify the main stalk by converting a plant skeleton point cloud to a graph; and third, detect nodes by finding the intersection between the main stalk and branches. Main stalk length was calculated by accumulating the distance between two adjacent points from the lowest to the highest point of the main stalk. Experimental results based on 26 plants showed that the proposed method could accurately measure plant main stalk length and detect nodes; the average R2 and mean absolute percentage error were 0.94 and 4.3% for the main stalk length measurements and 0.7 and 5.1% for node counting, respectively, for point numbers between 80,000 and 150,000 for each plant. Three-dimensional point cloud-based high throughput phenotyping may expedite breeding technologies to improve crop production.

A DIGITAL ARCHIVE OF BOROBUDUR BASED ON 3D POINT CLOUDS

Abstract. Three-dimensional point clouds are becoming popular representations for digital archives of cultural heritage sites. The Borobudur Temple, located in Central Java, Indonesia, was built in the 8th century. Borobudur is considered one of the greatest Buddhist monuments in the world and was listed as a UNESCO World Heritage site. We are developing a virtual reality system as a digital archive of the Borobudur Temple. This research is a collaboration between Ritsumeikan University, Japan, the Indonesian Institute of Sciences (LIPI), and the Borobudur Conservation Office, Indonesia. In our VR system, the following three data sources are integrated to form a 3D point cloud: (1) a 3D point cloud of the overall shape of the temple acquired by photogrammetry using a camera carried by a UAV, (2) a 3D point cloud obtained from precise photogrammetric measurements of selected parts of the temple building, and (3) 3D data of the hidden relief panels recovered from the archived 2D monocular photos using deep learning. Our VR system supports both the first-person view and the bird’s eye view. The first-person view allows immersive observation and appreciation of the cultural heritage. The bird’s eye view is useful for understanding the whole picture. A user can easily switch between the two views by using a user-friendly VR user interface constructed by a 3D game engine.

CHANGE DETECTION OF TIME-SERIES 3D POINT CLOUDS USING ROBUST PRINCIPAL COMPONENT ANALYSIS

Abstract. The chances of acquiring three-dimensional (3D) point clouds have recently increased with the emergence of laser scanners. Hence, 3D monitoring of various objects through the accumulation of “time-series 3D point clouds,” which are point clouds of the same place at different times, is possible. Change detection is a task that is indispensable in 3D monitoring. One of the most common change detection method of 3D point clouds is simple subtraction between two data. However, this method is vulnerable to various errors. Therefore, change detection methods that are robust to errors are required. In this study, we developed robust principal component analysis, which has become popular in the background modelling of video images, to robustly recognize changes in time-series 3D point clouds. We first applied the proposed method to time-series depth images and confirmed its accuracy. We then applied the method to the digital elevation models of Mt. Unzen, which were acquired between 2003 and 2016, to recognize yearly elevation changes. The results show that the proposed method robustly recognizes elevation changes with a properly set parameter.

EVALUATION OF TUNNEL EXCAVATION COMBINING TERRESTRIAL LASER SCANNING POINT CLOUDS AND DESIGN MODELS

Abstract. The quality of tunnel excavation is evaluated by comparing the excavated tunnel and the design model. Terrestrial laser scanning (TLS) provides surveyors with dense and accurate three-dimensional (3D) point clouds for excavation model reconstruction. However, sufficient attention has not been paid to incorporating design models for tunnel point cloud processing. In this paper, a technical framework that combines TLS point clouds and the design model for tunnel excavation evaluation is proposed. Firstly, the point clouds are sliced into cross-sections and the feature points are accordingly extracted. Then, considering the structure of the design model, feature point deficiencies are repaired by topological and parametric model interpolation. Finally, the excavation quality is evaluated in terms of the deviation of centerlines and 3D models. This method is validated in the case study. Experiments show that the deviation of centerline azimuth is acceptable but there remain considerable overbreak and underbreak, which respectively account for 20.6% and 11.2% of the design excavation volume.

Multispectral LiDAR Point Cloud Classification Using SE-PointNet++

A multispectral light detection and ranging (LiDAR) system, which simultaneously collects spatial geometric data and multi-wavelength intensity information, opens the door to three-dimensional (3-D) point cloud classification and object recognition. Because of the irregular distribution property of point clouds and the massive data volume, point cloud classification directly from multispectral LiDAR data is still challengeable and questionable. In this paper, a point-wise multispectral LiDAR point cloud classification architecture termed as SE-PointNet++ is proposed via integrating a Squeeze-and-Excitation (SE) block with an improved PointNet++ semantic segmentation network. PointNet++ extracts local features from unevenly sampled points and represents local geometrical relationships among the points through multi-scale grouping. The SE block is embedded into PointNet++ to strengthen important channels to increase feature saliency for better point cloud classification. Our SE-PointNet++ architecture has been evaluated on the Titan multispectral LiDAR test datasets and achieved an overall accuracy, a mean Intersection over Union (mIoU), an F1-score, and a Kappa coefficient of 91.16%, 60.15%, 73.14%, and 0.86, respectively. Comparative studies with five established deep learning models confirmed that our proposed SE-PointNet++ achieves promising performance in multispectral LiDAR point cloud classification tasks.

Automated point cloud classification using an image-based instance segmentation for structure from motion

Point cloud constantly gains popularity as a visualization tool in numerous fields including civil infrastructure scope. However, automatic point cloud classification for civil infrastructures such as piers is challenging due to untidy scenes, gigantic sizes, and image feature-rich objects that can generate many cloud points. Moreover, the lack of training point clouds and unrealistic synthetic data preventing deep learning to fully support the three-dimensional point cloud classification. This paper proposes Point cloud Classification based on image-based Instance Segmentation (PCIS), an automated point cloud classification based on two-dimensional digital images from a daily work basis. These images are processed into the pre-trained network in PCIS to generate mask images, which are later used to create three-dimensional masks based on the projection from the solved camera parameters. The cloud points located inside these masks are classified as the cloud point of interest. The experiment result showed that PCIS correctly classified the point cloud and achieved up to 0.96 F1-score from a one-class classification sample and 0.83 F1-score from a six-class classification sample in our validation process. Our study has proved that normal digital images can also be used to train deep learning to classify the three-dimensional point cloud.