Surface Point Cloud Research Articles

PHOTOGRAMMETRY AND COMPUTED TOMOGRAPHY POINT CLOUD REGISTRATION USING VIRTUAL CONTROL POINTS

Abstract. In this paper we propose a virtual control point based method for the registration of photogrammetry and computed tomography (CT) data. Because of the fundamentally different two data sources, conventional registration methods, such as manual control points registration or 3D local feature-based registration, are not suitable. The registration objective of our application is about 3D reconstructions of gyroscopes, which contain abundant geometric primitives to be fitted in the point clouds. In the first place, photogrammetry and CT scanning are applied, respectively, for 3D reconstructions. Secondly, our workflow implements a segmentation after obtaining the surface point cloud from the complete CT volumetric data. Then geometric primitives are fitted in this point cloud benefitting from the less complex cluster segments. In the next step, intersection operations of the parametrized primitives generates virtual points, which are utilized as control points for the transformation parameters estimation. A random sample consensus (RANSAC) method is applied to find the correspondences of both virtual control point sets using corresponding descriptors and calculates the transformation matrix as an initial alignment for further refining the registration. The workflow is invariant to pose, resolution, completeness and noise within our validation process.

A method for calculating the leaf inclination of soybean canopy based on 3D point clouds

ABSTRACT In order to achieve high-efficiency, low-cost and non-destructive measurement of soybean leaf inclination, three soybean varieties (Fudou-6, Kangxian-9 and Kangxian-13) were taken as research objects, a calculation method of soybean leaf inclination based on 3D point clouds was proposed. First, the original 3D point cloud data of soybean plants were obtained by Kinect 2.0 depth camera. Second, the grid method, depth threshold filtering and statistical filtering were used to pre-process the original 3D point cloud data. Third, the k-means clustering algorithm was used to segment the leaf point clouds. Further, the Delaunay triangulation was applied in reconstructing the surface of discrete point clouds. Finally, the ratio of leaf area to projected area was obtained by calculating the area of triangular mesh, so as to realize the calculation of soybean leaf inclination. At the same time, the calculated values were compared with those obtained by multispectral three-dimensional laser scanning device. The average relative error of soybean leaf inclination was 3.21%. The coefficient of determination (R 2) of the three varieties were 0.8317, 0.9075 and 0.9186, respectively. The results showed that the proposed method could meet the needs of non-destructive and accurate measurement of soybean leaf inclination.

Facial bones segmentation from skull point clouds based on deviation angle

AbstractFacial bones segmentation is an important step to understanding a skull. In this paper, a method for segmenting facial bones from skull point clouds is proposed. The segmentation is based on the deviation angle features. The method consists of three phases: surface normal estimation, feature extraction, and point clouds classification. The method is applied to skull point clouds derived from computed tomography images. For evaluation, the method is compared with manual segmentation. The method has succeeded in segmenting facial bones with Precision = 0.836, Recall = 0.951, and F = 0.890.

Rapid safety monitoring and analysis of foundation pit construction using unmanned aerial vehicle images

With the large-scale development and construction of urban underground spaces, the safety monitoring of foundation pit construction has gained much attention. Current safety monitoring of foundation pits is often achieved through manual onsite measurements with complex equipment and relies on complicated methods, which are labour intensive, time consuming, and tend to ignore the risk of accidental collapses caused by serious local deformation. To address this issue, this study develops a rapid safety monitoring and analysis method for foundation pit construction using Unmanned Aerial Vehicle (UAV) images. The safe inclination angle of the foundation pit slope was proposed as the safety-monitoring index. Taking the images of the foundation pit captured by the UAV as input, point cloud reconstruction and surface fitting were carried out. The local deformation distribution was introduced to evaluate the local safety state of the foundation pit. The experimental results validate the potential advantages of the method.

SUAS for 3D Tree Surveying: Comparative Experiments on a Closed-Canopy Earthen Dam

Defined as “personal remote sensing”, small unmanned aircraft systems (sUAS) have been increasingly utilized for landscape mapping. This study tests a sUAS procedure of 3D tree surveying of a closed-canopy woodland on an earthen dam. Three DJI drones—Mavic Pro, Phantom 4 Pro, and M100/RedEdge-M assembly—were used to collect imagery in six missions in 2019–2020. A canopy height model was built from the sUAS-extracted point cloud and LiDAR bare earth surface. Treetops were delineated in a variable-sized local maxima filter, and tree crowns were outlined via inverted watershed segmentation. The outputs include a tree inventory that contains 238 to 284 trees (location, tree height, crown polygon), varying among missions. The comparative analysis revealed that the M100/RedEdge-M at a higher flight altitude achieved the best performance in tree height measurement (RMSE = 1 m). However, despite lower accuracy, the Phantom 4 Pro is recommended as an optimal drone for operational tree surveying because of its low cost and easy deployment. This study reveals that sUAS have good potential for operational deployment to assess tree overgrowth toward dam remediation solutions. With 3D imaging, sUAS remote sensing can be counted as a reliable, consumer-oriented tool for monitoring our ever-changing environment.

A new single-view 3D pantograph reconstruction aided by prior CAD model

This paper proposes a novel 3D pantograph reconstruction method for the visualization and inspection based on the single-view scanned data and prior Computer-Aided Design (CAD) model. The proposed method uses the point cloud to reconstruct a scanned surface by the fusion of the point cloud and prior CAD surface. Then, a 3D model is reconstructed by integrating the scanned surface and the CAD model through an edge reconstruction process. Finally, the pantograph is inspected by the deviation analysis. The proposed method is validated by the pantograph data collected in a locomotive workshop with line structured light scanning measurement technology. Compared with the state-of-the-art methods, the proposed method can reconstruct the complete 3D model with high accuracy. Furthermore, the reconstructed 3D model has been successfully applied to the anomaly detection and visualization in the pantograph inspection, which is of great significance for improving the pantograph inspection efficiency and reducing traffic accidents.

A novel intelligent inspection robot with deep stereo vision for three-dimensional concrete damage detection and quantification

Crack assessment of reinforced concrete structures using stereo cameras is a potential way for increasing the efficiency and safety of infrastructure maintenance routines. However, existing damage methods for reinforced concrete structures are based on the segmentation of two-dimensional planes without consideration to the actual size of concrete damage. Furthermore, on-site structural monitoring requires the installation of a large number of contact-based sensing devices, resulting in the potentially excessive consumption of time and financial resources. Therefore, a new vision-based damage assessment method for reinforced concrete structures using a novel intelligent inspection robot with Internet of things–enabled data communication system is proposed in this article. In the first part of this article, the data acquisition system of the inspection robot and the algorithm for three-dimensional structural reconstruction using a stereo camera is discussed. The discussion is followed by a description of the method for crack quantification based on a new proposed deep-learning technique. Finally, to accomplish damage localization, the quantified concrete damage with actual size information is projected onto a three-dimensional surface point cloud reconstruction of the inspected structure. To verify the proposed method, a reinforced concrete column that has undergone cyclic loading failure is used as an inspection subject. The validation experiment demonstrated the ability of the proposed system to segment, localize, and quantify the damage in three-dimensional space with high accuracy.

Online detection of part size based on laser contour scanner

In aerospace die manufacturing, the processing quality of the processed workpiece is an important guarantee for the quality of the final product. Traditional detection methods have high labor intensity and low efficiency. Visual detection can only detect width, not depth information. Based on this, this paper proposes a non-contact laser scanning point cloud processing quality online detection method, through the laser contour scanner to obtain the actual machining parts of the surface point cloud information, and then in the Geomagic Control environment of the actual manufacturing parts point cloud information and Solid Works theoretical design model comparison analysis. Thus, the manufacturing error of the actual manufacturing parts can be obtained, which provides a fast and accurate judgment method for the quality control of the product manufacturing.

Smooth path generation method of laser cladding bit repair robot based on 3D automatic measurement of wear surface point cloud

In the work of repairing worn drill bits by laser cladding robots, the worn surface often presents irregular curved surfaces, which brings difficulties to cladding. Aiming at the generation of cladding path on irregular curved surfaces of worn drill bits, this paper proposes a method of searching interpolation points by equal defocusing. Firstly, perform point cloud slice processing on the 3D model of the worn drill bit. Then the point cloud model is fitted by the cubic B-spline curve. Finally, the proposed method is used to screen out the actual operating processing points of the robotic arm. By comparing the coincidence degree between the actual cladding path and the curved surface, the feasibility of using the method of searching interpolation points by equal defocusing on the fitted path to obtain a smooth path is verified.

Errors of Airborne Bathymetry LiDAR Detection Caused by Ocean Waves and Dimension-Based Laser Incidence Correction

Ocean waves are a vital environmental factor that affects the accuracy of airborne laser bathymetry (ALB) systems. As the regional water surface undulates with randomness, the laser propagation direction through the air–water surface will change and impact the underwater topographic result from the ALB system, especially for the small laser divergence system. However, the natural ocean surface changes rapidly over time, and uneven ocean surface point clouds from ALB scanning will cause an uncertain estimation of the laser propagation direction; therefore, a self-adaptive correction method based on the characteristics of the partial wave surface is key to improving the accuracy and applicability of the ALB system. In this paper, we focused on the issues of spatial position deviation caused by surface waves and position correction of the underwater laser footprint, and the dimension-based adaptive method is applied to attempt to correct the laser incidence angle. Simulation experiments and analysis of the actual measurement data from different ALB systems verified that the method can effectively suppress the influence of ocean waves. Furthermore, the inversion result of sea surface inclination changes is consistent with the surface wind wave reanalysis products. Based on the laser underwater propagation model in the strategy, we also quantitatively analyzed the influence of surface waves on laser bathymetry, which can guide the operation selection and data processing of the ALB system at specific water depths and under dynamic ocean conditions.

Fracture Extraction from Smooth Rock Surfaces Using Depth Image Segmentation

Due to the diversity of mineral particle features in rock, including sizes, shapes, arrangements, and structural connections, the color image of the rock sample surface at laboratory scale is quite complex. Thus, unacceptable misjudgement and information loss often occur when the traditional image processing algorithms are adopted. To improve the accuracy of fracture extraction, a method based on image processing algorithms is proposed in this paper to extract fractures from 3D point clouds. First, a high-precision original depth image is generated by gridding the rock surface point cloud data with the Kriging interpolation. The hill shading method is then applied to further clarify the fractures. Finally, the fractures are extracted from depth images and compared with color images. The results show that the integrity of the fracture skeleton is significantly improved and the error rate is reduced. In combination with color images, the physical fractures and other fracture-like features can be distinguished. The proposed method provides a new idea for extracting fractures in various destructive experiments on rocks, and may be developed for recognition of discontinuities at typical engineering scale.

ROBUST SURFACE-MATCHING REGISTRATION BASED ON THE STRUCTURE INFORMATION FOR IMAGE-GUIDED NEUROSURGERY SYSTEM

Image-to-patient space registration is to make the accurate alignment between the actual operating space and the image space. Although the image-to-patient space registration using paired-point is used in some image-guided neurosurgery systems, the current paired-point registration method has some drawbacks and usually cannot achieve the best registration result. Therefore, surface-matching registration is proposed to solve this problem. This paper proposes a surface-matching method that accomplishes image-to-patient space registration automatically. We represent the surface point clouds by the Gaussian Mixture Model (GMM), which can smoothly approximate the probability density distribution of an arbitrary point set. We also use mutual information as the similarity measure between the point clouds and take into account the structure information of the points. To analyze the registration error, we introduce a method for the estimation of Target Registration Error (TRE) by generating simulated data. In the experiments, we used the point sets of the cranium surface and the model of the human head determined by a CT and laser scanner. The TRE was less than 2[Formula: see text]mm, and the TRE had better accuracy in the front and the posterior region. Compared to the Iterative Closest Point algorithm, the surface registration based on GMM and the structure information of the points proved superior in registration robustness and accurate implementation of image-to-patient registration.

Single molecule network analysis identifies structural changes to caveolae and scaffolds due to mutation of the caveolin-1 scaffolding domain

Caveolin-1 (CAV1), the caveolae coat protein, also associates with non-caveolar scaffold domains. Single molecule localization microscopy (SMLM) network analysis distinguishes caveolae and three scaffold domains, hemispherical S2 scaffolds and smaller S1B and S1A scaffolds. The caveolin scaffolding domain (CSD) is a highly conserved hydrophobic region that mediates interaction of CAV1 with multiple effector molecules. F92A/V94A mutation disrupts CSD function, however the structural impact of CSD mutation on caveolae or scaffolds remains unknown. Here, SMLM network analysis quantitatively shows that expression of the CAV1 CSD F92A/V94A mutant in CRISPR/Cas CAV1 knockout MDA-MB-231 breast cancer cells reduces the size and volume and enhances the elongation of caveolae and scaffold domains, with more pronounced effects on S2 and S1B scaffolds. Convex hull analysis of the outer surface of the CAV1 point clouds confirms the size reduction of CSD mutant CAV1 blobs and shows that CSD mutation reduces volume variation amongst S2 and S1B CAV1 blobs at increasing shrink values, that may reflect retraction of the CAV1 N-terminus towards the membrane, potentially preventing accessibility of the CSD. Detection of point mutation-induced changes to CAV1 domains highlights the utility of SMLM network analysis for mesoscale structural analysis of oligomers in their native environment.

Analyzing Nonparametric Part-to-Part Variation in Surface Point Cloud Data

Surface point cloud data from three-dimensional optical scanners provide rich information about the surface geometry of scanned parts and potential variation in the surfaces from part-to-part. It is challenging, however, to make full use of these data for statistical process control purposes to identify sources of variation that manifest in a more complex nonparametric manner than variation in some prespecified set of geometric features of each part. We develop a framework for identifying nonparametric variation patterns that uses dissimilarity representation of the data and dissimilarity-based manifold learning, which helps discover a low-dimensional implicit manifold parameterization of the variation. Visualizing how the parts change as the manifold parameters are varied helps build an understanding of the physical characteristic of the variation. We also discuss using the nominal surface of parts when it is accessible to improve the computational expense and visualization aspects of the framework. Our approaches clearly reveal the nature of the variation patterns in a real cylindrical-part machining example and a simulated square head bolt example.

Extraction of Spectral Information from Airborne 3D Data for Assessment of Tree Species Proportions

With the rapid development of photogrammetric software and accessible camera technology, land surveys and other mapping organizations now provide various point cloud and digital surface model products from aerial images, often including spectral information. In this study, methods for colouring the point cloud and the importance of different metrics were compared for tree species-specific estimates at a coniferous hemi-boreal test site in southern Sweden. A total of three different data sets of aerial image-based products and one multi-spectral lidar data set were used to estimate tree species-specific proportion and stem volume using an area-based approach. Metrics were calculated for 156 field plots (10 m radius) from point cloud data and used in a Random Forest analysis. Plot level accuracy was evaluated using leave-one-out cross-validation. The results showed small differences in estimation accuracy of species-specific variables between the colouring methods. Simple averages of the spectral metrics had the highest importance and using spectral data from two seasons improved species prediction, especially deciduous proportion. Best tree species-specific proportion was estimated using multi-spectral lidar with 0.22 root mean square error (RMSE) for pine, 0.22 for spruce and 0.16 for deciduous. Corresponding RMSE for aerial images was 0.24, 0.23 and 0.20 for pine, spruce and deciduous, respectively. For the species-specific stem volume at plot level using image data, the RMSE in percent of surveyed mean was 129% for pine, 60% for spruce and 118% for deciduous.

Mapping and Semantic Modeling of Underground Parking Lots Using a Backpack LiDAR System

Presented in this paper is a novel method for the mapping and semantic modeling of an underground parking lot using 3D point clouds collected by a low-cost Backpack Laser Scanning (BLS) or LiDAR system. Our method consists of two parts: a Simultaneous Localization and Mapping (SLAM) algorithm based on Sparse Point Clouds (SPC) and a semantic modeling algorithm based on a modified PointNet model. The main contributions of this paper are as follows: (1) a probability frontend framework for the alignment of point clouds using the local point cloud surface variance as the weight of registration, which modifies registration failure caused by the lack of features in sparse point clouds, (2) a robust submap-based strategy for loop closure detection and back-end optimization under sparse point clouds, and (3) a modified PointNet model for classifying the point clouds of underground parking lots into four categories: ceiling, floor, wall, others. Experimental results show that our SPC-SLAM algorithm achieves centimeter-level accuracy (0.09% trajectory error rate) after closed loop processing in a Global Navigation Satellite System (GNSS)-denied underground parking lot, and precision of 84.8% in semantic segmentation.

Real-time tool to layer distance estimation for robotic subretinal injection using intraoperative 4D OCT.

The emergence of robotics could enable ophthalmic microsurgical procedures that were previously not feasible due to the precision limits of manual delivery, for example, targeted subretinal injection. Determining the distance between the needle tip, the internal limiting membrane (ILM), and the retinal pigment epithelium (RPE) both precisely and reproducibly is required for safe and successful robotic retinal interventions. Recent advances in intraoperative optical coherence tomography (iOCT) have opened the path for 4D image-guided surgery by providing near video-rate imaging with micron-level resolution to visualize retinal structures, surgical instruments, and tool-tissue interactions. In this work, we present a novel pipeline to precisely estimate the distance between the injection needle and the surface boundaries of two retinal layers, the ILM and the RPE, from iOCT volumes. To achieve high computational efficiency, we reduce the analysis to the relevant area around the needle tip. We employ a convolutional neural network (CNN) to segment the tool surface, as well as the retinal layer boundaries from selected iOCT B-scans within this tip area. This results in the generation and processing of 3D surface point clouds for the tool, ILM and RPE from the B-scan segmentation maps, which in turn allows the estimation of the minimum distance between the resulting tool and layer point clouds. The proposed method is evaluated on iOCT volumes from ex-vivo porcine eyes and achieves an average error of 9.24 µm and 8.61 µm measuring the distance from the needle tip to the ILM and the RPE, respectively. The results demonstrate that this approach is robust to the high levels of noise present in iOCT B-scans and is suitable for the interventional use case by providing distance feedback at an average update rate of 15.66 Hz.

Satellite Remote Sensing Image Stereoscopic Positioning Accuracy Promotion Based on Joint Block Adjustment With ICESat-2 Laser Altimetry Data

The next generation Ice, Cloud, and Land Elevation Satellite-2 (ICESat-2)/Advanced Topographic Laser Altimeter System (ATLAS) was launched in 2018 to provide large amounts of spaceborne laser altimetry data. The joint block adjustment with laser altimetry data and satellite remote sensing images can promote limited stereoscopic positioning accuracy without ground control points (GCPs). However, there are two problems in the joint block adjustment, that is, the reliable ATLAS laser altimetry points (LAPs) selection and the discrepancies of image-object points. To solve the above problems, a surface point cloud extraction algorithm based on wavelet reconstruction was proposed. Then, the LAPs selection constraint model was established, so the terrain influence was resolved. Finally, a block joint adjustment method has been proposed with remote sensing images and the reliable LAPs considering error of their plane coordinates, which could achieve the accurate corresponding image points of the reliable LAPs, remarkably decrease the discrepancies of image-object points. Fourteen pairs of ATLAS data, seven ZY3-02 stereo images and eleven Mapping Satellite-1 stereo images in Zhengzhou, China were collected to validate the method performance. Experiment results have shown that the consistency of the image-object points and the positioning accuracy of stereoscopic images without GCPs have been improved. Compared with the free net block adjustment, the height, east and north positioning accuracy of ZY3-02 and Mapping Satellite-1 stereo images have increased by 61%, 56%, 60% and 56%, 38%, 37%, respectively.

Rock Mass Discontinuity Extraction Method Based on Multiresolution Supervoxel Segmentation of Point Cloud

Discontinuity sets play an essential and pivotal role in the deformation monitoring and stability analysis of the rock mass, but there are still many challenges for accurately and rapidly extracting discontinuity. In this study, an extraction and characterization method of discontinuity sets based on point cloud supervoxel segmentation was proposed, which consists of four parts: 1) a multiresolution supervoxel segmentation (MRSS) algorithm was developed to classify unstructured point cloud into multiresolution facets and discrete points; 2) to extract the individual discontinuity, the single supervoxel that having spatial connectivity, similar planarity, and parallelism was clustered; 3) the orientation of individual discontinuity was calculated, respectively, based on the plane fitting parameters; and 4) for comprehensively analyzing the stability of rock mass, the improved K -means clustering algorithm is utilized to constructing the discontinuity sets that having similar orientation information. The novel method has been successfully tested on two practical cases (a rock cut and a side slope point cloud captured by the terrestrial laser scanner). A comparison with existing methods shows that the deviation of the discontinuity orientation for rock cut is less than 1°, and the time efficiency is increased by 2.6 times. In addition, the orientation variation of the seven principle discontinuity in the five temporal side slope point cloud is relatively small, the dip direction and angle are within 2° and 1°, respectively. We can conclude that the proposed method can efficiently obtain the full extent of every individual discontinuity from rock mass surface point cloud and accurately analyze their orientation information.

A study on 3D visual identification of the top structure of assembly building based on BIM

In order to overcome the problem of low recognition rate of building image when constructing three-dimensional model with traditional methods, a new idea of top shape and structure model of assembly building based on BIM is proposed in this paper. The method calculates and identifies the structural parameters of the top contour of assembly building according to BIM technology, retrieves the internal model of the top contour structure of assembly building from the database which is consistent with the result of feature recognition, and obtains the three-dimensional construction of image surface and the three-dimensional modelling of scattered point cloud surface through digital camera and point cloud data, so as to build the final three-dimensional model. Finally, the three-dimensional visualisation of assembly is realised. The experimental results show that the proposed method has high recognition rate.