Point Cloud Model Research Articles

Computer vision for road imaging and pothole detection: a state-of-the-art review of systems and algorithms

Abstract Computer vision algorithms have been utilized for 3-D road imaging and pothole detection for over two decades. Nonetheless, there is a lack of systematic survey articles on state-of-the-art (SoTA) computer vision techniques, especially deep learning models, developed to tackle these problems. This article first introduces the sensing systems employed for 2-D and 3-D road data acquisition, including camera(s), laser scanners and Microsoft Kinect. It then comprehensively reviews the SoTA computer vision algorithms, including (1) classical 2-D image processing, (2) 3-D point cloud modelling and segmentation and (3) machine/deep learning, developed for road pothole detection. The article also discusses the existing challenges and future development trends of computer vision-based road pothole detection approaches: classical 2-D image processing-based and 3-D point cloud modelling and segmentation-based approaches have already become history; and convolutional neural networks (CNNs) have demonstrated compelling road pothole detection results and are promising to break the bottleneck with future advances in self/un-supervised learning for multi-modal semantic segmentation. We believe that this survey can serve as practical guidance for developing the next-generation road condition assessment systems.

A Multiscale Multi-Feature Deep Learning Model for Airborne Point-Cloud Semantic Segmentation

In point-cloud scenes, semantic segmentation is the basis for achieving an understanding of a 3D scene. The disorderly and irregular nature of 3D point clouds makes it impossible for traditional convolutional neural networks to be applied directly, and most deep learning point-cloud models often suffer from an inadequate utilization of spatial information and of other related point-cloud features. Therefore, to facilitate the capture of spatial point neighborhood information and obtain better performance in point-cloud analysis for point-cloud semantic segmentation, a multiscale, multi-feature PointNet (MSMF-PointNet) deep learning point-cloud model is proposed in this paper. MSMF-PointNet is based on the classical point-cloud model PointNet, and two small feature-extraction networks called Mini-PointNets are added to operate in parallel with the modified PointNet; these additional networks extract multiscale, multi-neighborhood features for classification. In this paper, we use the spherical neighborhood method to obtain the local neighborhood features of the point cloud, and then we adjust the radius of the spherical neighborhood to obtain the multiscale point-cloud features. The obtained multiscale neighborhood feature point set is used as the input of the network. In this paper, a cross-sectional comparison analysis is conducted on the Vaihingen urban test dataset from the single-scale and single-feature perspectives.

Research on the evaluation method of lunar soil disturbance based on the fitting of Gaussian surface

It is crucial to maintain the layer sequence information on in situ lunar soil during the lunar soil sampling. In this paper, a discrete element model for direct push sampling of the lunar weathering layer is established, and the relationship between the perturbation rate of the lunar soil layer sequence information and the sampling depth is obtained. A quantitative evaluation method of the perturbation rate of the structural information on the lunar soil layer sequence based on the Gaussian curvature is proposed by reconstructing the center-of-mass point cloud model of the marked layer lunar soil particles in MATLAB. The proposed method can accurately describe the perturbation rate of lunar soil layer sequence information and can analyze the degree of lunar soil disturbance according to the displacement of the sampling tube and mechanical parameters in real-time inversion during the sampling process. When the sampling depth is 40 mm, the minimum and maximum perturbation rates of sampling along the penetration direction of the machine are 5.261 % and 27.649 %, respectively. The perturbation rate shows a trend of first decreasing and then increasing. The method proposed in this study could effectively solve the problem that the disturbance of lunar soil bedding information was difficult to evaluate in the sampling process of lunar regolith.

3DVerifier: efficient robustness verification for 3D point cloud models

3D point cloud models are widely applied in safety-critical scenes, which delivers an urgent need to obtain more solid proofs to verify the robustness of models. Existing verification method for point cloud model is time-expensive and computationally unattainable on large networks. Additionally, they cannot handle the complete PointNet model with joint alignment network that contains multiplication layers, which effectively boosts the performance of 3D models. This motivates us to design a more efficient and general framework to verify various architectures of point cloud models. The key challenges in verifying the large-scale complete PointNet models are addressed as dealing with the cross-non-linearity operations in the multiplication layers and the high computational complexity of high-dimensional point cloud inputs and added layers. Thus, we propose an efficient verification framework, 3DVerifier, to tackle both challenges by adopting a linear relaxation function to bound the multiplication layer and combining forward and backward propagation to compute the certified bounds of the outputs of the point cloud models. Our comprehensive experiments demonstrate that 3DVerifier outperforms existing verification algorithms for 3D models in terms of both efficiency and accuracy. Notably, our approach achieves an orders-of-magnitude improvement in verification efficiency for the large network, and the obtained certified bounds are also significantly tighter than the state-of-the-art verifiers. We release our tool 3DVerifier via https://github.com/TrustAI/3DVerifier for use by the community.

Point-Cloud Self-Adaptive Pose Transfer Based on Skinning Deformation

A point cloud pose transfer method based on self-adaptive weights skinning deformation is proposed to avoid the problems such as model tearing, distortion and insufficient pose transfer caused by the global smoothing coefficients of DDM. Firstly, the isomorphic skeletons of the source point cloud and the reference point cloud are extracted through the improved Laplacian contraction skeleton extraction method, the locations of the joints are optimized via clustering, and the geometric transformations of the joints between two isomorphic skeletons are calculated according to the isomorphic skeletons. Then, the per-vertex smoothing weights of DDM are improved based on the deformation degree of the vertex and clustering partition, and the rigid skinning weights of the vertices of source model are determined by using the skeleton hierarchy information. Finally, pose transfer is achieved via reformulating skinning as solving the rigid transformation matrix. Skeleton extraction and skinning deformation are carried out on the existing human point cloud models from MPI DYNA and animal point cloud models from MIT. The isomorphic skeletons generated by the proposed method are without redundant branches and joints. The experiment results show that the reference pose is studied sufficiently and the details of the source are kept well.

No-Reference Quality Assessment for 3D Colored Point Cloud and Mesh Models

To improve the viewer's Quality of Experience (QoE) and optimize computer graphics applications, 3D model quality assessment (3D-QA) has become an important task in the multimedia area. Point cloud and mesh are the two most widely used digital representation formats of 3D models, the visual quality of which is quite sensitive to lossy operations like simplification and compression. Therefore, many related studies such as point cloud quality assessment (PCQA) and mesh quality assessment (MQA) have been carried out to measure the visual quality degradations of 3D models. However, a large part of previous studies utilize full-reference (FR) metrics, which indicates they can not predict the quality level with the absence of the reference 3D model. Furthermore, few 3D-QA metrics consider color information, which significantly restricts their effectiveness and scope of application. In this paper, we propose a no-reference (NR) quality assessment metric for colored 3D models represented by both point cloud and mesh. First, we project the 3D models from 3D space into quality-related geometry and color feature domains. Then, the 3D natural scene statistics (3D-NSS) and entropy are utilized to extract quality-aware features. Finally, machine learning is employed to regress the quality-aware features into visual quality scores. Our method is validated on the colored point cloud quality assessment database (SJTU-PCQA), the Waterloo point cloud assessment database (WPC), and the colored mesh quality assessment database (CMDM). The experimental results show that the proposed method outperforms most compared NR 3D-QA metrics with competitive computational resources and greatly reduces the performance gap with the state-of-the-art FR 3D-QA metrics. The code of the proposed model is publicly available now to facilitate further research.

Semantic maps for cross-view relocalization of terrestrial to UAV point clouds

• The semantic maps from point clouds favor re-localization. • Design rules to punish the predicted results for a novel template matching. • The proposed method can even address a single TLS scan to a global map. To generate a complete 3D point cloud model for complex urban environments, data must be captured from both top-view and street-view sides due to the limited field of view of the sensors. Unmanned aerial vehicle-laser scanning (UAV-LS) point clouds are captured from a top view, and they often contain information about building roofs and tree crowns. In contrast, terrestrial point clouds, collected by terrestrial laser scanners (TLS) and mobile laser scanners (MLS), are captured from a street view and contain information about building facades and tree stems. These two types of point clouds are challenging to register without manual efforts or target-based assistance because of (1) very limited overlapping regions caused by the cross-view and (2) the wide search of regions for the automatic localization of terrestrial scans in a large-scale scene from a UAV scan. Consequently, we highlight that a semantic map can be developed by utilizing the invariant semantic and geometric relationships to automatically locate from a local scan to a global scan. In our tasks, the buildings and tree regions are detected and assigned as semantic objects and input to generate a 2.5D semantic map related to their actual location information. We propose a novel template matching strategy combined with a penalty system to realize the localization. Specifically, the templates can be produced by the obtained semantic maps, and then the best localization can be achieved by a graded punishment method based on the developed rules. We emphasize that the proposed method can handle the integration of multiple-aligned scans and even a single scan by using a terrestrial scanner to a UAV scan owing to the stability and continuity of semantic information. In the experiments, we used a UAV to capture the lidar point clouds over the campus of the Southwest Jiaotong university and generated a global semantic map offline. Keeping the global map in hand, we collected the data from the street views by TLS and MLS also the campus and then implemented the localization from TLS/MLS point clouds to the UAV point clouds. The evaluations are grouped into two grades related to the degree of difficulty: (1) the regular group, which contains the multiple-registered TLS scans and MLS scans, and (2) the challenging group, which has independent TLS scans. In general, from the experimental results, for the proposed method, the success rate of localization is up to 100% in the regular group and 90% in the challenging group, indicating that only one in ten tests fails for the challenging group.

DOPNet: Achieving Accurate and Efficient Point Cloud Registration Based on Deep Learning and Multi-Level Features.

Point cloud registration aims to find a rigid spatial transformation to align two given point clouds; it is widely deployed in many areas of computer vision, such as target detection, 3D localization, and so on. In order to achieve the desired results, registration error, robustness, and efficiency should be comprehensively considered. We propose a deep learning-based point cloud registration method, called DOPNet. DOPNet extracts global features of point clouds with a dynamic graph convolutional neural network (DGCNN) and cascading offset-attention modules, and the transformation is predicted by a multilayer perceptron (MLP). To enhance the information interaction between the two branches, the feature interaction module is inserted into the feature extraction pipeline to implement early data association. We compared DOPNet with the traditional method of using the iterative closest point (ICP) algorithm along with four learning-based registration methods on the Modelnet40 data set. In the experiments, the source and target point clouds were generated by sampling the original point cloud twice independently; we also conducted additional experiments with asymmetric objects. Further evaluation experiments were conducted with point cloud models from Stanford University. The results demonstrated that our DOPNet method outperforms these comparative methods in general, achieving more accurate and efficient point cloud registration.

3D MODEL OF BUILDING BASED ON MULTI-SOURCE DATA FUSION

Abstract. The process of building a digital city is the construction of a three-dimensional city model. The construction of 3 d data model is the process of multi-source spatial information collection and fusion. The current means of spatial information collection are mainly divided into three-dimensional laser technology and UAV photogrammetry technology. The three-dimensional laser technology is mainly based on ground laser, which has high acquisition accuracy for the bottom part of the building, but it is insufficient for the information collection at the top of the building. UAV photogrammetry has a large area and high efficiency spatial information collection method. At the same time, a large three-dimensional city model of the building can also be constructed. However, due to practical problems such as flight height and ground occlusion, the model constructed by UAV photogrammetry has a perfect top and poor bottom quality. The two methods are combined to build a more perfect point cloud model. Aiming at the above problems, a three-dimensional modeling method of buildings under multi-source data fusion is proposed. Firstly, the point cloud data and image data obtained by 3D laser scanner and UAV are preprocessed, and the two data forms are transformed into high-precision point cloud data. Then two sets of point cloud data are fused to generate the whole point cloud data. Finally, based on the patch generated by the whole point cloud, the point cloud model is reverse modeled. Get the corresponding building entity model, in order to get a more perfect 3D city model.

A Feature Extraction Algorithm Based on Optimal Neighborhood Size

The feature extraction of a point cloud fragment model is the basis of fragment splicing, which provides the technical support for research on the segmentation, splicing, and restoration of fragment surfaces. High-quality feature extraction, however, is a complicated process due to the diversity of the surface information of a fragment model. For this subject, a high-efficient point cloud feature extraction method was proposed to address a new method for extracting feature lines. First, the projection distance feature of the point cloud model was calculated to identify the potential feature points. Furthermore, the local information of the possible feature points was used to construct the adaptive neighborhoods for identifying the feature points based on neighborhoods of the model. The clustering fusion of the feature points was proposed according to the discrimination threshold values of the feature points. Finally, the Laplace operator was utilized to refine and connect the feature points to form smooth feature lines. The experimental results showed that the proposed method was automatic, highly efficient, and with good adaptability that could effectively extract the detailed features and construct the complete feature lines. Moreover, results showed that the provided framework could extract the features of simple structure models and be feasible to a certain extent for fragment models with abundant features.

Rotation Estimation Based on Serial Network and Application in Cave Buddha Statues

Rotation estimation is one of the crucial elements of pose estimation. Accurate pose estimation facilitates the observation of objects in the metaverse and is also the most fundamental part of automated archaeological mapping. Many studies have been conducted on 6D pose estimation, but no systematic study has only been conducted on 3D rotation estimation. We systematically study the rotation estimation problem for 3D point cloud models. Firstly, we discuss the representation of rotation and how it affects the training of the neural network. After that, the dataset is created. A new method of generating labels is also proposed, in which two points are used as labels, with the two points being able to recover the initial orientation. Finally, a new serial network is proposed, which can be used to improve the performance of rotational estimation systems. The research described in this paper has been successfully applied to the restoration of the Huangze Temple Grotto Buddha statue in Guangyuan, Sichuan Province, providing a reliable scheme for solving the rotation estimation problem with faster convergence and higher accuracy.

Multistage Adaptive Point-Growth Network for Dense Point Cloud Completion

The point cloud data from actual measurements are often sparse and incomplete, making it difficult to apply them directly to visual processing and 3D reconstruction. The point cloud completion task can predict missing parts based on a sparse and incomplete point cloud model. However, the disordered and unstructured characteristics of point clouds make it difficult for neural networks to obtain detailed spatial structures and topological relationships, resulting in a challenging point cloud completion task. Existing point cloud completion methods can only predict the rough geometry of the point cloud, but cannot accurately predict the local details. To address the shortcomings of existing point cloud complementation methods, this paper describes a novel network for adaptive point cloud growth, MAPGNet, which generates a sparse skeletal point cloud using the skeletal features in the composite encoder, and then adaptively grows the local point cloud in the spherical neighborhood of each point using the growth features to complement the details of the point cloud in two steps. In this paper, the Offset Transformer module is added in the process of complementation to enhance the contextual connection between point clouds. As a result, MAPGNet improves the quality of the generated point clouds and recovers more local detail information. Comparing our algorithm with other state-of-the-art algorithms in different datasets, experimental results show that our algorithm has advantages in dense point cloud completion.

Analysis of the Application of Deep Learning in Model Reconstruction of Ancient Buildings

With the rapid development of interactive 3D graphics technology, as well as the growing demand for virtual reality, digital urbanization and digital cultural heritage protection and time-consuming and inefficient traditional artificial building modeling methods have been far from meeting the rapid and intelligent needs of the application market and automatic. Architectural modeling methods have been paid more and more attention. Architectural modeling is an application-oriented comprehensive research field. According to different application scenarios, its research methods cover many technical fields and disciplines. This paper introduces a method of modeling ancient buildings using depth image estimation, spherical projection mapping, 3D adversarial generation network, and other techniques. The characteristics of architectural modeling methods are discussed from different disciplinary and technical perspectives. Second, the three major schools of architectural modeling technology, mainly the process modeling method, image modeling method, and point cloud modeling method, as well as the inverse process modeling method, which has attracted much attention and challenges in recent years, are summarized in detail. Then, the problem of building modeling is discussed. The problems and challenges of building modeling technology are analyzed, and the future development trend is predicted.

A Reverse Modeling Method Based on CAD Model Prior and Surface Modeling

Reverse engineering is an important process of real model digitization. However, the existing methods are limited by the characteristics of the modeling object; the modeling is low efficient, has poor versatility, and unstable accuracy. To complete the reverse engineering of a real workpiece with irregular deformation based on the ideal computer-aided design (CAD) model, a high-precision reverse engineering method of the workpiece based on the CAD model prior was presented. Through the registration of the ideal CAD model and the point cloud model of the real workpiece, the geometric feature position information and feature constraint information contained in the CAD model are transmitted to the modeling process, which helps to improve the accuracy and efficiency of reverse engineering. This method is applied to the reverse engineering process of the engine compartment with slight irregular deformation, the modeling accuracy reaches 0.04 mm, and the high-precision reverse engineering of the engine compartment is successfully realized.

Multi-Objective Location and Mapping Based on Deep Learning and Visual Slam.

Simultaneous localization and mapping (SLAM) technology can be used to locate and build maps in unknown environments, but the constructed maps often suffer from poor readability and interactivity, and the primary and secondary information in the map cannot be accurately grasped. For intelligent robots to interact in meaningful ways with their environment, they must understand both the geometric and semantic properties of the scene surrounding them. Our proposed method can not only reduce the absolute positional errors (APE) and improve the positioning performance of the system but also construct the object-oriented dense semantic point cloud map and output point cloud model of each object to reconstruct each object in the indoor scene. In fact, eight categories of objects are used for detection and semantic mapping using coco weights in our experiments, and most objects in the actual scene can be reconstructed in theory. Experiments show that the number of points in the point cloud is significantly reduced. The average positioning error of the eight categories of objects in Technical University of Munich (TUM) datasets is very small. The absolute positional error of the camera is also reduced with the introduction of semantic constraints, and the positioning performance of the system is improved. At the same time, our algorithm can segment the point cloud model of objects in the environment with high accuracy.

Deep-Learning-Guided Point Cloud Modeling with Applications in Intelligent Manufacturing.

Machining feature recognition is a key technology to realize CAD/CAPP/CAM system integration. Aiming at high robustness of traditional processing feature recognition in image reasoning, an automatic processing shape recognition method based on fuzzy learning of processing surrounding point black data is proposed. The Cloud RNN in the PointNet stage strongly demonstrates that the framework originates from convolutional neural spider webs. Protector shape for detailed discoloration data on constructed prominence surfaces for automatic rifle recognition is conducted. Spot staining data sample library is also constructed. The prosecuting feature recognizer gained advantages through sample training, which realized robot-style notification of 36 processing shapes. This is conducted with a recognition accuracy rate of over 90%. The method is simple and efficient, although it is not suitable for point cloud data with backlash and defects. It is sensible and still has usable robustness and confirmation performance against mischief around shape peripheries due to shape intersections.

An Adaptive Irregular Grid Generation Method for Scattering Simulation of Electrically Large Terrains

This communication provides a novel adaptive irregular grid model (AIGM) to improve the scattering simulation efficiency of electrically large terrains. This model distinguishes the steep and flat regions of the measured topographic point cloud models by a threshold. To determine the optimal threshold, we present the height gradient as the criterion to describe the topographic undulation. The optimal threshold is located at the inflection point of the curve describing topographic similarity in the range of height gradient. This generated AIGM can reduce the grid quantity and maintain consistency with the original point cloud model. It is also valuable in the popular coherent and incoherent (C&I) scattering model of global navigation satellite system reflectometry (GNSS-R). In this communication, the coherent component can be expressed by a simple product of the electric field of the conductor grid, the characteristic function of roughness, and the Fresnel reflection coefficient. The incoherent component comes from the diffuse scattering simulated by the integral equation method (IEM). Compared with the traditional solution to C&I scattering by using the Kirchhoff approximation (KA), the modified C&I model has a more extensive application. The precision of the AIGM-based-C&I model is verified by traditional physical optics (PO). The excellent simulation efficiency of the AIGM-based-C&I model is also demonstrated.

Retrieval of a 3D CAD Model of a Transformer Substation Based on Point Cloud Data

When constructing a three-dimensional model of a transformer substation, it is critical to quickly find the 3D CAD model corresponding to the current point cloud data from a large number of transformer substation model libraries (due to the complexity and variety of models in the model base). In response to this problem, this paper proposes a method to quickly retrieve a 3D CAD model. Firstly, a 3D CAD model that shares the same size as the current point cloud model bounding box is extracted from the model library by the double-layer bounding box screening method. Then, the selected 3D CAD model is finely compared with the point cloud model by the multi-view method. The 3D CAD model that has the highest degree of corresponding to the point cloud data is acquired. The proposed algorithm, compared to other similar methods, has the advantages of retrieval accuracy and high efficiency.

Indoor and outdoor multi-source 3D data fusion method for ancient buildings

Ancient buildings carry important information, such as ancient politics, economy, culture, customs. However, with the course of time, ancient buildings are often damaged to different degrees, so the restoration of ancient buildings is of great importance from the historical point of view. There are three commonly used non-contact measurement methods, including UAV-based oblique photogrammetry, terrestrial laser scanning, and close-range photogrammetry. These methods can provide integrated three-dimensional surveys of open spaces, indoor and outdoor surfaces for ancient buildings. Theoretically, the combined use of the three measurement methods can provide 3D (three-dimensional) data support for the protection and repair of ancient buildings. However, data from the three methods need to be fused urgently, because if the image data is not used, it will lead to a lack of real and intuitive texture information, and if only image matching point clouds are used, their accuracy will be lower than that of terrestrial laser scanning point clouds, and it will also lead to a lack of digital expression for components with high indoor historical value of ancient buildings. Therefore, in this paper, a data fusion method is proposed to achieve multi-source and multi-scale 3D data fusion of indoor and outdoor surfaces. It takes the terrestrial laser point cloud as the core, and based on fine component texture features and building outline features, respectively, the ground close-range image matching point cloud and UAV oblique image matching point cloud are registered with the terrestrial laser point cloud. This method unifies the data from three measurements in the point cloud and realizes the high-precision fusion of these three data. Based on the indoor and outdoor 3D full-element point cloud formed by the proposed method, it will constitute a visual point cloud model in producing plans, elevations, sections, orthophotos, and other elements for the study of ancient buildings.

Statistical Estimation of Blast Fragmentation by Applying 3D Laser Scanning to Muck Pile

Run-of-mine fragmentation is an important aspect of mine productivity optimization, as it affects all mine-to-mill processes. The current blasting fragmentation calculation methods do not consider the 3D geometric information. Therefore, their calculation results are imprecise. 3D laser scanning is a technique for extracting the 3D geometric information of an object by constructing a 3D point cloud model, with which extra information on the geometrical characteristics of an object could be captured than with the technique of 2D image processing. In this paper, 3D laser scanning technology was utilized for the calculation of the rock blocks on the surface of a muck pile, and the information about the surface blocks was utilized as the samples for the statistical estimation of blasting fragmentation of muck pile (BFMP). Monte Carlo simulation was utilized as the statistical estimation method for the BFMP. In the lab experiment, results from 2D image processing technique and from 3D laser scanning technique combined with statistical estimate were compared with the physical measurements utilizing a water tank, which show that results with 3D laser scanning are more similar to the physical measurement. Finally, the applicability of 3D laser scanning technology combined with statistical methods to the calculation of blast fragmentation was estimated through field tests in Biesikuduke and Santanghu mine, two open-pit coal mines in Xinjiang Province of western China. Results show that the accuracy of the statistical estimation results of BFMP has a particle size deviation of 1–3 cm.