Unstructured Point Clouds Research Articles

BIM generation from 3D point clouds by combining 3D deep learning and improved morphological approach

The demand for the models of BIM has increased due to the numerous applications in emergency response and location-based services. Creating a semantically rich 3D interior model has traditionally been a very manual process. Although the latest LiDAR scanning and photogrammetry techniques efficiently capture the existing building, the automatic reconstruction of complete volumetric and functional interior models is still an intensive and challenging research topic. This paper presents a novel parametric modeling method for reconstructing semantic volumetric building interiors from the unstructured point cloud of a building. Unlike existing partitioning-based methods, our proposed method overcomes the limitations by providing a flexible framework for combining 3D Deep Learning and an improved morphological approach for inverse BIM modeling. By employing a flexible and robust three-step mechanism, the proposed method first classifies the point cloud into thirteen types using a 3D Deep Learning method, then extracts the surfaces and generates the initial feature space. In regularizing the space and modeling BIM, the space boundaries are optimized by formulating the subordination between the grid cells generated by walls and the segmented boundary shapes generated by the gridded floor plan through energy minimization. Then, a grammar-enhanced point-line polygon and parametric description are designed to generate the final BIM model. By using a robust space partitioning and optimization method, three main goals are achieved: recovery of geometric and semantic information, robustness in different data sources (especially in non-Manhattan scenes), parametric BIM modeling in our proposed method. The paper demonstrates the potential of our algorithms for both RGB-D and LiDAR point clouds acquired from scenes in Manhattan and outside Manhattan. Experimental results show that this method is capable of automatically generating a geometrically and semantically consistent BIM model that is competitive with the existing method in terms of geometric accuracy and model completeness.

3D Rock Structure Digital Characterization Using Airborne LiDAR and Unmanned Aerial Vehicle Techniques for Stability Analysis of a Blocky Rock Mass Slope

Airborne light detection and ranging (LiDAR) and unmanned aerial vehicle-structure from motion (UAV-SfM) provide point clouds with unprecedented resolution and accuracy that are well suited for the digital characterization of rock outcrops where direct contact measurements cannot be obtained due to terrain or safety constraints. Today, however, how to better apply these techniques to the practice of geostructural analysis is a topic of research that must be further explored. This study presents a processing procedure for extracting three-dimensional (3D) rock structure parameters directly from point clouds using open-source software and a three-dimensional distinct element code-assisted (3DEC) simulation of slope failure based on carbonate rock cliffs in the Jiuzhaigou Scenic Area. The procedure involves (1) processing point clouds obtained with different remote sensing techniques; (2) using the Hough transform to estimate normals for the hue, saturation, and value (HSV) rendering of unstructured point clouds; (3) automatically clustering and extracting the set-based point clouds; (4) estimating set-based geometric parameters; and (5) performing a subsequent stability analysis based on rock structure parameters. The results show that integrating different remote sensing techniques and rock structure computing can provide a quick way for slope engineers to assess the safety of blocky rock masses.

3D Object Detection for Autonomous Driving: A Survey

Autonomous driving is regarded as one of the most promising remedies to shield human beings from severe crashes. To this end, 3D object detection serves as the core basis of perception stack especially for the sake of path planning, motion prediction, and collision avoidance etc.. Taking a quick glance at the progress we have made, we attribute challenges to visual appearance recovery in the absence of depth information from images, representation learning from partially occluded unstructured point clouds, and semantic alignments over heterogeneous features from cross modalities. Despite existing efforts, 3D object detection for autonomous driving is still in its infancy. Recently, a large body of literature have been investigated to address this 3D vision task. Nevertheless, few investigations have looked into collecting and structuring this growing knowledge. We therefore aim to fill this gap in a comprehensive survey, encompassing all the main concerns including sensors, datasets, performance metrics and the recent state-of-the-art detection methods, together with their pros and cons. Furthermore, we provide quantitative comparisons with the state of the art. A case study on fifteen selected representative methods is presented, involved with runtime analysis, error analysis, and robustness analysis. Finally, we provide concluding remarks after an in-depth analysis of the surveyed works and identify promising directions for future work.

DFT-Net: Deep Feature Transformation Based Network for Object Categorization and Part Segmentation in 3-Dimensional Point Clouds.

Unlike 2-dimensional (2D) images, direct 3-dimensional (3D) point cloud processing using deep neural network architectures is challenging, mainly due to the lack of explicit neighbor relationships. Many researchers attempt to remedy this by performing an additional voxelization preprocessing step. However, this adds additional computational overhead and introduces quantization error issues, limiting an accurate estimate of the underlying structure of objects that appear in the scene. To this end, in this article, we propose a deep network that can directly consume raw unstructured point clouds to perform object classification and part segmentation. In particular, a Deep Feature Transformation Network (DFT-Net) has been proposed, consisting of a cascading combination of edge convolutions and a feature transformation layer that captures the local geometric features by preserving neighborhood relationships among the points. The proposed network builds a graph in which the edges are dynamically and independently calculated on each layer. To achieve object classification and part segmentation, we ensure point order invariance while conducting network training simultaneously—the evaluation of the proposed network has been carried out on two standard benchmark datasets for object classification and part segmentation. The results were comparable to or better than existing state-of-the-art methodologies. The overall score obtained using the proposed DFT-Net is significantly improved compared to the state-of-the-art methods with the ModelNet40 dataset for object categorization.

GEOMETRIC PRIMITIVE EXTRACTION FROM SEMANTICALLY ENRICHED POINT CLOUDS

Abstract. 3D point clouds are robust representations of real-world objects and usually contain information about the shape, size, position and radiometry of the scene. However, unstructured point clouds do not directly exploit the full potential of such information and thus, further analysis is commonly required. Especially when dealing with cultural heritage objects which are, typically, described by complex 3D geometries, semantic segmentation is a fundamental step for the automatic identification of shapes, erosions, etc. This paper focuses on the efficient extraction of semantic classes that would support the generation of geometric primitives such as planes, spheres, cylinders, etc. Our semantic segmentation approach relies on supervised learning using a Random Forest algorithm, while the geometric shapes are identified and extracted with the RANSAC model fitting algorithm. In this way the parametric modelling procedure in a HBIM environment is easily enabled. Our experiments show the efficient label transferability of our 3D semantic segmentation approach across different Doric Greek temples, with qualitatively and quantitatively evaluations.

Growing Neural Gas with Different Topologies for 3D Space Perception

Three-dimensional space perception is one of the most important capabilities for an autonomous mobile robot in order to operate a task in an unknown environment adaptively since the autonomous robot needs to detect the target object and estimate the 3D pose of the target object for performing given tasks efficiently. After the 3D point cloud is measured by an RGB-D camera, the autonomous robot needs to reconstruct a structure from the 3D point cloud with color information according to the given tasks since the point cloud is unstructured data. For reconstructing the unstructured point cloud, growing neural gas (GNG) based methods have been utilized in many research studies since GNG can learn the data distribution of the point cloud appropriately. However, the conventional GNG based methods have unsolved problems about the scalability and multi-viewpoint clustering. In this paper, therefore, we propose growing neural gas with different topologies (GNG-DT) as a new topological structure learning method for solving the problems. GNG-DT has multiple topologies of each property, while the conventional GNG method has a single topology of the input vector. In addition, the distance measurement in the winner node selection uses only the position information for preserving the environmental space of the point cloud. Next, we show several experimental results of the proposed method using simulation and RGB-D datasets measured by Kinect. In these experiments, we verified that our proposed method almost outperforms the other methods from the viewpoint of the quantization and clustering errors. Finally, we summarize our proposed method and discuss the future direction on this research.

Shape registration with learned deformations for 3D shape reconstruction from sparse and incomplete point clouds.

Shape reconstruction from sparse point clouds/images is a challenging and relevant task required for a variety of applications in computer vision and medical image analysis (e.g. surgical navigation, cardiac motion analysis, augmented/virtual reality systems). A subset of such methods, viz. 3D shape reconstruction from 2D contours, is especially relevant for computer-aided diagnosis and intervention applications involving meshes derived from multiple 2D image slices, views or projections. We propose a deep learning architecture, coined Mesh Reconstruction Network (MR-Net), which tackles this problem. MR-Net enables accurate 3D mesh reconstruction in real-time despite missing data and with sparse annotations. Using 3D cardiac shape reconstruction from 2D contours defined on short-axis cardiac magnetic resonance image slices as an exemplar, we demonstrate that our approach consistently outperforms state-of-the-art techniques for shape reconstruction from unstructured point clouds. Our approach can reconstruct 3D cardiac meshes to within 2.5-mm point-to-point error, concerning the ground-truth data (the original image spatial resolution is ∼1.8×1.8×10mm3). We further evaluate the robustness of the proposed approach to incomplete data, and contours estimated using an automatic segmentation algorithm. MR-Net is generic and could reconstruct shapes of other organs, making it compelling as a tool for various applications in medical image analysis.

Fast and Accurate Normal Estimation for Point Clouds Via Patch Stitching

This paper presents an effective normal estimation method adopting multi-patch stitching for an unstructured point cloud. The majority of learning-based approaches encode a local patch around each point of a whole model and estimate the normals in a point-by-point manner. In contrast, we suggest a more efficient pipeline, in which we introduce a patch-level normal estimation architecture to process a series of overlapping patches. Additionally, a multi-normal selection method based on weights, dubbed as multi-patch stitching, integrates the normals from the overlapping patches. To reduce the adverse effects of sharp corners or noise in a patch, we introduce an adaptive local feature aggregation layer to focus on an anisotropic neighborhood. We then utilize a multi-branch planar experts module to break the mutual influence between underlying piecewise surfaces in a patch. At the stitching stage, we use the learned weights of multi-branch planar experts and distance weights between points to select the best normal from the overlapping parts. Furthermore, we put forward constructing a sparse matrix representation to reduce large-scale retrieval overheads for the loop iterations dramatically. Extensive experiments demonstrate that our method achieves SOTA results with the advantage of lower computational costs and higher robustness to noise over most existed approaches.

Automatic recognition of cylinders and planes from unstructured point clouds

Automatic recognition of cylinders and planes from unstructured point clouds

Geometry Guided Deep Surface Normal Estimation

We propose a geometry-guided neural network architecture for robust and detail-preserving surface normal estimation for unstructured point clouds. Previous deep normal estimators usually estimate the normal directly from the neighbors of a query point, which lead to poor performance. The proposed network is composed of a weight learning sub-network (WL-Net) and a lightweight normal learning sub-network (NL-Net). WL-Net first predicates point-wise weights for generating an optimized point set (OPS) from the input. Then, NL-Net estimates a more accurate normal from the OPS especially when the local geometry is complex. To boost the weight learning ability of the WL-Net, we introduce two geometric guidance in the network. First, we design a weight guidance using the deviations between the neighbor points and the ground truth tangent plane of the query point. This deviation guidance offers a “ground truth” for weights corresponding to some reliable inliers and outliers determined by the tangent plane. Second, we integrate the normals of multiple scales into the input. Its performance and robustness are further improved without relying on multi-branch networks, which are employed in previous multi-scale normal estimators. Thus our method is more efficient. Qualitative and quantitative evaluations demonstrate the advantages of our approach over the state-of-the-art methods, in terms of estimation accuracy, model size and inference time. Code is available at https://github.com/2429581027/local-geometric-guided.

Exploiting Structured CNNs for Semantic Segmentation of Unstructured Point Clouds from LiDAR Sensor

Accurate semantic segmentation of 3D point clouds is a long-standing problem in remote sensing and computer vision. Due to the unstructured nature of point clouds, designing deep neural architectures for point cloud semantic segmentation is often not straightforward. In this work, we circumvent this problem by devising a technique to exploit structured neural architectures for unstructured data. In particular, we employ the popular convolutional neural network (CNN) architectures to perform semantic segmentation of LiDAR data. We propose a projection-based scheme that performs an angle-wise slicing of large 3D point clouds and transforms those slices into 2D grids. Accounting for intensity and reflectivity of the LiDAR input, the 2D grid allows us to construct a pseudo image for the point cloud slice. We enhance this image with low-level image processing techniques of normalization, histogram equalization, and decorrelation stretch to suit our ultimate object of semantic segmentation. A large number of images thus generated are used to induce an encoder-decoder CNN model that learns to compute a segmented 2D projection of the scene, which we finally back project to the 3D point cloud. In addition to a novel method, this article also makes a second major contribution of introducing the enhanced version of our large-scale public PC-Urban outdoor dataset which is captured in a civic setup with an Ouster LiDAR sensor. The updated dataset (PC-Urban_V2) provides nearly 8 billion points including over 100 million points labeled for 25 classes of interest. We provide a thorough evaluation of our technique on PC-Urban_V2 and three other public datasets.

Two-Dimensional Shape Analysis of Complex Geometry Based on Photogrammetric Models of Iconostases

Three-dimensional digitization technologies have been proved as reliable methods for detailed and accurate spatial data collection from existing cultural heritage. In addition, the point segmentation techniques are particularly relevant for contour detection and classification of the unstructured point cloud. This paper describes an approach to obtain 2D CAD-like visualizations of complex geometry from photogrammetric models so that the detected contours of particular object elements can be used for 2D shape analysis. The work process uses the point clouds derived from photogrammetric models to create the plane visualization of the object’s geometry by segmenting points based on the verticality geometric feature. The research presented is on the case studies of iconostases as the specific art and architectural elements of the Christian Orthodox church that can be appreciated only in situ. To determine relations between the characteristics of the particular shapes and the iconostases’ style origins, the mathematical method of shape analysis was applied. This study aims to numerically describe the stylistic characteristics of the shapes of the main parts of the iconostasis concerning the artistic period to which it belongs to. The concept was based on the consideration of global shape descriptors and associated shape measurements which were used to analyze and classify the stylistic characteristics of the iconostases. The methodology was applied to the representative examples of three iconostases from the Baroque and Classicism art movements. The results illustrated that the proposed methods and techniques, with certain improvements, could be helpful for CAD visualization and shape analysis of complex geometry.

Fast occlusion-based point cloud exploration

Large-scale unstructured point cloud scenes can be quickly visualized without prior reconstruction by utilizing levels-of-detail structures to load an appropriate subset from out-of-core storage for rendering the current view. However, as soon as we need structures within the point cloud, e.g., for interactions between objects, the construction of state-of-the-art data structures requires O(NlogN) time for N points, which is not feasible in real time for millions of points that are possibly updated in each frame. Therefore, we propose to use a surface representation structure which trades off the (here negligible) disadvantage of single-frame use for both output-dominated and near-linear construction time in practice, exploiting the inherent 2D property of sampled surfaces in 3D. This structure tightly encompasses the assumed surface of unstructured points in a set of bounding depth intervals for each cell of a discrete 2D grid. The sorted depth samples in the structure permit fast surface queries, and on top of that an occlusion graph for the scene comes almost for free. This graph enables novel real-time user operations such as revealing partially occluded objects, or scrolling through layers of occluding objects, e.g., walls in a building. As an example application we showcase a 3D scene exploration framework that enables fast, more sophisticated interactions with point clouds rendered in real time.

Free-form scanning of non-planar appearance with neural trace photography

We propose neural trace photography, a novel framework to automatically learn high-quality scanning of non-planar, complex anisotropic appearance. Our key insight is that free-form appearance scanning can be cast as a geometry learning problem on unstructured point clouds, each of which represents an image measurement and the corresponding acquisition condition. Based on this connection, we carefully design a neural network, to jointly optimize the lighting conditions to be used in acquisition, as well as the spatially independent reconstruction of reflectance from corresponding measurements. Our framework is not tied to a specific setup, and can adapt to various factors in a data-driven manner. We demonstrate the effectiveness of our framework on a number of physical objects with a wide variation in appearance. The objects are captured with a light-weight mobile device, consisting of a single camera and an RGB LED array. We also generalize the framework to other common types of light sources, including a point, a linear and an area light.

A CAPSNETS APPROACH TO PAVEMENT CRACK DETECTION USING MOBILE LASER SCANNNING POINT CLOUDS

Abstract. Routine pavement inspection is crucial to keep roads safe and reduce traffic accidents. However, traditional practices in pavement inspection are labour-intensive and time-consuming. Mobile laser scanning (MLS) has proven a rapid way for collecting a large number of highly dense point clouds covering roadway surfaces. Handling a huge amount of unstructured point clouds is still a very challenging task. In this paper, we propose an effective approach for pavement crack detection using MLS point clouds. Road surface points are first converted into intensity images to improve processing efficiency. Then, a Capsule Neural Network (CapsNet) is developed to classify the road points for pavement crack detection. Quantitative evaluation results showed that our method achieved the recall, precision, and F1-score of 95.3%, 81.1%, and 88.2% in the testing scene, respectively, which demonstrated the proposed CapsNet framework can accurately and robustly detect pavement cracks in complex urban road environments.

Enabling Viewpoint Learning through Dynamic Label Generation

AbstractOptimal viewpoint prediction is an essential task in many computer graphics applications. Unfortunately, common viewpoint qualities suffer from two major drawbacks: dependency on clean surface meshes, which are not always available, and the lack of closed‐form expressions, which requires a costly search involving rendering. To overcome these limitations we propose to separate viewpoint selection from rendering through an end‐to‐end learning approach, whereby we reduce the influence of the mesh quality by predicting viewpoints from unstructured point clouds instead of polygonal meshes. While this makes our approach insensitive to the mesh discretization during evaluation, it only becomes possible when resolving label ambiguities that arise in this context. Therefore, we additionally propose to incorporate the label generation into the training procedure, making the label decision adaptive to the current network predictions. We show how our proposed approach allows for learning viewpoint predictions for models from different object categories and for different viewpoint qualities. Additionally, we show that prediction times are reduced from several minutes to a fraction of a second, as compared to state‐of‐the‐art (SOTA) viewpoint quality evaluation. Code and training data is available at https://github.com/schellmi42/viewpoint_learning, which is to our knowledge the biggest viewpoint quality dataset available.

Truncated octree and its applications

Octree is a hierarchical data structure with many applications, especially in encoding unstructured point clouds. The depth of an octree is dependent of the scale of the input data and the desired resolution of the smallest voxels in the leaf nodes as well. Thus, it often requires a deep octree to maintain low level of geometric errors for large-scale sparse point clouds, which leads to high memory requirement and low access speed. This paper presents a new structure called truncated octree or T-Octree that truncates the octree by adaptively pruning the top hierarchy and represents the deep octree by a set of shallow sub-octrees. The structure is further extended to support random access of nodes and out-of-core streaming of large data sets. We also propose a variable length addressing scheme to adaptively choose the length of an octree’s node address based on the truncation level. As a result, T-Octree provides highly efficient query performance and can save storage without losing the original structure for sparse or clustered models and scenes. We demonstrate the efficacy and efficiency of the new structure on point cloud compression and scene query tasks for sparse or clustered data.

One-Stage Anchor-Free 3D Vehicle Detection from LiDAR Sensors.

Recent one-stage 3D detection methods generate anchor boxes with various sizes and orientations in the ground plane, then determine whether these anchor boxes contain any region of interest and adjust the edges of them for accurate object bounding boxes. The anchor-based algorithm calculates the classification and regression label for each anchor box during the training process, which is inefficient and complicated. We propose a one-stage, anchor-free 3D vehicle detection algorithm based on LiDAR point clouds. The object position is encoded as a set of keypoints in the bird’s-eye view (BEV) of point clouds. We apply the voxel/pillar feature extractor and convolutional blocks to map an unstructured point cloud to a single-channel 2D heatmap. The vehicle’s Z-axis position, dimension, and orientation angle are regressed as additional attributes of the keypoints. Our method combines SmoothL1 loss and IoU (Intersection over Union) loss, and we apply as angle regression labels, which achieve high average orientation similarity (AOS) without any direction classification tricks. During the target assignment and bounding box decoding process, our framework completely avoids any calculations related to anchor boxes. Our framework is end-to-end training and stands at the same performance level as the other one-stage anchor-based detectors.

Direction-induced convolution for point cloud analysis

Point cloud analysis becomes a fundamental but challenging problem in the field of 3D scene understanding. To deal with unstructured and unordered point clouds in the embedded 3D space, we propose a novel direction-induced convolution (DIConv) to obtain the hierarchical representations of point clouds and then boost the performance of point cloud analysis. Specifically, we first construct a direction set as the basis of spatial direction information, where its entries can denote these latent direction components of 3D points. For each neighbor point, we can project its direction information into the constructed direction set for achieving an array of direction-dependent weights, then transform its features into the canonical ordered direction set space. After that, the standard image-like convolution can be leveraged to encode the unordered neighborhood regions of point cloud data. We further develop a residual DIConv (Res_DIConv) module and a farthest point sampling residual DIConv (FPS_Res_DIConv) module for jointly capturing the hierarchical features of input point clouds. By alternately stacking Res_DIConv modules and FPS_Res_DIConv modules, a direction-induced convolution network (DICNet) can be built to perform point cloud analysis in an end-to-end fashion. Comprehensive experiments on three benchmark datasets (including ModelNet40, ShapeNet Part, and S3DIS) demonstrate that the proposed DIConv method achieves encouraging performance on both point cloud classification and semantic segmentation tasks.

Multiscale Feature Line Extraction From Raw Point Clouds Based on Local Surface Variation and Anisotropic Contraction

Recent 3-D scanning techniques can produce various kinds of digitized 3-D data. Most of these scanned data are in a format of unstructured point clouds. Such low-level representation of 3-D data usually contains only geometric properties (point positions), while lacking higher level structure cues, for example, feature lines. Feature lines can be defined as a visually prominent characteristic of the shape, including edges, ridges, and valley lines in multiple scales, which can support a lot of downstream applications, such as shape reconstruction and analysis. We present a two-phase algorithm for extracting line-type features on point clouds. To extract both large-scale and shallow feature lines, we first define a statistical metric to detect all potential feature points while immune to the noise to some extent. Then, for correctly reconstructing the feature lines from these identified coarse feature points, we introduce an anisotropic contracting scheme to force feature points lying on the underlying real feature lines. To illustrate the reliability of our method, various experiments have been conducted on both synthetic and raw data. Both visual and quantitative comparisons show that our method is robust to noise and can correctly extract multiscale feature lines. In addition, our method is generally applicable to robotic picking. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article was motivated by the problem of the feature line extraction for real scanned point clouds. Feature lines, as one kind of the most important structure information, depict the basic shape of the real object in our life. Extracting this kind of shape features from the unstructured point clouds can facilitate a variety of downstream practical applications, such as product design, workpiece manufacturing, and robotic grasping. Existing approaches to detect features either heavily rely on differential quantities, which are sensitive to the noise, or need an elaborately designed local descriptor but fail to recognize small-scale features. These challenges motivate us to design a new approach aiming at extracting multiscale feature lines while keeping robustness to heavy noise. The technique developed in this work can produce high-quality feature points and feature lines, which would serve as higher level structural information and facilitate many applications. Additional applications in 6-degree-of-freedom (6-DoF) pose estimation demonstrate the potential of our method for robotic picking.