Set Of 3D Points Research Articles

Multi-GPU 3D k-nearest neighbors computation with application to ICP, point cloud smoothing and normals computation

The k-Nearest Neighbors algorithm is a fundamental algorithm that finds applications in many fields like Machine Learning, Computer Graphics, Computer Vision, and others. The algorithm determines the closest points (d-dimensional) of a reference set R according to a query set of points Q under a specific metric (Euclidean, Mahalanobis, Manhattan, etc.). This work focuses on the utilization of multiple Graphical Processing Units for the acceleration of the k-Nearest Neighbors algorithm with large or very large sets of 3D points. With the proposed approach the space of the reference set is divided into a 3D grid which is used to facilitate the search for the nearest neighbors. The search in the grid is performed in a multiresolution manner starting from a high-resolution grid and ending up in a coarse one, thus accounting for point clouds that may have non-uniform sampling and/or outliers. Three important algorithms in reverse engineering are revisited and new multi-GPU versions are proposed based on the introduced KNN algorithm. More specifically, the new multi-GPU approach is applied to the Iterative Closest Point algorithm, to the point cloud smoothing, and to the point cloud normal vectors computation and orientation problem. A series of tests and experiments have been conducted and discussed in the paper showing the merits of the proposed multi-GPU approach.

BallMerge: High‐quality Fast Surface Reconstruction via Voronoi Balls

AbstractWe introduce a Delaunay‐based algorithm for reconstructing the underlying surface of a given set of unstructured points in 3D. The implementation is very simple, and it is designed to work in a parameter‐free manner. The solution builds upon the fact that in the continuous case, a closed surface separates the set of maximal empty balls (medial balls) into an interior and exterior. Based on discrete input samples, our reconstructed surface consists of the interface between Voronoi balls, which approximate the interior and exterior medial balls. An initial set of Voronoi balls is iteratively processed, merging Voronoi‐ball pairs if they fulfil an overlapping error criterion. Our complete open‐source reconstruction pipeline performs up to two quick linear‐time passes on the Delaunay complex to output the surface, making it an order of magnitude faster than the state of the art while being competitive in memory usage and often superior in quality. We propose two variants (local and global), which are carefully designed to target two different reconstruction scenarios for watertight surfaces from accurate or noisy samples, as well as real‐world scanned data sets, exhibiting noise, outliers, and large areas of missing data. The results of the global variant are, by definition, watertight, suitable for numerical analysis and various applications (e.g., 3D printing). Compared to classical Delaunay‐based reconstruction techniques, our method is highly stable and robust to noise and outliers, evidenced via various experiments, including on real‐world data with challenges such as scan shadows, outliers, and noise, even without additional preprocessing.

Self-supervised learning of rotation-invariant 3D point set features using transformer and its self-distillation

Invariance against rotations of 3D objects is an important property in analyzing 3D point set data. Conventional 3D point set DNNs having rotation invariance typically obtain accurate 3D shape features via supervised learning by using labeled 3D point sets as training samples. However, due to the rapid increase in 3D point set data and the high cost of labeling, a framework to learn rotation-invariant 3D shape features from numerous unlabeled 3D point sets is required. This paper proposes a novel self-supervised learning framework for acquiring accurate and rotation-invariant 3D point set features at object-level. Our proposed lightweight DNN architecture decomposes an input 3D point set into multiple global-scale regions, called tokens, that preserve the spatial layout of partial shapes composing the 3D object. We employ a self-attention mechanism to refine the tokens and aggregate them into an expressive rotation-invariant feature per 3D point set. Our DNN is effectively trained by using pseudo-labels generated by a self-distillation framework. To facilitate the learning of accurate features, we propose to combine multi-crop and cut-mix data augmentation techniques to diversify 3D point sets for training. Through a comprehensive evaluation, we empirically demonstrate that, (1) existing rotation-invariant DNN architectures designed for supervised learning do not necessarily learn accurate 3D shape features under a self-supervised learning scenario, and (2) our proposed algorithm learns rotation-invariant 3D point set features that are more accurate than those learned by existing algorithms.

A Generalized Full-to-Partial Registration Framework of 3D Point Sets for Computer-Aided Orthopedic Surgery.

The precise alignment of full and partial 3D point sets is a crucial technique in computer-aided orthopedic surgery, but remains a significant challenge. This registration process is complicated by the partial overlap between the full and partial 3D point sets, as well as the susceptibility of 3D point sets to noise interference and poor initialization conditions. To address these issues, we propose a novel full-to-partial registration framework for computer-aided orthopedic surgery that utilizes reinforcement learning. Our proposed framework is both generalized and robust, effectively handling the challenges of noise, poor initialization, and partial overlap. Moreover, this framework demonstrates exceptional generalization capabilities for various bones, including the pelvis, femurs, and tibias. Extensive experimentation on several bone datasets has demonstrated that the proposed method achieves a superior C.D. error of 8.211 e-05 and our method consistently outperforms state-of-the-art registration techniques. Hence, our proposed method is capable of achieving precise bone alignments for computer-aided orthopedic surgery.

Long-Short Range Adaptive Transformer With Dynamic Sampling for 3D Object Detection

3D object detection in point cloud aims at simultaneously localizing and recognizing 3D objects from a 3D point set. However, since point clouds are usually sparse, unordered, and irregular, it is challenging to learn robust point representations and sample high-quality object queries. To deal with the above issues, we propose a Long-short rangE Adaptive transformer with Dynamic sampling (LeadNet), including a point representation encoder, a dynamic object query sampling decoder, and an object detection decoder in a unified architecture for 3D object detection. Specifically, in the point representation encoder, we combine an attention layer and a channel attentive kernel convolution layer to consider the local structure and the long-range context simultaneously. In the dynamic object query sampling decoder, we utilize multiple dynamic prototypes to adapt to various point clouds. In the object detection decoder, we incorporate a dynamic Gaussian weight map into the cross-attention mechanism to help the detection decoder focus on the proper visual regions near the object, further accelerating the training process. Extensive experimental results on two standard benchmarks show that our LeadNet outperforms the 3DETR baseline by 11.6% mAP <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">50</sub> on the ScanNet v2 dataset and achieves the new state-of-the-art results on ScanNet v2 and SUN RGB-D benchmarks for the geometric-only approaches.

RBF-Based Camera Model Based on a Ray Constraint to Compensate for Refraction Error.

A camera equipped with a transparent shield can be modeled using the pinhole camera model and residual error vectors defined by the difference between the estimated ray from the pinhole camera model and the actual three-dimensional (3D) point. To calculate the residual error vectors, we employ sparse calibration data consisting of 3D points and their corresponding 2D points on the image. However, the observation noise and sparsity of the 3D calibration points pose challenges in determining the residual error vectors. To address this, we first fit Gaussian Process Regression (GPR) operating robustly against data noise to the observed residual error vectors from the sparse calibration data to obtain dense residual error vectors. Subsequently, to improve performance in unobserved areas due to data sparsity, we use an additional constraint; the 3D points on the estimated ray should be projected to one 2D image point, called the ray constraint. Finally, we optimize the radial basis function (RBF)-based regression model to reduce the residual error vector differences with GPR at the predetermined dense set of 3D points while reflecting the ray constraint. The proposed RBF-based camera model reduces the error of the estimated rays by 6% on average and the reprojection error by 26% on average.

Cooperative 3D tunnel measurement based on 2D–3D registration of omnidirectional laser light

AbstractIn this study, we proposed a high‐density three‐dimensional (3D) tunnel measurement method, which estimates the pose changes of cameras based on a point set registration algorithm regarding 2D and 3D point clouds. To detect small deformations and defects, high‐density 3D measurements are necessary for tunnel construction sites. The line‐structured light method uses an omnidirectional laser to measure a high‐density cross‐section point cloud from camera images. To estimate the pose changes of cameras in tunnels, which have few textures and distinctive shapes, cooperative robots are useful because they estimate the pose by aggregating relative poses from the other robots. However, previous studies mounted several sensors for both the 3D measurement and pose estimation, increasing the size of the measurement system. Furthermore, the lack of 3D features makes it difficult to match point clouds obtained from different robots. The proposed measurement system consists of a cross‐section measurement unit and a pose estimation unit; one camera was mounted for each unit. To estimate the relative poses of the two cameras, we designed a 2D–3D registration algorithm for the omnidirectional laser light, and implemented hand‐truck and unmanned aerial vehicle systems. In the measurement of a tunnel with a width of 8.8 m and a height of 6.4 m, the error of the point cloud measured by the proposed method was 162.8 and 575.3 mm along 27 m, respectively. In a hallway measurement, the proposed method generated less errors in straight line shapes with few distinctive shapes compared with that of the 3D point set registration algorithm with Light Detection and Ranging.

A generalized closed-form solution for 3D registration of two-point sets under isotropic and anisotropic scaling

Fitting of two-point sets under non-uniform scaling involves several applications, including ultrasound-guided surgery, where ultrasound images generally show anisotropy between its lateral and depth resolution. However, the current state-of-the-art only accomplishes the closed-form solution under the assumption of isotropic scaling, and only the iterative form of the solution is available for registration regardless of scaling type. A generalized closed-form solution, including anisotropic scaling, could robust the computation in the applications of multi-modality image fusion and surgical navigation based on ultrasound images. In this paper, a generalized closed-form solution (GCFS) that accounts for isotropic scaling and anisotropic scaling is proposed. GCFS finds the least-squares solution of rotation, translation, and scaling based on the singular value decomposition (SVD). The method was demonstrated to 1) register two 3D point sets under the effect of anisotropic scaling and 2) register two coplanar 3D point sets under the effect of anisotropic scaling. This study evaluates 3D registration between the preoperative model and intraoperative ultrasonic scanning of a femur and 2D registration of ultrasound images to the phantom for ultrasound calibration. In addition, the conditions that affect the transformation estimations like estimating reflection and noise degeneracy were also considered in the evaluation.

Best (orthogonal) fitting ellipsoid with quaternions

The aim of this study is the determination of the best fit ellipsoid to given points by quaternions. The problem of the fitting ellipsoid is frequently encountered in image processing, computer games, medicine, engineering and science applications, geodesy, etc. The ellipsoid fitting problem is the process of determining the ellipsoid that best fits a given set of points in 3D. In the fitting process, it is generally done over two models. The first of these is the algebraic method and the second one is orthogonal (geometric) method. In this study, we tried to solve the problem of algebraic and orthogonal ellipsoid fitting based on Euler angles for the first time over quaternions. The superiority of quaternions over Euler rotation angles is well known. In addition, the variance–covariance matrix of the parameters of the fitted ellipsoid will also be calculated. Numerical applications show that the proposed method can be used successfully.

A Fast and Reliable Solution to PnP, Using Polynomial Homogeneity and a Theorem of Hilbert

One of the most-extensively studied problems in three-dimensional Computer Vision is “Perspective-n-Point” (PnP), which concerns estimating the pose of a calibrated camera, given a set of 3D points in the world and their corresponding 2D projections in an image captured by the camera. One solution method that ranks as very accurate and robust proceeds by reducing PnP to the minimization of a fourth-degree polynomial over the three-dimensional sphere S3. Despite a great deal of effort, there is no known fast method to obtain this goal. A very common approach is solving a convex relaxation of the problem, using “Sum Of Squares” (SOS) techniques. We offer two contributions in this paper: a faster (by a factor of roughly 10) solution with respect to the state-of-the-art, which relies on the polynomial’s homogeneity; and a fast, guaranteed, easily parallelizable approximation, which makes use of a famous result of Hilbert.

COST-EFFECTIVE CAMERA LOCALIZATION AIDED BY PRIOR POINT CLOUDS MAPS FOR LEVEL 3 AUTONOMOUS DRIVING VEHICLES

Abstract. Precise and robust localization is critical for many navigation tasks, especially autonomous driving systems. The most popular localization approach is global navigation satellite systems (GNSS). However, it has several shortcomings such as multipath and nonline- of-sight reception. Vision-based localization is one of the approaches without using GNSS which is based on vision. This paper used visual localization with a prior 3D LiDAR map. Compared to common methods for visual localization using camera-acquired maps, this paper used the method that tracks the image feature and poses of a monocular camera to match with prior 3D LiDAR maps. This paper reconstructs the image feature to several sets of 3D points by a local bundle adjustment-based visual odometry system. Those 3D points matched with the prior 3D point cloud map to track the globe pose of the user. The visual localization approach has several advantages. (1) Since it only relies on matching geometry, it is robust to changes in ambient luminosity appearance. (2) Also, it uses the prior 3D map to provide viewpoint invariance. Moreover, the proposed method only requires users to use low-cost and lightweight camera sensors.

Deep Neural Implicit Representation of Accessibility for Multi-Axis Manufacturing

One of the main concerns in design and process planning for multi-axis additive and subtractive manufacturing is collision avoidance between moving objects (e.g., tool assemblies) and stationary objects (e.g., part unified with fixtures). The collision measure for various pairs of relative rigid translations and rotations between the two pointsets can be conceptualized by a compactly supported scalar field over the 6D non-Euclidean configuration space. Explicit representation and computation of this field is costly in both time and space. If we fix O(m) sparsely sampled rotations (e.g., tool orientations), computation of the collision measure field as a convolution of indicator functions of the 3D pointsets over a uniform grid (i.e., voxelized geometry) of resolution O(n3) via fast Fourier transforms (FFTs) scales as in O(mn3logn) in time and O(mn3) in space. In this paper, we develop an implicit representation of the collision measure field via deep neural networks (DNNs). We show that our approach is able to accurately interpolate the collision measure from a sparse sampling of rotations, and can represent the collision measure field with a small memory footprint. Moreover, we show that this representation can be efficiently updated through fine-tuning to more efficiently train the network on multi-resolution data, as well as accommodate incremental changes to the geometry (such as might occur in iterative processes such as topology optimization of the part subject to CNC tool accessibility constraints).

A New Approach for Structural Monitoring Based on Terrestrial Laser Scan Data Using Control Planes

Laser scanning is a survey method that enables the obtaining of several data points on surfaces through the observation of horizontal and vertical angles and electronic distance measurements. The result of these observations is a set of 3D points named point clouds, which can be obtained from scans using total stations or the Terrestrial Laser Scanner (TLS). Currently, the use of TLS in geodetic structural deformation monitoring activities is under evaluation, since this technique can provide a greater number of points in a shorter period, when compared to scans performed by total stations. In this research, the feasibility of using control planes for geodetic monitoring was investigated in a controlled laboratory environment, exploring the current trend of parameterization of point clouds. From the development of an Experimental Plane Control (EPC) containing a flat surface that could be inclined in a controlled way, simultaneous scans with TLS and total station were carried out at different inclinations of the control plane. Both surveys were done in frontal scan mode (minimum plane inclination of 0º53’ and maximum plane inclination of 3º28’) and in oblique mode (plane inclination of 1º09’), where the plane inclination angles were generated by the rotation in EPC base, considering the first plane position as reference to inclination calculating. From parametrization of point clouds, a new approach was developed to test the significance of the estimated inclination angle of the plane in relation to a reference position. The results showed that it is possible to confirm that the inclined angles were statistically significant, but that the TLS scanning position can interfere in the determination of these data for monitoring purposes.

A Chebyshev metamodel based BnB approach to efficiently search global optimum for 3D ICP point set registration

Iterative Closest Point (ICP) algorithm for 3D point set registration has many robust properties that make it widely used. Regardless of the initialization, the Go-ICP method was first presented to search global optimum for 3D point set registration using Branch-and-Bound (BnB) scheme, but it could be time consuming. In this paper, an efficient modification, called Ego-ICP, is presented. The core idea is: (1) SE(3) feasible domain, L2-norm registration model of ICP and corresponding local ICP algorithm are reviewed, (2) Chebyshev metamodel is implemented to approximate the registration model with specific rotation, (3) the Chebyshev metamodel based single nested BnB approach is then presented to efficiently search global optimum for 3D point set registration. Furthermore, three experimental targets for different Datasets are demonstrated and shown that the presented approach, as compared with the BNB based method, the geometry based methods and the deep learning based method, (1) can be faster than or close to the efficiency of other compared methods, (2) can be reliable regardless of the initialization, and (3) can be 100% successful even if noise and outlier are existed. Lastly, several insights are provided to further improve the efficiency of our method.

Multi-view triangulation without correspondences

We introduce a novel method to solve the joint problem of correspondence and triangulation of points from multiple calibrated perspective views and show its application to counting the number of leaves present on plants photographed from multiple angles in phenotyping facilities. Assuming there is a set of n calibrated views of an object, the input to our algorithm is a set of 2D points (e.g., leaf tips, fruits, flowers) detected in each of the n views, and the output is a set of 3D points corresponding to the 2D points when re-projected on views. Our approach is robust to noise and occlusion. In particular, it is not required for all points to be visible from one of the views, as our algorithm can infer that a point is occluded by reasoning on the 3D geometry of the scene. Our algorithm is suitable for many points (approx. thousands) reconstructed from a reduced set of views (up to 15). For example, our implementation finds the correspondence of 20 points captured by six cameras in about one second on consumer hardware. We evaluate the performance on synthetic data as well as real examples. We show that the accuracy of leaf counting from multi-view images is drastically improved by our algorithm.

Multi-view Image Color Consistency Enhancement using 3D Point Cloud Registration

Color inconsistency in multi-view images degrades the performance of various consumer electronics applications that use multi-view images, such as panoramic image stitching and 3D reconstruction for VR contents. In this paper, a novel approach to multi-view image color consistency enhancement is proposed. In contrast to conventional methods that try to find a nonlinear transfer function for each color channel, in the proposed method, the color distribution of the source image is non-rigidly transformed into that of the reference image using a 3D point set registration algorithm. In order to significantly reduce the computation of the 3D registration process, representative color points are utilized for the 3D registration instead of all the pixel color points. The number of representatives is much less than the number of image pixels, but they effectively approximate the color distribution of the image. Through extensive experiments, we show that the proposed method outperforms conventional methods in color consistency enhancement both subjectively and objectively, and the proposed representative color point set approximation improves the computational efficiency.

Research on a Map-Based Cooperative Navigation System for Spraying–Dosing Robot Group

To solve the problem encountered when the spraying robot has run out of medicine even though the spraying task on the field is not complete, we developed a spraying–dosing robot group and proposed a collaborative navigation system based on an orchard map. Firstly, we constructed a 3D orchard point cloud map and set up navigation path points on the projected map. Secondly, we developed a master–slave command-based cooperative navigation strategy, where the spraying robot was the master and the dosing robot was the slave. Finally, the spraying robot and the dosing robot completed the cooperative navigation on the constructed map by using the pure pursuit algorithm and D-A control algorithm, respectively. To validate the cooperative navigation system, we conducted field tests on the separate communication and navigation control. The results of communication experiments demonstrated that the packet loss rate was less than 5%, which satisfied communication requirements. The experimental results of the navigation control demonstrated that the maximum value of the absolute lateral error is 24.9 cm for the spraying robot and 29.7 cm for the dosing robot. The collaborative navigation system proposed in this research can meet the automatic navigation requirements of the spraying–dosing robot group for collaborative tasks in traditional orchards.

Spatio-Temporal Calibration of Multiple Kinect Cameras Using 3D Human Pose.

RGB and depth cameras are extensively used for the 3D tracking of human pose and motion. Typically, these cameras calculate a set of 3D points representing the human body as a skeletal structure. The tracking capabilities of a single camera are often affected by noise and inaccuracies due to occluded body parts. Multiple-camera setups offer a solution to maximize coverage of the captured human body and to minimize occlusions. According to best practices, fusing information across multiple cameras typically requires spatio-temporal calibration. First, the cameras must synchronize their internal clocks. This is typically performed by physically connecting the cameras to each other using an external device or cable. Second, the pose of each camera relative to the other cameras must be calculated (Extrinsic Calibration). The state-of-the-art methods use specialized calibration session and devices such as a checkerboard to perform calibration. In this paper, we introduce an approach to the spatio-temporal calibration of multiple cameras which is designed to run on-the-fly without specialized devices or equipment requiring only the motion of the human body in the scene. As an example, the system is implemented and evaluated using Microsoft Azure Kinect. The study shows that the accuracy and robustness of this approach is on par with the state-of-the-art practices.

3D RECONSTRUCTION OF SIMPLE BUILDINGS FROM POINT CLOUDS USING NEURAL NETWORKS WITH CONTINUOUS CONVOLUTIONS (CONVPOINT)

Abstract. The automatic reconstruction of 3D building models from airborne laser scanning point clouds or aerial imagery data in a model-driven fashion most often consists of a recognition of standardized building primitives with typically rectangular footprints and parameterized roof shapes based on a pre-defined collection, and a parameter estimation so that the selected primitives best fit the input data. For more complex buildings that consist of multiple parts, several such primitives need to be combined. This paper focuses on the reconstruction of such simple buildings, and explores the potential of Deep Learning by presenting a neural network architecture that takes a 3D point cloud of a single building as input and outputs the geometric information needed to construct a 3D building model in half-space representation with up to four roof faces like saddleback, hip, and pyramid roof. The proposed neural network architecture consists of a roof face segmentation module implemented with continuous convolutions as used in ConvPoint, which performs feature extraction directly from a set of 3D points, and four PointNet modules that predict from sampled subsets of the feature-enriched points the presence of four roof faces and their slopes. Trained with the RoofN3D dataset, which contains roof point segmentations and geometric information for 3D reconstruction purposes for about 118,000 simple buildings, the neural network achieves a performance of about 80% intersection over union (IoU) for roof face segmentation, 1.8° mean absolute error (MAE) for roof slope angles, and 95% overall accuracy (OA) for predicting the presence of faces.

Efficient and Outlier-Robust Simultaneous Pose and Correspondence Determination by Branch-and-Bound and Transformation Decomposition.

Estimating the pose of a calibrated camera relative to a 3D point set from one image is an important task in computer vision. Perspective-n-Point algorithms are often used if perfect 2D-3D correspondences are known. However, it is difficult to determine 2D-3D correspondences perfectly, and then the simultaneous pose and correspondence determination problem is needed to be solved. Early methods aimed to solve this problem by local optimization. Recently, several new methods are proposed to globally solve this problem by using branch-and-bound (BnB) method, but they tend to be slow because the time complexity of the BnB-based methods is exponential to the dimensionality of the parameter space, and they directly search the 6D parameter space. In this paper, we propose to decompose the joint searching into two separate searching processes by introducing a rotation invariant feature (RIF). Specifically, we construct RIFs from the original 3D and 2D point sets and search for the globally optimal translation to match these two RIFs first. Then, the original 3D point set is translated and matched with the 2D point set to find a globally optimal rotation. Experiments on challenging data show that the proposed method outperforms state-of-the-art methods in terms of both speed and accuracy.