Observed Point Cloud Research Articles

Refined Prior Guided Category-Level 6D Pose Estimation and Its Application on Robotic Grasping

Estimating the 6D pose and size of objects is crucial in the task of visual grasping for robotic arms. Most current algorithms still require the 3D CAD model of the target object to match with the detected points, and they are unable to predict the object’s size, which significantly limits the generalizability of these methods. In this paper, we introduce category priors and extract high-dimensional abstract features from both the observed point cloud and the prior to predict the deformation matrix of the reconstructed point cloud and the dense correspondence between the reconstructed and observed point clouds. Furthermore, we propose a staged geometric correction and dense correspondence refinement mechanism to enhance the accuracy of regression. In addition, a novel lightweight attention module is introduced to further integrate the extracted features and identify potential correlations between the observed point cloud and the category prior. Ultimately, the object’s translation, rotation, and size are obtained by mapping the reconstructed point cloud to a normalized canonical coordinate system. Through extensive experiments, we demonstrate that our algorithm outperforms existing methods in terms of performance and accuracy on commonly used benchmarks for this type of problem. Additionally, we implement the algorithm in robotic arm-grasping simulations, further validating its effectiveness.

PA-Pose: Partial point cloud fusion based on reliable alignment for 6D pose tracking

Learning-based 6-DOF (6D) pose tracking, serving as a basis for most real-time applications such as augmented reality and robot manipulation, receives attention transiting from 2D to 3D vision, with the popularity of depth sensors. However, the irregular nature of 3D point clouds challenges this task, especially since the lack of explicit alignments hinders the interaction and fusion between the observed point clouds. Therefore, this paper proposes a novel approach named PA-Pose to achieve 6D pose tracking in point clouds. It takes the forward-predicted dense correspondences within an overlap as reliable alignments, to guide the feature fusion of the partial-to-partial point clouds. Then, the relative transformation pose of adjacent observations is continuously regressed from the point-wisely fused features by confidence scoring, avoiding non-differentiable pose fitting. In addition, a shifted point convolution (SPConv) operation is introduced in the fusion process, to further promote the local context interaction of the observed point cloud pair in the expanded alignment field. Extensive experiments on two benchmark datasets (YCB-Video and YCBInEOAT) demonstrate that our method achieves state-of-the-art performance. Even though only 3D point clouds are taken as input, our PA-Pose is still competitive with those methods fully utilizing RGB-D information in the single view. Finally, experiments in the real scene for tracking industrial objects also validates the effectiveness of the proposed method.

Near-optimal bounds for generalized orthogonal Procrustes problem via generalized power method

Given multiple point clouds, how to find the rigid transform (rotation, reflection, and shifting) such that these point clouds are well aligned? This problem, known as the generalized orthogonal Procrustes problem (GOPP), has found numerous applications in statistics, computer vision, and imaging science. While one commonly-used method is finding the least squares estimator, it is generally an NP-hard problem to obtain the least squares estimator exactly due to the notorious nonconvexity. In this work, we apply the semidefinite programming (SDP) relaxation and the generalized power method to solve this generalized orthogonal Procrustes problem. In particular, we assume the data are generated from a signal-plus-noise model: each observed point cloud is a noisy copy of the same unknown point cloud transformed by an unknown orthogonal matrix and also corrupted by additive Gaussian noise. We show that the generalized power method (equivalently alternating minimization algorithm) with spectral initialization converges to the unique global optimum to the SDP relaxation, provided that the signal-to-noise ratio is high. Moreover, this limiting point is exactly the least squares estimator and also the maximum likelihood estimator. Our theoretical bound is near-optimal in terms of the information-theoretic limit (only loose by a factor of the dimension and a log factor). Our results significantly improve the state-of-the-art results on the tightness of the SDP relaxation for the generalized orthogonal Procrustes problem, an open problem posed by Bandeira et al. (2014) [8].

Learning geometric consistency and discrepancy for category-level 6D object pose estimation from point clouds

Category-level 6D object pose estimation aims to predict the position and orientation of unseen object instances, which is a fundamental problem in robotic applications. Previous works mainly focused on exploiting visual cues from RGB images, while depth images received less attention. However, depth images contain rich geometric attributes about the object’s shape, which are crucial for inferring the object’s pose. This work achieves category-level 6D object pose estimation by performing sufficient geometric learning from depth images represented by point clouds. Specifically, we present a novel geometric consistency and geometric discrepancy learning framework called CD-Pose to resolve the intra-category variation, inter-category similarity, and objects with complex structures. Our network consists of a Pose-Consistent Module and a Pose-Discrepant Module. First, a simple MLP-based Pose-Consistent Module is utilized to extract geometrically consistent pose features of objects from the pre-computed object shape priors for each category. Then, the Pose-Discrepant Module, designed as a multi-scale region-guided transformer network, is dedicated to exploring each instance’s geometrically discrepant features. Next, the NOCS model of the object is reconstructed according to the integration of consistent and discrepant geometric representations. Finally, 6D object poses are obtained by solving the similarity transformation between the reconstruction and the observed point cloud. Experiments on the benchmark datasets show that our CD-Pose produces superior results to state-of-the-art competitors.

MSSPA-GC: Multi-Scale Shape Prior Adaptation with 3D Graph Convolutions for Category-Level Object Pose Estimation

Category-level object pose estimation aims to predict the 6D object pose and size of arbitrary objects from known categories. It remains a challenge due to the large intra-class shape variation. Recently, the introduction of the shape prior adaptation mechanism into the normalized canonical coordinates (i.e., NOCS) reconstruction process has been shown to be effective in mitigating the intra-class shape variation. However, existing shape prior adaptation methods simply map the observed point cloud to the normalized object space, and the extracted object descriptors are not sufficient for the perception of the object pose. As a result, they fail to predict the pose of objects with complex geometric structures (e.g., cameras). To this end, this paper proposes a novel shape prior adaption method named MSSPA-GC for category-level object pose estimation. Specifically, our main network takes the observed instance point cloud converted from the RGB-D image and the prior shape point cloud pre-trained on the object CAD models as inputs. Then, a novel 3D graph convolution network and a PointNet-like MLP network are designed to extract pose-aware object features and shape-aware object features from these two inputs, respectively. After that, the two-stream object features are aggregated through a multi-scale feature propagation mechanism to generate comprehensive 3D object descriptors that maintain both pose-sensitive geometric stability and intra-class shape consistency. Finally, by leveraging object descriptors aware of both object pose and shape when reconstructing the NOCS coordinates, our approach elegantly achieves state-of-the-art performance on the widely used REAL275 and CAMERA25 datasets using only 25% of the parameters compared with existing shape prior adaptation models. Moreover, our method also exhibits decent generalization ability on the unconstrained REDWOOD75 dataset.

Category-Level 6D Pose Estimation Using Geometry-Guided Instance-Aware Prior and Multi-Stage Reconstruction

Category-level object 6D pose estimation is essential for robotic manipulation, augmented reality and 3D scene understanding. It aims to accurately predict the translation and rotation of arbitrary shape instances from a given set of object classes without models of each instance. However, such estimation is cumbersome owing to the intra-class variation and the difficulty of accurately regressing dense correspondences between an observed point cloud and the reconstructed instance model from high-order and complex features. In this study, we propose a framework comprising geometry-guided instance-aware prior and multi-stage reconstruction networks to address these challenges. The geometry-guided instance-aware prior network can avoid unstable RGB data interference and robustly learn implicit relations on semantic and geometric information between the observed point cloud and prior. We take advantage of these potential relations using a transformer network to cope with the intra-class variation. Furthermore, the proposed multi-stage reconstruction network is designed to learn the residual of the preliminary reconstruction and improve the accuracy of predicting dense correspondences originating from the complex feature. The results of extensive experiments on a well-acknowledged benchmark for category-level 6D pose estimation demonstrate that our proposed method achieves a significant improvement in performance compared to previous methods.

Real-Time Occlusion-Robust Deformable Linear Object Tracking With Model-Based Gaussian Mixture Model.

Tracking and manipulating deformable linear objects (DLOs) has great potential in the industrial world. However, estimating the object's state is crucial and challenging, especially when dealing with heavy occlusion situations and physical properties of different objects. To address these problems, we introduce a novel tracking algorithm to observe and estimate the states of DLO. The proposed tracking algorithm is based on the Coherent Point Drift (CPD), which registers the observed point cloud, and the finite element method (FEM) model encodes physical properties. The Gaussian mixture model with CPD regularization generates constraints to deform a given FEM model into desired shapes. The FEM model encodes the local structure, the global topology, and the material property to better approximate the deformation process in the real world without using simulation software. A series of simulations and real data tracking experiments have been conducted on deformable objects, such as rope and iron wire, to demonstrate the robustness and accuracy of our method in the presence of occlusion.

6D-ViT: Category-Level 6D Object Pose Estimation via Transformer-Based Instance Representation Learning.

This paper presents 6D vision transformer (6D-ViT), a transformer-based instance representation learning network suitable for highly accurate category-level object pose estimation based on RGB-D images. Specifically, a novel two-stream encoder-decoder framework is dedicated to exploring complex and powerful instance representations from RGB images, point clouds, and categorical shape priors. The whole framework consists of two main branches, named Pixelformer and Pointformer. Pixelformer contains a pyramid transformer encoder with an all-multilayer perceptron (MLP) decoder to extract pixelwise appearance representations from RGB images, while Pointformer relies on a cascaded transformer encoder and an all-MLP decoder to acquire the pointwise geometric characteristics from point clouds. Then, dense instance representations (i.e., correspondence matrix and deformation field) for NOCS model reconstruction are obtained from a multisource aggregation (MSA) network with shape prior, appearance and geometric information as inputs. Finally, the instance 6D pose is computed by solving the similarity transformation between the observed point clouds and the reconstructed NOCS representations. Extensive experiments with synthetic and real-world datasets demonstrate that the proposed framework achieves state-of-the-art performance for both datasets. Code is available at https://github.com/luzzou/6D-ViT.

Estimating Control Points for B-Spline Surfaces Using Fully Populated Synthetic Variance–Covariance Matrices for TLS Point Clouds

A flexible approach for geometric modelling of point clouds obtained from Terrestrial Laser Scanning (TLS) is by means of B-splines. These functions have gained some popularity in the engineering geodesy as they provide a suitable basis for a spatially continuous and parametric deformation analysis. In the predominant studies on geometric modelling of point clouds by B-splines, uncorrelated and equally weighted measurements are assumed. Trying to overcome this, the elementary errors theory is applied for establishing fully populated covariance matrices of TLS observations that consider correlations in the observed point clouds. In this article, a systematic approach for establishing realistic synthetic variance–covariance matrices (SVCMs) is presented and afterward used to model TLS point clouds by B-splines. Additionally, three criteria are selected to analyze the impact of different SVCMs on the functional and stochastic components of the estimation results. Plausible levels for variances and covariances are obtained using a test specimen of several dm—dimension. It is used to identify the most dominant elementary errors under laboratory conditions. Starting values for the variance level are obtained from a TLS calibration. The impact of SVCMs with different structures and different numeric values are comparatively investigated. Main findings of the paper are that for the analyzed object size and distances, the structure of the covariance matrix does not significantly affect the location of the estimated surface control points, but their precision in terms of the corresponding standard deviations. Regarding the latter, properly setting the main diagonal terms of the SVCM is of superordinate importance compared to setting the off-diagonal ones. The investigation of some individual errors revealed that the influence of their standard deviation on the precision of the estimated parameters is primarily dependent on the scanning distance. When the distance stays the same, one-sided influences on the precision of the estimated control points can be observed with an increase in the standard deviations.

Skeletonization of Plant Point Cloud Data Using Stochastic Optimization Framework

Skeleton extraction from 3D plant point cloud data is an essential prior for myriads of phenotyping studies. Although skeleton extraction from 3D shapes have been studied extensively in the computer vision and graphics literature, handling the case of plants is still an open problem. Drawbacks of the existing approaches include the zigzag structure of the skeleton, nonuniform density of skeleton points, lack of points in the areas having complex geometry structure, and most importantly the lack of biological relevance. With the aim to improve existing skeleton structures of state-of-the-art, we propose a stochastic framework which is supported by the biological structure of the original plant (we consider plants without any leaves). Initially we estimate the branching structure of the plant by the notion of β-splines to form a curve tree defined as a finite set of curves joined in a tree topology with certain level of smoothness. In the next phase, we force the discrete points in the curve tree to move toward the original point cloud by treating each point in the curve tree as a center of Gaussian, and points in the input cloud data as observations from the Gaussians. The task is to find the correct locations of the Gaussian centroids by maximizing a likelihood. The optimization technique is iterative and is based on the Expectation Maximization (EM) algorithm. The E-step estimates which Gaussian the observed point cloud was sampled from, and the M-step maximizes the negative log-likelihood that the observed points were sampled from the Gaussian Mixture Model (GMM) with respect to the model parameters. We experiment with several real world and synthetic datasets and demonstrate the robustness of the approach over the state-of-the-art.

Vehicle Tracking and Speed Estimation From Roadside Lidar

Vehicle speed is a key variable for the calibration, validation, and improvement of traffic emission and air quality models. Lidar technologies have significant potential in vehicle tracking by scanning the surroundings in 3-D frequently, hence can be used as traffic flow monitoring sensors for accurate vehicle counting and speed estimation. However, the characteristics of lidar-based vehicle tracking and speed estimation, such as attainable accuracy, remain as open questions. This research therefore proposes a tracking framework from roadside lidar to detect and track vehicles with the aim of accurate vehicle speed estimation. Within this framework, on-road vehicles are first detected from the observed point clouds, after which a centroid-based tracking flow is implemented to obtain initial vehicle transformations. A tracker, utilizing the unscented Kalman Filter and joint probabilistic data association filter, is adopted in the tracking flow. Finally, vehicle tracking is refined through an image matching process to improve the accuracy of estimated vehicle speeds. The effectiveness of the proposed approach has been evaluated using lidar data obtained from two different panoramic 3-D lidar sensors, a RoboSense RS-LiDAR-32 and a Velodyne VLP-16, at a traffic light and a road intersection, respectively, in order to account for real-world scenarios. Validation against reference data obtained by a test vehicle equipped with accurate positioning systems shows that more than 94% of vehicles could be detected and tracked, with a mean speed accuracy of 0.22 m/s.

Hypergraph Spectral Clustering for Point Cloud Segmentation

Hypergraph spectral analysis has emerged as an effective tool processing complex data structures in data analysis. The surface of a three-dimensional (3D) point cloud, and the multilateral relationship among their points can be naturally captured by the high-dimensional hyperedges. This work investigates the power of hypergraph spectral analysis in unsupervised segmentation of 3D point clouds. We estimate, and order the hypergraph spectrum from observed point cloud coordinates. By trimming the redundancy from the estimated hypergraph spectral space based on spectral component strengths, we develop a clustering-based segmentation method. We apply the proposed method to various point clouds, and analyze their respective spectral properties. Our experimental results demonstrate the effectiveness and efficiency of the proposed segmentation method.

Coarse Alignment for Model Fitting of Point Clouds Using a Curvature-Based Descriptor

This paper presents a method for coarse alignment of point clouds by introducing a new descriptor based on the local curvature. The method is developed for model fitting a CAD model for use in robotic assembly. The method is based on selecting keypoints depending on shape factors calculated from the local covariance matrix of the surface. A descriptor is then calculated for each keypoint by fitting two spheres that describe the curvature of the surface. The spheres are calculated using conformal geometric algebra, which gives a convenient and efficient description of the geometry. The keypoint descriptors for the model and the observed point cloud are then compared to estimate the corresponding keypoints, which are used to calculate the displacement. The method is tested in several experiments. One experiment is for robotic assembly, where objects are placed on a table and their position and orientation is estimated using a 3-D CAD model. Note to Practitioners — 3-D cameras can be used in robotic assembly for recognizing objects, and for determining the position and orientation of parts to be assembled. In such applications, 3-D CAD models will be available for the objects, and point clouds representing each object can be generated for comparison with the observed point clouds from the 3-D camera. It is not straightforward to use existing descriptors in this paper, as the point cloud from the CAD model and the observed point cloud may differ due to different viewpoints and potential occlusions. The method proposed in this paper is intended to be easy to apply to industrial assembly problems where there is a need for a robust estimation of the displacement of an object, either as a coarse estimate for use in grasping or as an initial guess to use in fine registration for demanding assembly operations with close tolerances. The method exploits the curvature of the point clouds to accurately describe the surrounding surface of each point. This method serves as a basis for future industrial implementations.

Evaluation of Geometric Error Sources for Terrestrial Laser Scanner

As 3D geospatial information is demanded, terrestrial laser scanners which can obtain 3D model of objects have been applied in various fields such as Building Information Modeling (BIM), structural analysis, and disaster management. To acquire precise data, performance evaluation of a terrestrial laser scanner must be conducted. While existing 3D surveying equipment like a total station has a standard method for performance evaluation, a terrestrial laser scanner evaluation technique for users is not established. This paper categorizes and analyzes error sources which generally occur in terrestrial laser scanning. In addition to the prior researches about categorizing error sources of terrestrial Laser scanning, this paper evaluates the error sources by the actual field tests for the smooth in-situ applications. The error factors in terrestrial laser scanning are categorized into interior error caused by mechanical errors in a terrestrial laser scanner and exterior errors affected by scanning geometry and target property. Each error sources were evaluated by simulation and actual experiments. The 3D coordinates of observed target can be distortedby the biases in distance and rotation measurement in scanning system. In particular, the exterior factors caused significant geometric errors in observed point cloud. The noise points can be generated by steep incidence angle, mixed-pixel and crosstalk. In using terrestrial laser scanner, elaborate scanning plan and proper post processing are required to obtain valid and accurate 3D spatial information.

Estimation of linear target-layer trajectories using cluttered point cloud data

The problem of estimating a target-layer trajectory, modeled by a straight line, in 2D point clouds that contain target locations and overwhelming clutter is studied. These point clouds are generated by an image-based pre-processing tool, termed ATR, operating on SONAR image data that results in: (1) point locations and (2) an ATR score: a measure of the “target-likeness” for each point. The model of choice assumes that the observed point cloud is a superposition of two spatial processes: (1) a 1D Poisson process along the target-layer line, corrupted by 2D Gaussian noise, denoting target locations and (2) a 2D Poisson process denoting clutter. It is further assumed that the target-likeness measure follows known probability distributions for both target locations and clutter. The line is parameterized by distance from the origin and the angle with respect to a horizontal axis, and the likelihood of these parameters for observed data is derived. Using a maximum-likelihood approach, a gradient-based estimate for line parameters and other nuisance parameters is developed. A formal procedure that tests for the presence of a target-layer trajectory in the point cloud data is additionally developed. The success of this method in both simulated and real datasets collected by NSWC PCD is demonstrated.

Detection, classification and estimation of individual shapes in 2D and 3D point clouds

The problems of detecting, classifying, and estimating shapes in point cloud data are important due to their general applicability in image analysis, computer vision, and graphics. They are challenging because the data is typically noisy, cluttered, and unordered. We study these problems using a fully statistical model where the data is modeled using a Poisson process on the object’s boundary (curves or surfaces), corrupted by additive noise and a clutter process. Using likelihood functions dictated by the model, we develop a generalized likelihood ratio test for detecting a shape in a point cloud. This ratio test is based on optimizing over some unknown parameters, including the pose and scale associated with hypothesized objects, and an empirical evaluation of the log-likelihood ratio distribution. Additionally, we develop a procedure for estimating most likely shapes in observed point clouds under given shape hypotheses. We demonstrate this framework using examples of 2D and 3D shape detection and estimation in both real and simulated data, and a usage of this framework in shape retrieval from a 3D shape database.

Noisy Depth Maps Fusion for Multiview Stereo Via Matrix Completion

This paper introduces a general framework to fuse noisy point clouds from multiview images of the same object. We solve this classical vision problem using a newly emerging signal processing technique known as matrix completion. With this framework, we construct the initial incomplete matrix from the observed point clouds by all the cameras, with the invisible points by any camera denoted as unknown entries. The observed points corresponding to the same object point are put into the same row. When properly completed, the recovered matrix should have rank one, since all the columns describe the same object. Therefore, an intuitive approach to complete the matrix is by minimizing its rank subject to consistency with observed entries. In order to improve the fusion accuracy, we propose a general noisy matrix completion method called log-sum penalty completion (LPC), which is particularly effective in removing outliers. Based on the majorization–minimization algorithm (MM), the non-convex LPC problem is effectively solved by a sequence of convex optimizations. Experimental results on both point cloud fusion and MVS reconstructions verify the effectiveness of the proposed framework and the LPC algorithm.