Multi-view Point Cloud Research Articles

Digital image correlation calculation method for RGB-D camera multi-view matching using variable template

Multi-view matching is an indispensable method for digital image correlation 3D reconstruction. However, as the camera viewing angle changes, traditional DIC methods can lead to the accumulation of matching errors and an increase in mismatches, thereby affecting the 3D reconstruction quality. This paper proposes a digital image correlation method for multi-view RGB-D images based on variable template matching. Initial positioning is achieved by dynamically adjusting the size and sampling direction of the variable template. Sub-pixel registration is then performed using the inverse compositional algorithm, followed by multi-view point cloud fusion using the Iterative Closest Point (ICP) registration method. Experimental results show that the iteration speed of point cloud registration is increased by 30%, and the cumulative error is reduced by 62.5%. These improvements demonstrate that the proposed method is suitable for multi-viewpoint matching scenarios, achieving excellent results in multi-view feature point matching and multi-view point cloud fusion, significantly enhancing 3D reconstruction accuracy and efficiency.

Multi-View Point Cloud Registration Based on Improved NDT Algorithm and ODM Optimization Method

Multi-View Point Cloud Registration Based on Improved NDT Algorithm and ODM Optimization Method

A Hybrid Improved SAC-IA with a KD-ICP Algorithm for Local Point Cloud Alignment Optimization

To overcome incomplete point cloud data obtained from laser scanners scanning complex surfaces, multi-viewpoint cloud data needs to be aligned for use. A hybrid improved SAC-IA with a KD-ICP algorithm is proposed for local point cloud alignment optimization. The scanned point cloud data is preprocessed with statistical filtering, as well as uniform down-sampling. The sampling consistency initial alignment (SAC-IA) algorithm is improved by introducing a dissimilarity vector for point cloud initial alignment. In addition, the iterative closest point (ICP) algorithm is improved by incorporating bidirectional KD-tree to form the KD-ICP algorithm for fine point cloud alignment. Finally, the algorithms are compared in terms of runtime and alignment accuracy. The implementation of the algorithms is based on the Visual Studio 2013 software configurating point cloud library environment for testing experiments and practical experiments. The overall alignment method can be 40%~50% faster in terms of running speed. The improved SAC-IA algorithm provides better transformed poses, combined with the KD-ICP algorithm to select the corresponding nearest neighbor pairs, which improves the accuracy, as well as the applicability of the alignment.

Multi-view 3D reconstruction of seedling using 2D image contour

3D reconstruction of seedling can provide comprehensive and quantitative spatial structure information, offering an effective digital tool for breeding research. However, accurate and efficient reconstruction of seedling is still a challenging work due to limited performance of depth sensor for seedling with small-size stem and unavoidable error for multi-view point cloud registration. Therefore, in this paper, we propose an accurate multi-view 3D reconstruction method for seedling using 2D image contour to constrain 3D point cloud. The rotation axis is calibrated and optimised by minimising point-to-contour distance between 2D image contour and projected exterior points from 3D point cloud. Then, to remove outliers and noise, we introduce the seedling mask of 2D image to constrained and delete projected outlier points of 3D model from corresponding view. Furthermore, we propose a residual-guided method to recognise missing region for 3D model and complete 3D model of small-size stem. Finally, we can obtain an accurate 3D model of seedling. The reconstruction accuracy is evaluated by average distance between projected contour of 3D model and 2D image contour of all views (0.3185 mm). Then, the phenotypic parameters were calculated from 3D model and the results are close to manual measurements (Plant height: R2 = 0.98, RMSE = 2.3 mm, rRMSE = 1.52%; Petioles inclination angle: R2 = 0.99, RMSE = 0.73°, rRMSE = 1.41%; Leaf area: R2 = 0.66, RMSE = 1.05 cm2, rRMSE = 7.63%; Leaf inclination angle: R2 = 0.99, RMSE = 1.01°, rRMSE = 1.72%; Stem diameter: R2 = 0.95, RMSE = 0.12 mm, rRMSE = 5.43%). Breeders can improve the selection of more resilient varieties and cultivars to different growing conditions starting from the dynamic analysis of their phenotype.

An image segmentation and point cloud registration combined scheme for sensing of obscured tree branches

Automated robots are emerging as a solution for labor-intensive fruit orchard management. Three-dimensional (3D) reconstruction of tree branches is a fundamental requirement for robots to perform tasks like pruning and fruit harvesting. Current branch sensing methods often rely on planar segmentation with limited 3D information or computationally expensive point cloud segmentation, which may not be suitable for natural orchards with obscured tree branches. This study proposes a novel scheme that reconstructs occluded branches from RGB-D (Red-Green-Blue-Depth) images by integrating the point clouds converted from planar segmentation masks and depth images. The proposed approach extends the existing 2D branch sensing techniques to 3D, leveraging multi-view information. The deep learning models DeeplabV3+ and Pix2pix are employed to generate the segmentation masks, separately. And the Fast Global Registration (FGR) is used to register the multi-view point clouds. The results demonstrate that the output point clouds have at least a 24 % increase in the number of corresponding points after FGR. Furthermore, the time cost per hundred corresponding points is reduced by 85 % and 69 % when using the DeepLabV3 + and Pix2pix-based schemes, respectively, compared to the PointNet++ approach. These findings indicate that the proposed scheme significantly improves the sensing of occluded branches in terms of output richness and computational efficiency, making it applicable to natural orchard working spaces.

MSCS-ICP: point cloud registration method using multi-view spatial coordinate system–ICP

The effectiveness of point cloud registration critically determines three-dimensional (3D) reconstruction accuracy involving multi-view sensors. We introduce a multi-view point cloud registration method based on multi-view spatial coordinate system–ICP to solve the problem of 3D point cloud registration from different viewpoints. By integrating a spatial rotation axis line, our method successfully establishes the spatial coordinate system tailored for multi-view sensors, ensuring that 3D point clouds derived from various perspectives are optimally positioned initially. We employ the ICP technique for point cloud merging, facilitating a seamless transition from coarse to refined registration of these multi-view 3D point clouds. During the process of spatial rotation axis line fitting, we present a Ransac-based algorithm tailored for axis line fitting that effectively removes outliers, thus significantly improving the fitting precision. Experimental results from a standard sphere reconstruction reveal that within a measurement scope of 1.3–1.9 m, our proposed method boasts a maximum error of just 0.069 mm, an average absolute error of 0.039 mm, and a root mean square error of 0.043 mm. The speed of our point cloud registration outpaces that of alternative methods. Our method notably elevates the precision and velocity of 3D point cloud registration across diverse views, demonstrating commendable adaptability and resilience.

Multi-View consistency-based point cloud registration method with low overlap rate

Accurate and efficient workpiece measurement is crucial for workpiece processing and quality monitoring. Non-contact optical measurement methods have gained more attention due to their simplicity, efficiency, and flexibility compared to complicated and inefficient contact measurement methods. Multi-view registration of measurement data is a key issue in workpiece measurement, as it relies on the system’s geometric accuracy and motion stability, presenting challenges such as the insufficient overlap of multi-viewpoint cloud data and cumulative error. To address these challenges, this paper proposes a multi-view planning and registration algorithm with a low overlap rate. The multi-view planning algorithm employs a greedy method to plan the scanning viewpoints of the workpiece to obtain complete point cloud data efficiently. The multi-view registration algorithm extracts features using a multi-scale geometric feature extraction network, matches the features based on the Hungarian algorithm, builds a graph, and optimizes workpiece positions based on the G2O algorithm for multi-view registration, effectively reducing cumulative error. Measurement experiments on blade workpieces confirm the feasibility of the proposed algorithms.

Fast Multi-View 3D reconstruction of seedlings based on automatic viewpoint planning

Three-dimensional models of plants provide valuable phenotypic information. Phenotypic measurements of plant structures are essential for monitoring plant growth and understanding plant responses to environmental changes. Existing 3D reconstruction techniques have achieved accurate reconstruction models of some plants such as corn and soybeans. However, several drawbacks exist in current plant 3D reconstruction systems, including high cost, fixed capture perspectives and complex operation. To address these issues and considering the influence of camera capture angles on model reconstruction accuracy, we investigated a viewpoint planning reconstruction method. We designed a plant seedling reconstruction system that utilizes a consumer-grade L515 LiDAR sensor and a precision turntable. We establish a viewpoint rule based on the camera imaging model of the three-dimensional spatial structure of leaves, constructing minimum constraints on the leaf normal vector and the camera optical axis. This enables to determine the optimal capture viewpoint for each leaf to obtain the maximum leaf information. The angle β between the leaf normal vector and the camera optical axis serves as the basis for the turntable rotation, systematically planning the rotation of the turntable to enable the camera to capture the maximum useful leaf information. Unlike static camera-based turntable reconstruction systems that capture images at fixed angle intervals, we plan the rotation of the turntable based on the acquired information. This allows us to obtain more effective point cloud information of plant seedlings from the same or even fewer images, reducing the collection of redundant images and reconstructing more accurate plant seedling 3D models. Additionally, we measured commonly used leaf traits in plant phenotypic studies, such as leaf length, leaf width, and leaf area. The leaf area measured based on the reconstructed seedling model exhibited high accuracy (R2 > 0.99). The results of this study demonstrate that the consumer-grade L515 LiDAR sensor combined with the proposed viewpoint planning method can effectively reconstruct accurate seedling 3D models and measure phenotypic information. Due to the absence of error accumulation caused by adjacent view registration, this approach offers advantages such as shorter reconstruction time, higher efficiency and increased accuracy compared to some multi-view point cloud registration and reconstruction methods. Therefore, this system can be applied to three-dimensional reconstruction and phenotype analysis of plant seedlings due to its high efficiency and accuracy.

MVGR: Mean-Variance Minimization Global Registration Method for Multiview Point Cloud in Robot Inspection

MVGR: Mean-Variance Minimization Global Registration Method for Multiview Point Cloud in Robot Inspection

Registration of Multiview Point Clouds with Unknown Overlap

Registration of Multiview Point Clouds with Unknown Overlap

A novel 3D vision-based robotic welding path extraction method for complex intersection curves

With the rapid development of robot technology, robotic welding technology has gradually been applied in the complex co-main intersection pipeline welding field. However, due to the existence of clamping errors and processing dimensional errors in intersecting pipes of the same specification, the actual welding path deviates from the taught path and cannot meet the accuracy requirements of robot welding, which further hinders the development and application of robotic welding in this field. To solve these problems, we propose a 3D vision-based robotic welding path extraction method for complex intersecting curves without programming and teaching. We begin by selecting the complex co-main intersection pipeline as the research object and building a robotic welding system based on 3D vision. Through mathematical analysis of the complex intersecting pipe model and an analysis of the internal coordinate transformation relationship within the welding system, we establish a mathematical model of the complex intersection curves at any clamping position in the manipulator coordinate system. On this basis, We propose a multi-view point cloud stitching method using Eye-in-Hand calibration and the ICP algorithm to obtain a complete co-main intersection pipeline point cloud. Subsequently, we propose a mathematical model parameter-solving algorithm based on the co-main intersection pipeline point cloud. Finally, the obtained actual parameters of the workpiece are combined with the established mathematical model to extract the welding path of the robot’s current position. Experimental results demonstrate that the proposed method can automatically extract welding seams for complex intersection curves without the need for programming and teaching.

Three-Dimensional Structure Measurement for Potted Plant Based on Millimeter-Wave Radar

Potted plant canopy extraction requires a fast, accurate, stable, and affordable detection system for precise pesticide application. In this study, we propose a new method for extracting three-dimensional canopy information of potted plants using millimeter-wave radar and evaluate the system on plants in static, rotating, and rotating-while-spraying states. The position and rotation speed of the rotating platform are used to compute the rotation–translation matrix between point clouds, enabling the multi-view point clouds to be overlaid on the world coordinate system. Point cloud extraction is performed by applying the Density-Based Spatial Clustering of Applications with Noise algorithm (DBSCAN), while an Alpha-shape algorithm is used for three-dimensional reconstruction of the canopy. Our measurement results for the 3D reconstruction of plants at different growth stages showed that the reconstruction model has higher accuracy under the rotation condition than that under the static condition, with average relative errors of 41.61% and 10.21%, respectively. The significant correlation between the sampling data with and without spray reached 0.03, indicating that the effect of the droplets on radar detection during the spray process can be neglected. This study provides guidance for plant canopy detection using millimeter-wave radar for advanced agricultural informatization and automation.

Reconstruction-Based Hand–Eye Calibration Using Arbitrary Objects

This paper introduces a flexible hand-eye calibration technique for 3D sensor from reconstruction perspective, with no need for specialized and accurate calibration rig. Our intention is to find the hand-eye relation which simultaneously aligns multi-view point clouds of a common scene into the robot base frame, namely simultaneous calibration and reconstruction. To achieve this goal, a novel variant of iterative closest point (ICP) algorithm based on Gauss-Newton method and Lie algebra is proposed, which iteratively transforms multi-view point clouds into robot base frame, estimates point-to-point correspondences between point clouds then refines the hand-eye relation to minimize the Euclidean distance between corresponding points. In addition to the calibration result, it returns a preliminary reconstruction as byproduct. Cases of degeneracy and applicable condition are given and proved. Using arbitrary daily objects with no prior information and real robotic eye-in-hand system, we verify our method feasible and effective.

Multi-View Point Cloud Registration Based on Evolutionary Multitasking With Bi-Channel Knowledge Sharing Mechanism

Multi-view point cloud registration is fundamental in 3D reconstruction. Since there are close connections between point clouds captured from different viewpoints, registration performance can be enhanced if these connections be harnessed properly. Therefore, this article models the registration problem as multi-task optimization, and proposes a novel bi-channel knowledge sharing mechanism for effective and efficient problem solving. The modeling of multi-view point cloud registration as multi-task optimization are twofold. By simultaneously considering the local accuracy of two point clouds as well as the global consistency posed by all the point clouds involved, a fitness function with an adaptive threshold is derived. Also a framework of the co-evolutionary search process is defined for the concurrent optimization of multiple fitness functions belonging to related tasks. To enhance solution quality and convergence speed, the proposed bi-channel knowledge sharing mechanism plays its role. The intra-task knowledge sharing introduces aiding tasks that are much simpler to solve, and useful information is shared across aiding tasks and the original tasks, accelerating the search process. The inter-task knowledge sharing explores commonalities buried among the original tasks, aiming to prevent tasks from getting stuck to local optima. Comprehensive experiments conducted on model object as well as scene point clouds show the efficacy of the proposed method.

Mushroom Detection and Three Dimensional Pose Estimation from Multi-View Point Clouds.

Agricultural robotics is an up and coming field which deals with the development of robotic systems able to tackle a multitude of agricultural tasks efficiently. The case of interest, in this work, is mushroom collection in industrial mushroom farms. Developing such a robot, able to select and out-root a mushroom, requires delicate actions that can only be conducted if a well-performing perception module exists. Specifically, one should accurately detect the 3D pose of a mushroom in order to facilitate the smooth operation of the robotic system. In this work, we develop a vision module for 3D pose estimation of mushrooms from multi-view point clouds using multiple RealSense active-stereo cameras. The main challenge is the lack of annotation data, since 3D annotation is practically infeasible on a large scale. To address this, we developed a novel pipeline for mushroom instance segmentation and template matching, where a 3D model of a mushroom is the only data available. We evaluated, quantitatively, our approach over a synthetic dataset of mushroom scenes, and we, further, validated, qualitatively, the effectiveness of our method over a set of real data, collected by different vision settings.

Robust Multipoint-Sets Registration for Free-Form Surface Based on Probability

Free-form surface reconstruction using point clouds is a common issue in manufacturing. In this article, a robust joint registration approach for multiview point clouds is proposed to address the problems brought by coarse initialization, outliers, and noise. The basic idea is that minimizing the L2 distance between probability distributions of integrated and standard models, such that a robust initialization is provided for fine registration to avoid local minima. The fine registration is formulated as a joint closet point problem, which is implicitly constrained by closed-loop consistency. In addition, a Lie algebra solution is derived to enforce rigid transformations. The robust initialization is judged by the simulated annealing algorithm. Finally, a probabilistic distance is defined and a maximum likelihood estimation of multiview transformations is designed to resist noise. The experiment on simulated and real data illustrate better robustness of our method to initial errors, outliers, and noise.

High-fidelity standard model reconstruction and verification of an airliner based on point clouds.

Computational fluid dynamics (CFD) numerical simulation as the primary research tool is particularly essential for its credibility during the aerodynamic design of aircraft. To further promote CFD verification and validation on the airliner, a high-fidelity model reconstruction of the airliner is fundamental. Based on this, we put forward a novel framework, to our best knowledge, to reconstruct a high-fidelity standard model for an airliner efficiently, and the feasibility and accuracy of these reconstructed models are accessed by the CFD simulation-based validation method. First and foremost, a laser scanner was placed at each station around the airliner to scan and acquire multiview point clouds. Afterwards, the truncated least-squares-based algorithm was adopted to register these point clouds entirely. Additionally, we fitted the nonuniform rational basis spline surface based on the least-squares progressive and iterative approximation algorithm. Finally, CFD simulation results were compared and analyzed with the aerodynamic data obtained by the aircraft manufacturer under the same Mach number of the uniform model. It turns out that the coincidence between them is high, and the changing trend is basically consistent. Hence, this method is highly feasible and can establish a high-fidelity standard model of an airliner with unified high and low speeds so that its appearance, test data, and research results can be adopted as the standard data for CFD verification and validation.

Novel optical-markers-assisted point clouds registration for panoramic 3D shape measurement

Panoramic 3D shape measurement for larger-scale and complex objects inevitably requires multi-view point clouds registration. However, traditional registration methods often use expensive devices or inflexible physical markers. In this paper, a novel optical-markers-assisted registration (OMAR) method is proposed, which substitutes traditional physical markers with optical markers for the registration. Specifically, a markers design strategy is first proposed to generate the desired optical markers according to different requirements. The designed markers can be projected by using a low-cost projector or a laser with grating. The camera captures these projected markers from the object surface, and the 3D sensor collaboratively acquires the corresponding 3D point clouds. Finally, a markers-guided algorithm is specially designed to suit optical markers, which can make full use of the optical markers and recover the 360-degree 3D shape. The provided experimental results demonstrate both the accuracy and efficiency of the proposed OMAR in measuring large-scale, complex and texture-less objects. The proposed OMAR also enables flexible and non-destructive registration without losing accuracy compared with traditional registration methods.

Multi-View Point Clouds Registration Method Based on Overlap-Area Features and Local Distance Constraints for the Optical Measurement of Blade Profiles

The accurate and efficient measurement of blade profiles plays an important role in blade processing and quality monitoring. Currently, increasing attention is being paid to the optical measurement method for blade profiles due to its easy accessibility, high efficiency, and flexibility. However, as a key issue of blade profile measurement, the registration of multiview measured data still relies on the geometric accuracy and motion stability of the measurement system. Besides, multiple factors, including the various density and insufficient overlaps of multiview point cloud data, make the registration a challenging problem. To overcome these problems and realize accurate measurement of blade profiles, a fine registration algorithm is proposed in this article, which mainly comprises two novel constraint mechanisms: overlap-area features and local distance constraints. Two constraint matrices based on the features and distance information are constructed and integrated based on the Hadamard Product of matrices. In this way, the underlying corresponding mapping between multiview point clouds can be effectively recovered and the registration accuracy can be improved. The results of measurement experiment on three typical blades demonstrate the accuracy and feasibility of the proposed method.

An Improved Point Clouds Model for Displacement Assessment of Slope Surface by Combining TLS and UAV Photogrammetry

TLS can quickly and accurately capture object surface coordinates. However, TLS point clouds cannot cover the entire surface of the target object, due to block of view and limitation of measurement condition. Thus, using it to monitor deformation of slope reduces the detection accuracy of slope surface deformation. To overcome the drawbacks, a method to improve TLS point clouds by UAV photogrammetric point clouds is proposed. The two kinds of point clouds are registered as the new multi-view point clouds by PCA and ICP. The locations of monitoring points are extracted based on HSL color space recognition method from the new multi-view point clouds to analyze the surface displacement. At present, the proposed method has applied in a highway slope in Yunnan Province, and complete point clouds were successfully constructed. A RTK survey was used to compare and verify the proposed method. The verification result demonstrate that the difference of displacement between two measurement methods is less than 10 mm. Comprehensive experiments demonstrate that the proposed method is reliable and meets the slope displacement monitoring standard.