3D Data Registration Research Articles

Alignment of a 360° image with posed color images for locally accurate texturing of 3D mesh

With the popularity of 3D content like virtual tours, the challenges of 3D data registration have become increasingly significant. The registration of heterogeneous data obtained from 2D and 3D sensors is required to create photo-realistic 3D models. However, the alignment of 2D images with 3D models introduces a significant challenge due to their inherent differences. This article introduces a rigorous mathematical approach to align a 360° image with its corresponding 3D model generated from images with known camera poses followed by texture projection on the model. We use Scale-Invariant Feature Transform (SIFT) feature descriptors enhanced with a homography-based metric to establish correspondences between the faces of a cubemap and the posed images. To achieve optimal alignment, we use a non-linear least squares optimization technique with a custom objective function. Subsequently, the outcomes of the alignment process are evaluated through texturing using a customized raytracing algorithm. The resulting projections are compared against the original textures, with a comprehensive assessment of the alignment's fidelity and precision.

Cotton morphological traits tracking through spatiotemporal registration of terrestrial laser scanning time-series data.

Understanding the complex interactions between genotype-environment dynamics is fundamental for optimizing crop improvement. However, traditional phenotyping methods limit assessments to the end of the growing season, restricting continuous crop monitoring. To address this limitation, we developed a methodology for spatiotemporal registration of time-series 3D point cloud data, enabling field phenotyping over time for accurate crop growth tracking. Leveraging multi-scan terrestrial laser scanning (TLS), we captured high-resolution 3D LiDAR data in a cotton breeding field across various stages of the growing season to generate four-dimensional (4D) crop models, seamlessly integrating spatial and temporal dimensions. Our registration procedure involved an initial pairwise terrain-based matching for rough alignment, followed by a bird's-eye view adjustment for fine registration. Point clouds collected throughout nine sessions across the growing season were successfully registered both spatially and temporally, with average registration errors of approximately 3cm. We used the generated 4D models to monitor canopy height (CH) and volume (CV) for eleven cotton genotypes over two months. The consistent height reference established via our spatiotemporal registration process enabled precise estimations of CH (R 2=0.95, RMSE = 7.6cm). Additionally, we analyzed the relationship between CV and the interception of photosynthetically active radiation (IPAR f ), finding that it followed a curve with exponential saturation, consistent with theoretical models, with a standard error of regression (SER) of 11%. In addition, we compared mathematical models from the Richards family of sigmoid curves for crop growth modeling, finding that the logistic model effectively captured CH and CV evolution, aiding in identifying significant genotype differences. Our novel TLS-based digital phenotyping methodology enhances precision and efficiency in field phenotyping over time, advancing plant phenomics and empowering efficient decision-making for crop improvement efforts.

Retracted: 3D Point Cloud Data Registration Algorithm Based on Augmented Reality Technology

Retracted: 3D Point Cloud Data Registration Algorithm Based on Augmented Reality Technology

Building 3D Generative Models from Minimal Data.

We propose a method for constructing generative models of 3D objects from a single 3D mesh and improving them through unsupervised low-shot learning from 2D images. Our method produces a 3D morphable model that represents shape and albedo in terms of Gaussian processes. Whereas previous approaches have typically built 3D morphable models from multiple high-quality 3D scans through principal component analysis, we build 3D morphable models from a single scan or template. As we demonstrate in the face domain, these models can be used to infer 3D reconstructions from 2D data (inverse graphics) or 3D data (registration). Specifically, we show that our approach can be used to perform face recognition using only a single 3D template (one scan total, not one per person). We extend our model to a preliminary unsupervised learning framework that enables the learning of the distribution of 3D faces using one 3D template and a small number of 2D images. Our approach is motivated as a potential model for the origins of face perception in human infants, who appear to start with an innate face template and subsequently develop a flexible system for perceiving the 3D structure of any novel face from experience with only 2D images of a relatively small number of familiar faces.

Robust H-K Curvature Map Matching for Patient-to-CT Registration in Neurosurgical Navigation Systems

Image-to-patient registration is a coordinate system matching process between real patients and medical images to actively utilize medical images such as computed tomography (CT) during surgery. This paper mainly deals with a markerless method utilizing scan data of patients and 3D data from CT images. The 3D surface data of the patient are registered to CT data using computer-based optimization methods such as iterative closest point (ICP) algorithms. However, if a proper initial location is not set up, the conventional ICP algorithm has the disadvantages that it takes a long converging time and also suffers from the local minimum problem during the process. We propose an automatic and robust 3D data registration method that can accurately find a proper initial location for the ICP algorithm using curvature matching. The proposed method finds and extracts the matching area for 3D registration by converting 3D CT data and 3D scan data to 2D curvature images and by performing curvature matching between them. Curvature features have characteristics that are robust to translation, rotation, and even some deformation. The proposed image-to-patient registration is implemented with the precise 3D registration of the extracted partial 3D CT data and the patient’s scan data using the ICP algorithm.

3D Point Cloud Data Registration Algorithm Based on Augmented Reality Technology

Point cloud data registration is one of the key steps in 3D laser scanning data processing. At present, point cloud data registration has the problems of error and is too much time-consuming. In order to solve the above problems, a 3D point cloud data registration algorithm based on augmented reality technology is proposed, and the 3D data registration model is constructed by combining augmented reality technology, build the evaluation index of 3D point cloud data registration, and carry out the initial registration of 3D point cloud based on the evaluation index. The experiment shows that the 3D point cloud data registration algorithm based on augmented reality technology can more effectively improve the accuracy of data registration and avoid the problem of low efficiency of data registration in practice.

Multi-line laser structured light fast visual positioning system with assist of TOF and CAD

As a reliable and high-precision three-dimensional (3D) vision method, line laser structured light vision has been widely used in the engineering field. However, to obtain overall 3D shape of target to be measured, scanning mechanisms need to be added. Multi-line laser structured light vision can obtain multi-line array 3D shape of workpiece without scanning; however, the encoding of multi-line lasers is a difficult task. This paper proposes a composite vision system, which is composed of time-of-flight(TOF) camera, color camera and vertical cross multi-line laser structured light. Color camera and multi-line laser form a multi-line array 3D vision system, while the coded signal of multi-line laser stripes is provided by TOF camera and corresponding coding algorithm is also proposed in this paper. Based on this system, automatic positioning of welding seam is realized by registration of CAD data and multi-line laser 3D data. The registration is divided into two steps, which are coarse registration and precise registration. Coarse registration is performed by using CAD point cloud data and original point cloud data of multi-line laser; while precise registration is performed with CAD welding seam point cloud data and welding seam point cloud data of multi-line laser. This paper also analyzes accuracy of vision system based on constructing structured light vision model. The measurement accuracy of vision system in x, y and z directions is 0.25, 0.06 and 0.29 mm respectively. This composite vision system can realize precise positioning of target within a single exposure time, which provides possibility for positioning under movement.

Does the 3D Feature Descriptor Impact The Registration Accuracy in Laparoscopic Liver Surgery?

Abstract In laparoscopic liver surgery (LLS) image-guided navigation systems could support the surgeon by providing subsurface information such as the positions of tumors and vessels. For this purpose, one option is to perform a registration of preoperative 3D data and 3D surface patches reconstructed from laparoscopic images. Part of an automatic 3D registration pipeline is the feature description, which takes into account various geometric and spatial information. Since there is no leading feature descriptor in the field of LLS, two feature descriptors are compared in this paper: The Fast Point Feature Histogram (FPFH) and Triple Orthogonal Local Depth Images (TOLDI). To evaluate their performance, three perturbations were induced: varying surface patch sizes, spatial displacement, and Gaussian deformation. Registration was performed using the RANSAC algorithm. FPFH outperformed TOLDI for small surface patches and in case of Gaussian deformations in terms of registration accuracy. In contrast, TOLDI showed lower registration errors for patches with spatial displacement. While developing a 3D-3D registration pipeline, the choice of the feature descriptor is of importance, consequently a careful choice suitable for the application in LLS is necessary.

Simple and efficient registration of 3D point cloud and image data for an indoor mobile mapping system.

Registration of 3D lidar point clouds with optical images is critical in the combination of multisource data. Geometric misalignment originally exists in the pose data between lidar point clouds and optical images. To improve the accuracy of the initial pose and the applicability of the integration of 3D points and image data, we develop a simple but efficient registration method. We first extract point features from lidar point clouds and images: point features are extracted from single-frame lidar and point features are extracted from images using a classical Canny operator. The cost map is subsequently built based on Canny image edge detection. The optimization direction is guided by the cost map, where low cost represents the desired direction, and loss function is also considered to improve the robustness of the proposed method. Experiments show positive results.

Airborne high-resolution image motion target detection combined with hyperspectral features

Aiming at moving object detection in Airborne Hyperspectral Remote Sensing Images under complex background, this paper proposes a background removal method based on Mixture Gauss Model. Firstly, in order to eliminate the influence of the dithering of airborne platform on remote sensing image, this paper uses affine transform-based image registration algorithm to achieve the image registration of 3D data. Then, this paper selects specific spectral images, establishes the Mixture Gauss background model, and subtracts the background to achieve the extraction of moving objects. The experimental results show that the method can extract moving objects in Airborne Hyperspectral Remote Sensing Images effectively.

Fast and coarse registration of point cloud in ICP three-dimensional space

The traditional ICP algorithm has a large amount of computation in the process of searching for corresponding points[1].This is the bottleneck of the traditional ICP algorithm[2],and the standard ICP is looking for the corresponding point[3].The point where the Euclidean distance is closest is the corresponding point[4].Unreasonable,there will be a certain number of wrong points[5].In order to improve the accuracy and speed of 3D point cloud data registration,this paper combines the coarse registration of the center of the box and the circular domain search method of the rasterized small cube to further improve and optimize the traditional ICP algorithm[6].

A technical solution for 3D crowdsourced cadastral surveys

The development of 3D property registration systems is indispensable for the spatial determination of property Rights, Restrictions and Responsibilities (RRR), the sustainable operation of property markets and the safeguarding of ownership in the highly urbanized world. Several developments in computer graphics, 3D modelling and rendering techniques have attracted the growing global interest in 3D cadastre. Since 2012, the Land Administration Domain Model (LADM ISO 19152) is adopted as the international standard for 2D and 3D cadastral data modelling. Developed legal procedures and prototype systems for 3D property rights registration provide interesting solutions for data acquisition and visualization but such approaches are time and cost demanding. In parallel, 2D cadastral surveying procedures have progressed significantly in reducing required time and cost, utilizing crowdsourcing methodology and mobile services.This paper presents a technical solution under development, aiming to provide the beginning for a new era in the initial acquisition, registration and representation of 3D crowdsourced cadastral data, appropriate for both the developed and the developing world. The proposed solution consists of two complementary parts, the technical framework and the crowdsourced methodology to be followed. The technical framework consists of two interconnected parts, the server-side and the client-side. The server-side refers to the Database Management System (DBMS) where the collected data are stored. The client-side refers to the data capturing tool, which in this case is the mobile device. For the server-side, a prototype system based on model driven architecture practices and LADM is developed; while for the client-side, an open-source mobile application for the acquisition of 3D crowdsourced cadastral data, 3D modelling and visualization of 3D property units as block models (LoD1) on a mobile’s phone screen in real-time, is developed.This research is focused on investigating technical aspects for capturing and integrating data about legal rights on physical objects. Detailed investigation of legal issues is not within the objectives of this research. Two case-studies for testing the proposed technical solution are made. An assessment of the current stage of development of the technical solution is presented. The main conclusions converge that the proposed technical solution has huge potentials for the fast, economic implementation of 3D cadastral surveys as it can produce an accurate and reliable 3D information model, depending on the accuracy of the available basemaps. The developed application is easy-to-use as users may not have the necessary level of 3D modelling skills in order to contribute to the registration procedure.

UAV-assisted autonomous mobile robot navigation for as-is 3D data collection and registration in cluttered environments

Abstract This paper discusses enabling autonomous mobile robots to operate in unstructured terrain and environments and gather data in a time-efficient manner. In rugged environments such as construction sites and disaster sites, spatiotemporal data is difficult to acquire since these environments are potentially hazardous to humans. Autonomous robot systems represent a reasonable solution to provide topographic and surveillance data that can assist human activity in these environments, whether for exploration, mapping, or search and rescue. However, automatically operating the mobile laser scanning robots at cluttered environments is challenging because as-is geometrical conditions of the site are difficult to comprehend from the ground level due to the blocked line-of-site views. To address the issue, this paper introduces a new framework for operating mobile robots equipped with a laser scanning system in cluttered outdoor environments with the aid of an unmanned aerial vehicle (UAV). To obtain an initial map from the current field, this method first deploys UAV to collect photographic images of a site and builds a point cloud of a3D terrain of the site including obstacle information. A voxel grid is then created from the UAV-generated point cloud, and simulation for laser scan planning is conducted to determine the stationary laser scan positions at which a mobile robot can collect data with less occluded views while capturing crucial geometric information as much as it can. Finally, optimal paths for the mobile robot to navigate among the estimated scan positions are generated. Promising test results were obtained from a real-world outdoor structural material test yard as an example of a cluttered environment. It is expected that the proposed UAV-assisted robotic approach can significantly reduce human intervention and time for data collection and processing and provide technologies to enable cluttered environments to be frequently monitored, updated, and analyzed to support timely decision-making.

A multi-level 3D data registration approach for supporting reliable spatial change classification of single-pier bridges

The reliability of condition assessment of bridges using 3D imagery data, such as 3D laser scanning point clouds, relies on inspectors’ structural engineering knowledge and skills of 3D data processing. A challenge of 3D-data-based structural condition assessment lies in the difficulties of reliably comparing 3D imagery data sets collected at different times for analyzing spatial changes of the structures and finding anomalous deformations. Spatial changes of structures could occur at multiple levels of details and be of different types: (1) rigid body motions (e.g., translations and rotations) at the structure or structural element levels; (2) deformations (e.g., bending of girders) at the levels of structural elements. Unfortunately, existing 3D imagery data-based change analysis methods only produce deviations between two 3D data sets without distinguishing deviations caused by various changes at multiple levels. Significant rigid body motions of structures and structural elements often cause large deviations that “overwhelm” deviation patterns caused by smaller element-level deformations so that engineers could hardly recognize local deformations. Unreliable deformation analysis of structural elements can lead to incorrect condition assessments.This paper presents a new multi-level 3D data registration and spatial change classification approach that automate the analysis of both element-level deformations and interactions between the motions of multiple elements based on deviations calculated between two 3D data sets. This approach uses a multi-level data registration method augmented by formalized knowledge for representing spatial changes using deviation maps between two 3D datasets. This knowledge will guide pattern analysis methods to reveal how various changes of structures collectively lead to structural systems behaviors. More specifically, this 3D data registration and spatial change classification approach eliminates deviations caused by rigid body motions before assessing deformations of structural elements. The authors conducted annual 3D imagery data collection for two single pier bridges in July 2015 and June 2016, and use those 3D data to characterize the performance of the new approach in identifying relative motions between and deformations of structural elements. The results indicate that the new approach can reliably identify relative motions between and deformations of bridge elements, such as angular changes between elements, and torsions of girders. Finally, the authors validated the change analysis results generated by the developed approach against the traditional change analysis results obtained by a knowledgeable structural engineering researcher and change analyses in multiple single-pier bridge research studies.

Scale-Free Registrations in 3D: 7 Degrees of Freedom with Fourier Mellin SOFT Transforms

Fourier Mellin SOFT (FMS) as a novel method for global registration of 3D data is presented. It determines the seven degrees of freedom (7-DoF) transformation, i.e., the 6-DoF rigid motion parameters plus 1-DoF scale, between two scans, i.e., two noisy, only partially overlapping views on objects or scenes. It is based on a sequence of the 3D Fourier transform, the Mellin transform and the SO(3) Fourier transform. This combination represents a non-trivial complete 3D extension of the well known Fourier-Mellin registration for 2D images. It is accordingly based on decoupling rotation and scale from translation. First, rotation—which is the main challenge for the extension to 3D data - is tackled with a SO(3) Fourier Transform (SOFT) based on Spherical Harmonics. In a second step, scale is determined via a 3D Mellin transform. Finally, translation is calculated by Phase-Matching. Experiments are presented with simulated data sets for ground truth comparisons and with real world data including object recognition and localization in Magnetic Resonance Tomography (MRT) data, registration of 2.5D RGBD scans from a Microsoft Kinect with a scale-free 3D model generated by Multi-View Vision, and 3D mapping by registration of a sequence of consecutive scans from a low-cost actuated Laser Range Finder. The results show that the method is fast and that it can robustly handle partial overlap, interfering structures, and noise. It is also shown that the method is a very interesting option for 6-DoF registration, i.e., when scale is known.

3D object recognition using scale-invariant features

As 3D scanning technology develops, it becomes easier to acquire various 3D surface data; thus, there is a growing need for 3D data registration and recognition technology. Many existing studies use local descriptors using local surface patches, and most of them use a fixed support radius, so they cannot cope perfectly when the model and scene have different scales. In this study, we propose a perfectly scale-invariant feature selection algorithm by extending the 2D SIFT algorithm (Lowe in Int J Comput Vis 60(2):91–110, 2004) to a 3D mesh. The feature selection method proposed in this study can obtain highly repeatable feature points and support radii regardless of mesh scale. The selected features can effectively describe the local information by the new shape descriptor proposed in this study. Unlike existing shape descriptors, it is possible to perform scale-invariant 3D object recognition and achieve a high recognition rate when combined with the feature point selection algorithm proposed in this study by using the gradients of the scalar functions defined on the 3D surface. We also reduced the searching space and lowered the false positive rate by suggesting a new RANSAC-based transformation hypotheses generation algorithm. Our 3D object recognition algorithm achieves recognition rates of 100 and 98.5%, respectively, when tested on the U3OR and CFVD datasets, exceeding the results of previous studies.

An Iterative Closest Points Algorithm for Registration of 3D Laser Scanner Point Clouds with Geometric Features.

The Iterative Closest Points (ICP) algorithm is the mainstream algorithm used in the process of accurate registration of 3D point cloud data. The algorithm requires a proper initial value and the approximate registration of two point clouds to prevent the algorithm from falling into local extremes, but in the actual point cloud matching process, it is difficult to ensure compliance with this requirement. In this paper, we proposed the ICP algorithm based on point cloud features (GF-ICP). This method uses the geometrical features of the point cloud to be registered, such as curvature, surface normal and point cloud density, to search for the correspondence relationships between two point clouds and introduces the geometric features into the error function to realize the accurate registration of two point clouds. The experimental results showed that the algorithm can improve the convergence speed and the interval of convergence without setting a proper initial value.

3D-2D face recognition with pose and illumination normalization

In this paper, we propose a 3D-2D framework for face recognition that is more practical than 3D-3D, yet more accurate than 2D-2D. For 3D-2D face recognition, the gallery data comprises of 3D shape and 2D texture data and the probes are arbitrary 2D images. A 3D-2D system (UR2D) is presented that is based on a 3D deformable face model that allows registration of 3D and 2D data, face alignment, and normalization of pose and illumination. During enrollment, subject-specific 3D models are constructed using 3D+2D data. For recognition, 2D images are represented in a normalized image space using the gallery 3D models and landmark-based 3D-2D projection estimation. A method for bidirectional relighting is applied for non-linear, local illumination normalization between probe and gallery textures, and a global orientation-based correlation metric is used for pairwise similarity scoring. The generated, personalized, pose- and light- normalized signatures can be used for one-to-one verification or one-to-many identification. Results for 3D-2D face recognition on the UHDB11 3D-2D database with 2D images under large illumination and pose variations support our hypothesis that, in challenging datasets, 3D-2D outperforms 2D-2D and decreases the performance gap against 3D-3D face recognition. Evaluations on FRGC v2.0 3D-2D data with frontal facial images, demonstrate that the method can generalize to databases with different and diverse illumination conditions.

Innovative Methodology for Multi-View Point Cloud Registration in Robotic 3D Object Scanning and Reconstruction

The paper is concerned with the problem of multi-view three-dimensional (3D) point cloud registration. A novel global registration method is proposed to accurately register two series of scans into an object model underlying 3D imaging digitization by using the proposed oriented bounding box (OBB) regional area-based descriptor. A robot 3D scanning strategy is nowadays employed to generate the complete set of point cloud of physical objects by using 3D robot multi-view scanning and data registration. The automated operation has to successively digitize view-dependent area-scanned point cloud from complex-shaped objects by simultaneous determination of the next best robot pose and multi-view point cloud registration. To achieve this, the OBB regional area-based descriptor is employed to determine an initial transformation matrix and is then refined employing the iterative closest point (ICP) algorithm. The key technical breakthrough can resolve the commonly encountered difficulty in accurately merging two neighboring area-scanned images when no coordinate reference exists. To verify the effectiveness of the strategy, the developed method has been verified through some experimental tests for its registration accuracy. Experimental results have preliminarily demonstrated the feasibility and applicability of the developed method.

A novel approach for a 2D/3D image registration routine for medical tool navigation in minimally invasive vascular interventions

Minimally invasive interventions are frequently aided by 2D projective image guidance. To facilitate the navigation of medical tools within the patient, information from preoperative 3D images can supplement interventional data. This work describes a novel approach to perform a 3D CT data registration to a single interventional native fluoroscopic frame. The goal of this procedure is to recover and visualize a current 2D interventional tool position in its corresponding 3D dataset.A dedicated routine was developed and tested on a phantom. The 3D position of a guidewire inserted into the phantom could successfully be reconstructed for varying 2D image acquisition geometries. The scope of the routine includes projecting the CT data into the plane of the fluoroscopy. A subsequent registration of the real and virtual projections is performed with an accuracy within the range of 1.16±0.17mm for fixed landmarks. The interventional tool is extracted from the fluoroscopy and matched to the corresponding part of the projected and transformed arterial vasculature. A root mean square error of up to 0.56mm for matched point pairs is reached. The desired 3D view is provided by backprojecting the matched guidewire through the CT array.Due to its potential to reduce patient dose and treatment times, the proposed routine has the capability of reducing patient stress at lower overall treatment costs.