3D Object Registration Research Articles

Deep Learning for 3D Reconstruction, Augmentation, and Registration: A Review Paper.

The research groups in computer vision, graphics, and machine learning have dedicated a substantial amount of attention to the areas of 3D object reconstruction, augmentation, and registration. Deep learning is the predominant method used in artificial intelligence for addressing computer vision challenges. However, deep learning on three-dimensional data presents distinct obstacles and is now in its nascent phase. There have been significant advancements in deep learning specifically for three-dimensional data, offering a range of ways to address these issues. This study offers a comprehensive examination of the latest advancements in deep learning methodologies. We examine many benchmark models for the tasks of 3D object registration, augmentation, and reconstruction. We thoroughly analyse their architectures, advantages, and constraints. In summary, this report provides a comprehensive overview of recent advancements in three-dimensional deep learning and highlights unresolved research areas that will need to be addressed in the future.

Match Normalization: Learning-Based Point Cloud Registration for 6D Object Pose Estimation in the Real World.

In this work, we tackle the task of estimating the 6D pose of an object from point cloud data. While recent learning-based approaches have shown remarkable success on synthetic datasets, we have observed them to fail in the presence of real-world data. We investigate the root causes of these failures and identify two main challenges: The sensitivity of the widely-used SVD-based loss function to the range of rotation between the two point clouds, and the difference in feature distributions between the source and target point clouds. We address the first challenge by introducing a directly supervised loss function that does not utilize the SVD operation. To tackle the second, we introduce a new normalization strategy, Match Normalization. Our two contributions are general and can be applied to many existing learning-based 3D object registration frameworks, which we illustrate by implementing them in two of them, DCP and IDAM. Our experiments on the real-scene TUD-L Hodan et al. 2018, LINEMOD Hinterstoisser et al. 2012 and Occluded-LINEMOD Brachmann et al. 2014 datasets evidence the benefits of our strategies. They allow for the first-time learning-based 3D object registration methods to achieve meaningful results on real-world data. We therefore expect them to be key to the future developments of point cloud registration methods.

Self-training based augmented reality for robust 3D object registration and task assistance

Typical augmented reality (AR) technologies superimpose virtual information onto the real environment for task guidance and assistance by tracking fiducial markers. However, these have narrow recognition ranges due to the markers, and it is difficult to track markers when occlusion occurs. Therefore, it is challenging to seamlessly register real objects in AR scenes captured by the camera without requiring fiducial markers. Deep learning–based approaches to 3D object registration and task assistance have recently indicated promising directions for superimposing 3D virtual objects onto corresponding real objects or their surroundings seamlessly. However, since supervised learning-based object registration requires many pose annotations on images for the locations and orientations of real objects, this process is a time-consuming and labor-intensive task. Thus, this paper proposes a new self-training based augmented reality method for seamless real object registration and task assistance without time-consuming pose annotations. As the industrial components or objects have their 3D geometric models, the proposed approach utilizes them as their synthetic data. The automatically generated synthetic dataset is used for training the deep learning model, which outputs a colored coordinate map of objects to be detected. 2D–3D key point matching between the 2D colored coordinate map and 3D colored coordinate model is then performed for 3D object registration in an AR environment. In particular, self-training is performed for fine-tuning the deep learning model by generating pseudo-labels with the highest probability for mask intersection over union (IoU) in the unlabeled real object dataset. We compared the proposed approach with previous studies to confirm its effectiveness and advantages. Also, we performed ablation studies by training a custom dataset based on various industrial 3D virtual models. Finally, we present AR-based task guidance and assistance, such as industrial robot manipulation.

Dual spin-image: A bi-directional spin-image variant using multi-scale radii for 3D local shape description

Obtaining good feature descriptors for 3D local shape under different conditions, such as noise, clutter, occlusion, and limited overlap, is still a challenging task in computer vision. In this paper, we present a variant of spin-image called dual spin-image (Dual-SI) for 3D local shape description that encodes bi-directional information between an oriented point and its surrounding neighbors within a given radius. By accumulating the parameters that encode the 3D point information into a 2D space bidirectionally, we can generate two different signatures. These signatures are then concatenated to form the Dual-SI feature descriptor. Finally, we propose a multi-scale radius approach to address the problem of occlusion and make use of a weighted kernel to address the noise problem. We tested our Dual-SI feature descriptor on popular datasets addressing 3D object registration, 3D object recognition, and shape retrieval scenarios. We also conduct experiments for 3D point cloud registration to further evaluate the effectiveness of our method. Consensus experimental results show that our Dual-SI achieves outstanding performance on datasets with various nuisances and application contexts.

Device-based manipulation technique with separated control structures for 3D object translation and rotation in handheld mobile AR

Abstract 3D object manipulation is one of the most critical tasks for handheld mobile Augmented Reality (AR), which can contribute towards its practical potential, especially for real-world assembly support. In this context, the study of techniques which are used to manipulate 3D objects is an important research area. This study has developed an improved device-based technique within handheld mobile AR interfaces, to solve the large-range 3D object rotation problem, as well as issues related to position and orientation deviations in manipulating 3D objects. We firstly enhanced the existing device-based 3D object rotation technique (named as HoldAR) with an innovative control structure (named as TiltAR) that utilizes the handheld mobile device tilting and skewing amplitudes to determine the rotation axes and directions of the 3D object. Whenever the device is tilted or skewed in a way that exceeds the threshold values of the amplitudes, the 3D object rotation will start continuously with a pre-defined angular speed per second, to prevent over-rotation of the handheld mobile device. This over-rotation is a common occurrence when using the existing technique to perform large-range 3D object rotations. The problem of over-rotation of the handheld mobile devices needs to be solved since it causes a 3D object registration error and a 3D object display issue, where the 3D object does not appear consistent within the user's range of view. Secondly, restructuring the existing device-based 3D object manipulation technique (named as DI) was done by separating the degrees of freedom (DOF) of the 3D object translation and rotation, to prevent deviations of the 3D object position and orientation, caused by the DOF integration that utilizes the same control structure, which is HoldAR, for both tasks. Next, an improved device-based manipulation technique (named as DS), with better performance on task completion time for 3D object manipulation within handheld mobile AR interfaces, was developed. A pilot test was carried out before other main tests to determine several pre-defined values designed in the control structure of TiltAR. A series of 3D manipulation tasks were designed and developed to benchmark DS (the proposed manipulation technique) with DI (the existing technique) on task completion time (s). Sixteen participants aged 19–24 years old were selected. Each participant had to complete twelve trials, which came to a total 192 trials per experiment for all the participants. Repeated measure analysis was used to analyze the data. The results obtained have statistically proven that DS markedly outpaced DI with significant shorter task completion times in all tasks consisting of different difficulty levels and rotation amounts. Based on the findings, an improved device-based 3D object manipulation technique has been successfully developed to address the insufficient functionalities of the current technique.

3D Surface Matching by a Voxel-Based Buffer-Weighted Binary Descriptor

3D surface matching by local feature descriptors is a fundamental task in 3D object registration, recognition, and retrieval. In view of 2D projected descriptors’ information compression and 3D directly voxelized descriptors’ sensitivity to density variation and boundary, this paper first proposes a voxel-based buffer-weighted binary descriptor, named VBBD. After local surfaces of detected keypoints are voxelized, each voxel’s buffer region is established, and the buffer-weighted Gaussian kernel density is calculated. If the current voxel’s buffer-weighted density is larger than its local surface’s average density, the voxel is labeled 1, otherwise, it is 0. The proposed descriptor has several merits: 1) direct acquisition of 3D information without projection, the neighbor information is less compressed. 2) voxels are labeled and binarized according to the buffer-weighted density, which improves the robustness to boundary effect, noise and density variation. Based on the VBBD, a global optimal surface matching method based on a Kuhn_Munkres (KM) algorithm is adopted. The strategy has several advantages: 1) by calculating the descriptors of large-scaled surfaces which are down sampled, the number of the point cloud is reduced, and meanwhile, large-scaled information is obtained; and 2) KM algorithm is adopted to find matching pairs to achieve final maximum weight sum of all matching pairs, which can efficiently avoid local optimum. The experimental results show that, compared with other state-of-the-art descriptors, the VBBD has better descriptiveness and robustness, and the surface matching strategy by the VBBD can achieve both high recall and precision.

Adaptive 3D object registration based on point cloud distribution for mobile robot

An efficient three-dimensional (3D) object registration method considering the point cloud distribution for a mobile robot platform is suggested. This technique adjusts overlapping areas between two consecutive scan data according to the property of the observed object's surface. The proposed technique can reduce computational load and increase the accuracy of 3D object registration with less total number of scan data. The improved performance is verified by comparing the estimated pose with the ground truth pose for observation robots.

In brief

PAGE 778 Researchers from the USA have demonstrated a novel way of fabricating UV photodetectors – by printing them using an ink made from salmon DNA. The cheap-to-produce biopolymer product shows good conductivity and adheres well to a large variety of surfaces. PAGE 738 An implantable CMOS device that monitors blood glucose levels continually has been developed by researchers from Japan. The device, an improvement on their previous work, measures the intensity of light produced by glucose-responsive fluorescent hydrogel. The DNA based ink adheres well to a variety of materials PAGE 752 Researchers from Korea have developed a new 3D object registration technique for mobile robot platforms. The technique uses point cloud distribution to allow robots to register objects with fewer scans, while reducing the computational load and increasing the accuracy. The device is less invasive and can be injected with a commercially available syringe PAGE 769 A miniaturised dual-band branch-line coupler has been developed by researchers from Iceland. The coupler achieves a size reduction of 75% compared to the conventional structure, with minimal performance impact. This reduction in size could lead to miniaturised components for radar and polarisation atennas. The researchers tested the technique experimentally using three observation robots PAGE 733 In Japan, researchers have investigated and verified the feasibility of a digitally tunable metamaterials-based antenna. Using a leaky-wave antenna with periodic J-shaped metamaterial patches, the researchers demonstrated that they could pseudo continuously control the arrays dispersion pattern. This could lead to use in smart cities with the Internet of Things. The topography of the coupler allows for the drastic size reduction The new array opens and closes gaps between the J-shapes to alter the radiation pattern

A computer vision-based assistant system for the assembly of narrow cabin products

Due to the narrowness of space and the complexity of structure, the assembly of cabin parts has become one of the major bottlenecks in the whole manufacturing process. This paper presents a computer vision-based assistant system that integrates assembly planning, assembly training and guidance, assembly status inspection, and assembly quality evaluation to improve the assembly efficiency and quality of cabin products. By using a novel real-time 3D object registration approach, the mixed reality technology is applied to assembly training and guidance of cabin parts. A machine vision-based custom-made inspection system is designed for assembly surplus detection and gap measurement in half-enclosed cabin space. A cabin assembly quality model is proposed on the basis of Markov chain to quantitatively evaluate the cabin assembly quality. A case study is given to demonstrate the practical application of the proposed system.

Evaluation of a 3D object registration method for analysis of humeral kinematics

In 3D image-based studies of joint kinematics, 3D registration methods should be automatic, insensitive to segmentation inconsistencies and use coordinate systems that have clinically relevant orientations and locations because this is important for analyzing rotation angles and translation directions. We developed and evaluated a registration method, which is based on the cylindrical geometry of the humerus shaft and an analysis of the inertia moments of the humerus head, in order to consistently and automatically orient the humerus coordinate system according to its anatomy. Registration techniques must be thoroughly evaluated. In this study we used a well-detectable marker as reference, from which coordinate system determination errors of a 3D object could be measured. This allowed us to quantify by means of unique error analysis the translational and rotational errors in terms of how much and about/along which humeral axis errors occurred. The evaluation experiments were performed using virtual rotations of 3D humeral binary image, a humerus model and a 3D image of a volunteer's shoulder. They indicated that the humeral coordinate system determination errors primarily originated from segmentation inconsistencies, which influenced mostly the humeral transverse axes orientation. The error analysis revealed that the developed registration method reduced the effect of manual segmentation inconsistencies on the orientation of the humeral transverse axes up to 37%, in comparison to the commonly used inertia registration.

Evaluation of a 3D object registration method for analysis of humeral kinematics

Evaluation of a 3D object registration method for analysis of humeral kinematics

A robust iterative hypothesis testing design of the repeated genetic algorithm

The genetic algorithm is a simple and interesting optimization method for a wide variety of computer vision problems. However, its performance is often brittle and degrades drastically with increasing input problem complexity. While this problem is difficult to overcome due to the stochastic nature of the algorithm, this paper shows that a robust statistical design using sequential sampling, repeated independent trials and hypothesis testing can be used to greatly alleviate the degradation. The working principle is as follows: The probability of success P of a stochastic algorithm A (in this case A is the genetic algorithm) can be estimated by running N copies of A simultaneously or running A repeatedly N times. Such a scheme is generally referred to as the parallel or repeated (genetic) algorithm. By hypothesis testing, P can be tested with a required figure of merit (i.e. the level of significance). This is used in turn to adjust N in an iterative scheme to maintain a constant P repeated, achieving a robust feedback loop. Experimental results on both synthetic and real images are reported on the application of this novel algorithm to an affine object detection problem and a free form 3D object registration problem.

Generating normative data of the cranium using spherical harmonics

The paper describes a method for estimation of the orientation of 3D objects without point correspondence information. It is based on decomposition of the object onto a basis of spherical harmonics. Tensors are obtained, and their normalization provides the orientation of the object. Theoretical and experimental results show that the approach is more accurate than the classical method based on the diagonalization of the inertia matrix. Fast registration of 3D objects is a problem of practical interest in domains such as robotics and medical imaging, where it helps to compare multimodal data.

Determination of the Orientation of 3D Objects Using Spherical Harmonics

The paper describes a method for estimation of the orientation of 3D objects without point correspondence information. It is based on decomposition of the object onto a basis of spherical harmonics. Tensors are obtained, and their normalization provides the orientation of the object. Theoretical and experimental results show that the approach is more accurate than the classical method based on the diagonalization of the inertia matrix. Fast registration of 3D objects is a problem of practical interest in domains such as robotics and medical imaging, where it helps to compare multimodal data.

Registration of 3D objects and surfaces

An approach to the problem of comparative analysis of objects and their surfaces that arises in medical imaging is presented and illustrated with an application to postsurgical bone-graft separation. A common requirement in medical imaging applications is the registration of multiple representations of the object prior to mensuration. The modeling of the registration process and the registration system are described. The measurement of change in volume of bone grafts implanted inside the human body in a corrective surgical procedure is discussed, and the results are evaluated. >