Pose Transformation Research Articles

A hand-eye calibration algorithm of binocular stereo vision based on multi-pixel 3D geometric centroid relocalization

The vision-guided robotic machining accuracy highly depends on the hand-eye calibration accuracy between robot and vision equipment. In order to address the problem of less parameter constraints in existing hand-eye calibration methods, in this paper a hand-eye calibration algorithm of binocular stereo vision is proposed based on multi-pixel 3D geometric centroid relocalization. The algorithm mainly includes three steps:1) the checkerboard relocalization images of multiple sets of fixed-point pose transformations are captured by the binocular stereo vision; 2) the robot tool center point (TCP) coordinates in the binocular coordinate system are obtained by an iterative reweighted least squares algorithm based on sub-pixel corner extraction, and 3) the hand-eye transformation matrix between the binocular system and the robot is obtained by the singular value decomposition (SVD). The experimental results show that both the average error and the mean square error of the proposed hand-eye calibration algorithm can reach 0.45mm and 0.21mm, respectively, which are much smaller than the existing algorithms and can meet the accuracy requirements of robotic positioning and machining.

SAR image classification method based on improved capsule network

Aiming at the problem that the convolutional neural network is difficult to recognize the posture transformation of the image, which requires a large number of label samples for training. This paper proposes an improved synthetic aperture radar image classification method based on the combination of dilated convolution and capsule network. We use dilated convolution to expand the receptive field while reducing the amount of calculation, and use the capsule network to improve the classification effect when there are fewer training samples. The experimental results show that the proposed method can still achieve high classification performance after the training sample is small and the test sample undergoes pose transformation, and at the same time, it has a low computational load.

Loop-Box: Multiagent Direct SLAM Triggered by Single Loop Closure for Large-Scale Mapping.

In this article, we present a multiagent framework for real-time large-scale 3-D reconstruction applications. In SLAM, researchers usually build and update a 3-D map after applying nonlinear pose graph optimization techniques. Moreover, many multiagent systems are prevalently using odometry information from additional sensors. These methods generally involve extensive computer vision algorithms and are tightly coupled with various sensors. We develop a generic method for the key challenging scenarios in multiagent 3-D mapping based on different camera systems. The proposed framework performs actively in terms of localizing each agent after the first loop closure between them. It is shown that the proposed system only uses monocular cameras to yield real-time multiagent large-scale localization and 3-D global mapping. Based on the initial matching, our system can calculate the optimal scale difference between multiple 3-D maps and then estimate an accurate relative pose transformation for large-scale global mapping.

LiPMatch: LiDAR Point Cloud Plane Based Loop-Closure

This letter presents a point cloud based loop-closure method to correct long-term drifts in Light Detection and Ranging based Simultaneous Localization and Mapping systems. In the method, we formulate each keyframe as a fully-connected graph with nodes representing planes. To detect loop-closures, the proposed method employs geometric restrictions to define a similarity metric to match current keyframe and those in the map. After similarity assessment, the candidate keyframes which comply with the geometric restrictions are further checked out successively by normal constraints of planes, and validated by an improved Iterative Closest Point method. The latter also provides relative pose transformation estimation between the current keyframe and the matched keyframe in the global reference frame. Experimental results demonstrate that the proposed method is able to fulfill fast and reliable loop-closure. To benefit the community by serving a benchmark for loop-closure, the entire system is made open source on GitHub.

Weighted Calibration Algorithm of Multiple Laser Scanning Sensors Based on Measurement Error Correction

The calibration of pose transformation relationship among multiple laser scanning sensors or between one sensor and a reference coordinate system is of great importance to the accuracy of 3D laser scanning system. In some applications, such as section inspection of Metro tunnels, the sparsity of measuring points and high demand for accuracy make the calibration process very difficult. This paper proposes an automatically calibration algorithm, named GA-ICP-WHT, based on an improved ICP algorithm with point-wise measurement error corrections. The algorithm consists of stages: coarse registration with GA and fine registration with ICP and weighted Hong-Tan algorithm. Mathematical simulation and application experiments show that the proposed algorithm is accurate and robust for sparse point cloud registration.

Lidar-based system for high-precision localization and real-time 3D map construction

Simultaneous localization and mapping (SLAM) is considered as the core of building high-precision three-dimensional environmental maps. For this, we proposed a high-precision laser-based SLAM system. Aiming at the problem that the point cloud data obtained from laser is not representative and the iterative closest point method for calculating pose transformation is time-consuming, we proposed an efficient and stable matching algorithm. It uses fused feature points to align with the occupancy grid submaps with less registration error and is less time-consuming. Then, to address the problem in which the registration result falls into the local optimum early, we proposed a quadratic registration algorithm. This method effectively improves the initial value of the registration process. Finally, a time consistency and global consistency loop detection algorithm are used to reduce the cumulative error. The system we proposed has been tested on the University of Michigan North Campus Long-Term dataset and Cartographer dataset. Experiments show that our system has good accuracy under low-speed motion conditions (speeds ranging from 1 to 2 m / s).

Accuracy Improvement Calibrations for the Double-Position 4-PPPS Aircraft Docking System

A double-positions 4-PPPS parallel mechanism is used for the aircraft fuselage assembly process to improve the docking efficiency and reduce the labor intensity. However, the accuracy is hard to guarantee, for the mechanism is large and redundant and has manufacturing and assembly errors. To improve the accuracy of the 4-PPPS parallel aircraft fuselage docking system, firstly, an averaging iteration method is proposed to calibrate the datum points in the airplane coordinate which are the references of the entire docking system. And secondly, a kinematic calibration method based on the derivative of the spatial pose transformation is proposed to calibrate up to 42 kinematic parameters. By these two methods, the final maximum position error reduced from 2.2 mm to 0.035 mm and the maximum pointing error reduced from 0.08 degree to 0.018 degree. The accuracy measurement and docking experiment prove the efficiency of the proposed methods.

An assembling algorithm for fixture in an assembly process planning system

Three-dimensional assembly process planning is a precondition for achieving full product lifecycle management based on three-dimensional modelling. Information expression and management operations of fixture models are key to three-dimensional assembly process planning systems. To reasonably and efficiently achieve the goal of planning assembly processes with a fixture model, the application of a fixture information model to a three-dimensional assembly process planning system is investigated. First, to manage tooling information, a fixture information model is defined that consists of management information, display information, geometric information, assembly constraints and degrees of freedom. Then, based on the analysis of the degrees of freedom of components and the assembly constraint relationship, a method of solving assembly constraints based on degrees of freedom reasoning is proposed. Finally, by using the method of solving assembly positioning based on geometric constraints, the pose transformation and assembly positioning of parts are achieved. Through the development of a tooling function module, the three-dimensional assembly process planning system is improved. The feasibility of the above method is verified using part of a spacecraft as an example.

3D digital breast cancer models with multimodal fusion algorithms

Breast cancer image fusion consists of registering and visualizing different sets of a patient synchronized torso and radiological images into a 3D model. Breast spatial interpretation and visualization by the treating physician can be augmented with a patient-specific digital breast model that integrates radiological images. But the absence of a ground truth for a good correlation between surface and radiological information has impaired the development of potential clinical applications.A new image acquisition protocol was designed to acquire breast Magnetic Resonance Imaging (MRI) and 3D surface scan data with surface markers on the patient’s breasts and torso. A patient-specific digital breast model integrating the real breast torso and the tumor location was created and validated with a MRI/3D surface scan fusion algorithm in 16 breast cancer patients.This protocol was used to quantify breast shape differences between different modalities, and to measure the target registration error of several variants of the MRI/3D scan fusion algorithm. The fusion of single breasts without the biomechanical model of pose transformation had acceptable registration errors and accurate tumor locations. The performance of the fusion algorithm was not affected by breast volume. Further research and virtual clinical interfaces could lead to fast integration of this fusion technology into clinical practice.

Self-Recognition Grasping Operation with a Vision-Based Redundant Manipulator System

For unstructured environment applications, the ability of self-recognition for grasping operations should be guaranteed for manipulators. For this purpose, a grasping process, including instance segmentation, pose estimation, and pose transformation, is proposed herein to achieve autonomous object detection, location detection, and grasp planning. An inverse solution in position form is derived for pose transformation to guarantee redundant manipulator adaption. The inverse solution requires no default initial configuration and can obtain all feasible solutions for grasping. Additionally, the optimal grasp can be selected by introducing an optimal factor, such as manipulability. Besides, the process is programmed with high computational efficiency, making it a better choice for manipulators to achieve self-recognized grasping operation. Experiments are carried out herein to verify the necessity of instance segmentation, pose estimation, and pose transformation in achieving self-recognized grasping operation. The inverse solution in the position form is also proven to be efficient and adaptable for the pose transformation of redundant manipulators.

DeepIM: Deep Iterative Matching for 6D Pose Estimation

Estimating the 6D pose of objects from images is an important problem in various applications such as robot manipulation and virtual reality. While direct regression of images to object poses has limited accuracy, matching rendered images of an object against the observed image can produce accurate results. In this work, we propose a novel deep neural network for 6D pose matching named DeepIM. Given an initial pose estimation, our network is able to iteratively refine the pose by matching the rendered image against the observed image. The network is trained to predict a relative pose transformation using an untangled representation of 3D location and 3D orientation and an iterative training process. Experiments on two commonly used benchmarks for 6D pose estimation demonstrate that DeepIM achieves large improvements over state-of-the-art methods. We furthermore show that DeepIM is able to match previously unseen objects.

Metric Learning for Patch-Based 3-D Image Registration

Patch-based image registration is a challenging problem in visual geometry, the crucial component of which is the selection of an appropriate similarity measure. The similarity measure participates in the objective calculation of the pose optimization, which determines the optimization convergence performance. In this paper, we propose learning a similarity metric of patches from reference and target images such that the pairwise patches with a small projection error receive high similarity scores. To achieve this objective, we designed and trained the classification, regression, and rank networks separately based on self-collected data sets. The network can directly output the projection error according to the patches, which is sensitive to the deviation of the pose transformation. We also designed evaluation criteria and validated the superior performance of the network’s outputs compared with the performance of traditional methods, such as the sum of absolute difference and the sum of squared differences. Note to Practitioners —Patch-based image registration is a basic field of study in visual geometry. The collected data sets, trained networks, and designed evaluation criteria in this paper can be used to conduct related research. In visual odometry or 3-D reconstruction, the pose optimization between frames is involved, and the network output can be used as an optimization objective. The patch size can be adjusted according to the speed and accuracy requirements.

Calibration of mobile robot odometry and camera based on hand-eye calibration

The relative pose transformation among several sensors in mobile robots has a certain impact on the pose estimation of the robot. The traditional hand-eye calibration works out the matrix transformation relationship between the camera coordinate system and the robot end effector or its base coordinate system. As for the household sweeping robot, which use monocular camera for localization and mapping. A novel method is proposed calibrating the chassis odometry and the top-view camera. First, we get chassis poses from wheel encoder and camera poses from Perspective-n-Point(PnP).Then we use two types of poses to solve the formula we build which is similar to hand-eye calibration. We make some tests to ensure this method is feasible.

Human Action Transfer Based on 3D Model Reconstruction

We present a practical and effective method for human action transfer. Given a sequence of source action and limited target information, we aim to transfer motion from source to target. Although recent works based on GAN or VAE achieved impressive results for action transfer in 2D, there still exists a lot of problems which cannot be avoided, such as distorted and discontinuous human body shape, blurry cloth texture and so on. In this paper, we try to solve these problems in a novel 3D viewpoint. On the one hand, we design a skeleton-to-3D-mesh generator to generate the 3D model, which achieves huge improvement on appearance reconstruction. Furthermore, we add a temporal connection to improve the smoothness of the model. On the other hand, instead of directly utilizing the image in RGB space, we transform the target appearance information into UV space for further pose transformation. Specially, unlike conventional graphics render method directly projects visible pixels to UV space, our transformation is according to pixel’s semantic information. We perform experiments on Human3.6M and HumanEva-I to evaluate the performance of pose generator. Both qualitative and quantitative results show that our method outperforms methods based on generation method in 2D. Additionally, we compare our render method with graphic methods on Human3.6M and People-snapshot. The comparison results show that our render method is more robust and effective.

3D Human Pose Machines with Self-Supervised Learning.

Driven by recent computer vision and robotic applications, recovering 3D human poses has become increasingly important and attracted growing interests. In fact, completing this task is quite challenging due to the diverse appearances, viewpoints, occlusions and inherently geometric ambiguities inside monocular images. Most of the existing methods focus on designing some elaborate priors /constraints to directly regress 3D human poses based on the corresponding 2D human pose-aware features or 2D pose predictions. However, due to the insufficient 3D pose data for training and the domain gap between 2D space and 3D space, these methods have limited scalabilities for all practical scenarios (e.g., outdoor scene). Attempt to address this issue, this paper proposes a simple yet effective self-supervised correction mechanism to learn all intrinsic structures of human poses from abundant images. Specifically, the proposed mechanism involves two dual learning tasks, i.e., the 2D-to-3D pose transformation and 3D-to-2D pose projection, to serve as a bridge between 3D and 2D human poses in a type of "free" self-supervision for accurate 3D human pose estimation. The 2D-to-3D pose implies to sequentially regress intermediate 3D poses by transforming the pose representation from the 2D domain to the 3D domain under the sequence-dependent temporal context, while the 3D-to-2D pose projection contributes to refining the intermediate 3D poses by maintaining geometric consistency between the 2D projections of 3D poses and the estimated 2D poses. Therefore, these two dual learning tasks enable our model to adaptively learn from 3D human pose data and external large-scale 2D human pose data. We further apply our self-supervised correction mechanism to develop a 3D human pose machine, which jointly integrates the 2D spatial relationship, temporal smoothness of predictions and 3D geometric knowledge. Extensive evaluations on the Human3.6M and HumanEva-I benchmarks demonstrate the superior performance and efficiency of our framework over all the compared competing methods.

An Image Mosaic Algorithm based on Coordinate Transformation

In this paper, an image mosaic algorithm is presented to realize the high-precision geometric measurement of large parts through the vision measurement. First, the visual acquisition system moves along the planned trajectory path and grabs the high-resolution, small-viewing field image sequence successively, then a backward mapping algorithm is adopted to resample the above-mentioned images in the normalized focal plane in order to eliminate the image distortion. Finally, with the aid of pose transformation relations, the mapping images are precisely tiled into a larger field of view image. This image mosaic algorithm has high mosaic accuracy and is suitable for a wide range of applications due to its independency on characteristics information of the image to be mosaicked. Experimental results show that the algorithm has good application value.

A Self-Supervised Learning Manipulator Grasping Approach Based on Instance Segmentation

Automatic grasping is playing an important role in robotics, and traditional grasping approaches cannot deal with both occlusions and texture-less objects well. To improve the stability and accuracy of grasping in a novel and occlusion environment, a grasping approach based on instance segmentation and self-supervised learning pose estimation network is proposed to grasp objects with the manipulator in this paper. The approach can be divided into three phases: instance segmentation, pose estimation, and pose transformation. Instance segmentation predicts classification masks for each pixel. Masks which stand for the contour of objects provide a heuristic knowledge for the self-supervised learning pose estimation network. Pose estimation network has two fully connected layers and regards estimation as a self-supervised classification problem. Then, the pose can be transformed from value in pixel coordinates to actual value relative to the base coordinate of the manipulator. As a result, the manipulator can be operated to grasp by the given actual value of pose. With the help of the approach proposed in this paper, we improve the grasping accuracy by 35%, compared to the former grasping approach based on the pose estimation network on the grasping dataset of CMU. Besides, grasping with this approach on hardware also shows a high success rate. Therefore, the proposed approach is a more robust and more accurate way of grasping.

Calibration and using a laser profile scanner for 3D robotic welding

This paper describes first functions of a developed cognitive module for 3D robotic welding using TIG or laser technology. This area is constantly evolving with the needs to solve complex problems of welding automatisation and robotisation, and also thanks to the continuous advances in measurement technology and robotics. Besides, the use of welding robots for serial production with dedicated tightly defined trajectories, systems for automatic welding of a previously undefined path or paths, which the operator cannot manually define because of its complexity, are developed. This paper covers the general description of the cognitive module and its required functions. Then necessary knowledge about a pose representation and transformation is presented. After that, procedures for a calibration of a profile scanner and its using for 3D model construction based on a depth map are described in more detail. The cognitive module prototype is tested in the task of automatic cavity repair.

Study on Assembly Precision Reliability Assessment for Industrial Sensing Devices

This paper investigates the method of assembly precision reliability assessment for industrial sensing devices. The part variation is formulated by small displacement torsor. The assembly error model is developed through the approach of pose transformation matrix, then the error transfer between the parts is analyzed and the assembly error is calculated. The concept of assembly precision reliability is proposed. Monte Carlo simulation is adopted to calculate the reliability, after that the procedure of assembly precision reliability is determined. Case study of a laser sensor is provided therefore the effectiveness of the proposed models is verified.

Reproduction method of the relative position and posture of vehicle ends on the test bench

An inter-vehicle relationship test bench which can simulate the relative motion of the adjacent vehicle ends is introduced. The test bench is driven by 14 actuators. A three-dimensional model including track, train, and test bench is established to solve the control instructions of the 14 actuators. The posture of the train running on the track is converted to the control instructions of the actuators based on the theory of space coordinates position and pose transformation. A virtual prototype model of the test bench is established before bench test to check the control instructions of the actuators and solve the relative position and posture of the adjacent vehicle ends. After determining the safety and reliability of the test, the bench test is carried out to measure the relative position and posture of the adjacent vehicle ends and compare with the results of the simulation. The error of the test and simulation is small, which proves the correctness of the test. The test proves that the inter-vehicle relationship test bench has the ability to simulate the actual running state of two adjacent vehicle ends and can provide design reference for the design of the inter-vehicle connections of new vehicle model.