6-DOF Pose Research Articles

Efficient Inverse Kinematics and Planning of a Hybrid Active and Passive Cable-Driven Segmented Manipulator

A cable-driven segmented manipulator (CDSM) has superior dexterity for operations in confined space due to its light-slender body and redundant degree of freedoms (DOFs). However, its inverse kinematics resolving and configuration planning are very challenging due to the complex structure and strict constraints. In this article, we propose a two-layer geometric iteration (TLGI) method for inverse kinematics resolving and configuration-constrained Cartesian path planning. The computation efficiency is largely improved and singularities are avoided. First, the end-effector attitude is decomposed into a direction vector and a rotation angle. The former and the end-effector position are combined into state variables of the inner layer, and the latter is treated separately as the state variable of the outer layer. Then, the TLGI method enables to rapidly reach the desired 6-DOF pose by two-layer iterations, i.e., the inner and outer loop iteration. Second, during the inner loop iteration, the CDSM is modeled as an equivalent articulated arm whose end-effector position and direction is the same as that of CDSM, but its links length and joint angles depend on the current configuration of CDSM. Then, the efficient forward and backward reaching inverse kinematics (FABRIKs) method is extended to apply on CDSM so that it can fast reach the inner state variables. During the outer loop iteration, three different rotation cases, i.e., the rotating around the end, root, and both end and root, are designed to switch automatically to reach the outer state variable iteratively. Moreover, by parameterizing geometric constraints of the environment, a TLGI-based configuration-pose simultaneous planning method is also put forward to efficiently achieve additional configuration constraints for operations of CDSM in confined space. Finally, the proposed method is verified by both the simulations and experiments.

The impact of domain randomization on cross-device monocular deep 6DoF detection

• We propose an evaluation of synthetic datas influence when training deep learning models to handle unseen scenarios. • We provide an analysis of how the models behave when faced with different environmental challenges and cameras. • We describe use cases for 3D printed objects and how the method facilitates the development of real-world applications. • We introduce the eC3Po dataset with over 110,000 annotated frames (real and synthetic), that will be made publicly available. This work evaluates the use of synthetic data to train deep 6DoF pose estimation models that use a monocular RGB camera as input. We have compared different training strategies combining real and synthetic data (with domain randomization) to investigate how to better handle real-world challenges. We show that it is possible to obtain accurate models using less real data and suggest how to utilize this strategy. In this work, we have captured and made available two datasets: one real and one synthetic, totaling over 110,000 annotated frames. These datasets are organized according to the different cameras used and the challenges present in the sequences, all featuring textureless 3D printed objects. We also show that synthetic data can help models generalize, handling challenges such as fast motion, occlusion, illumination changes, color variation, scale changes, and unexpected geometry. Finally, we evaluated 70 different models to understand how a model trained for one camera sensor performs when used with a different sensor. To this end, we also suggest how to handle this challenge better by using synthetic simulations to supplement training.

HMD-EgoPose: head-mounted display-based egocentric marker-less tool and hand pose estimation for augmented surgical guidance.

The success or failure of modern computer-assisted surgery procedures hinges on the precise six-degree-of-freedom (6DoF) position and orientation (pose) estimation of tracked instruments and tissue. In this paper, we present HMD-EgoPose, a single-shot learning-based approach to hand and object pose estimation and demonstrate state-of-the-art performance on a benchmark dataset for monocular red-green-blue (RGB) 6DoF marker-less hand and surgical instrument pose tracking. Further, we reveal the capacity of our HMD-EgoPose framework for performant 6DoF pose estimation on a commercially available optical see-through head-mounted display (OST-HMD) through a low-latency streaming approach. Our framework utilized an efficient convolutional neural network (CNN) backbone for multi-scale feature extraction and a set of subnetworks to jointly learn the 6DoF pose representation of the rigid surgical drill instrument and the grasping orientation of the hand of a user. To make our approach accessible to a commercially available OST-HMD, the Microsoft HoloLens 2, we created a pipeline for low-latency video and data communication with a high-performance computing workstation capable of optimized network inference. HMD-EgoPose outperformed current state-of-the-art approaches on a benchmark dataset for surgical tool pose estimation, achieving an average tool 3D vertex error of 11.0mm on real data and furthering the progress towards a clinically viable marker-free tracking strategy. Through our low-latency streaming approach, we achieved a round trip latency of 199.1ms for pose estimation and augmented visualization of the tracked model when integrated with the OST-HMD. Our single-shot learned approach, which optimized 6DoF pose based on the joint interaction between the hand of a user and a rigid surgical drill, was robust to occlusion and complex surfaces and improved on current state-of-the-art approaches to marker-less tool and hand pose estimation. Further, we presented the feasibility of our approach for 6DoF object tracking on a commercially available OST-HMD.

A novel robotic 6DOF pose measurement strategy for large-size casts based on stereo vision

PurposeThis paper aims to study six degrees-of-freedom (6DOF) pose measurement of reflective metal casts by machine vision, analyze the problems existing in the positioning of metal casts by stereo vision sensor in unstructured environment and put forward the visual positioning and grasping strategy that can be used in industrial robot cell.Design/methodology/approachA multikeypoints detection network Binocular Attention Hourglass Net is constructed, which can complete the two-dimensional positioning of the left and right cameras of the stereo vision system at the same time and provide reconstruction information for three-dimensional pose measurement. Generate adversarial networks is introduced to enhance the image of local feature area of object surface, and the three-dimensional pose measurement of object is completed by combining RANSAC ellipse fitting algorithm and triangulation method.FindingsThe proposed method realizes the high-precision 6DOF positioning and grasping of reflective metal casts by industrial robots; it has been applied in many fields and solves the problem of difficult visual measurement of reflective casts. The experimental results show that the system exhibits superior recognition performance, which meets the requirements of the grasping task.Research limitations/implicationsBecause of the chosen research approach, the research results may lack generalizability. The proposed method is more suitable for objects with plane positioning features.Originality/valueThis paper realizes the 6DOF pose measurement of reflective casts by vision system, and solves the problem of positioning and grasping such objects by industrial robot.

Spatial feature mapping for 6DoF object pose estimation

This work aims to estimate 6Dof (6D) object pose in background clutter. Considering the strong occlusion and background noise, we propose to utilize the spatial structure for better tackling this challenging task. Observing that the 3D mesh can be naturally abstracted by a graph, we build the graph using 3D points as vertices and mesh connections as edges. We construct the corresponding mapping from 2D image features to 3D points for filling the graph and fusion of the 2D and 3D features. Afterward, a Graph Convolutional Network (GCN) is applied to help the feature exchange among objects’ points in 3D space. To address the problem of rotation symmetry ambiguity for objects, a spherical convolution is utilized and the spherical features are combined with the convolutional features that are mapped to the graph. Predefined 3D keypoints are voted and the 6DoF pose is obtained via the fitting optimization. Two scenarios of inference, one with the depth information and the other without it are discussed. Tested on the datasets of YCB-Video and LINEMOD, the experiments demonstrate the effectiveness of our proposed method.

Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational Settings

In this paper, we address the challenge of estimating the 6DoF pose of objects in 2D equirectangular images. This solution allows the transition to the objects’ 3D model from their current pose. In particular, it finds application in the educational use of 360° videos, where it enhances the learning experience of students by making it more engaging and immersive due to the possible interaction with 3D virtual models. We developed a general approach usable for any object and shape. The only requirement is to have an accurate CAD model, even without textures of the item, whose pose must be estimated. The developed pipeline has two main steps: vehicle segmentation from the image background and estimation of the vehicle pose. To accomplish the first task, we used deep learning methods, while for the second, we developed a 360° camera simulator in Unity to generate synthetic equirectangular images used for comparison. We conducted our tests using a miniature truck model whose CAD was at our disposal. The developed algorithm was tested using a metrological analysis applied to real data. The results showed a mean difference of 1.5° with a standard deviation of 1° from the ground truth data for rotations, and 1.4 cm with a standard deviation of 1.5 cm for translations over a research range of ±20° and ±20 cm, respectively.

Vision-based trajectory monitoring for assembly alignment of precast concrete bridge components

The prefabricated structures are popularly used because of their high efficiency and it is important to ensure their installation quality. The current installation is guided by visual observation and total station, which is inaccurate and time-consuming. This study presents a stereo-vision-based pose measurement technique for guiding the installation to address the problems. The contributions are as follows: 1) A framework of a vision-based 6-DOF pose monitoring system is designed to track the precast member continuously; 2) A two-level adjustment strategy is suggested by transferring the relative pose measurement between the precast member and existing structure to the process to determine the relative relations of two coordinate systems; 3) A robust deep-learning-based ellipse detection method is proposed to identify and classify the artificial targets accurately. The proposed technique is evaluated in both indoor experiments and the construction of an expressway viaduct to demonstrate its accuracy and effectiveness.

POSE ESTIMATION THROUGH MASK-R CNN AND VSLAM IN LARGE-SCALE OUTDOORS AUGMENTED REALITY

Abstract. Deep Learning (DL) ingrained into Mobile Augmented Reality (MAR) enables a new information-delivery paradigm. In the context of 6 DoF pose estimation, powerful DL networks could provide a direct solution for AR systems. However, their concurrent operation requires a significant number of computations per frame and yields to both misclassifications and localization errors. In this paper, a hybrid and lightweight solution on 3D tracking of arbitrary geometry for outdoor MAR scenarios is presented. The camera pose information obtained by ARCore SDK and vSLAM algorithm is combined with the semantic and geometric output of a CNN-object detector to validate and improve tracking performance in large-scale and uncontrolled outdoor environments. The methodology involves three main steps: i) training of the Mask-R CNN model to extract the class, bounding box and mask predictions, ii) real-time detection, segmentation and localization of the region of interest (ROI) in camera frames, and iii) computation of 2D-3D correspondences to enhance pose estimation of a 3D overlay. The dataset holds 30 images of the rock of St. Modestos – Modi in Meteora, Greece in which the ROI is an area with characteristic geological features. The comparative evaluation between the prototype system and the original one, as well as with R-CNN and FAST-R CNN detectors demonstrates higher precision accuracy and stable visualization at half a kilometre distance, while tracking time has decreased at 42% during far-field AR session.

HCCG: Efficient high compatibility correspondence grouping for 3D object recognition and 6D pose estimation in cluttered scenes

Recognizing and localizing queried objects in point clouds is a critical technique for robotic manipulation and bin picking tasks. Even though it has been steadily studied, it is still a challenging task for scenes with heavy occlusion and clutter. To copy with this intractable problem, this paper presents a fast and robust 3D object recognition framework, especially an efficient high compatibility correspondence grouping (HCCG) technique achieved by the correspondence ranking, clustering and expanding operations. Utilizing the HCCG technique, we first cluster the compatible correspondences into several high compatibility correspondence groups. Then, for each group, a 6DoF pose hypothesis is generated by using a point pair feature constraints (PPFCs)-based outlier removal module and a local reference frame (LRF)-based pose estimation algorithm. Finally, a robust pose verification operator is carried out to reject the false positive pose hypotheses and pick up the correct target object pose. Our approach is not only suitable for the multi-object recognition task, but also inherently yields the capability of detecting multiple instances thanks to the efficient HCCG technique. Extensive experiments on four challenging datasets show that our approach yields efficient and timely solutions and its advantages are further verified by comparing with the latest methods.

Pose Measurement at Small Scale by Spectral Analysis of Periodic Patterns

The retrieval of an observed object’s pose is an essential computer vision problem. The challenge arises in many different fields, among them robotics control, contactless metrology, or augmented reality. When the observed object shrinks from the macroscopic scale to the microscopic, pose estimation is further complicated by the weaker perspective of imaging macroscale lenses down to the quasi-orthographic projection inherent to microscope objectives. This paper tackles this issue of microscale pose estimation in two complementary steps that rely on the use of planar periodic targets. We first consider the orthographic projection case as a means of presenting the theory of the method and showing how the pose of periodic patterns can be directly retrieved from the Fourier frequency spectrum of a given image. We then address the perspective case with long focal lengths, in which the full six-degrees of freedom (6-DOF) pose can be retrieved without ambiguities by following the same theoretical background. In addition to theoretically justifying pose retrieval via Fourier analysis of acquired images, this paper demonstrates the method’s actual performance. Both simulations and experimentation are conducted to validate the method and confirm an experimental resolution lower than \(1/1000{\mathrm{th}}\) of a pixel for translations. For orientation measurement, resolutions below 1 \(\upmu \)rad. for in-plane orientation, and below 100 \(\upmu \)rad. for off-axis orientations can be achieved.

EgoNN: Egocentric Neural Network for Point Cloud Based 6DoF Relocalization at the City Scale

The paper presents a deep neural network-based method for global and local descriptors extraction from a point cloud acquired by a rotating 3D LiDAR. The descriptors can be used for two-stage 6DoF relocalization. First, a course position is retrieved by finding candidates with the closest global descriptor in the database of geo-tagged point clouds. Then, the 6DoF pose between a query point cloud and a database point cloud is estimated by matching local descriptors and using a robust estimator such as RANSAC. Our method has a simple, fully convolutional architecture based on a sparse voxelized representation. It can efficiently extract a global descriptor and a set of keypoints with local descriptors from large point clouds with tens of thousand points. Our code and pretrained models are publicly available on the project website.

A Practical Robotic Grasping Method by Using 6-D Pose Estimation With Protective Correction

The pose estimation is the critical technology in industrial robot. Nowadays, many machine vision-based approaches have applied the technology and achieved excellent results. However, the rapid detection of the pose estimation in complex multiscene environments is still a challenge, due to the interference of multiangle light and multibackground. To address these issues, this article proposes a practical robotic grasping method by using the 6-D pose estimation with protective correction. In this method, the synthetic dataset by self-production is used to train the improved deep object pose estimation network and then use the standard perspective-n-point algorithm to estimate the 6-DoF pose for each object instance. Meanwhile, in order to prevent grasp collisions cause by misrecognition, we propose the corrected grasping pose algorithm for protective correction by measured translation and predicted translation. Finally, the proposed grasping method has an average grasping success rate of 83.3&#x0025; for the three objects under normal light, and the network for single-image detection speed has been to 1.490 frames/s. The code is available at <uri>https://github.com/aimiplus/Practical_Robotic_Grasping_Method</uri>.

Vision-Only Robot Navigation in a Neural Radiance World

Neural Radiance Fields (NeRFs) have recently emerged as a powerful paradigm for the representation of natural, complex 3D scenes. Neural Radiance Fields (NeRFs) represent continuous volumetric density and RGB values in a neural network, and generate photo-realistic images from unseen camera viewpoints through ray tracing. We propose an algorithm for navigating a robot through a 3D environment represented as a NeRF using only an onboard RGB camera for localization. We assume the NeRF for the scene has been pre-trained offline, and the robot&#x2019;s objective is to navigate through unoccupied space in the NeRF to reach a goal pose. We introduce a trajectory optimization algorithm that avoids collisions with high-density regions in the NeRF based on a discrete time version of differential flatness that is amenable to constraining the robot&#x2019;s full pose and control inputs. We also introduce an optimization based filtering method to estimate 6DoF pose and velocities for the robot in the NeRF given only an onboard RGB camera. We combine the trajectory planner with the pose filter in an online replanning loop to give a vision-based robot navigation pipeline. We present simulation results with a quadrotor robot navigating through a jungle gym environment, the inside of a church, and Stonehenge using only an RGB camera. We also demonstrate an omnidirectional ground robot navigating through the church, requiring it to reorient to fit through a narrow gap.

Semantic keypoint-based pose estimation from single RGB frames

This paper presents an approach to estimating the continuous 6-DoF pose of an object from a single RGB image. The approach combines semantic keypoints predicted by a convolutional network (convnet) with a deformable shape model. Unlike prior investigators, we are agnostic to whether the object is textured or textureless, as the convnet learns the optimal representation from the available training-image data. Furthermore, the approach can be applied to instanceand class-based pose recovery. Additionally, we accompany our main pipeline with a technique for semi-automatic data generation from unlabeled videos. This procedure allows us to train the learnable components of our method with minimal manual intervention in the labeling process. Empirically, we show that our approach can accurately recover the 6-DoF object pose for both instance- and class-based scenarios even against a cluttered background. We apply our approach both to several, existing, large-scale datasets - including PASCAL3D+, LineMOD-Occluded, YCB-Video, and TUD-Light - and, using our labeling pipeline, to a new dataset with novel object classes that we introduce here. Extensive empirical evaluations show that our approach is able to provide pose estimation results comparable to the state of the art.

Pose Estimation of a Small Connector Attached to the Tip of a Cable Sticking Out of a Circuit Board

In this study, we present the pose estimation of a small connector attached to the tip of a cable sticking out from a circuit board. Since such connectors are generally small and float in various configurations on their workpieces, it is difficult to achieve automation of grasping and inserting the connector into the corresponding socket. We focus on the task of grasping a connector and propose methods for detecting the location of the connector and estimating its 6DoF pose. In this regard, we use a high-precision three-dimensional digitizer to capture the object point cloud and combine several methods, such as deep learning and registration, to perform data processing. We conducted grasp experiments on several connectors using an actual industrial robot and confirmed the effectiveness of our approach in terms of pose estimation.

A Wide-Area, Low-Latency, and Power-Efficient 6-DoF Pose Tracking System for Rigid Objects

Position sensitive detectors (PSDs) offer possibility to track single active marker’s two (or three) degrees of freedom (DoF) position with a high accuracy, while having a fast response time with high update frequency and low latency, all using a very simple signal processing circuit. However they are not particularly suitable for 6-DoF object pose tracking system due to lack of orientation measurement, limited tracking range, and sensitivity to environmental variation. We propose a novel 6-DoF pose tracking system for a rigid object tracking requiring a single active marker. The proposed system uses a stereo-based PSD pair and multiple Inertial Measurement Units (IMUs). This is done based on a practical approach to identify and control the power of Infrared-Light Emitting Diode (IR-LED) active markers, with an aim to increase the tracking work space and reduce the power consumption. Our proposed tracking system is validated with three different work space sizes and for static and dynamic positional accuracy using robotic arm manipulator with three different dynamic motion patterns. Results demonstrate that the static position accuracy root-mean-square (RMS) error is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$0.6~ {{mm}}$ </tex-math></inline-formula> ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$0.22~ {{mm}}$ </tex-math></inline-formula> precision RMS error) at the mid range of the workspace and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.3 ~{{mm}}$ </tex-math></inline-formula> ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$0.44~ {{mm}}$ </tex-math></inline-formula> precision RMS error) at the largest workspace. The dynamic position RMS error is 0.7- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$0.9 ~{{mm}}$ </tex-math></inline-formula> . The orientation RMS error is between 0.04° and 0.9° at varied dynamic motion. Overall, our proposed tracking system is capable of tracking a rigid object pose with sub-millimeter accuracy at the mid range of the work space and sub-degree accuracy for all work space under a lab setting.

A dynamic keypoint selection network for 6DoF pose estimation

6 DoF pose estimation problem aims to estimate the rotation and translation parameters between two coordinates, such as object world coordinate and camera world coordinate. Although some advances are made with the help of deep learning, how to full use scene information is still a problem. Prior works tackle the problem by pixel-wise feature fusion but need to randomly select numerous points from images, which can not satisfy the demands of fast inference simultaneously and accurate pose estimation. In this work, we present a novel deep neural network based on dynamic keypoint selection designed for 6DoF pose estimation from a single RGBD image. Our network includes three parts, instance semantic segmentation, edge points detection and 6DoF pose estimation. Given an RGBD image, our network is trained to predict pixel category and the translation to edge points and center points. Then, a least-square fitting manner is applied to estimate the 6DoF pose parameters. Specifically, we propose a dynamic keypoint selection algorithm to choose keypoints from the foreground feature map. It allows us to leverage geometric and appearance information. During 6DoF pose estimation, we utilize the instance semantic segmentation result to filter out background points and only use foreground points to finish edge points detection and 6DoF pose estimation. Experiments on two commonly used 6DoF estimation benchmark datasets, YCB-Video and LineMoD, demonstrate that our method outperforms the state-of-the-art methods and achieves significant improvements over other same category methods time efficiency.

A high precision stereo-vision robotic system for large-sized object measurement and grasping in non-structured environment

Handling and loading of large-sized objects represent a challenging task in industrial environments, especially when the object is a metal object with reflective surface features. Active stereo vision technology is not good for measuring the posture of reflective metal object, a high-precision pose measurement system based on passive stereo vision is proposed to automatic measurement, handling and loading of large objects by industrial robots in industrial environments. The system adopts advanced coarse and fine stereo vision positioning strategy, and realises the high-precision positioning of the measured target for the premise of ensuring stability. For coarse positioning, an improved multi-models template matching method based on machine learning is proposed to robust recognition of the multiple objects in complex background. A RANSAC-based method for the ellipse fitting and multi-points plane fitting is proposed for the 6-DOF pose of object accurately obtained in fine positioning step. Compared with the classical CAD-views method, experiments show that the method proposed in this paper has better performance in positioning accuracy and recognition robustness.

로봇 물체 조작 작업을 위한 동적 3차원 장면 그래프 생성

In this study, we proposed a novel dataset and a deep learning model that can generate three-dimensional (3D) dynamic scene graphs for robotic manipulation tasks. First, we defined a new 3D scene graph to effectively represent the dynamics of a robotic manipulation task environment. Subsequently, we collected a series of input sensory data by conducting multiple manipulation tasks in a simulated environment. Based on the collected sensory data and the corresponding 3D scene graphs, we constructed a dataset, namely, D3DSG, for training and validating a scene graph generation model. In addition, we proposed a ST-GCN based context reasoning module that can utilize both rich spatial and temporal contexts, after which an effective 3D scene graph generation model, namely, SG4RMT, which consisted of a 6DoF pose estimation module and a spatio-temporal context reasoning module, was presented. The superiority and high performance of the proposed SG4RMT model were demonstrated by performing multiple experiments using the D3DSG dataset.

An intelligent in-situ visual inspection system for preventive conservation of cultural heritages

Preventive conservation is an advanced protection scheme that is widely recognized and constantly pursued by the worldwide conservation community of cultural heritages. Following the minimum interruption principle, preventive protection can effectively inhibit the deterioration of cultural relics by finely inspecting the symptoms, inferring the causes of diseases, and accurately controlling relevant pathogenesis factors. Reliable inspection of symptoms is the starting point and basis of preventive conservation. At present, many technologies and data have been accumulated for environmental factors. Despite the plenty techniques and accumulated data for the perception of environmental factors, the feasible, accurate, and reliable in-situ inspection of the status of cultural heritages still remains an open problem due to slow changing speed, subtle change scale, and versatile occurrence environments, which seriously hinders the development and promotion of preventive protection. In this paper, we present the first intelligent in-situ visual inspection system for the preventive conservation of cultural heritages. By physically reproducing the cameras 6-DoF pose, effective image-guided detection of fine-grained changes for cultural heritages is realized. Till now, this system has been used in more than 10 cultural heritage sites in China, such as the Dunhuang Mogao Grottoes, the Imperial Palace, and the Summer Palace.