6-DOF Pose Research Articles

Automated Data Annotation for 6-DoF AI-Based Navigation Algorithm Development.

Accurately estimating the six degree of freedom (6-DoF) pose of objects in images is essential for a variety of applications such as robotics, autonomous driving, and autonomous, AI, and vision-based navigation for unmanned aircraft systems (UAS). Developing such algorithms requires large datasets; however, generating those is tedious as it requires annotating the 6-DoF relative pose of each object of interest present in the image w.r.t. to the camera. Therefore, this work presents a novel approach that automates the data acquisition and annotation process and thus minimizes the annotation effort to the duration of the recording. To maximize the quality of the resulting annotations, we employ an optimization-based approach for determining the extrinsic calibration parameters of the camera. Our approach can handle multiple objects in the scene, automatically providing ground-truth labeling for each object and taking into account occlusion effects between different objects. Moreover, our approach can not only be used to generate data for 6-DoF pose estimation and corresponding 3D-models but can be also extended to automatic dataset generation for object detection, instance segmentation, or volume estimation for any kind of object.

Mobile3DScanner: An Online 3D Scanner for High-quality Object Reconstruction with a Mobile Device.

We present a novel online 3D scanning system for high-quality object reconstruction with a mobile device, called Mobile3DScanner. Using a mobile device equipped with an embedded RGBD camera, our system provides online 3D object reconstruction capability for users to acquire high-quality textured 3D object models. Starting with a simultaneous pose tracking and TSDF fusion module, our system allows users to scan an object with a mobile device to get a 3D model for real-time preview. After the real-time scanning process is completed, the scanned 3D model is globally optimized and mapped with multi-view textures as an efficient postprocess to get the final textured 3D model on the mobile device. Unlike most existing state-of-the-art systems which can only scan homeware objects such as toys with small dimensions due to the limited computation and memory resources of mobile platforms, our system can reconstruct objects with large dimensions such as statues. We propose a novel visual-inertial ICP approach to achieve real-time accurate 6DoF pose tracking of each incoming frame on the front end, while maintaining a keyframe pool on the back end where the keyframe poses are optimized by local BA. Simultaneously, the keyframe depth maps are fused by the optimized poses to a TSDF model in real-time. Especially, we propose a novel adaptive voxel resizing strategy to solve the out-of-memory problem of large dimension TSDF fusion on mobile platforms. In the post-process, the keyframe poses are globally optimized and the keyframe depth maps are optimized and fused to obtain a final object model with more accurate geometry. The experiments with quantitative and qualitative evaluation demonstrate the effectiveness of the proposed 3D scanning system based on a mobile device, which can successfully achieve online high-quality 3D reconstruction of natural objects with larger dimensions for efficient AR content creation.

Instant Visual Odometry Initialization for Mobile AR.

Mobile AR applications benefit from fast initialization to display world-locked effects instantly. However, standard visual odometry or SLAM algorithms require motion parallax to initialize (see Figure 1) and, therefore, suffer from delayed initialization. In this paper, we present a 6-DoF monocular visual odometry that initializes instantly and without motion parallax. Our main contribution is a pose estimator that decouples estimating the 5-DoF relative rotation and translation direction from the 1-DoF translation magnitude. While scale is not observable in a monocular vision-only setting, it is still paramount to estimate a consistent scale over the whole trajectory (even if not physically accurate) to avoid AR effects moving erroneously along depth. In our approach, we leverage the fact that depth errors are not perceivable to the user during rotation-only motion. However, as the user starts translating the device, depth becomes perceivable and so does the capability to estimate consistent scale. Our proposed algorithm naturally transitions between these two modes. Our second contribution is a novel residual in the relative pose problem to further improve the results. The residual combines the Jacobians of the functional and the functional itself and is minimized using a Levenberg-Marquardt optimizer on the 5-DoF manifold. We perform extensive validations of our contributions with both a publicly available dataset and synthetic data. We show that the proposed pose estimator outperforms the classical approaches for 6-DoF pose estimation used in the literature in low-parallax configurations. Likewise, we show our relative pose estimator outperforms state-of-the-art approaches in an odometry pipeline configuration where we can leverage initial guesses. We release a dataset for the relative pose problem using real data to facilitate the comparison with future solutions for the relative pose problem. Our solution is either used as a full odometry or as a pre-SLAM component of any supported SLAM system (ARKit, ARCore) in world-locked AR effects on platforms such as Instagram and Facebook.

A Feature Based Laser SLAM Using Rasterized Images of 3D Point Cloud

An accurate and computationally efficient SLAM algorithm is vital for autonomous vehicles. Most modern SLAM systems use feature detection approaches to limit computational requirements. Feature detection through a 3D point cloud can be a computationally challenging task. In this paper, we propose a feature-based SLAM algorithm using 2D image projections of the 3D laser point cloud. We use a camera parameters matrix to rasterize the 3D point cloud to an image. Then ORB feature detector is applied to these images. The proposed method gives repeatable and stable features in a variety of environments. Based on such features, we can estimate the 6dof pose of the robot. For loop detection, we employ a 2-step approach, i.e., nearest key-frame detection and loop candidate verification by matching features extracted from rasterized LIDAR images. We evaluate the proposed system with implementation on the KITTI dataset. Through experimental results, we show that the algorithm presented in this paper can substantially reduce the computational cost of feature detection from the point cloud and the whole SLAM system while giving accurate results.

Fast and High-Quality, GPU-based, Deliberative, Object-Pose Estimation

Pose estimation of recognized objects is fundamental to tasks such as robotic grasping and manipulation. The need for reliable grasping imposes stringent accuracy requirements on pose estimation in cluttered, occluded scenes in dynamic environments. Modern methods employ large sets of training data to learn features and object templates in order to find correspondence between models and observed data. However, these methods require extensive annotation of ground-truth poses. An alternative is to use algorithms, such as PERCH (PErception Via SeaRCH) that seek an optimal explanation of the observed scene in a space of possible rendered versions. While PERCH offers strong guarantees on accuracy, the initial formulation suffers from poor scalability owing to its high runtime. In this work, we present PERCH 2.0, a deliberative approach that takes advantage of GPU acceleration and RGB data by formulating pose estimation as a single-shot, fully parallel approach. We show that PERCH 2.0 achieves a two orders of magnitude speedup (∼100X) over the hierarchical PERCH by evaluating thousands of poses in parallel. In addition, we propose a combined deliberative and discriminative framework for 6-DoF pose estimation that doesn’t require any ground-truth pose-annotation. Our work shows that PERCH 2.0 achieves, on the YCB-Video Dataset, a higher accuracy than DenseFusion, a state-of-the-art, end-to-end, learning-based approach. We also demonstrate that our work leads directly to an extension of deliberative pose estimation methods like PERCH to new domains, such as conveyor picking, which was previously infeasible due to high runtime. Our code is available at https://sbpl-cruz.github.io/perception/

A 3D Keypoints Voting Network for 6DoF Pose Estimation in Indoor Scene

This paper addresses the problem of instance-level 6DoF pose estimation from a single RGBD image in an indoor scene. Many recent works have shown that a two-stage network, which first detects the keypoints and then regresses the keypoints for 6d pose estimation, achieves remarkable performance. However, the previous methods concern little about channel-wise attention and the keypoints are not selected by comprehensive use of RGBD information, which limits the performance of the network. To enhance RGB feature representation ability, a modular Split-Attention block that enables attention across feature-map groups is proposed. In addition, by combining the Oriented FAST and Rotated BRIEF (ORB) keypoints and the Farthest Point Sample (FPS) algorithm, a simple but effective keypoint selection method named ORB-FPS is presented to avoid the keypoints appear on the non-salient regions. The proposed algorithm is tested on the Linemod and the YCB-Video dataset, the experimental results demonstrate that our method outperforms the current approaches, achieves ADD(S) accuracy of 94.5% on the Linemod dataset and 91.4% on the YCB-Video dataset.

6D Pose Estimation for Subsea Intervention in Turbid Waters

Manipulation tasks on subsea instalments require extremely precise detection and localization of objects of interest. This problem is referred to as “pose estimation”. In this work, we present a framework for detecting and predicting 6DoF pose for relevant objects (fish-tail, gauges, and valves) on a subsea panel under varying water turbidity. A deep learning model that takes 3D vision data as an input is developed, providing a more robust 6D pose estimate. Compared to the 2D vision deep learning model, the proposed method reduces rotation and translation prediction error by (−Δ0.39∘) and translation (−Δ6.5 mm), respectively, in high turbid waters. The proposed approach is able to provide object detection as well as 6D pose estimation with an average precision of 91%. The 6D pose estimation results show 2.59∘ and 6.49 cm total average deviation in rotation and translation as compared to the ground truth data on varying unseen turbidity levels. Furthermore, our approach runs at over 16 frames per second and does not require pose refinement steps. Finally, to facilitate the training of such model we also collected and automatically annotated a new underwater 6D pose estimation dataset spanning seven levels of turbidity.

Improve the Estimation of Monocular Vision 6-DOF Pose Based on the Fusion of Camera and Laser Rangefinder

Monocular vision is one of the most commonly used noncontact six-degrees-of-freedom (6-DOF) pose estimation methods. However, the large translational DOF measurement error along the optical axis of the camera is one of its main weaknesses, which greatly limits the measurement accuracy of monocular vision measurement. In this paper, we propose a novel monocular camera and 1D laser rangefinder (LRF) fusion strategy to overcome this weakness and design a remote and ultra-high precision cooperative targets 6-DOF pose estimation sensor. Our approach consists of two modules: (1) a feature fusion module that precisely fuses the initial pose estimated from the camera and the depth information obtained by the LRF. (2) An optimization module that optimizes pose and system parameters. The performance of our proposed 6-DOF pose estimation method is validated using simulations and real-world experiments. The experimental results show that our fusion strategy can accurately integrate the information of the camera and the LRF. Further optimization carried out on this basis effectively reduces the measurement error of monocular vision 6-DOF pose measurement. The experimental results obtained from a prototype show that its translational and rotational DOF measurement accuracy can reach up to 0.02 mm and 15″, respectively, at a distance of 10 m.

An RGB-D Camera Based Visual Positioning System for Assistive Navigation by a Robotic Navigation Aid

There are about 253 million people with visual impairment worldwide. Many of them use a white cane and/or a guide dog as the mobility tool for daily travel. Despite decades of efforts, electronic navigation aid that can replace white cane is still research in progress. In this paper, we propose an RGB-D camera based visual positioning system (VPS) for real-time localization of a robotic navigation aid (RNA) in an architectural floor plan for assistive navigation. The core of the system is the combination of a new 6-DOF depth-enhanced visual-inertial odometry (DVIO) method and a particle filter localization (PFL) method. DVIO estimates RNA's pose by using the data from an RGB-D camera and an inertial measurement unit (IMU). It extracts the floor plane from the camera's depth data and tightly couples the floor plane, the visual features (with and without depth data), and the IMU's inertial data in a graph optimization framework to estimate the device's 6-DOF pose. Due to the use of the floor plane and depth data from the RGB-D camera, DVIO has a better pose estimation accuracy than the conventional VIO method. To reduce the accumulated pose error of DVIO for navigation in a large indoor space, we developed the PFL method to locate RNA in the floor plan. PFL leverages geometric information of the architectural CAD drawing of an indoor space to further reduce the error of the DVIO-estimated pose. Based on VPS, an assistive navigation system is developed for the RNA prototype to assist a visually impaired person in navigating a large indoor space. Experimental results demonstrate that: 1) DVIO method achieves better pose estimation accuracy than the state-of-the-art VIO method and performs real-time pose estimation (18 Hz pose update rate) on a UP Board computer; 2) PFL reduces the DVIO-accrued pose error by 82.5% on average and allows for accurate wayfinding (endpoint position error ≤ 45 cm) in large indoor spaces.

Outdoor SLAM Using Monocular Vision-Based Localization with LIDAR-Aided Mapping for service robot in Highway

This paper designed an intelligent service robot system in highway based on multi-sensor fusion. The mobile robot attempts to fuse the lidar information and monocular vision information to estimate the pose of itself and obtain an environmental map. It adapts a new SLAM method which combines lidar and vision information. Lidar is used to obtain the 2D occupancy grid map and the monocular vision SLAM algorithm uses the Extended Kalman Filter (EKF) to magnify the pose estimation. The 3-DOF pose provided by lidar is obtained through Cartographer algorithm and the monocular vision SLAM who offers the 6-DOF pose is realized with ORB-SLAM. The experimental results show that the system is effective in application as an intelligent service robot of highway.

Uncalibrated stereo vision with deep learning for 6-DOF pose estimation for a robot arm system

This paper proposes a novel method for six degrees of freedom pose estimation of objects for the application of robot arm pick and place. It is based on the use of a stereo vision system, which does not require calibration. Using both cameras, four corner points of the object are detected. A deep-neural-network (DNN) is trained for the prediction of the 6 DOF pose of the object from the four detected corner points’ coordinates in each image of both cameras. The stereo vision used is a low-end vision system placed in a custom-made setup. Before the training phase of the DNN, the robot is set to auto collect data in a predefined workspace. This workspace is defined dependently on the spatial feasibility of the robot arm and the shared field of view of the stereo vision system. The collected data represent images of a 2D marker attached to the robot arm gripper. The 2D marker is used for data collection to ease the detection of the four corner points. The proposed method succeeds in estimating the six degrees of freedom pose of the object, without the need for the determination of neither the intrinsic nor the extrinsic parameters of the stereo vision system. The optimum design of the proposed DNN is obtained after comparing different activation functions and optimizers associated with the DNN.The proposed uncalibrated DNN-based method performance is compared to that of the traditional calibration-based method. In the calibration-based method, the rotational matrix relating the robot coordinates to the stereo vision coordinates is computed using two approaches. The first approach uses Singular Value Decomposition (SVD) while the second approach uses a novel proposed modification of particle swarm optimization (PSO) called Hyper particle Scouts optimization (HPSO). HPSO outperforms other metaheuristic optimization algorithms such as PSO and genetic algorithm (GA).Exhaustive tests are performed, and the proposed DNN-based method is shown to outperform all tested alternatives.

LHFF-Net: Local heterogeneous feature fusion network for 6DoF pose estimation

LHFF-Net: Local heterogeneous feature fusion network for 6DoF pose estimation

Can We Localize an Autonomous Vehicle From a Single Image? Deep-Geometric Six Degrees-of-Freedom Localization in Topo-Metric Maps

Abstract We describe a deep-geometric localizer that is able to estimate the full six degrees-of-freedom (DOF) global pose of the camera from a single image in a previously mapped environment. Our map is a topo-metric one, with discrete topological nodes whose 6DOF poses are known. Each topo-node in our map also comprises of a set of points, whose 2D features and 3D locations are stored as part of the mapping process. For the mapping phase, we utilize a stereo camera and a regular stereo visual SLAM pipeline. During the localization phase, we take a single camera image, localize it to a topological node using deep learning, and use a geometric algorithm (perspective-n-point (PnP)) on the matched 2D features (and their 3D positions in the topo map) to determine the full 6DOF globally consistent pose of the camera. Our method divorces the mapping and the localization algorithms and sensors (stereo and mono) and allows accurate 6DOF pose estimation in a previously mapped environment using a single camera. With results in simulated and real environments, our hybrid algorithm is particularly useful for autonomous vehicles (AVs) and shuttles that might repeatedly traverse the same route.

Attention Voting Network with Prior Distance Augmented Loss for 6DoF Pose Estimation

Attention Voting Network with Prior Distance Augmented Loss for 6DoF Pose Estimation

Robotic Grasping With Multi-View Image Acquisition and Model-Based Pose Estimation

Due to recent advances on hardware and software technologies, industrial automation has been significantly improved in the past few decades. For random bin picking applications, it is a future trend to use machine vision based approaches to estimate the 3D poses of workpieces. In this work, we present a robotic grasping system with multi-view depth image acquisition. First, RANSAC and an outlier filter are adopted for noise removal and multi-object segmentation. A voting scheme is then used for preliminary pose computation, followed by the ICP algorithm to derive a more precise target orientation. A model-based registration approach using a genetic algorithm with parameter minimization is proposed for 6-DOF pose estimation. Finally, the grasping efficiency is increased by disturbance detection, which reduces the number of 3D data scanning for multiple operations. The experiments are carried out in the real scene environment, and the performance evaluation has demonstrated the feasibility of the proposed technique.

From IR Images to Point Clouds to Pose: Point Cloud-Based AR Glasses Pose Estimation.

In this paper, we propose two novel AR glasses pose estimation algorithms from single infrared images by using 3D point clouds as an intermediate representation. Our first approach “PointsToRotation” is based on a Deep Neural Network alone, whereas our second approach “PointsToPose” is a hybrid model combining Deep Learning and a voting-based mechanism. Our methods utilize a point cloud estimator, which we trained on multi-view infrared images in a semi-supervised manner, generating point clouds based on one image only. We generate a point cloud dataset with our point cloud estimator using the HMDPose dataset, consisting of multi-view infrared images of various AR glasses with the corresponding 6-DoF poses. In comparison to another point cloud-based 6-DoF pose estimation named CloudPose, we achieve an error reduction of around 50%. Compared to a state-of-the-art image-based method, we reduce the pose estimation error by around 96%.

Optimizing RGB-D Fusion for Accurate 6DoF Pose Estimation

Today's standard object localization systems often do not meet the industry's demands regarding 2D and 3D accuracy for digital manufacturing applications. Two targets are considered: digital-based assistance and robotic inspection. 2D precision is necessary to provide accurate assistance whilst 3D precision is crucial to get an inspection as much close to the object's true state. In this letter, we propose a new pose estimation system which ensures highest both 2D and 3D precision. While most RGB-based solutions focus on obtaining best 2D accuracy, RGBD-based systems mainly use depth information to maximize 3D accuracy. Very few solutions propose a way to jointly optimize both constraints. Nonetheless, pose estimation should produce high accuracy as a slight 2D error can result in a large 3D error (and inversely). To address this problem, we present a new system which uses RGB-D to fully take advantage of the depth information. A new 3D primitive is proposed in order to minimize the effect of RGB-D noise on 3D coordinates accuracy. CNN Keypoint Detector (KPD) method is used to localize this new primitive in order to achieve pose estimation task. Finally, we propose a novel refinement method which ensures optimal precision as both RGB and depth information are fused. We show the results of our experimentation on widely-used and challenging Linemod and Occlusion datasets. We demonstrate that our solution outperforms state-of-the-art methods when taking into account both 3D and 2D accuracy.

SAC-COT: Sample Consensus by Sampling Compatibility Triangles in Graphs for 3-D Point Cloud Registration

Six-degree-of-freedom (6-DOF) pose estimation from feature correspondences remains a popular and robust approach for 3-D registration. However, heavy outliers that existed in the initial correspondence set pose a great challenge to this problem. This article presents a simple yet effective estimator called SAmple Consensus by sampling COmpatibility Triangles in graphs (SAC-COT) for robust 6-DOF pose estimation and 3-D registration. The key novelty is a guided three-point sampling approach. It is based on a novel correspondence sample representation, i.e., COmpatibility Triangle (COT). We first model the correspondence set as a graph with nodes connecting compatible correspondences. Then, by ranking and sampling COTs formed by ternary loops, we show that correct hypotheses can be generated in early iteration stage. Finally, the hypothesis generated by the COT yielding to the maximum consensus is the output of SAC-COT. Extensive experiments on six data sets and extensive comparisons with the state-of-the-art estimators confirm that: 1) SAC-COT can achieve accurate registrations with a few iterations and 2) SAC-COT outperforms all competitors and is ultrarobust when confronted with Gaussian noise, data decimation, holes, clutter, partial overlap, varying scales of input correspondences, and data modality variation.

Triangulate geometric constraint combined with visual-flow fusion network for accurate 6DoF pose estimation

Estimating the 6D object pose based on a monocular RGB image is a challenging task in computer vision, which produces false positives under the influence of occlusion or cluttered environments. In addition, the prediction of translation is affected by changes of the image size. In this work, we present a novel two-stage method TGCPose6D for robust 6DoF object pose estimation which is composed of 2D keypoint detection and translation refinement. In the first stage, the 2D keypoint regression space is constrained by triangulate geometric feature vectors, and the low-quality prediction is suppressed by the center-heatmap weighted loss function, thereby the performance of keypoint detection is significantly improved. In the second stage, the Visual-Flow Fusion network (VFFNet) is used to extract the visual feature and optical flow feature of the rendered image and the observed image, and to predict the relative translation based on the difference of features. Specifically, the VFFNet is trained iteratively to gain the ability to predict the relative translation deviation. Extensive experiments are conducted to demonstrate the effectiveness of the proposed TGCPose6D method. Our overall pose estimation pipeline outperforms state-of-the-art object pose estimation methods on several benchmarks.

End-to-End 6DoF Pose Estimation From Monocular RGB Images

We present a conceptually simple framework for 6DoF object pose estimation, especially for autonomous driving scenarios. Our approach can efficiently detect the traffic participants from a monocular RGB image while simultaneously regressing their 3D translation and rotation vectors. The proposed method 6D-VNet, extends the Mask R-CNN by adding customised heads for predicting vehicle's finer class, rotation and translation. It is trained end-to-end compared to previous methods. Furthermore, we show that the inclusion of translational regression in the joint losses is crucial for the 6DoF pose estimation task, where object translation distance along longitudinal axis varies significantly, e.g., in autonomous driving scenarios. Additionally, we incorporate the mutual information between traffic participants via a modified non-local block to capture the spatial dependencies among the detected objects. As opposed to the original non-local block implementation, the proposed weighting modification takes the spatial neighbouring information into consideration whilst counteracting the effect of extreme gradient values. We evaluate our method on the challenging real-world Pascal3D+ dataset and our 6D-VNet reaches the 1st place in ApolloScape challenge 3D Car Instance task (Apolloscape, 2018), (Huang et al., 2018).