6-DOF Pose Research Articles

MANTRA: An Effective System Based on Augmented Reality and Infrared Thermography for Industrial Maintenance

In recent years, the benefits of both Augmented Reality (AR) technology and infrared thermography (IRT) have been demonstrated in the industrial maintenance sector, allowing maintenance operations to be carried out in a safer, faster, and more efficient manner. However, there still exists no solution that optimally combines both technologies. In this work, we propose a new AR system—MANTRA—with specific application to industrial maintenance. The system can automatically align virtual information and temperature on any 3D object, in real time. This is achieved through the joint use of an RGB-D sensor and an IRT camera, leading to high accuracy and robustness. To achieve this objective, a pose estimation method that combines a deep-learning-based object detection method, YOLOV4, together with the template-based LINEMOD pose estimation method, as well as a model-based 6DOF pose tracking technique, was developed. The MANTRA system is validated both quantitatively and qualitatively through a real use-case, demonstrating the effectiveness of the system compared to traditional methods and those using only AR.

3D Point Cloud-Based Indoor Mobile Robot in 6-DoF Pose Localization Using a Wi-Fi-Aided Localization System

Six-DoF (Six-degree-of-freedom) pose localization based on 3D point clouds is a challenging task for LBSs (localization-based services). This paper proposes a robust and efficient method that uses multimodal information (vision and Wi-Fi signal information) to estimate the 6-DoF pose of an RGBD camera on a robot with respect to complex 3D textured models of the indoor environment that can contain more than 650,000,000 points. Our developed method narrows the search scope, which delimits boundaries initially using the Wi-Fi location system and applies an environment-adaptive approach to determine the radius of the search sphere based on the signal stability of the Wi-Fi location system. In addition, we propose an algorithm for estimating a novel correspondence between local points with a 3D submap by combining 3D points and surface normals to acquire absolute poses from noisy and outlier-contaminated matching point sets for RGBD sensors in dynamic indoor scenes. Then, a novel two-level spatial verification strategy is used to estimate an accurate pose, which includes the use of a RANSAC (Random Sample Consensus) algorithm for identification and a direct least-square method to acquire the pose from the inliers. The proposed method has been implemented and tested extensively in various indoor scenes. The experimental results demonstrate that the Wi-Fi-aided localization system can efficiently localize a mobile robot in a variety of large-scale 3D point cloud datasets to realize stable time consumption and similar performance to state-of-the-art methods.

Extrinsic Visual–Inertial Calibration for Motion Distortion Correction of Underwater 3D Scans

Underwater 3D laser scanners are an essential type of sensor used by unmanned underwater vehicles (UUVs) for operations such as navigation, inspection, and object recognition and manipulation. Scanners that acquire 3D data by sweeping a laser plane across the scene can provide very high lateral resolution. However, their data may suffer from rolling shutter effect if the change of pose of the robot with respect to the scanned target during the sweep is not negligible. In order to compensate for motion-related distortions without the need for point cloud postprocessing, the 6-DoF pose at which the scanner acquires each line needs to be accurately known. In the underwater domain, autonomous vehicles are often equipped with a high-end inertial navigation system (INS) that provides reliable navigation data. Nonetheless, the relative pose of the 3D scanner with respect to the inertial reference frame of the robot is not usually known a priori. Therefore, this paper uses an ego-motion-based calibration algorithm to calibrate the extrinsic parameters of the visual-inertial sensor pair. Simulations are performed to quantify how miscalibration affects motion-related distortion. The method is also evaluated experimentally in laboratory conditions.

Visual Reconstruction and Localization-Based Robust Robotic 6-DoF Grasping in the Wild

The intelligent grasping expects that the manipulator has the ability to grasp objects with high degree of freedom in a wild (unstructured) environment. Due to low perception ability in handing targets and environments, most industrial robots are limited to top-down 4-DoF grasping. In this work, we propose a novel low-cost coarse to fine robotic grasping framework. First, we design a global localization based environment perception method, which enables the manipulator to roughly and automatically locate work space. Then, constrained by the above initial localization, a 3D point cloud reconstruction based 6-DoF pose estimation method is proposed for the manipulator further fine locating grasping target. Finally, our framework realizes full function of visual 6DoF robotic grasping, which includes two different visual servoing and grasp planning strategies for different objects grasping. Meanwhile, it also can integrate various state-of-arts 6DoF pose estimation algorithms to facilitate various practical grasping applications or researches. Experimental results show that our method achieves autonomous robotic grasping with high degree of freedom in an unknown environment. Especially for objects with occlusion, singular shape or small scale, our method can still maintain robust grasping.

RnR: Retrieval and Reprojection Learning Model for Camera Localization

Camera localization is an essential technique in many applications, such as robot navigation, mixed reality, and unmanned vehicle. We are committed to solving the problem of predicting the 6-DoF pose of cameras from a single color image in a given three-Dimensional (3D) environment. In this paper, we proposed a robust learning model for it. Basically, our proposed methodology consists of two steps: image retrieval and space reprojection. The former is in charge of simultaneous localization and mapping based on pre-captured reference images that rely on the correspondence between pixel points and scene coordinates; whereas the latter carries out camera calibration between the 2D image plane and the 3D scene. Given a two-Dimensional (2D) image, the initial localization is accomplished rapidly by matching a reference image using Siamese networks. More precise localization is achieved by camera calibration between the 2D image and the 3D scene using a fully convolutional network. The experimental results on the public dataset show that our model is more robust and expandable than the previous methods. At the end of this paper, we also apply the system to Unmanned Aerial Vehicle (UAV) localization and achieve good results.

6DOF Object Positioning and Grasping Approach for Industrial Robots Based on Boundary Point Cloud Features

For the three-dimensional (3D) pose estimation of metal blank casts estimate in industrial production process, a novel 6-degree-of-freedom (6DOF) positioning and grasping approach for industrial robots based on boundary features and combining structured light 3D vision and the point cloud matching method is proposed. The proposed approach first uses the Gray code plus grating phase-shift algorithm to reconstruct the 3D surface information of the object. Subsequently, an improved point pair feature (PPF) matching location method based on point cloud boundary extraction is proposed to realize the accurate 6DOF pose estimation of the object. In this method, the point cloud boundary feature is used to optimize the PPF algorithm. Finally, by combining with the point cloud preprocessing process and the improved PPF matching location method, the 6DOF pose positioning of metal blank casting by industrial robots is realized. The proposed approach can accurately complete pose measurement and positioning of objects placed randomly in an open environment. The experimental results demonstrate that the proposed PPF matching location method significantly improves both matching speed and accuracy compared to the traditional PPF algorithm.

Real‐time keypoints detection for autonomous recovery of the unmanned ground vehicle

The combination of a small unmanned ground vehicle (UGV) and a large unmanned carrier vehicle allows more flexibility in real applications such as rescue in dangerous scenarios. The autonomous recovery system, which is used to guide the small UGV back to the carrier vehicle, is an essential component to achieve a seamless combination of the two vehicles. This paper proposes a novel autonomous recovery framework with a low-cost monocular vision system to provide accurate positioning and attitude estimation of the UGV during navigation. First, we introduce a light-weight convolutional neural network called UGV-KPNet to detect the keypoints of the small UGV from the images captured by a monocular camera. UGV-KPNet is computationally efficient with a small number of parameters and provides pixel-level accurate keypoints detection results in real-time. Then, six degrees of freedom pose is estimated using the detected keypoints to obtain positioning and attitude information of the UGV. Besides, we are the first to create a large-scale real-world keypoints dataset of the UGV. The experimental results demonstrate that the proposed system achieves state-of-the-art performance in terms of both accuracy and speed on UGV keypoint detection, and can further boost the 6-DoF pose estimation for the UGV.

6DoF Pose Estimation of Transparent Object from a Single RGB-D Image.

6DoF object pose estimation is a foundation for many important applications, such as robotic grasping, automatic driving, and so on. However, it is very challenging to estimate 6DoF pose of transparent object which is commonly seen in our daily life, because the optical characteristics of transparent material lead to significant depth error which results in false estimation. To solve this problem, a two-stage approach is proposed to estimate 6DoF pose of transparent object from a single RGB-D image. In the first stage, the influence of the depth error is eliminated by transparent segmentation, surface normal recovering, and RANSAC plane estimation. In the second stage, an extended point-cloud representation is presented to accurately and efficiently estimate object pose. As far as we know, it is the first deep learning based approach which focuses on 6DoF pose estimation of transparent objects from a single RGB-D image. Experimental results show that the proposed approach can effectively estimate 6DoF pose of transparent object, and it out-performs the state-of-the-art baselines by a large margin.

Mobile3DRecon: Real-time Monocular 3D Reconstruction on a Mobile Phone.

We present a real-time monocular 3D reconstruction system on a mobile phone, called Mobile3DRecon. Using an embedded monocular camera, our system provides an online mesh generation capability on back end together with real-time 6DoF pose tracking on front end for users to achieve realistic AR effects and interactions on mobile phones. Unlike most existing state-of-the-art systems which produce only point cloud based 3D models online or surface mesh offline, we propose a novel online incremental mesh generation approach to achieve fast online dense surface mesh reconstruction to satisfy the demand of real-time AR applications. For each keyframe of 6DoF tracking, we perform a robust monocular depth estimation, with a multi-view semi-global matching method followed by a depth refinement post-processing. The proposed mesh generation module incrementally fuses each estimated keyframe depth map to an online dense surface mesh, which is useful for achieving realistic AR effects such as occlusions and collisions. We verify our real-time reconstruction results on two mid-range mobile platforms. The experiments with quantitative and qualitative evaluation demonstrate the effectiveness of the proposed monocular 3D reconstruction system, which can handle the occlusions and collisions between virtual objects and real scenes to achieve realistic AR effects.

On the development of a collaborative robotic system for industrial coating cells

For remaining competitive in the current industrial manufacturing markets, coating companies need to implement flexible production systems for dealing with mass customization and mass production workflows. The introduction of robotic manipulators capable of mimicking with accuracy the motions executed by highly skilled technicians is an important factor in enabling coating companies to cope with high customization. However, there are some limitations associated with the usage of a fully automated system for coating applications, especially when considering customized products of large dimensions and complex geometry. This paper addresses the development of a collaborative coating cell to increase the flexibility and efficiency of coating processes. The robot trajectory is taught with an intuitive programming by demonstration system, in which an icosahedron marker with multicoloured LEDs is attached to the coating tool for tracking its trajectories using a stereoscopic vision system. For avoiding the construction of fixtures and allowing the operator to freely place products within the coating work cell, a modular 3D perception system was developed, relying on principal component analysis for performing the initial point cloud alignment and on the iterative closest point algorithm for 6 DoF pose estimation. Furthermore, to enable safe and intuitive human-robot collaboration, a non-intrusive zone monitoring safety system was employed to track the position of the operator in the cell.

Seeing Through the Occluders: Robust Monocular 6-DOF Object Pose Tracking via Model-Guided Video Object Segmentation

To deal with occlusion is one of the most challenging problems for monocular 6-DOF object pose tracking. In this letter, we propose a novel 6-DOF object pose tracking method which is robust to heavy occlusions. When the tracked object is occluded by another object, instead of trying to detect the occluder, we seek to see through it, as if the occluder doesnt exist. To this end, we propose to combine a learning-based video object segmentation module with an optimization-based pose estimation module in a closed loop. Firstly, a model-guided video object segmentation network is utilized to predict the accurate and full mask of the object (including the occluded part). Secondly, a non-linear 6-DOF pose optimization method is performed with the guidance of the predicted full mask. After solving the current object pose, we render the 3D object model to obtain a refined, model-constrained mask of the current frame, which is then fed back to the segmentation network for processing the next frame, closing the whole loop. Experiments show that the proposed method outperforms the state-of-arts by a large margin for dealing with heavy occlusions, and could handle extreme cases which previous methods would fail.

Incomplete Orientation Mapping for Teleoperation With One DoF Master-Slave Asymmetry

Teleoperation systems require a human-centered approach in which the kinematic mapping is intuitive and straightforward for the operators. However, a mismatch in the degrees-of-freedom (DoFs) between master and slave could result in an asymmetrical teleoperation system. That is an obstacle for intuitive kinematic mapping. In particular, it is even more challenging when the missing DoF is a pure rotational DoF, since the rotation group SO(3) is a nonlinear Riemannian manifold. This letter is concerned with an asymmetric teleoperation system, where the master subsystem can provide 6-DoF pose sensing while the slave subsystem only has 5 DoFs. The rotation along the missing DoF, which is configuration-dependent, is mapped to a geodesic curve in SO(3). We define and prove the closed-form solution of the perpendicular curve to the geodesic curve. Based on the perpendicular curve, we develop a novel Incomplete Orientation Mapping (IOM) approach to avoid the motion in the missing DoF. By comparing with two baseline methods, the experimental results demonstrate that the proposed method can discard the rotational motion along the missing DoF for all configurations, while preserving the remaining rotations.

An Occlusion‐aware Edge‐Based Method for Monocular 3D Object Tracking using Edge Confidence

AbstractWe propose an edge‐based method for 6DOF pose tracking of rigid objects using a monocular RGB camera. One of the critical problem for edge‐based methods is to search the object contour points in the image corresponding to the known 3D model points. However, previous methods often produce false object contour points in case of cluttered backgrounds and partial occlusions. In this paper, we propose a novel edge‐based 3D objects tracking method to tackle this problem. To search the object contour points, foreground and background clutter points are first filtered out using edge color cue, then object contour points are searched by maximizing their edge confidence which combines edge color and distance cues. Furthermore, the edge confidence is integrated into the edge‐based energy function to reduce the influence of false contour points caused by cluttered backgrounds and partial occlusions. We also extend our method to multi‐object tracking which can handle mutual occlusions. We compare our method with the recent state‐of‐art methods on challenging public datasets. Experiments demonstrate that our method improves robustness and accuracy against cluttered backgrounds and partial occlusions.

EL net: Ensemble Learning in end-to-end learning

Kednell. et al. [1] propose a robust and real-time monocular six degree of freedom (6-DOF) relocalization system which uses the end-to-end method to regress the 6-DOF pose in the RGB image. However, when the scenes change, PoseNet has huge variance. Ensemble learning can deal with this problem. We propose to use a new end to end camera relocalization method which combines the ensemble learning with PoseNet. In this model we modify the structure of PoseNet and get several different weak models. After that we train these different models and then use ensemble learning to do them together. Experiments show the effectiveness of ensemble learning, compared to previous PoseNet, using ensemble learning can get better result.

POSE ESTIMATION OF A MOVING CAMERA WITH LOW-COST, MULTI-GNSS DEVICES

Abstract. Without additional prior information, the pose of a camera estimated with computer vision techniques is expressed in a local coordinate frame attached to the camera’s initial location. Albeit sufficient in many cases, such an arbitrary representation is not convenient for employment in certain applications and has to be transformed to a coordinate system external to the camera before further use. Assuming a camera that is firmly mounted on a moving platform, this paper describes a method for continuously tracking the pose of that camera in a projected coordinate system. By combining exterior orientation from a known target with incremental pose changes inferred from accurate multi-GNSS positioning, the full 6 DoF pose of the camera is updated with low processing overhead and without requiring the continuous visual tracking of ground control points. Experimental results of applying the proposed method to a moving vehicle and a mobile port crane are reported, demonstrating its efficacy and potential.

Pose Estimation of Primitive-Shaped Objects from a Depth Image Using Superquadric Representation

This paper presents a method for estimating the six Degrees of Freedom (6DoF) pose of texture-less primitive-shaped objects from depth images. As the conventional methods for object pose estimation require rich texture or geometric features to the target objects, these methods are not suitable for texture-less and geometrically simple shaped objects. In order to estimate the pose of the primitive-shaped object, the parameters that represent primitive shapes are estimated. However, these methods explicitly limit the number of types of primitive shapes that can be estimated. We employ superquadrics as a primitive shape representation that can represent various types of primitive shapes with only a few parameters. In order to estimate the superquadric parameters of primitive-shaped objects, the point cloud of the object must be segmented from a depth image. It is known that the parameter estimation is sensitive to outliers, which are caused by the miss-segmentation of the depth image. Therefore, we propose a novel estimation method for superquadric parameters that are robust to outliers. In the experiment, we constructed a dataset in which the person grasps and moves the primitive-shaped objects. The experimental results show that our estimation method outperformed three conventional methods and the baseline method.

MAPPING WITHOUT DYNAMIC: ROBUST LiDAR-SLAM FOR UGV MOBILE MAPPING IN DYNAMIC ENVIRONMENTS

Abstract. Detection And Tracking of Moving Objects (DATMO) is essential and necessary for mobile mapping system to generate clean and accurate point clouds maps since dynamic targets in real-world scenarios will deteriorate the performance of whole system. In this research, a robust LiDAR-SLAM system is presented incorporated with a real-time dynamic objects removal module to improve the accuracy of 6 DOF pose estimation and precision of maps. The key idea of the proposed method is to efficiently cluster the sparse point clouds of moving objects and then track them independently so as to relieve their influence on the odometry and mapping results. In the back-end, in order to further refine the point clouds maps, a valid probabilistic map fusion method is performed based on the free-space theory. We have evaluated our system on the dataset collected from daily crowded environments full of moving objects, providing competitive results with the state-of-the-art system both on the pose estimation and point cloud mapping.

HDMI-Loc: Exploiting High Definition Map Image for Precise Localization via Bitwise Particle Filter

In this letter, we propose a method for accurately estimating the 6-Degree Of Freedom (DOF) pose in an urban environment when a High Definition (HD) map is available. An HD map expresses 3D geometric data with semantic information in a compressed format and thus is more memory-efficient than point cloud maps. The small capacity of HD maps can be a significant advantage for autonomous vehicles in terms of map storage and updates within a large urban area. Unfortunately, existing approaches failed to sufficiently exploit HD maps by only estimating partial pose. In this study, we present a full 6-DOF localization against an HD map using an onboard stereo camera with semantic information from roads. We introduce an 8-bit representation for road information, which allow for effective bitwise operation when matching between query data and the HD map. For the pose estimation, we leverage a particle filter followed by a full 6-DOF pose optimization. Our experimental results show a median error of approximately 0.3 m in the lateral and longitudinal directions for a drive of approximately 11 km. These results can be used by autonomous vehicles to correct the global position without using Global Positioning System (GPS) data in highly complex urban environments. The median operation speed is approximately 60 msec supporting 10 Hz.

STRATEGIES TO INTEGRATE IMU AND LIDAR SLAM FOR INDOOR MAPPING

Abstract. In recent years, the importance of indoor mapping increased in a wide range of applications, such as facility management and mapping hazardous sites. The essential technique behind indoor mapping is simultaneous localization and mapping (SLAM) because SLAM offers suitable positioning estimates in environments where satellite positioning is not available. State-of-the-art indoor mobile mapping systems employ Visual-based SLAM or LiDAR-based SLAM. However, Visual-based SLAM is sensitive to textureless environments and, similarly, LiDAR-based SLAM is sensitive to a number of pose configurations where the geometry of laser observations is not strong enough to reliably estimate the six-degree-of-freedom (6DOF) pose of the system. In this paper, we present different strategies that utilize the benefits of the inertial measurement unit (IMU) in the pose estimation and support LiDAR-based SLAM in overcoming these problems. The proposed strategies have been implemented and tested using different datasets and our experimental results demonstrate that the proposed methods do indeed overcome these problems. We conclude that IMU observations increase the robustness of SLAM, which is expected, but also that the best reconstruction accuracy is obtained not with a blind use of all observations but by filtering the measurements with a proposed reliability measure. To this end, our results show promising improvements in reconstruction accuracy.

Vision-Based Robotic Traversal of Textureless Smooth Surfaces

This investigation focuses on servoing a robot's tool with respect to a smooth workpiece surface by making use of the surface's characteristic local differential properties. A novel formulation for 6 degree-of-freedom (DoF) textureless visual servoing based on these properties is proposed, which extends an existing 3-DoF scheme. Our approach naturally combines the geometric tools of computer-aided design and machining (CAD/CAM) theory with the manipulator control tools of visual servoing synergistically to achieve full 6-DoF pose control. A novel family of observed feature sets and their associated interaction matrices are presented. A geometric condition on the surface shape is derived under which local asymptotic stability for 6-DoF is guaranteed. Validation of the proposed method is performed in simulation and experiment using an articulated desktop robot equipped with only a monocular camera and 16 laser pointers.