Full 3D Pose Research Articles

PolarMesh: A Star-Convex 3D Shape Approximation for Object Pose Estimation

In this letter, we introduce PolarMesh as a star-convex approximation of a 3D object based on spherical projection and can be applied to monocular object pose and shape estimation. The proposed PolarMesh can be stored in a discrete 2D map that allows a trivial conversion between it and the object surface, as the adjacent map elements form faces of the 3D model. The fixed-grid 2D nature of this object surface approximation ensures that it can be estimated from an image patch of the object with an auto-encoder neural network. Moreover, the PolarMesh not only encodes the object shape but also its orientation. Consequently, it is used for accurate estimation of object orientation and can be extended to the full 6D pose estimation using a projection-based method for translation. An exhaustive quantitative evaluation on LineMod, LineMod-Occlusion and HomebrewedDB shows the competitive performance of the proposed method compared to other monocular RGB pose estimation pipelines. Furthermore, we tested the generalization capabilities of the representation on the recent NOCS dataset for category-level pose estimation and achieved very compelling results in terms of predicted rotations.

A Robust 6-D Pose Tracking Approach by Fusing a Multi-Camera Tracking Device and an AHRS Module

Optical tracking technique based on multiple cameras is widely employed in robots, virtual reality, and industrial measurements due to its excellent position accuracy. However, its 6-dimensional (6D) tracking practicability is impaired by the line-of-sight issue and the complicated tracker based on multiple markers. Hence, we implemented a custom-built hybrid tracking system consisting of four-camera equipment and a 6D tracker comprising a single optical marker and an automatic heading reference system (AHRS). AHRS provides the 3D orientation of the tracked object directly and compensates for the 3D position when the optical tracking is occluded. A two-stage cascaded adaptive Unscented Kalman Filtering (CAUKF) was proposed to enhance real-time fusion tracking performance. The CAUKF not only provides a reliable frequency enhancement solution for the optical tracking adapting to various frequencies, but also improves the continuality of prediction on fusion state variables and the corresponding covariance matrix, which helps to initialize the occlusion tracking accurately. When an occlusion occurs, a learning-based adaptive Unscented Kalman Filter (LAUKF) module can adaptively adjust noise estimation matrices in the unscented transformation according to the AHRS data, thereby significantly reducing the position estimation error. Experimental results reveal that the proposed tracking approach achieved 6D tracking at 100 Hz with 0.32 mm position error (mean absolute error) and 0.1° orientation error. This study furnishes a novel implementation method for the full 6D pose tracking with a simplified tracker structure and improved accuracy, continuity, and stability.

Autonomous subsea intervention (SEAVENTION)

This paper presents the main results and latest developments in a 4-year project called autonomous subsea intervention (SEAVENTION). In the project we have developed new methods for autonomous inspection, maintenance and repair (IMR) in subsea oil and gas operations with Unmanned Underwater Vehicles (UUVs). The results are also relevant for offshore wind, aquaculture and other industries. We discuss the trends and status for UUV-based IMR in the oil and gas industry and provide an overview of the state of the art in intervention with UUVs. We also present a 3-level taxonomy for UUV autonomy: mission-level, task-level and vehicle-level. To achieve robust 6D underwater pose estimation of objects for UUV intervention, we have developed marker-less approaches with input from 2D and 3D cameras, as well as marker-based approaches with associated uncertainty. We have carried out experiments with varying turbidity to evaluate full 6D pose estimates in challenging conditions. We have also devised a sensor autocalibration method for UUV localization. For intervention, we have developed methods for autonomous underwater grasping and a novel vision-based distance estimator. For high-level task planning, we have evaluated two frameworks for automated planning and acting (AI planning). We have implemented AI planning for subsea inspection scenarios which have been analyzed and formulated in collaboration with the industry partners. One of the frameworks, called T-REX demonstrates a reactive behavior to the dynamic and potentially uncertain nature of subsea operations. We have also presented an architecture for comparing and choosing between mission plans when new mission goals are introduced.

Application of a Vision-Based Single Target on Robot Positioning System.

In this paper, we propose a Circular-ring visual location marker based on a global image-matching model to improve the positioning ability in the fiducial marker system of a single-target mobile robot. The unique coding information is designed according to the cross-ratio invariance of the projective theorem. To verify the accuracy of full 6D pose estimation using the Circular-ring marker, a 6 degree of freedom (DoF) robotic arm platform is used to design a visual location experiment. The experimental result shows in terms of small resolution images, different size markers, and long-distance tests that our proposed robot positioning method significantly outperforms AprilTag, ArUco, and Checkerboard. Furthermore, through a repeatable robot positioning experiment, the results indicated that the proposed Circular-ring marker is twice as accurate as the fiducial marker at 2–4 m. In terms of recognition speed, the Circular-ring marker processes a frame within 0.077 s. When the Circular-ring marker is used for robot positioning at 2–4 m, the maximum average translation error of the Circular-ring marker is 2.19, 3.04, and 9.44 mm. The maximum average rotation error is also 1.703°, 1.468°, and 0.782°.

Elevation Angle Estimations of Wide-Beam Acoustic Sonar Measurements for Autonomous Underwater Karst Exploration.

This paper proposes a solution for merging the measurements from two perpendicular profiling sonars with different beam-widths, in the context of underwater karst (cave) exploration and mapping. This work is a key step towards the development of a full 6D pose SLAM framework adapted to karst aquifer, where potential water turbidity disqualifies vision-based methods, hence relying on acoustic sonar measurements. Those environments have complex geometries which require 3D sensing. Wide-beam sonars are mandatory to cover previously seen surfaces but do not provide 3D measurements as the elevation angles are unknown. The approach proposed in this paper leverages the narrow-beam sonar measurements to estimate local karst surface with Gaussian process regression. The estimated surface is then further exploited to infer scaled-beta distributions of elevation angles from a wide-beam sonar. The pertinence of the method was validated through experiments on simulated environments. As a result, this approach allows one to benefit from the high coverage provided by wide-beam sonars without the drawback of loosing 3D information.

Temporally Refined Graph U-Nets for Human Shape and Pose Estimation From Monocular Videos

This work addresses a challenging problem of estimating the full 3D human shape and pose from monocular videos. Since real-world 3D mesh-labeled datasets are limited, most current methods in 3D human shape reconstruction only focus on single RGB images, losing all the temporal information. In contrast, we propose temporally refined Graph U-Nets, including an image-level module and a video-level module, to solve this problem. The image-level module is Graph U-Nets for human shape and pose estimation from images, where the Graph Convolutional Neural Network (Graph CNN) helps the information communication of neighboring vertices, and the U-Nets architecture enlarges the receptive field of each vertex and fuses high-level and low-level features. The video-level module is a small Residual Temporal Graph CNN (Residual TG-CNN), which learns temporal dynamics from both structural and temporal neighbors. The temporal dynamics of each vertex are continuous in the temporal dimension and highly relevant to the structural neighbors, so it is helpful to diminish the ambiguity of the body in single images by fusing temporal dynamics. Our algorithm makes full use of labels from image-level datasets and refines the image-level results through video-level module. Evaluated on Human3.6 M and 3DPW datasets, our model produces accurate 3D human meshes and achieves superior 3D human pose estimation accuracy when compared with state-of-the-art methods.

An image appearance based optimization scheme for monocular 6D pose estimation of SOR cabins

Monocular pose estimation for the surface of revolution (SOR) structure is a classical computer vision issue in aerospace applications. Existing pose estimation methods solve the 3-DoF translation and 2-DoF attitude reliably. However, they still require external constraints to solve the remained 1-DoF rotation and optimize the full 6D pose. This work proposes a general full 6D pose estimation method based on the image appearance feature and a non-linear optimization scheme. First, the position and attitude are solved from the imaged elliptic cross-sections. Then, an error objective function is designed to describe the appearance difference of the cross section caused by the pose error. Finally, the attitude is determined, and the full pose is solved by sorting and iteratively minimizing the objective function. Experiment results show that the proposed method is accurate, and is adaptive to view changes, dark illumination, and partial occlusion conditions. The average absolute translation and rotation error is 0.434 mm and 0.661° respectively, outperforming the keypoint-based baseline. The mean image difference on real images is 0.306, yielding a 17% error reduction from that of the fiducial marker benchmark.

6D interaction control with aerial robots: The flying end-effector paradigm

This paper presents a novel paradigm for physical interactive tasks in aerial robotics allowing reliability to be increased and weight and costs to be reduced compared with state-of-the-art approaches. By exploiting its tilted propeller actuation, the robot is able to control the full 6D pose (position and orientation independently) and to exert a full-wrench (force and torque independently) with a rigidly attached end-effector. Interaction is achieved by means of an admittance control scheme in which an outer loop control governs the desired admittance behavior (i.e., interaction compliance/stiffness, damping, and mass) and an inner loop based on inverse dynamics ensures full 6D pose tracking. The interaction forces are estimated by an inertial measurement unit (IMU)-enhanced momentum-based observer. An extensive experimental campaign is performed and four case studies are reported: a hard touch and slide on a wooden surface, called the sliding surface task; a tilted peg-in-hole task, i.e., the insertion of the end-effector in a tilted funnel; an admittance shaping experiment in which it is shown how the stiffness, damping, and apparent mass can be modulated at will; and, finally, the fourth experiment is to show the effectiveness of the approach also in the presence of time-varying interaction forces.

Image‐Based 3D Pose Reconstruction of Surgical Needle for Robot‐Assisted Laparoscopic Suturing

We present a new method to solve the recognition and localisation problem of a surgical needle for robot-assisted laparoscopy. Based on the observation from a single monocular laparoscopic image, we propose a new modelling method to parametrise the full 3D pose of the surgical needle by constrained Degrees of freedom (DOFs) using only two generalised variables. To obtain effective image feedback for the modelling, a feature segmentation algorithm is introduced from probabilistic linear constraints in RGB colour space, constructed from typical laparoscopic images. An iterative algorithm using gradient descent rule is implemented to converge the computed needle's pose to its real one. Experiments demonstrate the feasibility of the proposed scheme using laparoscopic torso models.

Heterogeneous Multisensor Fusion for Mobile Platform Three-Dimensional Pose Estimation

Precise, robust, and consistent localization is an important subject in many areas of science such as vision-based control, path planning, and simultaneous localization and mapping (SLAM). To estimate the pose of a platform, sensors such as inertial measurement units (IMUs), global positioning system (GPS), and cameras are commonly employed. Each of these sensors has their strengths and weaknesses. Sensor fusion is a known approach that combines the data measured by different sensors to achieve a more accurate or complete pose estimation and to cope with sensor outages. In this paper, a three-dimensional (3D) pose estimation algorithm is presented for a unmanned aerial vehicle (UAV) in an unknown GPS-denied environment. A UAV can be fully localized by three position coordinates and three orientation angles. The proposed algorithm fuses the data from an IMU, a camera, and a two-dimensional (2D) light detection and ranging (LiDAR) using extended Kalman filter (EKF) to achieve accurate localization. Among the employed sensors, LiDAR has not received proper attention in the past; mostly because a two-dimensional (2D) LiDAR can only provide pose estimation in its scanning plane, and thus, it cannot obtain a full pose estimation in a 3D environment. A novel method is introduced in this paper that employs a 2D LiDAR to improve the full 3D pose estimation accuracy acquired from an IMU and a camera, and it is shown that this method can significantly improve the precision of the localization algorithm. The proposed approach is evaluated and justified by simulation and real world experiments.

Tracking for mobile devices: A systematic mapping study

Tracking is an important task that is used for several applications, such as navigation assistance and augmented reality. The improvement and popularization of mobile devices in recent years allowed these applications to be executed on such devices. Thus, several tracking techniques proposed lately take into consideration the benefits and limitations of handheld devices. Therefore, the goal of this work is to perform a systematic mapping in order to provide trends and classification regarding the recent publications in the area of tracking for mobile devices. This study collected 2276 papers from three scientific databases using an open-source crawler, from which 360 were selected to be classified according to four properties: tracking type, degree of freedom, tracking platform and research type. The analysis of these data resulted in a map of the research field, which was presented under three perspectives: the distribution and trends over time of each classification property and the relationship between them. Besides the visual map, the full list of classified papers is available through an open-source web-based catalog. The results showed that the number of publications is increasing every year, which shows a growing interest in this field. Moreover, most works use the device׳s sensors for tracking in location-based applications and almost all of them calculate a 2D or 2D + θ pose. There are also several papers about vision-based techniques to compute the device׳s pose and in the majority of them a full 6D pose is computed. Beyond that, there is a clear preference for systems that calculate the pose locally on the device and only a few use a remote server to assist in this task. Moreover, more than 92% of all papers propose a new technique or use existing ones to create a solution.

ON THE DESIGN AND EVALUATION OF A PRECISE, ROBUST AND SCALABLE FIDUCIAL MARKER FRAMEWORK

In this paper we present an improved color-based planar fiducial marker system. Our framework provides precise and robust full 3D pose estimation of markers with superior accuracy when compared with many fiducial systems in the literature, while color information encoding enables using over 65 000 distinct markers. Unlike most color-based fiducial frameworks, which requires prior classification training and color calibration, ours can perform reliably under illumination changes, requiring but a rough white balance adjustment. Our methodology provides good detection performance even under poor illumination conditions which typically compromise other marker identification techniques, thus avoiding the evaluation of otherwise falsely identified markers. Several experiments are presented and carefully analyzed, in order to validate our system and demonstrate the significant improvement in estimation accuracy of both position and orientation over traditional techniques.

3D SSD tracking with estimated 3D planes

We present a tracking method where full camera position and orientation is tracked from intensity differences in a video sequence. The camera pose is calculated based on 3D planes, and hence does not depend on point correspondences. The plane based formulation also allows additional constraints to be naturally added, e.g., perpendicularity between walls, floor and ceiling surfaces, co-planarity of wall surfaces etc. A particular feature of our method is that the full 3D pose change is directly computed from temporal image differences without making a commitment to a particular intermediate (e.g., 2D feature) representation. We experimentally compared our method with regular 2D SSD tracking and found it more robust and stable. This is due to 3D consistency being enforced even in the low level registration of image regions. This yields better results than first computing (and hence committing to) 2D image features and then from these compute 3D pose.