Pose Estimation Problem Research Articles

EHFusion: an efficient heterogeneous fusion model for group-based 3D human pose estimation

AbstractStimulated by its important applications in animation, gaming, virtual reality, augmented reality, and healthcare, 3D human pose estimation has received considerable attention in recent years. To improve the accuracy of 3D human pose estimation, most approaches have converted this challenging task into a local pose estimation problem by dividing the body joints of the human body into different groups based on the human body topology. The body joint features of different groups are then fused to predict the overall pose of the whole body, which requires a joint feature fusion scheme. Nevertheless, the joint feature fusion schemes adopted in existing methods involve the learning of extensive parameters and hence are computationally very expensive. This paper reports a new topology-based grouped method ‘EHFusion’ for 3D human pose estimation, which involves a heterogeneous feature fusion (HFF) module that integrates grouped pose features. The HFF module reduces the computational complexity of the model while achieving promising accuracy. Moreover, we introduce motion amplitude information and a camera intrinsic embedding module to provide better global information and 2D-to-3D conversion knowledge, thereby improving the overall robustness and accuracy of the method. In contrast to previous methods, the proposed new network can be trained end-to-end in one single stage. Experimental results not only demonstrate the advantageous trade-offs between estimation accuracy and computational complexity achieved by our method but also showcase the competitive performance in comparison with various existing state-of-the-art methods (e.g., transformer-based) when evaluated on two public datasets, Human3.6M and HumanEva. The data and code are available at doi:10.5281/zenodo.11113132

Certifiably optimal rotation and pose estimation based on the Cayley map

We present novel, convex relaxations for rotation and pose estimation problems that can a posteriori guarantee global optimality for practical measurement noise levels. Some such relaxations exist in the literature for specific problem setups that assume the matrix von Mises-Fisher distribution (a.k.a., matrix Langevin distribution or chordal distance) for isotropic rotational uncertainty. However, another common way to represent uncertainty for rotations and poses is to define anisotropic noise in the associated Lie algebra. Starting from a noise model based on the Cayley map, we define our estimation problems, convert them to Quadratically Constrained Quadratic Programs (QCQPs), then relax them to Semidefinite Programs (SDPs), which can be solved using standard interior-point optimization methods; global optimality follows from Lagrangian strong duality. We first show how to carry out basic rotation and pose averaging. We then turn to the more complex problem of trajectory estimation, which involves many pose variables with both individual and inter-pose measurements (or motion priors). Our contribution is to formulate SDP relaxations for all these problems based on the Cayley map (including the identification of redundant constraints) and to show them working in practical settings. We hope our results can add to the catalogue of useful estimation problems whose solutions can be a posteriori guaranteed to be globally optimal.

OneTip: A soft tactile interface for 6-D fingertip pose acquisition in human-computer interaction

Advances in display technology have created the need for more efficient and natural multi-degree-of-freedom interaction devices. The movement of a single fingertip has six degrees of freedom (DOFs), but traditional rigid touchscreens usually sense only 2-D information. This article proposes a new deformable tactile interface, OneTip, for single-fingertip human-computer interaction with 6 DOFs. It is manufactured based on the principle of vision-based tactile sensing using virtual stereoscopic cameras, and its size is about twice that of a thumb. The contact surface of OneTip has a specially designed structure and material to mimic the sensitivity and softness of human skin. Also, OneTip employs a new sensing method to address the problem of soft fingertip pose estimation (measuring relative change only) with incipient slip effects. Experiments show that OneTip has good 6-D pose estimation accuracy, with root mean square errors (RMSE) of translation and rotation not exceeding 0.1 mm and 2.6°, respectively, within the linear interval (x and y: −1.2–1.2 mm; z: 0–3 mm; yaw: −15–15 deg; pitch and roll: −40–40 deg). Experiments were also conducted to explore the application of OneTip in typical virtual manipulation tasks and the possibility of combining it with other interaction devices.

Hand-object pose estimation based on HOPE-net and attention mechanism

Abstract Hand-object pose estimation is an important part of studying hand pose, focusing on the joint detection of hand and object poses. Although CNN has shown advantages in various hand pose estimation, due to various problems such as the complex structure of the hand pose and self-occlusion, there are limitations in the extraction of image features and the amount of calculation. Existing studies have shown that GCNs have a good performance in dealing with such problems. In this paper, we propose to combine the attention module with two adaptive GCNs, and perform feature enhancement operations on hand features. While ensuring the accuracy of feature extraction, it can effectively improve the network calculation speed and avoid some questions such as self-occlusion. Experimental results show that, through end-to-end training, the algorithm has a great performance on the problem of hand-object pose estimation.

3D Human Pose Estimation Based on Wearable IMUs and Multiple Camera Views

The problem of 3D human pose estimation (HPE) has been the focus of research in recent years, yet precise estimation remains an under-explored challenge. In this paper, the merits of both multiview images and wearable IMUs are combined to enhance the process of 3D HPE. We build upon a state-of-the-art baseline while introducing three novelties. Initially, we enhance the precision of keypoint localization by substituting Gaussian kernels with Laplacian kernels in the generation of target heatmaps. Secondly, we incorporate orientation regularized network (ORN), which enhances cross-modal heatmap fusion by taking a weighted average of the top-scored values instead of solely relying on the maximum value. This not only improves robustness to outliers but also leads to higher accuracy in pose estimation. Lastly, we modify the limb length constraint in the conventional orientation regularized pictorial structure model (ORPSM) to improve the estimation of joint positions. Specifically, we devise a soft-coded binary term for limb length constraint, hence imposing a flexible and smoothed penalization and reducing sensitivity to hyperparameters. The experimental results using the TotalCapture dataset reveal a significant improvement, with a 10.3% increase in PCKh accuracy at the one-twelfth threshold and a 3.9 mm reduction in MPJPE error compared to the baseline.

Absolute Pose Estimation With a Known Direction by Motion Decoupling

This paper develops an extremely robust solution for absolute pose estimation with known prior gravity direction by motion decoupling. Absolute pose estimation is a fundamental problem in computer vision, and recently the prior known vertical direction is commonly applied to help solve the pose estimation problem. In this paper, we explore the geometrical constraints of the absolute pose estimation with a known direction. We find that the rigid pose can be decoupled with the help of the known direction. Thereby, absolute pose estimation algorithms, which decouple rigid motion, are proposed. Notably, in real applications, there may be imperfect inputs, i.e., outliers, due to incorrect 2D-3D matches. Unfortunately, these outliers may lead to unacceptable results. To suppress the outliers, the decoupled absolute pose estimation problem is solved by branch-and-bound algorithm and globally voting, which can provide the optimal solution with provable guarantees. Moreover, in extreme case, the proposed method can solve absolute pose estimation problem without knowing the 2D-3D correspondences, which is also known as simultaneous camera pose correspondence estimation. To demonstrate the feasibility and the superiority of the proposed methods, comprehensive comparison experiment are conduced. The source code is available at https://github.com/Liu-Yinlong/algorithm-for-PnP-with-known-vertical-direction.

Weakly Supervised Pose Estimation of Surgical Instrument from a Single Endoscopic Image.

Instrument pose estimation is a key demand in computer-aided surgery, and its main challenges lie in two aspects: Firstly, the difficulty of obtaining stable corresponding image feature points due to the instruments' high refraction and complicated background, and secondly, the lack of labeled pose data. This study aims to tackle the pose estimation problem of surgical instruments in the current endoscope system using a single endoscopic image. More specifically, a weakly supervised method based on the instrument's image segmentation contour is proposed, with the effective assistance of synthesized endoscopic images. Our method consists of the following three modules: a segmentation module to automatically detect the instrument in the input image, followed by a point inference module to predict the image locations of the implicit feature points of the instrument, and a point back-propagatable Perspective-n-Point module to estimate the pose from the tentative 2D-3D corresponding points. To alleviate the over-reliance on point correspondence accuracy, the local errors of feature point matching and the global inconsistency of the corresponding contours are simultaneously minimized. Our proposed method is validated with both real and synthetic images in comparison with the current state-of-the-art methods.

Vision-state Fusion: Improving Deep Neural Networks for Autonomous Robotics

Vision-based deep learning perception fulfills a paramount role in robotics, facilitating solutions to many challenging scenarios, such as acrobatic maneuvers of autonomous unmanned aerial vehicles (UAVs) and robot-assisted high-precision surgery. Control-oriented end-to-end perception approaches, which directly output control variables for the robot, commonly take advantage of the robot’s state estimation as an auxiliary input. When intermediate outputs are estimated and fed to a lower-level controller, i.e., mediated approaches, the robot’s state is commonly used as an input only for egocentric tasks, which estimate physical properties of the robot itself. In this work, we propose to apply a similar approach for the first time – to the best of our knowledge – to non-egocentric mediated tasks, where the estimated outputs refer to an external subject. We prove how our general methodology improves the regression performance of deep convolutional neural networks (CNNs) on a broad class of non-egocentric 3D pose estimation problems, with minimal computational cost. By analyzing three highly-different use cases, spanning from grasping with a robotic arm to following a human subject with a pocket-sized UAV, our results consistently improve the R2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{2}$$\\end{document} regression metric, up to +0.51, compared to their stateless baselines. Finally, we validate the in-field performance of a closed-loop autonomous cm-scale UAV on the human pose estimation task. Our results show a significant reduction, i.e., 24% on average, on the mean absolute error of our stateful CNN, compared to a State-of-the-Art stateless counterpart.

A branch-and-bound based globally optimal solution to 2D multi-robot relative pose estimation problems

The relative pose estimation problem is important for multi-robot systems when they execute a variety of tasks cooperatively, such as cooperative sensing, multi-robot search and rescue, formation control. The finding of reliable solutions to this problem is essential, however, algorithms that guarantee to provide globally optimal solutions are still in demand. In this paper, an alternative dimensionality reduced objective function is proposed for the planar ground multi-robot relative pose estimation problem, and a new branch-and-bound (BnB) based method is developed, through which initialization is not required and the globally optimal solution is guaranteed under arbitrary noise levels. We demonstrate that the globally optimal solutions obtained via the algorithm proposed in this paper are superior than that obtained by existing classic methods such as SE-Sync algorithm when noise levels are high. Moreover, the method proposed is simple and easy to implement.

BDR6D: Bidirectional Deep Residual Fusion Network for 6D Pose Estimation

Six-dimensional (6D) pose estimation is an important branch in the field of robotics focused on enhancing the ability of robots to manipulate and grasp objects. The latest research trend in 6D pose estimation is to directly predict the positions of two-dimensional (2D) keypoints from a single red, green, and blue (RGB) image through convolutional neural networks (CNNs) and establish a corresponding relationship with the three-dimensional (3D) keypoints of the model. Then, the perspective-n-point (PnP) algorithm is used to recover the 6D pose parameters. Currently, two challenges are encountered in pose estimation based on an RGB image. On the one hand, an RGB image lacks depth information, and it is thus difficult to directly obtain the corresponding geometric object information. On the other hand, when depth information is available, it is difficult to efficiently fuse the features of the RGB image with the features of the corresponding depth image. In this paper, we propose a bidirectional depth residual fusion network with a depth prediction (DP) network to estimate the 6D poses of objects (BDR6D). The BDR6D network predicts the depth information of objects using an RGB image, converts the depth information into point cloud information, and performs feature extraction and representation together with the RGB information during the feature extraction and representation stages. Specifically, the RGB image is fed into the BDR6D network, the DP network predicts the depth information of the objects in the image, and the depth map and RGB image are input into a point cloud network (PCN) and CNN, respectively, for feature extraction and representation. We build the bidirectional depth residual (BDR) structure so that the CNN and PCN can share information during feature extraction and representation. This approach allows the two networks to use each other’s local and global information to improve feature extraction and representation. For the keypoint selection stage, we propose an effective 2D keypoint selection method that considers the appearance and geometric information of the object of interest. We evaluate the proposed method with three benchmark datasets and compare it with other 6D pose estimation algorithms. The experimental results show that our method outperforms the state-of-the-art approach. Finally, we deploy our proposed method in conjunction with the Universal Robots 5 manipulator (UR5) robot to grasp and manipulate objects. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The purpose of this paper is to solve the problem of 6D pose estimation for robot grasping. The existing RGB image-based pose estimation approach faces two challenges. On the one hand, a single RGB image lacks depth information, so that it is difficult to directly obtain the corresponding geometric object information. On the other hand, when depth information is available, it is difficult to efficiently fuse the features of the RGB image with the features of the corresponding depth image. To solve the above problems, a novel network that can predict the depth information of objects from an RGB image and fuse the depth information with the RGB information to estimate the 6D pose of objects is proposed. Furthermore, we propose an effective 2D keypoint selection method that considers the appearance and geometric information of objects of interest. We evaluate the proposed approach based on three benchmark datasets and the UR5 robot platform and verify that our method is effective.

AiPE: A Novel Transformer-Based Pose Estimation Method

AiPE: A Novel Transformer-Based Pose Estimation Method

UViT: Efficient and lightweight U-shaped hybrid vision transformer for human pose estimation

Pose estimation plays a crucial role in human-centered vision applications and has advanced significantly in recent years. However, prevailing approaches use extremely complex structural designs for obtaining high scores on the benchmark dataset, hampering edge device applications. In this study, an efficient and lightweight human pose estimation problem is investigated. Enhancements are made to the context enhancement module of the U-shaped structure to improve the multi-scale local modeling capability. With a transformer structure, a lightweight transformer block was designed to enhance the local feature extraction and global modeling ability. Finally, a lightweight pose estimation network— U-shaped Hybrid Vision Transformer, UViT— was developed. The minimal network UViT-T achieved a 3.9% improvement in AP scores on the COCO validation set with fewer model parameters and computational complexity compared with the best-performing V2 version of the MobileNet series. Specifically, with an input size of 384×288, UViT-T achieves an impressive AP score of 70.2 on the COCO test-dev set, with only 1.52 M parameters and 2.32 GFLOPs. The inference speed is approximately twice that of general-purpose networks. This study provides an efficient and lightweight design idea and method for the human pose estimation task and provides theoretical support for its deployment on edge devices.

Visual-Based Kinematics and Pose Estimation for Skid-Steering Robots

To build commercial robots, skid-steering mechanical design is of increased popularity due to its manufacturing simplicity and unique mechanism. However, these also cause significant challenges on software and algorithm design, especially for the pose estimation (i.e., determining the robot’s rotation and position) of skid-steering robots, since they change their orientation with an inevitable skid. To tackle this problem, we propose a probabilistic sliding-window estimator dedicated to skid-steering robots, using measurements from a monocular camera, the wheel encoders, and optionally an inertial measurement unit (IMU). Specifically, we explicitly model the kinematics of skid-steering robots by both track instantaneous centers of rotation (ICRs) and correction factors, which are capable of compensating for the complexity of track-to-terrain interaction, the imperfectness of mechanical design, terrain conditions and smoothness, etc. To prevent performance reduction in robots’ long-term missions, the time-and location-varying kinematic parameters are estimated online along with pose estimation states in a tightly-coupled manner. More importantly, we conduct in-depth observability analysis for different sensors and design configurations in this paper, which provides us with theoretical tools in making the correct choice when building real commercial robots. In our experiments, we validate the proposed method by both simulation tests and real-world experiments, which demonstrate that our method outperforms competing methods by wide margins. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper was motivated by the problem of long-term pose estimation of the commonly commercial-used skid-steering robots with only low-cost sensors. Skid-steering robots change their orientation with a skid, which poses a significant challenge for pose estimation when using the wheel encoders. We propose to online estimate the robot’s kinematics, which succeeds in compensating for the complexity of track-to-terrain interaction, due to the slippage, the imperfectness of mechanical design, terrain conditions and smoothness. It is critical to estimate the kinematics and poses jointly to prevent performance reduction in robots’ long-term missions. We further theoretically analyze whether the kinematics parameters can be estimated under different sensor configurations, and find out the special degrade motions that make the parameters unobservable.

TRI-POSE-Net: Adaptive 3D human pose estimation through selective kernel networks and self-supervision with trifocal tensors.

Accurate and flexible 3D pose estimation for virtual entities is a strenuous task in computer vision applications. Conventional methods struggle to capture realistic movements; thus, creative solutions that can handle the complexities of genuine avatar interactions in dynamic virtual environments are imperative. In order to tackle the problem of precise 3D pose estimation, this work introduces TRI-POSE-Net, a model intended for scenarios with limited supervision. The proposed technique, which is based on ResNet-50 and includes integrated Selective Kernel Network (SKNet) blocks, has proven to be efficient for feature extraction customised specifically to pose estimation scenarios. Furthermore, trifocal tensors and their trio-view geometry allow us to generate 3D ground truth poses from 2D poses, resulting in more refined triangulations. Through the proposed approach, the 3D poses can be estimated from a single 2D RGB image. Moreover, the proposed approach was evaluated on the HumanEva-I dataset yielding a Mean-Per-Joint-Position-Error (MPJPE) of 47.6 under self-supervision and an MPJPE of 29.9 under full supervision. In comparison with the other works, the proposed work has performed well in the self-supervision paradigm.

Tooth Motion Monitoring in Orthodontic Treatment by Mobile Device-based Multi-view Stereo.

Nowadays, orthodontics has become an important part of modern personal life to assist one in improving mastication and raising self-esteem. However, the quality of orthodontic treatment still heavily relies on the empirical evaluation of experienced doctors, which lacks quantitative assessment and requires patients to visit clinics frequently for in-person examination. To resolve the aforementioned problem, we propose a novel and practical mobile device-based framework for precisely measuring tooth movement in treatment, so as to simplify and strengthen the traditional tooth monitoring process. To this end, we formulate the tooth movement monitoring task as a multi-view multi-object pose estimation problem via different views that capture multiple texture-less and severely occluded objects (i.e. teeth). Specifically, we exploit a pre-scanned 3D tooth model and a sparse set of multi-view tooth images as inputs for our proposed tooth monitoring framework. After extracting tooth contours and localizing the initial camera pose of each view from the initial configuration, we propose a joint pose estimation scheme to precisely estimate the 3D pose of each individual tooth, so as to infer their relative offsets during treatment. Furthermore, we introduce the metric of Relative Pose Bias to evaluate the individual tooth pose accuracy in a small scale. We demonstrate that our approach is capable of reaching high accuracy and efficiency as practical orthodontic treatment monitoring requires.

Rethinking the Person Localization for Single-Stage Multi-Person Pose Estimation

Single-stage models for multi-person pose estimation have garnered significant attention due to their streamlined approach in generating person position localization and body structure perception in a single pass. These two parts, however, are processed individually by existing methods, leading to suboptimal results, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , candidates with high confidences for person localization while poor structure estimations. To this end, we propose a simple yet effective approach, namely Structure-guided Person Localization (SPL), jointly leveraging the advantages of the two aspects to solve the multi-person pose estimation problem, with two complementary novelties. First, we propose to incorporate body structure perception to guide person position localization, consequently, we introduce the Structure-guided Center Learning (SCL) to unify the quality of the body structure perception in the displacement map with the confidence of the person existence in the center map, thus achieving more accurate keypoint position localization results even with extreme poses. Second, to facilitate the end-to-end training of SPL, we propose the efficient Agency-based Scale-adaptive Learning (ASL). Specifically, we predict an agency map of the same size as the center map, which focuses on the foreground area and can adaptively adjust the scale size for each central area with the body structure perception confidence. Comprehensive experiments on challenging benchmarks including COCO and CrowdPose clearly verify the superiority of our framework, which achieves new state-of-the-art single-stage multi-person pose estimation results. Specifically, SPL obtains 72.1 AP scores and 69.5 AP scores in COCO test-dev2017 and CrowdPose test set, respectively.

HeadDiff: Exploring Rotation Uncertainty With Diffusion Models for Head Pose Estimation.

In this paper, we propose a probabilistic regression diffusion model for head pose estimation, dubbed HeadDiff, which typically addresses the rotation uncertainty, especially when faces are captured in wild conditions. Unlike conventional image-to-pose methods which cannot explicitly establish the rotational manifold of head poses, our HeadDiff aims to ensure the pose rotation via the diffusion process and in parallel, refine the mapping process iteratively. Specifically, we initially formulate the head pose estimation problem as a reverse diffusion process, defining a paradigm for progressive denoising on the manifold, which explores the uncertainty by decomposing the large gap into intermediate steps. Moreover, our HeadDiff is equipped with an isotropic Gaussian distribution by encoding the incoherence information in our rotation representation. Finally, we learn the facial relationship of nearest neighbors with a cycle-consistent constraint for robust pose estimation versus diverse shape variations. Experimental results on multiple datasets demonstrate that our proposed method outperforms existing state-of-the-art techniques without auxiliary data.

Lightweight 2D Human Pose Estimation Based on Joint Channel Coordinate Attention Mechanism

Traditional human pose estimation methods typically rely on complex models and algorithms. Lite-HRNet can achieve an excellent performance while reducing model complexity. However, its feature extraction scale is relatively single, which can lead to lower keypoints’ localization accuracy in crowded and complex scenes. To address this issue, we propose a lightweight human pose estimation model based on a joint channel coordinate attention mechanism. This model provides a powerful information interaction channel, enabling features of different resolutions to interact more effectively. This interaction can solve the problem of human pose estimation in complex scenes and improve the robustness and accuracy of the pose estimation model. The introduction of the joint channel coordinate attention mechanism enables the model to more effectively retain key information, thereby enhancing keypoints’ localization accuracy. We also redesign the lightweight basic module using the shuffle module and the joint channel coordinate attention mechanism to replace the spatial weight calculation module in the original Lite-HRNet model. By introducing this new module, we not only improve the network calculation speed and reduce the number of parameters of the entire model, but also ensure the accuracy of the model, thereby achieving a balance between performance and efficiency. We compare this model with current mainstream methods on the COCO and MPII dataset. The experimental results show that this model can effectively reduce the number of parameters and computational complexity while ensuring high model accuracy.

AttentionPose: Attention-driven end-to-end model for precise 6D pose estimation

Abstract Addressing the complex problem of 6D pose estimation from single RGB images is essential for robotics, augmented reality, and autonomous driving applications. The aim of this study is to overcome limitations in handling scenes with high object occlusion and clutter. We introduce an attention-driven end-to-end model that builds upon existing methods employing pixel-wise unit vectors and voting for object keypoints. Integrating attention mechanisms allows the model to focus computational resources on salient features, enhancing accuracy. Experimental results using the LINEMOD benchmark dataset demonstrate an accuracy rate of 99.73%, outperforming state-of-the-art approaches. The model also exhibits strong generalization capabilities, achieving an average accuracy of 97.36% on objects not included in the dataset. This work concludes that the attention mechanism significantly elevates the performance and robustness of 6D pose estimation, particularly in challenging environments, and opens new avenues for real-world applications.

Optimal Pose Estimation and Covariance Analysis with Simultaneous Localization and Mapping Applications

This work provides a theoretical analysis for optimally solving the pose estimation problem using total-least-squares for vector observations from landmark features, which is central to applications involving simultaneous localization and mapping. First, the optimization process is formulated with observation vectors extracted from point-cloud features. Then, error-covariance expressions are derived. The attitude and position estimates obtained via the derived optimization process are proven to reach the bounds defined by the Cramér–Rao lower bound under the small-angle approximation of attitude errors. A fully populated observation noise-covariance matrix is assumed as the weight in the cost function to cover the most general case of the sensor uncertainty. This includes more generic correlations in the errors than previous cases involving an isotropic noise assumption. The proposed solution is verified using Monte Carlo simulations, a Gazebo simulation in a robotics operating system, and an experiment with an actual light detection and ranging sensor to validate the error-covariance analysis.