3D Human Pose Research Articles

A Systematic Review of Recent Deep Learning Approaches for 3D Human Pose Estimation.

Three-dimensional human pose estimation has made significant advancements through the integration of deep learning techniques. This survey provides a comprehensive review of recent 3D human pose estimation methods, with a focus on monocular images, videos, and multi-view cameras. Our approach stands out through a systematic literature review methodology, ensuring an up-to-date and meticulous overview. Unlike many existing surveys that categorize approaches based on learning paradigms, our survey offers a fresh perspective, delving deeper into the subject. For image-based approaches, we not only follow existing categorizations but also introduce and compare significant 2D models. Additionally, we provide a comparative analysis of these methods, enhancing the understanding of image-based pose estimation techniques. In the realm of video-based approaches, we categorize them based on the types of models used to capture inter-frame information. Furthermore, in the context of multi-person pose estimation, our survey uniquely differentiates between approaches focusing on relative poses and those addressing absolute poses. Our survey aims to serve as a pivotal resource for researchers, highlighting state-of-the-art deep learning strategies and identifying promising directions for future exploration in 3D human pose estimation.

PosturePose: Optimized Posture Analysis for Semi-Supervised Monocular 3D Human Pose Estimation.

One motivation for studying semi-supervised techniques for human pose estimation is to compensate for the lack of variety in curated 3D human pose datasets by combining labeled 3D pose data with readily available unlabeled video data-effectively, leveraging the annotations of the former and the rich variety of the latter to train more robust pose estimators. In this paper, we propose a novel, fully differentiable posture consistency loss that is unaffected by camera orientation and improves monocular human pose estimators trained with limited labeled 3D pose data. Our semi-supervised monocular 3D pose framework combines biomechanical pose regularization with a multi-view posture (and pose) consistency objective function. We show that posture optimization was effective at decreasing pose estimation errors when applied to a 2D-3D lifting network (VPose3D) and two well-studied datasets (H36M and 3DHP). Specifically, the proposed semi-supervised framework with multi-view posture and pose loss lowered the mean per-joint position error (MPJPE) of leading semi-supervised methods by up to 15% (-7.6 mm) when camera parameters of unlabeled poses were provided. Without camera parameters, our semi-supervised framework with posture loss improved semi-supervised state-of-the-art methods by 17% (-15.6 mm decrease in MPJPE). Overall, our pose models compete favorably with other high-performing pose models trained under similar conditions with limited labeled data.

From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans

Great progress has been made in estimating 3D human pose and shape from images and video by training neural networks to directly regress the parameters of parametric human models like SMPL. However, existing body models have simplified kinematic structures that do not correspond to the true joint locations and articulations in the human skeletal system, limiting their potential use in biomechanics. On the other hand, methods for estimating biomechanically accurate skeletal motion typically rely on complex motion capture systems and expensive optimization methods. What is needed is a parametric 3D human model with a biomechanically accurate skeletal structure that can be easily posed. To that end, we develop SKEL, which re-rigs the SMPL body model with a biomechanics skeleton. To enable this, we need training data of skeletons inside SMPL meshes in diverse poses. We build such a dataset by optimizing biomechanically accurate skeletons inside SMPL meshes from AMASS sequences. We then learn a regressor from SMPL mesh vertices to the optimized joint locations and bone rotations. Finally, we re-parametrize the SMPL mesh with the new kinematic parameters. The resulting SKEL model is animatable like SMPL but with fewer, and biomechanically-realistic, degrees of freedom. We show that SKEL has more biomechanically accurate joint locations than SMPL, and the bones fit inside the body surface better than previous methods. By fitting SKEL to SMPL meshes we are able to "upgrade" existing human pose and shape datasets to include biomechanical parameters. SKEL provides a new tool to enable biomechanics in the wild, while also providing vision and graphics researchers with a better constrained and more realistic model of human articulation. The model, code, and data are available for research at https://skel.is.tue.mpg.de.

A Geometric Knowledge Oriented Single-Frame 2D-to-3D Human Absolute Pose Estimation Method

As a critical part of the 3D human pose estimation (HPE), establishing the 2D-to-3D lifting mapping is limited by depth ambiguity. Most current works generally lack the quantitative analysis of the relative depth expression and the depth ambiguity error expression in lifting mapping, resulting in low prediction efficiency and poor interpretability. To this end, this paper mines and leverages prior geometric knowledge of these expressions based on the pinhole imaging principle, decoupling the 2D-to-3D lifting mapping and simplifying the model training. Specifically, this paper proposes a prior geometric knowledge oriented pose estimation model with two-branch transformer architectures, explicitly introducing high-dimensional prior geometric features to improve model efficiency and interpretability. It converts the regression of spatial coordinates into the prediction of spatial direction vectors between joints to generate multiple feasible solutions further alleviate the depth ambiguity. Moreover, this paper raises a novel non-learning-based absolute depth estimation algorithm based on prior geometric relationship decoupling from relative depth expression for the first time. It establishes multiple independent depth mapping from non-root nodes to the root node to calculate the absolute depth candidate, which is parameter-free, plug-and-play, and interpretable. Experiments show that the proposed pose estimation model achieves state-of-the-art performance on Human 3.6M and MPI-INF-3DHP benchmarks with lower parameters and faster inference speed, and the proposed absolute depth estimation algorithm achieves similar performance to traditional methods without any network parameters. The source code are available at https://github.com/Humengxian/GKONet.

Human Motion Prediction Based on a Multi-Scale Hypergraph for Intangible Cultural Heritage Dance Videos

Compared to traditional dance, intangible cultural heritage dance often involves the isotropic extension of choreographic actions, utilizing both upper and lower limbs. This characteristic choreography style makes the remote joints lack interaction, consequently reducing accuracy in existing human motion prediction methods. Therefore, we propose a human motion prediction method based on the multi-scale hypergraph convolutional network of the intangible cultural heritage dance video. Firstly, this method inputs the 3D human posture sequence from intangible cultural heritage dance videos. The hypergraph is designed according to the synergistic relationship of the human joints in the intangible cultural heritage dance video, which is used to represent the spatial correlation of the 3D human posture. Then, a multi-scale hypergraph convolutional network is constructed, utilizing multi-scale transformation operators to segment the human skeleton into different scales. This network adopts a graph structure to represent the 3D human posture at different scales, which is then used by the single-scalar fusion operator to spatial features in the 3D human posture sequence are extracted by fusing the feature information of the hypergraph and the multi-scale graph. Finally, the Temporal Graph Transformer network is introduced to capture the temporal dependence among adjacent frames within the time domain. This facilitates the extraction of temporal features from the 3D human posture sequence, ultimately enabling the prediction of future 3D human posture sequences. Experiments show that we achieve the best performance in both short-term and long-term human motion prediction when compared to Motion-Mixer and Motion-Attention algorithms on Human3.6M and 3DPW datasets. In addition, ablation experiments show that our method can predict more precise 3D human pose sequences, even in the presence of isotropic extensions of upper and lower limbs in intangible cultural heritage dance videos. This approach effectively addresses the issue of missing segments in intangible cultural heritage dance videos.

Skeleton-based 3D human pose estimation with low-resolution infrared array sensor using attention based CNN-BiGRU

Skeleton-based 3D human pose estimation with low-resolution infrared array sensor using attention based CNN-BiGRU

JoyPose: Jointly learning evolutionary data augmentation and anatomy-aware global–local representation for 3D human pose estimation

Video-based 3D human pose estimation is an important yet challenging task for many human-involved pattern recognition systems. Existing deep learning-based 3D human pose estimation methods are faced with the problems of lacking large-scale training data and lacking effective solutions to represent the complicated human body structure. To this end, this paper proposes a jointly learning framework entitled JoyPose that simultaneously leverages both human pose data augmentation and human pose estimation. In particular, JoyPose consists of an evolutionary data augmentation module and an anatomy-aware global–local pose feature representation module for 3D human pose estimation. The evolution for data augmentation is guided by a reinforcement learning strategy in a probabilistic way according to pose estimation loss. The anatomy-aware global–local pose feature representation module separately captures global features and local features according to anatomical and kinematic patterns observed from pose estimation errors across different human joints. The performance of the final human pose estimation is leveraged by both data augmentation and anatomy-aware global–local feature representation. Extensive experiments on three real-world datasets demonstrate the superiority and robustness against state-of-the-art methods.

MPA-GNet: multi-scale parallel adaptive graph network for 3D human pose estimation

MPA-GNet: multi-scale parallel adaptive graph network for 3D human pose estimation

Wearable Motion Capture: Reconstructing and Predicting 3D Human Poses from Wearable Sensors.

Reconstructing and predicting 3D human walking poses in unconstrained measurement environments have the potential to use for health monitoring systems for people with movement disabilities by assessing progression after treatments and providing information for assistive device controls. The latest pose estimation algorithms utilize motion capture systems, which capture data from IMU sensors and third-person view cameras. However, third-person views are not always possible for outpatients alone. Thus, we propose the wearable motion capture problem of reconstructing and predicting 3D human poses from the wearable IMU sensors and wearable cameras, which aids clinicians' diagnoses on patients out of clinics. To solve this problem, we introduce a novel Attention-Oriented Recurrent Neural Network (AttRNet) that contains a sensor-wise attention-oriented recurrent encoder, a reconstruction module, and a dynamic temporal attention-oriented recurrent decoder, to reconstruct the 3D human pose over time and predict the 3D human poses at the following time steps. To evaluate our approach, we collected a new WearableMotionCapture dataset using wearable IMUs and wearable video cameras, along with the musculoskeletal joint angle ground truth. The proposed AttRNet shows high accuracy on the new lower-limb WearableMotionCapture dataset, and it also outperforms the state-of-the-art methods on two public full-body pose datasets: DIP-IMU and TotalCaputre.

Recognition Method with Deep Contrastive Learning and Improved Transformer for 3D Human Motion Pose

Three-dimensional (3D) human pose recognition techniques based on spatial data have gained attention. However, existing models and algorithms fail to achieve desired precision. We propose a 3D human motion pose recognition method using deep contrastive learning and an improved Transformer. The improved Transformer removes noise between human motion RGB and depth images, addressing orientation correlation in 3D models. Two-dimensional (2D) pose features are extracted from de-noised RGB images using a kernel generation module in a graph convolutional network (GCN). Depth features are extracted from de-noised depth images. The 2D pose features and depth features are fused using a regression module in the GCN to obtain 3D pose recognition results. The results demonstrate that the proposed method captures RGB and depth images, achieving high recognition accuracy and fast speed. The proposed method demonstrates good accuracy in 3D human motion pose recognition.

Vertex position estimation with spatial–temporal transformer for 3D human reconstruction

Reconstructing 3D human pose and body shape from monocular images or videos is a fundamental task for comprehending human dynamics. Frame-based methods can be broadly categorized into two fashions: those regressing parametric model parameters (e.g., SMPL) and those exploring alternative representations (e.g., volumetric shapes, 3D coordinates). Non-parametric representations have demonstrated superior performance due to their enhanced flexibility. However, when applied to video data, these non-parametric frame-based methods tend to generate inconsistent and unsmooth results. To this end, we present a novel approach that directly regresses the 3D coordinates of the mesh vertices and body joints with a spatial–temporal Transformer. In our method, we introduce a SpatioTemporal Learning Block (STLB) with Spatial Learning Module (SLM) and Temporal Learning Module (TLM), which leverages spatial and temporal information to model interactions at a finer granularity, specifically at the body token level. Our method outperforms previous state-of-the-art approaches on Human3.6M and 3DPW benchmark datasets.

GolfMate: Enhanced Golf Swing Analysis Tool through Pose Refinement Network and Explainable Golf Swing Embedding for Self-Training

Digital fitness has become a widely used tool for remote exercise guidance, leveraging artificial intelligence to analyze exercise videos and support self-training. This paper introduces a method for self-training in golf, a sport where automated posture analysis can significantly reduce the costs associated with professional coaching. Our system utilizes a pose refinement methodology and an explainable golf swing embedding for analyzing the swing motions of learners and professional golfers. By leveraging sequential coordinate information, we detect biased pose joints and refine the 2D and 3D human pose estimation results. Furthermore, we propose a swing embedding method that considers geometric information extracted from the swing pose. This approach enables not only the comparison of the similarity between two golf swing poses but also the visualization of different points, providing learners with specific and intuitive feedback on areas that require correction. Our experimental results demonstrate the effectiveness of our swing guide system in identifying specific body points that need adjustment to align more closely with a professional golfer’s swing. This research contributes to the digital fitness domain by enhancing the accuracy of posture analysis and providing a specialized and interpretable golf swing analysis system. Our proposed system offers a low-cost and time-efficient approach for users who wish to improve their golf swing, paving the way for broader applications of digital fitness technologies in self-training contexts.

Automatic labeling of Parkinson's Disease gait videos with weak supervision.

Motor dysfunction in Parkinson's Disease (PD) patients is typically assessed by clinicians employing the Movement Disorder Society's Unified Parkinson's Disease Rating Scale (MDS-UPDRS). Such comprehensive clinical assessments are time-consuming, expensive, semi-subjective, and may potentially result in conflicting labels across different raters. To address this problem, we propose an automatic, objective, and weakly-supervised method for labeling PD patients' gait videos. The proposed method accepts videos of patients and classifies their gait scores as normal (Gait score in MDS-UPDRS = 0) or PD (MDS-UPDRS ≥ 1). Unlike previous work, the proposed method does not require a priori MDS-UPDRS ratings for training, utilizing only domain-specific knowledge obtained from neurologists. We propose several labeling functions that classify patients' gait and use a generative model to learn the accuracy of each labeling function in a self-supervised manner. Since results depended upon the estimated values of the patients' 3D poses, and existing pre-trained 3D pose estimators did not yield accurate results, we propose a weakly-supervised 3D human pose estimation method for fine-tuning pre-trained models in a clinical setting. Using leave-one-out evaluations, the proposed method obtains an accuracy of 89% on a dataset of 29 PD subjects - a significant improvement compared to previous work by 7%-10% depending upon the dataset. The method obtained state-of-the-art results on the Human3.6M dataset. Our results suggest that the use of labeling functions may provide a robust means to interpret and classify patient-oriented videos involving motor tasks.

Reconstructing 3D Human Pose from RGB‐D Data with Occlusions

AbstractWe propose a new method to reconstruct the 3D human body from RGB‐D images with occlusions. The foremost challenge is the incompleteness of the RGB‐D data due to occlusions between the body and the environment, leading to implausible reconstructions that suffer from severe human‐scene penetration. To reconstruct a semantically and physically plausible human body, we propose to reduce the solution space based on scene information and prior knowledge. Our key idea is to constrain the solution space of the human body by considering the occluded body parts and visible body parts separately: modeling all plausible poses where the occluded body parts do not penetrate the scene, and constraining the visible body parts using depth data. Specifically, the first component is realized by a neural network that estimates the candidate region named the “free zone”, a region carved out of the open space within which it is safe to search for poses of the invisible body parts without concern for penetration. The second component constrains the visible body parts using the “truncated shadow volume” of the scanned body point cloud. Furthermore, we propose to use a volume matching strategy, which yields better performance than surface matching, to match the human body with the confined region. We conducted experiments on the PROX dataset, and the results demonstrate that our method produces more accurate and plausible results compared with other methods.

Anatomy-guided domain adaptation for 3D in-bed human pose estimation.

3D human pose estimation is a key component of clinical monitoring systems. The clinical applicability of deep pose estimation models, however, is limited by their poor generalization under domain shifts along with their need for sufficient labeled training data. As a remedy, we present a novel domain adaptation method, adapting a model from a labeled source to a shifted unlabeled target domain. Our method comprises two complementary adaptation strategies based on prior knowledge about human anatomy. First, we guide the learning process in the target domain by constraining predictions to the space of anatomically plausible poses. To this end, we embed the prior knowledge into an anatomical loss function that penalizes asymmetric limb lengths, implausible bone lengths, and implausible joint angles. Second, we propose to filter pseudo labels for self-training according to their anatomical plausibility and incorporate the concept into the Mean Teacher paradigm. We unify both strategies in a point cloud-based framework applicable to unsupervised and source-free domain adaptation. Evaluation is performed for in-bed pose estimation under two adaptation scenarios, using the public SLP dataset and a newly created dataset. Our method consistently outperforms various state-of-the-art domain adaptation methods, surpasses the baseline model by 31%/66%, and reduces the domain gap by 65%/82%. Source code is available at https://github.com/multimodallearning/da-3dhpe-anatomy.

View consistency aware holistic triangulation for 3D human pose estimation

The rapid development of multi-view 3D human pose estimation (HPE) is attributed to the maturation of monocular 2D HPE and the geometry of 3D reconstruction. However, 2D detection outliers in occluded views due to neglect of view consistency, and 3D implausible poses due to lack of pose coherence, remain challenges. To solve this, we introduce a Multi-View Fusion module to refine 2D results by establishing view correlations. Then, Holistic Triangulation is proposed to infer the whole pose as an entirety, and anatomy prior is injected to maintain the pose coherence and improve the plausibility. Anatomy prior is extracted by PCA whose input is skeletal structure features, which can factor out global context and joint-by-joint relationship from abstract to concrete. Benefiting from the closed-form solution, the whole framework is trained end-to-end. Our method outperforms the state of the art in both precision and plausibility which is assessed by a new metric.

3D Human Pose Estimation: A Survey

This comprehensive review article explores the latest research advancements in the realm of estimating 3D human pose. Traditional methods such as PSM, SVM are discussed. Besides, this review also talks about deep learning-based approaches, including direct approaches, 2D-to-3D lifting and volumetric model approach for single person, top-down approaches and bottom-up approaches for multi-person pose estimation. The analysis covers the strengths and challenges of various methods, encompassing issues such as model generalization, occlusion robustness, and computational efficiency. Current research issues are identified, and future directions are proposed. By summarizing and evaluating existing methods, this paper aims to provide valuable insights for researchers in both academia and industry, driving the evolution of 3D human pose estimation for better practical applications.

Prior-knowledge-based self-attention network for 3D human pose estimation

Prior-knowledge-based self-attention network for 3D human pose estimation

Extracting spatial knowledge from track and field broadcasts for monocular 3D human pose estimation

Collecting large datasets for investigations into human locomotion is an expensive and labor-intensive process. Methods for 3D human pose estimation in the wild are becoming increasingly accurate and could soon be sufficient to assist with the collection of datasets for analysis into running kinematics from TV broadcast data. In the domain of biomechanical research, small differences in 3D angles play an important role. More precisely, the error margins of the data collection process need to be smaller than the expected variation between athletes. In this work, we propose a method to infer the global geometry of track and field stadium recordings using lane demarcations. By projecting estimated 3D skeletons back into the image using this global geometry, we show that current state-of-the-art 3D human pose estimation methods are not (yet) accurate enough to be used in kinematics research.

Deep Learning-based Human Pose Estimation: A Survey

Human pose estimation aims to locate the human body parts and build human body representation (e.g., body skeleton) from input data such as images and videos. It has drawn increasing attention during the past decade and has been utilized in a wide range of applications including human-computer interaction, motion analysis, augmented reality, and virtual reality. Although the recently developed deep learning-based solutions have achieved high performance in human pose estimation, there still remain challenges due to insufficient training data, depth ambiguities, and occlusion. The goal of this survey article is to provide a comprehensive review of recent deep learning-based solutions for both 2D and 3D pose estimation via a systematic analysis and comparison of these solutions based on their input data and inference procedures. More than 260 research papers since 2014 are covered in this survey. Furthermore, 2D and 3D human pose estimation datasets and evaluation metrics are included. Quantitative performance comparisons of the reviewed methods on popular datasets are summarized and discussed. Finally, the challenges involved, applications, and future research directions are concluded. A regularly updated project page is provided: https://github.com/zczcwh/DL-HPE .