3D Human Pose Research Articles

Interactive pose and shape editing with simple sketches from different viewing angles

The challenge of generating and creating humanoid 3D models is nowadays omnipresent in animation films, video games, and many other applications. To enhance and speed up this costly process, we propose an interactive method to infer 3D human poses and shapes from simple free-hand sketches. In contrast to previous work, our sketches are easy and fast to draw even for inexperienced users. Reconstructing the human 3D pose and shape from a single sketch is difficult due to the perspective projection resulting in ambiguous representations. We approach this challenge by adding a novel multi-view sketch interface for human pose estimation. Our simple sketch style allows for fast and intuitive, iterative refinement of not only the pose but also the shape using simple 2D interactions. The pose is inferred from a pre-trained neural network that is trained on a self-generated data set of sketches, derived by sampling 3D STAR models and rendering them in a sketch style. We conduct a user study to evaluate the accuracy, intuitiveness, and usability of our system in a real-world scenario. With our novel multi-view input system, we can effectively resolve ambiguities inherent to single sketch approaches while still delivering a fast and intuitive user experience.

Motion Projection Consistency-Based 3-D Human Pose Estimation With Virtual Bones From Monocular Videos

Real-time 3D human pose estimation is crucial for human-computer interaction. It is cheap and practical to estimate 3D human pose only from monocular video. However, recent bone splicing based 3D human pose estimation method brings about the problem of cumulative error. In this paper, the concept of virtual bones is proposed to solve such a challenge. The virtual bones are imaginary bones between non-adjacent joints. They do not exist in reality, but they bring new loop constraints for the estimation of 3D human joints. The proposed network in this paper predicts real bones and virtual bones, simultaneously. The final length of real bones is constrained and learned by the loop constructed by the predicted real bones and virtual bones. Besides, the motion constraints of joints in consecutive frames are considered. The consistency between the 2D projected position displacement predicted by the network and the captured real 2D displacement by the camera is proposed as a new projection consistency loss for the learning of 3D human pose. The experiments on the Human3.6M dataset demonstrate the good performance of the proposed method. Ablation studies demonstrate the effectiveness of the proposed inter-frame projection consistency constraints and intra-frame loop constraints.

Deep Reconstruction of 3-D Human Poses From Video

Deep Learning advances have made it possible to recover full 3D meshes of human models from individual images. However, extension of this notion to videos for recovering temporally coherent poses is still under-explored. A major challenge in this direction is the lack of appropriately annotated video data for learning the desired computational models. Existing human pose datasets only provide 2D or 3D skeleton joint annotations, whereas the datasets are also insufficiently recorded in constrained environments. We first contribute a technique to synthesize monocular action videos with rich 3D annotations that are suitable for learning computational models for full mesh 3D human pose recovery. Compared to the existing methods which simply ‘`texture-map’' clothes onto the 3D human pose models, our approach incorporates Physics-based realistic cloth deformations with human body movements. The generated videos cover a large variety of human actions, poses, and visual appearances, while the annotations record accurate human pose dynamics and human body surface information. Our second major contribution is an end-to-end trainable Recurrent Neural Network for full pose mesh recovery from monocular video. Using the proposed video data and a Long-Short-Term-Memory recurrent structure, our network explicitly learns to model the temporal coherence in videos and imposes geometric consistency over the recovered meshes. We establish the effectiveness of the proposed model with quantitative and qualitative analysis using the proposed and benchmark datasets.

Ego-Body Pose Estimation via Ego-Head Pose Estimation

Estimating 3D human motion from an ego-centric video, which records the environment viewed from the first-person perspective with a front-facing monocular camera, is critical to applications in VR/AR. However, naively learning a mapping between egocentric videos and full-body human motions is challenging for two reasons. First, modeling this complex relationship is difficult; unlike reconstruction motion from third-person videos, the human body is often out of view of an egocentric video. Second, learning this mapping requires a large-scale, diverse dataset containing paired egocentric videos and the corresponding 3D human poses. Creating such a dataset requires meticulous instrumentation for data acquisition, and unfortunately, such a dataset does not currently exist. As such, existing works have only worked on small-scale datasets with limited motion and scene diversity (yuan20183d; yuan2019ego; luo2021dynamics).

Monocular 3D Human Pose Markerless Systems for Gait Assessment.

Gait analysis plays an important role in the fields of healthcare and sports sciences. Conventional gait analysis relies on costly equipment such as optical motion capture cameras and wearable sensors, some of which require trained assessors for data collection and processing. With the recent developments in computer vision and deep neural networks, using monocular RGB cameras for 3D human pose estimation has shown tremendous promise as a cost-effective and efficient solution for clinical gait analysis. In this paper, a markerless human pose technique is developed using motion captured by a consumer monocular camera (800 × 600 pixels and 30 FPS) for clinical gait analysis. The experimental results have shown that the proposed post-processing algorithm significantly improved the original human pose detection model (BlazePose)'s prediction performance compared to the gold-standard gait signals by 10.7% using the MoVi dataset. In addition, the predicted T2 score has an excellent correlation with ground truth (r = 0.99 and y = 0.94x + 0.01 regression line), which supports that our approach can be a potential alternative to the conventional marker-based solution to assist the clinical gait assessment.

Reconstructing 3D human pose and shape from a single image and sparse IMUs.

This article presents an approach to improve 3D human pose estimation by fusing a single image with sparse inertial measurement units (IMUs). Based on a dual-stream feature extract network, we design a model-attention network with a residual module to closely couple the dual-modal feature from a static image and sparse inertial measurement units. The final 3D pose and shape parameters are directly obtained by a regression strategy. Extensive experiments are conducted on two benchmark datasets for 3D human pose estimation. Compared to state-of-the-art methods, the per vertex error (PVE) of human mesh reduces by 9.4 mm on Total Capture dataset and the mean per joint position error (MPJPE) reduces by 7.8 mm on the Human3.6M dataset. The quantitative comparison demonstrates that the proposed method could effectively fuse sparse IMU data and images and improve pose accuracy.

SSP-Net: Scalable sequential pyramid networks for real-Time 3D human pose regression

In this paper we propose a highly scalable convolutional neural networks, end-to-end trainable, for real-time 3D human pose regression from still RGB images. We call this approach Scalable Sequential Pyramid Networks (SSP-Net) as it is trained with refined supervision at multiple scales in a sequential manner. Our network requires a single training procedure and is capable of producing its best predictions at 120 frames per second (FPS), or acceptable predictions at more than 200 FPS when cut at test time. We show that the proposed regression approach is invariant to the size of feature maps, allowing our method to perform multi-resolution intermediate supervisions and reaching results comparable to the state-of-the-art with very low resolution feature maps. We demonstrate the accuracy and the effectiveness of our method by providing extensive experiments on two of the most important publicly available datasets for 3D pose estimation, Human3.6M and MPI-INF-3DHP. Additionally, we provide relevant insights about our decisions on the network architecture and show its flexibility to meet the best precision-speed compromise.

A monocular 3D human pose estimation approach for virtual character skeleton retargeting

A monocular 3D human pose estimation approach for virtual character skeleton retargeting

PoseGU: 3D human pose estimation with novel human pose generator and unbiased learning

3D pose estimation has recently gained substantial interests in computer vision domain. Existing 3D pose estimation methods have a strong reliance on large size well-annotated 3D pose datasets, and they suffer poor model generalization on unseen poses due to limited diversity of 3D poses in training sets. In this work, we propose PoseGU, a novel human pose generator that generates diverse poses with access only to a small size of seed samples, while equipping the Counterfactual Risk Minimization to pursue an unbiased evaluation objective. Extensive experiments demonstrate PoseGU outperforms almost all the state-of-the-art 3D human pose methods under consideration over three popular benchmark datasets. Empirical analysis also proves PoseGU generates 3D poses with improved data diversity and better generalization ability.

3D human pose and shape estimation via de-occlusion multi-task learning

Three-dimensional human pose and shape estimation is to compute a full human 3D mesh given a single image. The contamination of features caused by occlusion usually degrades its performance significantly. Recent progress in this field typically addressed the occlusion problem implicitly. By contrast, in this paper, we address it explicitly using a simple yet effective de-occlusion multi-task learning network. Our key insight is that feature for mesh parameter regression should be noiseless. Thus, in the feature space, our method disentangles the occludee that represents the noiseless human feature from the occluder. Specifically, a spatial regularization and an attention mechanism are imposed in the backbone of our network to disentangle the features into different channels. Furthermore, two segmentation tasks are proposed to supervise the de-occlusion process. The final mesh model is regressed by the disentangled occlusion-aware features. Experiments on both occlusion and non-occlusion datasets are conducted, and the results prove that our method is superior to the state-of-the-art methods on two occlusion datasets, while achieving competitive performance on a non-occlusion dataset. We also demonstrate that the proposed de-occlusion strategy is the main factor to improve the robustness against occlusion. The code is available athttps://github.com/qihangran/De-occlusion_MTL_HMR.

Seeing the unseen: Wifi-based 2D human pose estimation via an evolving attentive spatial-Frequency network

Camera-based human pose estimation has become popular due to its wide applications and easy implementation. However, it is not applicable under several circumstances such as poor illumination, occlusion, and private protection. In this work, we utilize Wifi signals to estimate 2D human poses, which is challenging because Wifi signals are abstract and contain limited information. To address these challenges, we develop an evolving attentive spatial-frequency network to discover the relationship between signal variation and body movement for Wifi-based 2D human pose estimation. By first taking dilated CSI sequences as inputs, a spatial-frequency encoder is then introduced to effectively integrate static spatial information and dynamic frequency information from CSI signals. Finally, we design an evolving attention module to enable our model to attend to certain channels of features. Due to a lack of benchmarks, we propose two Wifi-based human pose estimation datasets, the General Pose Estimation dataset (GPE) and Specific Pose Estimation dataset (SPE), which have been released as a public download at project page. Extensive experiments on the proposed datasets show that our model outperforms the state-of-the-art method by at least 16% in terms of PCK@20 (percentage of correct keypoints).

Scene‐Aware 3D Multi‐Human Motion Capture from a Single Camera

AbstractIn this work, we consider the problem of estimating the 3D position of multiple humans in a scene as well as their body shape and articulation from a single RGB video recorded with a static camera. In contrast to expensive marker‐based or multi‐view systems, our lightweight setup is ideal for private users as it enables an affordable 3D motion capture that is easy to install and does not require expert knowledge. To deal with this challenging setting, we leverage recent advances in computer vision using large‐scale pre‐trained models for a variety of modalities, including 2D body joints, joint angles, normalized disparity maps, and human segmentation masks. Thus, we introduce the first non‐linear optimization‐based approach that jointly solves for the 3D position of each human, their articulated pose, their individual shapes as well as the scale of the scene. In particular, we estimate the scene depth and person scale from normalized disparity predictions using the 2D body joints and joint angles. Given the per‐frame scene depth, we reconstruct a point‐cloud of the static scene in 3D space. Finally, given the per‐frame 3D estimates of the humans and scene point‐cloud, we perform a space‐time coherent optimization over the video to ensure temporal, spatial and physical plausibility. We evaluate our method on established multi‐person 3D human pose benchmarks where we consistently outperform previous methods and we qualitatively demonstrate that our method is robust to in‐the‐wild conditions including challenging scenes with people of different sizes. Code: https://github.com/dluvizon/scene‐aware‐3d‐multi‐human

Bidirectional temporal feature for 3D human pose and shape estimation from a video

Abstract3D human pose and shape estimation is the foundation of analyzing human motion. However, estimating accurate and temporally consistent 3D human motion from a video remains a challenge. By now, most of the video‐based methods for estimating 3D human pose and shape rely on unidirectional temporal features and lack more comprehensive information. To solve this problem, we propose a novel model “bidirectional temporal feature for human motion recovery” (BTMR), which consists of a human motion generator and a discriminator. The transformer‐based generator effectively captures the forward and reverse temporal features to enhance the temporal correlation between frames and reduces the loss of spatial information. The motion discriminator based on Bi‐LSTM can distinguish whether the generated pose sequences are consistent with the realistic sequences of the AMASS dataset. In the process of continuous generation and discrimination, the model can learn more realistic and accurate poses. We evaluate our BTMR on 3DPW and MPI‐INF‐3DHP datasets. Without the training set of 3DPW, BTMR outperforms VIBE by 2.4 mm and 14.9 mm/s2 in PA‐MPJPE and Accel metrics and outperforms TCMR by 1.7 mm in PA‐MPJPE metric on 3DPW. The results demonstrate that our BTMR produces better accurate and temporal consistent 3D human motion.

Privacy preserving for AI-based 3D human pose recovery and retargeting

As an essential research task in artificial intelligence (AI), the estimation of 3D human poses has important application value in virtual reality, medical diagnosis, athlete training and other fields. However, human pose recovery and retargeting require the acquisition of detailed visual data containing private information, which has led to increasing concerns about user privacy and security. Therefore, we build a lightweight framework, called Human Motion Parameters Prediction (HMPP), which can infer the 3D mesh and 3D skeletal joint points of the human body while protecting the privacy of the user. The proposed method successfully reduces or suppresses privacy attributes while ensuring important features to perform human pose estimation. The 2D and 3D joints are used for supervision to improve the interpretability of the model at each stage. In addition, the prediction of the camera’s internal parameters is added so that the model can be augmented with projection supervision, thereby using more 2D datasets for training and improving the generalization ability of the model. Finally, the predicted motion parameters are used for 3D reconstruction and motion retargeting. Experiments show that our approach can achieve excellent evaluation results on multiple datasets and avoid inadvertently compromising private and sensitive data.

Multi-hypothesis representation learning for transformer-based 3D human pose estimation

Despite significant progress, estimating 3D human poses from monocular videos remains a challenging task due to depth ambiguity and self-occlusion. Most existing works attempt to solve both issues by exploiting spatial and temporal relationships. However, those works ignore the fact that it is an inverse problem where multiple feasible solutions (i.e., hypotheses) exist. To relieve this limitation, we propose a Multi-Hypothesis Transformer that learns spatio-temporal representations of multiple plausible pose hypotheses. In order to effectively model multi-hypothesis dependencies and build strong relationships across hypothesis features, we introduce a one-to-many-to-one three-stage framework: (i) Generate multiple initial hypothesis representations; (ii) Model self-hypothesis communication, merge multiple hypotheses into a single converged representation and then partition it into several diverged hypotheses; (iii) Learn cross-hypothesis communication and aggregate the multi-hypothesis features to synthesize the final 3D pose. Through the above processes, the final representation is enhanced and the synthesized pose is much more accurate. Extensive experiments show that the proposed method achieves state-of-the-art results on two challenging datasets: Human3.6M and MPI-INF-3DHP. The code and models are available at https://github.com/Vegetebird/MHFormer.

Three-Dimensional Human Pose Estimation with Spatial–Temporal Interaction Enhancement Transformer

Three-dimensional human pose estimation is a hot research topic in the field of computer vision. In recent years, significant progress has been made in estimating 3D human pose from monocular video, but there is still much room for improvement in this task owing to the issues of self-occlusion and depth ambiguity. Some previous work has addressed the above problems by investigating spatio-temporal relationships and has made great progress. Based on this, we further explored the spatio-temporal relationship and propose a new method, called STFormer. Our whole framework consists of two main stages: (1) extract features independently from the temporal and spatial domains; (2) modeling the communication of information across domains. The temporal dependencies were injected into the spatial domain to dynamically modify the spatial structure relationships between joints. Then, the results were used to refine the temporal features. After the preceding steps, both spatial and temporal features were strengthened, and the estimated final pose will be more precise. We conducted substantial experiments on a well-known dataset (Human3.6), and the results indicated that STFormer outperformed recent methods with an input of nine frames. Compared to PoseFormer, the performance of our method reduced the MPJPE by 2.1%. Furthermore, we performed numerous ablation studies to analyze and prove the validity of the various constituent modules of STFormer.

HSGNet: hierarchically stacked graph network with attention mechanism for 3D human pose estimation

HSGNet: hierarchically stacked graph network with attention mechanism for 3D human pose estimation

Evaluation metrics systematization for 2D human poses analysis models

This paper describes the systematization of evaluation metrics for 2D human pose analysis models. Some of the most popular tasks solved using machine learning (ML) methods are detection, tracking and recognition of human actions for various practical applications. There are a lot of different metrics that allow evaluating the model from one point or another. To evaluate a specific task, a certain set of metrics is used. However, as literature analysis shows, the vast number of metric definitions, as well as the use of different terms and multiple representations of the same ideas, causes problems of interpretation and comparison of different ML models and methods in detecting, tracking, and recognizing human actions. The purpose of this work is to analyze the metrics for evaluating methods for processing 2D human poses in video in order to facilitate the informed choice of the metrics. To improve the objectivity of evaluating the results of empirical studies of existing and newly developed methods and models for detecting, tracking, and recognizing human actions, a systematization of existing metrics into subgroups was proposed, depending on what task they evaluate. Four classes of evaluation metrics were introduced: classification metrics, key point’s detection, object tracking, and general metrics. Classification metrics are based on quality evaluation and matching values from predicted bounding boxes with ground truths. Key point’s detection metrics are oriented on the quality of found joints of the human body skeleton. Tracking metrics evaluate the object detection on each frame and the correctness of determining its trajectory. General metrics are not specifically related to any of the human 2D pose analysis tasks. The prototype of the application based on suggested metrics systematization, the purpose of which is to help data scientists in formalizing the choice of metrics for evaluating models depending on the ML problem being solved and the application area was developed. To evaluate and demonstrate the metrics, that were suggested in this application, Faster R-CNN, SSD and YOLOv3 object detection models were analyzed and compared in scope of 2D human pose analysis application area. The results of the analysis showed that Faster R-CNN and YOLOv3 have the most accurate responses, although they have the disadvantage of a high False positive rate. The implementation also showed that metrics that based on True negative values are uninformative in scope of working with bounding boxes, because of the specific of application area and inability to calculate True negatives on the image data.

Real-time 3D human pose estimation without skeletal a priori structures

This study is about real-time 2D-3D human pose estimation without using the a priori structure of the skeleton and with a low number of parameters for regression tasks. Current graph convolution-based 3D human pose tasks require structural knowledge of the skeleton, which limits the exploration of pose estimation for unknown structures. Inspired by tyre rotation and circular convolution, weights rotation is used to fully learn the potential connections between the human joints. We refer to this process as the weight cyclic sharing mechanism, a novel method for updating features. It does not require knowledge of the structure of the human skeleton and learns different constraints between joints with a low number of parameters. We propose an end-to-end weight circular sharing network (WCirSNet) based on the weight circular sharing mechanism. We propose a simple and efficient weighted residual block in this WCirSNet. The superiorities of the weight circular sharing mechanism and weighted residual block were verified by abundant ablation studies. Extensive evaluations on two challenging benchmark datasets (Human 3.6 M, MPI-INF-3DHP) show that the performance and generalization capabilities of our framework are superior to the results of many previously advanced methods.

Adaptive Multi-view and Temporal Fusing Transformer for 3D Human Pose Estimation.

This paper proposes a unified framework dubbed Multi-view and Temporal Fusing Transformer (MTF-Transformer) to adaptively handle varying view numbers and video length without camera calibration in 3D Human Pose Estimation (HPE). It consists of Feature Extractor, Multi-view Fusing Transformer (MFT), and Temporal Fusing Transformer (TFT). Feature Extractor estimates 2D pose from each image and fuses the prediction according to the confidence. It provides pose-focused feature embedding and makes subsequent modules computationally lightweight. MFT fuses the features of a varying number of views with a novel Relative-Attention block. It adaptively measures the implicit relative relationship between each pair of views and reconstructs more informative features. TFT aggregates the features of the whole sequence and predicts 3D pose via a transformer. It adaptively deals with the video of arbitrary length and fully unitizes the temporal information. The migration of transformers enables our model to learn spatial geometry better and preserve robustness for varying application scenarios. We report quantitative and qualitative results on the Human3.6M, TotalCapture, and KTH Multiview Football II. Compared with state-of-the-art methods with camera parameters, MTF-Transformer obtains competitive results and generalizes well to dynamic capture with an arbitrary number of unseen views.