3D Human Posture Research Articles

Three-Dimensional Posture Estimation of Vehicle Occupants Using Depth and Infrared Images.

Pose estimation is crucial for ensuring passenger safety and better user experiences in semi- and fully autonomous vehicles. Traditional methods relying on pose estimation from regular color images face significant challenges due to a lack of three-dimensional (3D) information and the sensitivity to occlusion and lighting conditions. Depth images, which are invariant to lighting issues and provide 3D information about the scene, offer a promising alternative. However, there is a lack of strong work in 3D pose estimation from such images due to the time-consuming process of annotating depth images with 3D postures. In this paper, we present a novel approach to 3D human posture estimation using depth and infrared (IR) images. Our method leverages a three-stage fine-tuning process involving simulation data, approximated data, and a limited set of manually annotated samples. This approach allows us to effectively train a model capable of accurate 3D pose estimation with a median error of under 10 cm across all joints, using fewer than 100 manually annotated samples. To the best of our knowledge, this is the first work focusing on vehicle occupant posture detection utilizing only depth and IR data. Our results demonstrate the feasibility and efficacy of this approach, paving the way for enhanced passenger safety in autonomous vehicle systems.

Proposal and validation of a new approach in tele-rehabilitation with 3D human posture estimation: a randomized controlled trial in older individuals with sarcopenia

ObjectiveThrough a randomized controlled trial on older adults with sarcopenia, this study compared the training effects of an AI-based remote training group using deep learning-based 3D human pose estimation technology with those of a face-to-face traditional training group and a general remote training group.MethodsSeventy five older adults with sarcopenia aged 60–75 from community organizations in Changchun city were randomly divided into a face-to-face traditional training group (TRHG), a general remote training group (GTHG), and an AI-based remote training group (AITHG). All groups underwent a 3-month program consisting of 24-form Taichi exercises, with a frequency of 3 sessions per week and each session lasting 40 min. The participants underwent Appendicular Skeletal Muscle Mass Index (ASMI), grip strength, 6-meter walking pace, Timed Up and Go test (TUGT), and quality of life score (QoL) tests before the experiment, during the mid-term, and after the experiment. This study used SPSS26.0 software to perform one-way ANOVA and repeated measures ANOVA tests to compare the differences among the three groups. A significance level of p < 0.05 was defined as having significant difference, while p < 0.01 was defined as having a highly significant difference.Results(1) The comparison between the mid-term and pre-term indicators showed that TRHG experienced significant improvements in ASMI, 6-meter walking pace, and QoL (p < 0.01), and a significant improvement in TUGT timing test (p < 0.05); GTHG experienced extremely significant improvements in 6-meter walking pace and QoL (p < 0.01); AITHG experienced extremely significant improvements in ASMI, 6-meter walking pace, and QoL (p < 0.01), and a significant improvement in TUGT timing test (p < 0.05). (2) The comparison between the post-term and pre-term indicators showed that TRHG experienced extremely significant improvements in TUGT timing test (p < 0.01); GTHG experienced significant improvements in ASMI and TUGT timing test (p < 0.05); and AITHG experienced extremely significant improvements in TUGT timing test (p < 0.01). (3) During the mid-term, there was no significant difference among the groups in all tests (p > 0.05). The same was in post-term tests (p > 0.05).ConclusionCompared to the pre-experiment, there was no significant difference at the post- experiment in the recovery effects on the muscle quality, physical activity ability, and life quality of patients with sarcopenia between the AI-based remote training group and the face-to-face traditional training group. 3D pose estimation is equally as effective as traditional rehabilitation methods in enhancing muscle quality, functionality and life quality in older adults with sarcopenia.Trial registrationThe trial was registered in ClinicalTrials.gov (NCT05767710).

Geometry-driven self-supervision for 3D human pose estimation

Although 3D human pose estimation has recently made strides, it is still difficult to precisely recreate a 3D human posture from a single image without the aid of 3D annotation for the following reasons. Firstly, the process of reconstruction inherently suffers from ambiguity, as multiple 3D poses can be projected onto the same 2D pose. Secondly, accurately measuring camera rotation without laborious camera calibration is a difficult task. While some approaches attempt to address these issues using traditional computer vision algorithms, they are not differentiable and cannot be optimized through training. This paper introduces two modules that explicitly leverage geometry to overcome these challenges, without requiring any 3D ground-truth or camera parameters. The first module, known as the relative depth estimation module, effectively mitigates depth ambiguity by narrowing down the possible depths for each joint to only two candidates. The second module, referred to as the differentiable pose alignment module, calculates camera rotation by aligning poses from different views. The use of these geometrically interpretable modules reduces the complexity of training and yields superior performance. By adopting our proposed method, we achieve state-of-the-art results on standard benchmark datasets, surpassing other self-supervised methods and even outperforming several fully-supervised approaches that heavily rely on 3D annotations.

Design of Basketball Teaching Aid System Based on Artificial Intelligence

Abstract Artificial intelligence has penetrated every aspect of people’s lives with the rapid development of science and technology. In basketball, how to effectively utilize artificial intelligence technology to improve the effectiveness of training has become a problem worth studying. A Mediapipe-based 3D human posture domain adaptive detection method is proposed in the study by combining statistics and human limb proportion simulation. The modified SVM classification algorithm is used to compare and classify the newly acquired key point data information of the system with the standard basketball action key point data information preset by the system to judge whether the basketball action is up to the standard or not, so as to design the basketball teaching aid system based on artificial intelligence. Based on this basis, after testing the recognition performance of the system, its practical application effect is explored. The total basketball scores improved after the experiment (86.26±2.228 for the experimental group and 81.34±9.672 for the control group), but there was no significant difference (P&gt;0.05). The control group was slower than the experimental group by 0.64 seconds and 0.77 seconds overall in the cross-step breakout time with the ball and the same-side step breakout time with the ball, respectively, indicating that the experimental group was more proficient in basketball technology. The system has been tested in various ways to meet the multiple needs of the users, meet the expected standards, and be able to be applied in real-world scenarios.

Research on the Application of Virtual Reality Technology in Motor Skills Training

Abstract The emerging efficacy of Virtual Reality (VR) technologies in sports training is explored in this study. We developed a VR-based motor skill training model utilizing a markerless optical motion capture system. This system employs a domain adaptive detection method in the 3D human posture realm, enabling precise identification of motor action postural feature points in students. Subsequently, the model incorporates a human dynamic motion tracking function that minimizes computational errors in motion capture data, facilitating the implementation of multi-feature fusion-based virtual digital human body motion capture technology. An empirical analysis was conducted using H College University as the case study to evaluate the effectiveness of virtual motor skill training. The findings demonstrate that the average accuracy of the proposed method for recognizing body gestures achieved a significant level of 0.834, enabling more accurate identification of students’ motor action characteristics. Moreover, the post-test results of the VR class showed marked improvements, ranging from 0.56 to 15.56 points over traditional lecture classes and 0.43 to 10.33 points over video teaching classes. These substantial variances not only underline the superiority of the VR-based training model in enhancing academic interest and physical fitness but also in advancing motor skills among learners. The study accentuates VR’s potential to augment motor skills and foresees its expansive application in future motor training domains.

3D Animation Production and Generation Based on Human Body Base 3D Models

Abstract With the gradual increase in the demand for parametric human 3D models, the requirements for the accuracy of modeling and the quality of generation are getting higher and higher. In this paper, the human body structure model is constructed by 3D human posture estimation and spatio-temporal graph convolutional network, and after normalizing the data of the human body base, the 3D human body model based on the human body base is generated by combining the free deformation method, and the local light model is used to make the human body model more three-dimensional and real. In addition, the quality of 3D animation production is investigated by combining blind quality evaluation and validation analysis of the human body-based 3D model. The 3D human body modeling simulation error assessment has an error assessment root-means-square error that falls within 1.5%. In the validation analysis of model generation quality, the quality evaluation model proposed in this paper has a quality correlation coefficient and prediction accuracy of 0.9456 and 0.9454 on the dataset, respectively, and the performance of the ablation experiments and generalization experiments is stable. The mean value of the error between the animation production character generation effect and the actual testers is not more than 10mm.Combined with the research in this paper, it effectively improves the accuracy of human body modeling and provides an evaluation method for the quality of human 3D animation production and generation, which is of practical significance for its development.

트랜스포머 기반의 다중 시점 3차원 인체자세추정

The technology of Three-dimensional human posture estimation is used in sports, motion recognition, and special effects of video media. Among various methods for this, multi-view 3D human pose estimation is essential for precise estimation even in complex real-world environments. But Existing models for multi-view 3D human posture estimation have the disadvantage of high order of time complexity as they use 3D feature maps. This paper proposes a method to extend an existing monocular viewpoint multi-frame model based on Transformer with lower time complexity to 3D human posture estimation for multi-viewpoints. To expand to multi-viewpoints our proposed method first generates an 8-dimensional joint coordinate that connects 2-dimensional joint coordinates for 17 joints at 4-vieiwpoints acquired using the 2-dimensional human posture detector, CPN(Cascaded Pyramid Network). This paper then converts them into 17×32 data with patch embedding, and enters the data into a transformer model, finally. Consequently, the MLP(Multi-Layer Perceptron) block that outputs the 3D-human posture simultaneously updates the 3D human posture estimation for 4-viewpoints at every iteration. Compared to Zheng[5]'s method the number of model parameters of the proposed method was 48.9%, MPJPE(Mean Per Joint Position Error) was reduced by 20.6 mm (43.8%) and the average learning time per epoch was more than 20 times faster.

Human Motion Prediction Based on a Multi-Scale Hypergraph for Intangible Cultural Heritage Dance Videos

Compared to traditional dance, intangible cultural heritage dance often involves the isotropic extension of choreographic actions, utilizing both upper and lower limbs. This characteristic choreography style makes the remote joints lack interaction, consequently reducing accuracy in existing human motion prediction methods. Therefore, we propose a human motion prediction method based on the multi-scale hypergraph convolutional network of the intangible cultural heritage dance video. Firstly, this method inputs the 3D human posture sequence from intangible cultural heritage dance videos. The hypergraph is designed according to the synergistic relationship of the human joints in the intangible cultural heritage dance video, which is used to represent the spatial correlation of the 3D human posture. Then, a multi-scale hypergraph convolutional network is constructed, utilizing multi-scale transformation operators to segment the human skeleton into different scales. This network adopts a graph structure to represent the 3D human posture at different scales, which is then used by the single-scalar fusion operator to spatial features in the 3D human posture sequence are extracted by fusing the feature information of the hypergraph and the multi-scale graph. Finally, the Temporal Graph Transformer network is introduced to capture the temporal dependence among adjacent frames within the time domain. This facilitates the extraction of temporal features from the 3D human posture sequence, ultimately enabling the prediction of future 3D human posture sequences. Experiments show that we achieve the best performance in both short-term and long-term human motion prediction when compared to Motion-Mixer and Motion-Attention algorithms on Human3.6M and 3DPW datasets. In addition, ablation experiments show that our method can predict more precise 3D human pose sequences, even in the presence of isotropic extensions of upper and lower limbs in intangible cultural heritage dance videos. This approach effectively addresses the issue of missing segments in intangible cultural heritage dance videos.

Multi-task artificial neural networks and their extrapolation capabilities to predict full-body 3D human posture during one- and two-handed load-handling activities

Machine-learning based human posture-prediction tools can potentially be robust alternatives to motion capture measurements. Existing posture-prediction approaches are confined to two-handed load-handling activities performed at heights below 120 cm from the floor and to predicting a limited number of body-joint coordinates/angles. Moreover, the extrapolating power of these tools beyond the range of the input dataset they were trained for (e.g., for underweight, overweight, or left-handed individuals) has not been investigated. In this study, we trained/validated/tested two posture-prediction (for full-body joint coordinates and angles) artificial neural networks (ANNs) using both 70%/15%/15% random-hold-out and leave-one-subject-out methods, based on a comprehensive kinematic dataset of forty-one full-body skin markers collected from twenty right-handed normal-weight (BMI = 18–26 kg/m2) subjects. Subjects performed 204 one- and two-handed unloaded activities at different vertical (0 to 180 cm from the floor) and horizontal (up to 60 cm lateral and/or anterior) destinations. Subsequently, the extrapolation capability of the trained/validated/tested ANNs was evaluated using data collected from fifteen additional subjects (unseen by the ANNs); three individuals in five groups: underweight, overweight, obese, left-handed individuals, and subjects with a hand-load. Results indicated that the ANNs predicted body joint coordinates and angles during various activities with errors of ∼ 25 mm and ∼ 10°, respectively; considerable improvements when compared to previous posture-prediction ANNs. Extrapolation errors of our ANNs generally remained within the error range of existing ANNs with obesity and being left-handed having, respectively, the most and least compromising effects on their accuracy. These easy-to-use ANNs appear, therefore, to be robust alternatives to common posture-measurement approaches.

A Vision-Based Human Digital Twin Modeling Approach for Adaptive Human–Robot Collaboration

Abstract Human–robot collaboration (HRC) has been identified as a highly promising paradigm for human-centric smart manufacturing in the context of Industry 5.0. In order to enhance both human well-being and robotic flexibility within HRC, numerous research efforts have been dedicated to the exploration of human body perception, but many of these studies have focused only on specific facets of human recognition, lacking a holistic perspective of the human operator. A novel approach to addressing this challenge is the construction of a human digital twin (HDT), which serves as a centralized digital representation of various human data for seamless integration into the cyber-physical production system. By leveraging HDT, performance and efficiency optimization can be further achieved in an HRC system. However, the implementation of visual perception-based HDT remains underreported, particularly within the HRC realm. To this end, this study proposes an exemplary vision-based HDT model for highly dynamic HRC applications. The model mainly consists of a convolutional neural network that can simultaneously model the hierarchical human status including 3D human posture, action intention, and ergonomic risk. Then, on the basis of the constructed HDT, a robotic motion planning strategy is further introduced with the aim of adaptively optimizing the robotic motion trajectory. Further experiments and case studies are conducted in an HRC scenario to demonstrate the effectiveness of our approach.

EGPose: Explicit and Geometric Self-Supervision for 3D Human Pose Estimation

Despite the recent progress of 3D human pose estimation, reconstructing an accurate 3D human posture from a single image without 3D annotation is yet challenging due to the following reasons. First, the reconstructing process is inherently ambiguous, as multiple 3D poses can be projected onto the same 2D pose. Second, camera rotation is difficult to measure precisely without laborious camera calibration. Some approaches resort to traditional computer vision algorithms to address these issues, but they are not differentiable and cannot be optimized through training. In this paper, we propose two geometrically explicit modules to solve the problems without any 3D ground-truth or camera parameters. Relative depth estimation module effectively mitigates depth ambiguity, reducing a number of possible depths for each joint to only two candidates. Differentiable pose alignment module calculates camera rotation via aligning poses from different views. The two modules are geometrically interpretable, reducing the training difficulty and leading to superior performance. Our method achieves state-of-the-art on the standard benchmark datasets among self-supervised methods and even outperforms several fully-supervised approaches that rely on 3D annotations.

Detection of 3D Human Posture Based on Improved Mediapipe

Based on the continuous development of motion capture technology for ordinary video images, unmarked optical motion capture has become the fastest human posture recognition technology. Compared with other technical products, Google’s 3D human body recognition framework—Mediapipe is the most mature representative in this field. However, Mediapipe also has many defects in the detection of 3D human posture. In this paper, firstly, to solve the problem of inaccurate detection of human posture by Mediapipe, the accuracy of 2D human posture detection is improved through the speed threshold correction method for each joint; According to the problem that the monocular camera can not detect the depth Z value in the human posture data accurately, the Z value of the joint point is corrected for the human tilt angle through statistics; Then, according to the inaccurate recognition of Z value caused by large body posture, the accurate correction of Z value of human posture under different body posture is realized by normalizing the simulation proportion of each body limb; Finally, in order to solve the problem of jitter, lag problem and periodic noise in multiple frames caused by the speed change of human joints, one euro filtering and mean filtering of joint data are carried out. This paper verifies that the accuracy of 3D human posture detection based on the improved Mediapipe is more than 90% through the multi-pose recognition test for people of different heights, weights, ages and gender.

3D Human Motion Posture Tracking Method Using Multilabel Transfer Learning

To overcome the high position and posture angle tracking error, long tracking loss time and posture tracking update response time, and low fitness problem of traditional human motion posture tracking methods, in this paper, a three-dimensional (3D) human motion posture tracking method using multilabel transfer learning is proposed. According to the human structure composition and degree of freedom constraints, the 3D human joint skeleton model is constructed to generate the 3D human pose image and perform the noise reduction operation. The background difference is used to detect the 3D human moving target. Using multilabel transfer learning, human motion posture features are extracted from joint position and joint angle, and the estimation results of 3D human motion posture are obtained. The tracking error of human motion posture is corrected by three-step search, and the visual 3D human motion posture tracking results are output. The results show that, compared with the traditional human motion posture tracking method, the position and posture angle tracking errors of the proposed method are 2.18 mm and 0.178 deg, respectively. The tracking loss time and posture tracking update response time are shorter, which proves that the proposed method has more advantages in tracking accuracy and higher adaptability.

Unified End-to-End YOLOv5-HR-TCM Framework for Automatic 2D/3D Human Pose Estimation for Real-Time Applications.

Three-dimensional human pose estimation is widely applied in sports, robotics, and healthcare. In the past five years, the number of CNN-based studies for 3D human pose estimation has been numerous and has yielded impressive results. However, studies often focus only on improving the accuracy of the estimation results. In this paper, we propose a fast, unified end-to-end model for estimating 3D human pose, called YOLOv5-HR-TCM (YOLOv5-HRet-Temporal Convolution Model). Our proposed model is based on the 2D to 3D lifting approach for 3D human pose estimation while taking care of each step in the estimation process, such as person detection, 2D human pose estimation, and 3D human pose estimation. The proposed model is a combination of best practices at each stage. Our proposed model is evaluated on the Human 3.6M dataset and compared with other methods at each step. The method achieves high accuracy, not sacrificing processing speed. The estimated time of the whole process is 3.146 FPS on a low-end computer. In particular, we propose a sports scoring application based on the deviation angle between the estimated 3D human posture and the standard (reference) origin. The average deviation angle evaluated on the Human 3.6M dataset (Protocol #1–Pro #1) is 8.2 degrees.

Determination of the 3D Human Spine Posture from Wearable Inertial Sensors and a Multibody Model of the Spine.

Determination of spine posture is of great interest for the effective prevention, evaluation, treatment and evolution monitoring of spinal disorders. Limitations of traditional imaging systems, including cost, radiation exposure (for X-ray based systems), projection volume issues and subject positioning requirements, etc., make non-invasive motion assessment tools effective alternatives for clinical and non-clinical use. In this work, a procedure was developed to obtain a subject-specific multibody model of the spine using either inertial or optical sensors and, based on this multibody model, to estimate the locations and orientations of the 17 vertebrae constituting the thoracolumbar spine. The number and calibration of the sensors, angular offsets, scaling difficulties and gender differences were addressed to achieve an accurate 3D-representation of the spine. The approach was validated by comparing the estimated positions of the sensors on 14 healthy subjects with those provided by an optical motion capture system. A mean position error of lower than 12 mm was obtained, thus showing that the proposed method can offer an effective non-invasive tool for the assessment of spine posture.

Human body 3D posture estimation using significant points and two cameras.

This paper proposes a three-dimensional (3D) human posture estimation system that locates 3D significant body points based on 2D body contours extracted from two cameras without using any depth sensors. The 3D significant body points that are located by this system include the head, the center of the body, the tips of the feet, the tips of the hands, the elbows, and the knees. First, a linear support vector machine- (SVM-) based segmentation method is proposed to distinguish the human body from the background in red, green, and blue (RGB) color space. The SVM-based segmentation method uses not only normalized color differences but also included angle between pixels in the current frame and the background in order to reduce shadow influence. After segmentation, 2D significant points in each of the two extracted images are located. A significant point volume matching (SPVM) method is then proposed to reconstruct the 3D significant body point locations by using 2D posture estimation results. Experimental results show that the proposed SVM-based segmentation method shows better performance than other gray level- and RGB-based segmentation approaches. This paper also shows the effectiveness of the 3D posture estimation results in different postures.

3D human posture segmentation by spectral clustering with surface normal constraint

In this paper, we propose a new algorithm for partitioning human posture represented by 3D point clouds sampled from the surface of human body. The algorithm is formed as a constrained extension of the recently developed segmentation method, spectral clustering (SC). Two folds of merits are offered by the algorithm: (1) as a nonlinear method, it is able to deal with the situation that data (point cloud) are sampled from a manifold (the surface of human body) rather than the embedded entire 3D space; (2) by using constraints, it facilitates the integration of multiple similarities for human posture partitioning, and it also helps to reduce the limitations of spectral clustering. We show that the constrained spectral clustering (CSC) still can be solved by generalized eigen-decomposition. Experimental results confirm the effectiveness of the proposed algorithm.

Nonlinear Synchronization for Automatic Learning of 3D Pose Variability in Human Motion Sequences

A dense matching algorithm that solves the problem of synchronizing prerecorded human motion sequences, which show different speeds and accelerations, is proposed. The approach is based on minimization of MRF energy and solves the problem by using Dynamic Programming. Additionally, an optimal sequence is automatically selected from the input dataset to be a time-scale pattern for all other sequences. The paper utilizes an action specific model which automatically learns the variability of 3D human postures observed in a set of training sequences. The model is trained using the public CMU motion capture dataset for the walking action, and a mean walking performance is automatically learnt. Additionally, statistics about the observed variability of the postures and motion direction are also computed at each time step. The synchronized motion sequences are used to learn a model of human motion for action recognition and full-body tracking purposes.

Markerless Human Motion Capture from Voxel Reconstruction with Simple Human Model

This paper investigates a human body posture estimation method based on the back projection of human silhouette images extracted from multi-camera images. The multi-camera system is based on a server-client system with local network of 1000Base-T to achieve a voxel 3D reconstruction of human body posture in real-time. In order to extract significant points of the human body such as head, neck, shoulders, elbow joints, hands, waist, knee joints, and toes in 3D, an articulated cylindrical human model is applied to the voxel reconstruction of human body. To evaluate the proposed human body posture estimation method, 3D reconstruction experiments of human body posture and extraction experiments of human body's significant points are carried out. The system runs in approximately real time (9 frames/sec with 50×50×50 voxel resolution) and the experimental results confirm both the feasibility and effectiveness of the proposed system in 3D human body posture estimation.

Retrieval and constraint-based human posture reconstruction from a single image

In this study, we present a novel model-based approach to reconstruct the 3D human posture from a single image. The approach is guided by a posture library and a set of constraints. Given a 2D human figure, i.e., a set of labeled body segments and estimated root orientation in the image, a 3D pivotal posture whose 2D projection is similar to the human figure is first retrieved from the posture library. To facilitate the retrieval process, a table-lookup technique is proposed to index postures according to their 2D projections with respect to designated view directions. Next physical and environmental constraints, including segment length ratios, joint angle limits, pivotal posture reference, and feet-floor contact, are automatically applied to reconstruct the 3D posture. Experimental results show the effectiveness of the proposed approach.