3D Human Motion Research Articles

Motion Recognition Based on Deep Learning Algorithm

Advanced scientific and technological measurement methods are the basis of sports research and the progress of sports research methods. It promotes the continuous improvement of sports technology to meet athletes’ pursuit of excellent sports achievements. Therefore, the method of sports scientific research must be changed from the traditional empirical training mode to the programmed training method, from the qualitative analysis of the training effect to the nuanced analysis of the training process. Based on this, this paper uses in-depth learning to study sports motion capture. First, two-dimensional human joints are extracted based on Mask R-CNN. Then, the 3D human motion skeleton is constructed by using the binocular vision system, and the Mask R-CNN human pose estimation algorithm is optimized. On this basis, a sports motion capture system is designed, and the system’s accuracy is verified. The error of the depth information obtained by the sports capture system constructed in this paper is less than 3% in the experiment of about 2[Formula: see text]m. It has strong practicability.

DAMO: A Deep Solver for Arbitrary Marker Configuration in Optical Motion Capture

Marker-based optical motion capture (mocap) systems are increasingly utilized for acquiring 3D human motion, offering advantages in capturing the subtle nuances of human movement, style consistency, and ease of obtaining desired motion. Motion data acquisition via mocap typically requires laborious marker labeling and motion reconstruction, recent deep-learning solutions have aimed to automate the process. However, such solutions generally presuppose a fixed marker configuration to reduce learning complexity, thereby limiting flexibility. To overcome the limitation, we introduce DAMO, an end-to-end deep solver, proficiently inferring arbitrary marker configurations and optimizing pose reconstruction. DAMO outperforms state-of-the-art like SOMA and MoCap-Solver in scenarios with significant noise and unknown marker configurations. We expect that DAMO will meet various practical demands such as facilitating dynamic marker configuration adjustments during capture sessions, processing marker clouds irrespective of whether they employ mixed or entirely unknown marker configurations, and allowing custom marker configurations to suit distinct capture scenarios. DAMO code and pretrained models are available at https://github.com/CritBear/damo

LiDARCapV2: 3D human pose estimation with human–object interaction from LiDAR point clouds

Human–object interactions in open environments are common in the real world. Estimating 3D human pose from data where objects occlude the human is a challenging task in biometrics. However, existing LiDAR-based human motion capture datasets lack occlusion scenarios between humans and objects. To overcome this limitation, we propose LiDARHuman51M, a new human–object interaction dataset captured by LiDAR in long-range outdoor scene. It includes human motion labels acquired by an IMU system and synchronous RGB images. Additionally, we present an occlusion-aware method, LiDARCapV2, for capturing human motion from LiDAR point clouds under human–object interaction settings. Our key insight is to overcome object interference in human feature extraction by introducing a module called AgNoise-Segment. A noise augmentation strategy introduced in the AgNoise-Segment module alleviates the dependency of the segmentation accuracy on the effectiveness of 3D human pose estimations. Furthermore, we propose a skeleton extraction module that integrates features learned from the AgNoise-Segment module and predicts the skeleton locations. Quantitative and qualitative experiments demonstrate that LiDARCapV2 can capture high-quality 3D human motion under human–object interaction settings. Experiments on the KITTI and Waymo datasets demonstrate that our method can be generalized to real-world open scenarios.

Tacking over-smoothing: Target-guide progressive dynamic graph learning for 3D skeleton-based human motion prediction

Graph Convolution Network-based (GCN-based) approaches show promising performance on 3D skeleton-based human motion prediction due to its natural graph representation and outstanding ability for spatial–temporal dependencies modeling for human motion. However, the existing GCN-based methods inherently have difficulties in designing deep GCN-based structures for human motion prediction due to the over-smoothing issue. Although the over-smoothing issue has been studied in many scenarios, few attempts focus on this problem in 3D human motion prediction tasks. Therefore, we propose a novel deep GCN architecture termed TPDGN (Target-guide Progressive Dynamic Graph Network) to solve this issue in a motion prediction context. The core of the proposed TPDGN is the progressive dynamic graph leaning block, which is developed to model the dynamic graph pattern. We claim that the key to solving the over-smoothing issue is to reduce the loss of motion features and boost graph representations. Accordingly, we further explore the progressive learning mode by the specific target guidance for the motion prediction task, which aims at learning rich motion dependencies to tackle over-smoothing. Empirical results show that our approach could improve the over-smoothing issue, achieving state-of-the-art results on three public datasets. The codes is available at https://github.com/1111szu/TPDGN.

Efficient neural implicit representation for 3D human reconstruction

High-fidelity digital human representations are increasingly in demand in the digital world, particularly for interactive telepresence, AR/VR, 3D graphics, and the rapidly evolving metaverse. Even though they work well in small spaces, conventional methods for reconstructing 3D human motion frequently require the use of expensive hardware and have high processing costs. This study presents HumanAvatar, an innovative approach that efficiently reconstructs precise human avatars from monocular video sources. At the core of our methodology, we integrate the pre-trained HuMoR, a model celebrated for its proficiency in human motion estimation. This is adeptly fused with the cutting-edge neural radiance field technology, Instant-NGP, and the state-of-the-art articulated model, Fast-SNARF, to enhance the reconstruction fidelity and speed. By combining these two technologies, a system is created that can render quickly and effectively while also providing estimation of human pose parameters that are unmatched in accuracy. We have enhanced our system with an advanced posture-sensitive space reduction technique, which optimally balances rendering quality with computational efficiency. In our detailed experimental analysis using both artificial and real-world monocular videos, we establish the advanced performance of our approach. HumanAvatar consistently equals or surpasses contemporary leading-edge reconstruction techniques in quality. Furthermore, it achieves these complex reconstructions in minutes, a fraction of the time typically required by existing methods. Our models achieve a training speed that is 110× faster than that of State-of-The-Art (SoTA) NeRF-based models. Our technique performs noticeably better than SoTA dynamic human NeRF methods if given an identical runtime limit. HumanAvatar can provide effective visuals after only 30 s of training. Please visit https://github.com/HZXu-526/Human-Avatar for further demo results and code.

FLK: A filter with learned kinematics for real-time 3D human pose estimation

There is a growing interest in adopting 3D human pose estimation in safety-critical systems, from healthcare to Industry 5.0. Nevertheless, when applied in such settings, these neural networks may suffer from estimation inaccuracy. Besides imprecise or inconsistent annotations in the training dataset, the inaccuracy is caused by poor image quality, rare poses, dropped frames, or heavy occlusions in the scene. In addition, these scenarios often require the software results to have temporal constraints, such as real-time and zero- or low-latency, which make many of the filtering solutions proposed in the literature inapplicable. This paper proposes FLK, a Filter with Learned Kinematics, to refine 3D human motion data in real-time and at zero/low latency. The temporal core combines a Kalman filter and a low-pass filter, which learns the motion model through a recurrent neural network. The spatial core takes advantage of the biomechanical constraints of the human body to provide spatial coherency between keypoints. The combination of the cores allows the filter to adequately address different types of noise, from jittering to dropped frames. We test the filter on motion data from multiple datasets and seven 3D human pose estimation backbones, improving accuracy up to 140 mm with non-Gaussian noise and 53 mm with missing information.

Evaluating Gesture Generation in a Large-scale Open Challenge: The GENEA Challenge 2022

This article reports on the second GENEA Challenge to benchmark data-driven automatic co-speech gesture generation. Participating teams used the same speech and motion dataset to build gesture-generation systems. Motion generated by all these systems was rendered to video using a standardised visualisation pipeline and evaluated in several large, crowdsourced user studies. Unlike when comparing different research articles, differences in results are here only due to differences between methods, enabling direct comparison between systems. The dataset was based on 18 hours of full-body motion capture, including fingers, of different persons engaging in a dyadic conversation. Ten teams participated in the challenge across two tiers: full-body and upper-body gesticulation. For each tier, we evaluated both the human-likeness of the gesture motion and its appropriateness for the specific speech signal. Our evaluations decouple human-likeness from gesture appropriateness, which has been a difficult problem in the field. The evaluation results show some synthetic gesture conditions being rated as significantly more human-like than 3D human motion capture. To the best of our knowledge, this has not been demonstrated before. On the other hand, all synthetic motion is found to be vastly less appropriate for the speech than the original motion-capture recordings. We also find that conventional objective metrics do not correlate well with subjective human-likeness ratings in this large evaluation. The one exception is the Fréchet gesture distance (FGD), which achieves a Kendall’s tau rank correlation of around -0.5. Based on the challenge results we formulate numerous recommendations for system building and evaluation.

Calibrationless monocular vision musculoskeletal simulation during gait

With computer vision technology and prediction of ground reaction forces (GRF), a previous study performed markerless motion capture and musculoskeletal simulation with two smartphones (OpenCap). A recent approach can reconstruct 3D human motion from a single video without calibration and it may further simplify the motion capture process. However it has not been combined with musculoskeletal simulation and the validity is unclear. Therefore, the purpose of this study was to determine the validity of the musculoskeletal simulation using a monocular vision approach. An open-source dataset that contains motion capture and video data during gait from 10 healthy participants was used. Human motion reconstruction with the skinned human (SMPL) model was performed on each video. Virtual marker data was generated by extracting the position data from the SMPL skin vertices. Inverse kinematics, GRF prediction (only for monocular vision approach), inverse dynamics and static optimization were performed using a musculoskeletal model for experimental motion capture data and the generated virtual markers from videos. Mean absolute errors (MAE) between motion capture based and monocular vision based simulation outcomes were calculated. The MAE were 8.4° for joint angles, 5.0 % bodyweight for GRF, 1.1 % bodyweight*height for joint moments and 0.11 for estimated muscle activations from 16 muscles. The entire MAE was larger but some were comparable to OpenCap. Using the monocular vision approach, motion capture and musculoskeletal simulation can be done with no preparations and is beneficial for clinicians to quantify the daily gait assessment.

3D human pose data augmentation using Generative Adversarial Networks for robotic-assisted movement quality assessment.

In the realm of human motion recognition systems, the augmentation of 3D human pose data plays a pivotal role in enriching and enhancing the quality of original datasets through the generation of synthetic data. This augmentation is vital for addressing the current research gaps in diversity and complexity, particularly when dealing with rare or complex human movements. Our study introduces a groundbreaking approach employing Generative Adversarial Networks (GANs), coupled with Support Vector Machine (SVM) and DenseNet, further enhanced by robot-assisted technology to improve the precision and efficiency of data collection. The GANs in our model are responsible for generating highly realistic and diverse 3D human motion data, while SVM aids in the effective classification of this data. DenseNet is utilized for the extraction of key features, facilitating a comprehensive and integrated approach that significantly elevates both the data augmentation process and the model's ability to process and analyze complex human movements. The experimental outcomes underscore our model's exceptional performance in motion quality assessment, showcasing a substantial improvement over traditional methods in terms of classification accuracy and data processing efficiency. These results validate the effectiveness of our integrated network model, setting a solid foundation for future advancements in the field. Our research not only introduces innovative methodologies for 3D human pose data enhancement but also provides substantial technical support for practical applications across various domains, including sports science, rehabilitation medicine, and virtual reality. By combining advanced algorithmic strategies with robotic technologies, our work addresses key challenges in data augmentation and motion quality assessment, paving the way for new research and development opportunities in these critical areas.

Research on 3D Human Motion Visualization for Immersive Teaching Mechanism - 3D Pose Reconstruction form the Tradition Motion Video for Teaching and Evaluation

Objective: Based on the three-dimensional (3D) visualization technology of human motion, the immersive teaching mechanism of human motion was proposed. At present, the training and learning of human motion could not observe the 3D information of its posture due to the issue of two-dimensional (2D) perspective, which had many shortcomings. Methods: This article proposed new teaching mechanisms, including personal immersive assisted instruction, multi person immersive interactive teaching, and immersive teaching evaluation method for motion learning. The design of the three methods were carried out in the background of artificial intelligence (AI). Results: Combining with current advanced AI algorithms, the concept of visualizing 3D human motion for immersive teaching method was conceived, and a specific implementation framework diagram and description of the main work of each teaching mechanism are provided. The personal immersive assisted instruction method would be a powerful tool for enhancing individual learning experiences in the realm of 3D human motion visualization. The multi-person immersive interactive teaching took 3D human motion visualization to the next level by facilitating collaborative learning experiences. The immersive teaching evaluation method for human motion training was pivotal for assessing the effectiveness of these immersive techniques. Conclusion: After the evaluation and analysis, this mechanism had been proven to be an effective teaching mechanism for pose learning. 3D human motion visualization for the immersive teaching held the promising prospects in the field of education and instructional practices.

Towards Detailed Text-to-Motion Synthesis via Basic-to-Advanced Hierarchical Diffusion Model

Text-guided motion synthesis aims to generate 3D human motion that not only precisely reflects the textual description but reveals the motion details as much as possible. Pioneering methods explore the diffusion model for text-to-motion synthesis and obtain significant superiority. However, these methods conduct diffusion processes either on the raw data distribution or the low-dimensional latent space, which typically suffer from the problem of modality inconsistency or detail-scarce. To tackle this problem, we propose a novel Basic-to-Advanced Hierarchical Diffusion Model, named B2A-HDM, to collaboratively exploit low-dimensional and high-dimensional diffusion models for high quality detailed motion synthesis. Specifically, the basic diffusion model in low-dimensional latent space provides the intermediate denoising result that to be consistent with the textual description, while the advanced diffusion model in high-dimensional latent space focuses on the following detail-enhancing denoising process. Besides, we introduce a multi-denoiser framework for the advanced diffusion model to ease the learning of high-dimensional model and fully explore the generative potential of the diffusion model. Quantitative and qualitative experiment results on two text-to-motion benchmarks (HumanML3D and KIT-ML) demonstrate that B2A-HDM can outperform existing state-of-the-art methods in terms of fidelity, modality consistency, and diversity.

MotionMix: Weakly-Supervised Diffusion for Controllable Motion Generation

Controllable generation of 3D human motions becomes an important topic as the world embraces digital transformation. Existing works, though making promising progress with the advent of diffusion models, heavily rely on meticulously captured and annotated (e.g., text) high-quality motion corpus, a resource-intensive endeavor in the real world. This motivates our proposed MotionMix, a simple yet effective weakly-supervised diffusion model that leverages both noisy and unannotated motion sequences. Specifically, we separate the denoising objectives of a diffusion model into two stages: obtaining conditional rough motion approximations in the initial T-T* steps by learning the noisy annotated motions, followed by the unconditional refinement of these preliminary motions during the last T* steps using unannotated motions. Notably, though learning from two sources of imperfect data, our model does not compromise motion generation quality compared to fully supervised approaches that access gold data. Extensive experiments on several benchmarks demonstrate that our MotionMix, as a versatile framework, consistently achieves state-of-the-art performances on text-to-motion, action-to-motion, and music-to-dance tasks.

Spatial-Related Sensors Matters: 3D Human Motion Reconstruction Assisted with Textual Semantics

Leveraging wearable devices for motion reconstruction has emerged as an economical and viable technique. Certain methodologies employ sparse Inertial Measurement Units (IMUs) on the human body and harness data-driven strategies to model human poses. However, the reconstruction of motion based solely on sparse IMU data is inherently fraught with ambiguity, a consequence of numerous identical IMU readings corresponding to different poses. In this paper, we explore the spatial importance of sparse sensors, supervised by text that describes specific actions. Specifically, uncertainty is introduced to derive weighted features for each IMU. We also design a Hierarchical Temporal Transformer (HTT) and apply contrastive learning to achieve precise temporal and feature alignment of sensor data with textual semantics. Experimental results demonstrate our proposed approach achieves significant improvements in multiple metrics compared to existing methods. Notably, with textual supervision, our method not only differentiates between ambiguous actions such as sitting and standing but also produces more precise and natural motion.

Fault Signal Perception of Nanofiber Sensor for 3D Human Motion Detection Using Multi-Task Deep Learning

Once a fault occurs in the nanofiber sensor, the scientific and reliable three-dimensional (3D) human motion detection results will be compromised. It is necessary to accurately and rapidly perceive the fault signals of the nanofiber sensor and determine the type of fault, to enable it to continue operating in a sustained and stable manner. Therefore, we propose a fault signal perception method for 3D human motion detection nanofiber sensor based on multi-task deep learning. First, through obtaining the fault characteristic parameters of the nanofiber sensor, the fault of the nanofiber sensor is reconstructed to complete the fault location of the nanofiber sensor. Second, the fault signal of the nanofiber sensor is mapped by the penalty function, and the feature extraction model of the fault signal of the nanofiber sensor is constructed by combining the multi-task deep learning. Finally, the multi-task deep learning algorithm is used to calculate the sampling frequency of the fault signal, and the key variable information of the fault of the nanofiber sensor is extracted according to the amplitude of the state change of the nanofiber sensor, to realize the perception of the fault signal of the nanofiber sensor. The results show that the proposed method can accurately perceive the fault signal of a nanofiber sensor in 3D human motion detection, the maximum sensor fault location accuracy is 97%, and the maximum noise content of the fault signal is only 5 dB, which shows that the method can be widely used in fault signal perception.

A Point-2s reinforcement learning biomimetic model for estimating and analyzing human 3D motion posture

With the rapid progress of computer vision and artificial intelligence, the accuracy of estimating and analyzing human body movements and postures has always been a highly focused research field. However, current methods still have some shortcomings in accurately estimating the pose of 3D human movements. This study aims to propose an effective method to accurately estimate the motion posture of 3D human activities using new technologies of deep learning and neural networks. Firstly, based on previous research, this paper analyzes the problems and challenges of insufficient accuracy in current 3D human motion pose estimation methods, limited capture of 3D spatial information in deep video data, and inability to capture subtle motion details. Then, in response to the problem of disorder in point clouds, an innovative SMPL human Point-2 s reinforcement learning framework was constructed using the VIBE network to estimate the pose of RGB in the NTU dataset. 24 Point-2 s biomimetic algorithm joints were sampled from a distance using the FPS of PointNet++in SMPL, and an index was established based on relative positions to ensure that other joint points are in the same position. Finally, these joint points were input into the 2 s-AGCN network to construct a complete Point-2 s model. The research results indicate that the proposed Point-2 s model has achieved good results in accurately estimating the motion posture of 3D human activities, and effectively solves the problem of disorder in point clouds converted from deep videos. Compared to traditional methods, this research model has significantly improved accuracy and stability. The practical application of this method will provide useful references and guidance for research and method development in related fields.

Multi-Semantics Aggregation Network Based on the Dynamic-Attention Mechanism for 3D Human Motion Prediction

Multi-Semantics Aggregation Network Based on the Dynamic-Attention Mechanism for 3D Human Motion Prediction

MarkerNet: A divide‐and‐conquer solution to motion capture solving from raw markers

AbstractMarker‐based optical motion capture (MoCap) aims to localize 3D human motions from a sequence of input raw markers. It is widely used to produce physical movements for virtual characters in various games such as the role‐playing game, the fighting game, and the action‐adventure game. However, the conventional MoCap cleaning and solving process is extremely labor‐intensive, time‐consuming, and usually the most costly part of game animation production. Thus, there is a high demand for automated algorithms to replace costly manual operations and achieve accurate MoCap cleaning and solving in the game industry. In this article, we design a divide‐and‐conquer‐based MoCap solving network, dubbed MarkerNet, to estimate human skeleton motions from sequential raw markers effectively. In a nutshell, our key idea is to decompose the task of direct solving of global motion from all markers into first modeling sub‐motions of local parts from the corresponding marker subsets and then aggregating sub‐motions into a global one. In this manner, our model can effectively capture local motion patterns w.r.t. different marker subsets, thus producing more accurate results compared to the existing methods. Extensive experiments on both real and synthetic data verify the effectiveness of the proposed method.

Multi-Condition Latent Diffusion Network for Scene-Aware Neural Human Motion Prediction.

Inferring 3D human motion is fundamental in many applications, including understanding human activity and analyzing one's intention. While many fruitful efforts have been made to human motion prediction, most approaches focus on pose-driven prediction and inferring human motion in isolation from the contextual environment, thus leaving the body location movement in the scene behind. However, real-world human movements are goal-directed and highly influenced by the spatial layout of their surrounding scenes. In this paper, instead of planning future human motion in a "dark" room, we propose a Multi-Condition Latent Diffusion network (MCLD) that reformulates the human motion prediction task as a multi-condition joint inference problem based on the given historical 3D body motion and the current 3D scene contexts. Specifically, instead of directly modeling joint distribution over the raw motion sequences, MCLD performs a conditional diffusion process within the latent embedding space, characterizing the cross-modal mapping from the past body movement and current scene context condition embeddings to the future human motion embedding. Extensive experiments on large-scale human motion prediction datasets demonstrate that our MCLD achieves significant improvements over the state-of-the-art methods on both realistic and diverse predictions.

Bridging Actions: Generate 3D Poses and Shapes In-Between Photos.

Generating realistic 3D human motion has been a fundamental goal of the game/animation industry. This work presents a novel transition generation technique that can bridge the actions of people in the foreground by generating 3D poses and shapes in-between photos, allowing 3D animators/novice users to easily create/edit 3D motions. To achieve this, we propose an adaptive motion network (ADAM-Net) that effectively learns human motion from masked action sequences to generate kinematically compliant 3D poses and shapes in-between given temporally-sparse photos. Three core learning designs underpin ADAM-Net. First, we introduce a random masking process that randomly masks images from an action sequence and fills masked regions in latent space by interpolation of unmasked images to simulate various transitions under given temporally-sparse photos. Second, we propose a long-range adaptive motion (L-ADAM) attention module that leverages visual cues observed from human motion to adaptively recalibrate the range that needs attention in a sequence, along with a multi-head cross-attention. Third, we develop a short-range adaptive motion (S-ADAM) attention module that weightedly selects and integrates adjacent feature representations at different levels to strengthen temporal correlation. By coupling these designs, the results demonstrate that ADAM-Net excels not only in generating 3D poses and shapes in-between photos, but also in classic 3D human pose and shape estimation.

From Method to Application: A Review of Deep 3D Human Motion Capture

From Method to Application: A Review of Deep 3D Human Motion Capture