Human Motion Prediction Research Articles

Learning full context feature for human motion prediction

Human motion prediction aims to predict the target poses given the previous poses. Most existing methods are devoted to extracting richer motion features from only the given previous poses to predict the target poses. However, we consider that the post poses after the target poses are helpful in acquiring the context feature and constraint between neighbor motions, which is also important for motion prediction. In this paper, we explore to make use of the post motion information for a powerful human motion prediction method. Specifically, we propose a human motion prediction model which learns the motion constraint from both the previous and post poses, in order to fully utilize the context features of the target poses. During training, the proposed memory dictionary module is used to learn the mapping from previous features to post features. In testing, the proposed memory dictionary module fully exploits the learned mapping to calculate the future motion feature according to the input previous feature. Thus, the context feature of human motion is enriched in our method. We evaluate the proposed method on two large-scale datasets, Human3.6M and CMU-Mocap. The experimental results demonstrate that our method improves the motion prediction performance, especially for long-term human motion.

Toward Realistic 3D Human Motion Prediction With a Spatio-Temporal Cross- Transformer Approach

Human motion prediction intends to predict how humans move given a historical sequence of 3D human motions. Recent transformer-based methods have attracted increasing attentions and demonstrated their promising performance in 3D human motion prediction. However, existing methods generally decompose the input of human motion information into spatial and temporal branches in a separate way and seldom consider their inherent coherence between the two branches, hence often failing to register the dynamic spatio-temporal information during the training process. Motivated by these issues, we propose a spatio-temporal cross-transformer network (STCT) for 3D human motion predictions. Specifically, we investigate various types of interaction methods (i.e., Concatenation Interaction, Msg token interaction, and Cross-transformer) to capture the coherence of the spatial and temporal branches. According to the obtained results, the proposed cross-transformer interaction method shows its superiority over other methods. Meanwhile, considering that most existing works treat the human body as a set of 3D human joint positions, the predicted human joints are proportionally less appropriate to the realistic human body due to unreasonable bone length and non-plausible poses as time progresses. We further resort to the bone constraints of human mesh to produce more realistic human motions. By fitting a parametric body model (i.e., SMPL-X model) to the predicted human joints, a reconstruction loss function is proposed to remedy the unreasonable bone length and pose errors. Comprehensive experiments on AMASS and Human3.6M datasets have demonstrated that our method achieves superior performance over compared methods.

Enhancing Human Motion Prediction through Joint-based Analysis and AVI Video Conversion

Enhancing Human Motion Prediction through Joint-based Analysis and AVI Video Conversion

Learning disentangled representations for controllable human motion prediction

Generative model-based motion prediction techniques have recently realized predicting controlled human motions, such as predicting multiple upper human body motions with similar lower-body motions. However, to achieve this, the state-of-the-art methods require either subsequently learning mapping functions to seek similar motions or training the model repetitively to enable control over the desired portion of body. In this paper, we propose a novel framework to learn disentangled representations for controllable human motion prediction. Our task is to predict multiple future human motions based on the past observed sequence, with the control of partial-body movements. Our network involves a conditional variational auto-encoder (CVAE) architecture to model full-body human motion, and an extra CVAE path to learn only the corresponding partial-body (e.g., lower-body) motion. Specifically, the inductive bias imposed by the extra CVAE path encourages two latent variables in two paths to respectively govern separate representations for each partial-body motion. With a single training, our model is able to provide two types of controls for the generated human motions: (i) strictly controlling one portion of human body and (ii) adaptively controlling the other portion, by sampling from a pair of latent spaces. Additionally, we extend and adapt a sampling strategy to our trained model to diversify the controllable predictions. Our framework also potentially allows new forms of control by flexibly customizing the input for the extra CVAE path. Extensive experimental results and ablation studies demonstrate that our approach is capable of predicting state-of-the-art controllable human motions both qualitatively and quantitatively.

Full-body Human Motion Reconstruction with Sparse Joint Tracking Using Flexible Sensors

Human motion tracking is a fundamental building block for various applications including computer animation, human-computer interaction, healthcare, and so on. To reduce the burden of wearing multiple sensors, human motion prediction from sparse sensor inputs has become a hot topic in human motion tracking. However, such predictions are non-trivial as (i) the widely adopted data-driven approaches can easily collapse to average poses, and (ii) the predicted motions contain unnatural jitters. In this work, we address the aforementioned issues by proposing a novel framework which can accurately predict the human joint moving angles from the signals of only four flexible sensors, thereby achieving the tracking of human joints in multi-degrees of freedom. Specifically, we mitigate the collapse to average poses by implementing the model with a Bi-LSTM neural network that makes full use of short-time sequence information; we reduce jitters by adding a median pooling layer to the network, which smooths consecutive motions. Although being bio-compatible and ideal for improving the wearing experience, the flexible sensors are prone to aging which increases prediction errors. Observing that the aging of flexible sensors usually results in drifts of their resistance ranges, we further propose a novel dynamic calibration technique to rescale sensor ranges, which further improves the prediction accuracy. Experimental results show that our method achieves a low and stable tracking error of 4.51 degrees across different motion types with only four sensors.

A human-like action learning process: Progressive pose generation for motion prediction

Human motion prediction is crucial for the human–robot inter-action and self-driving. We human beings learn an action with two stages, i.e., the approximation stage and the adjustment stage. While the first stage learns an approximate pose in general, the second stage learns more details. Based on this observation, this paper proposes a new two-stage framework to predict the upcoming human motion poses progressively based on a historical sequence. In the approximation stage, we adopt the motion attention network to derive an approximation of the upcoming poses. In the adjustment stage, we propose a novel fine-tuning network that can extract both spatial and temporal features to enhance the prediction accuracy. To imitate the action learning procedure of our humans, the adjustment stage in our progressive prediction framework continuously adjusts the pose many times via a series of fine-tuning networks. Extensive experiments on three benchmark datasets (i.e., Human3.6M, CMU-Mocap, and 3DPW) show that our proposed method can outperform state-of-the-art methods in both short-term and long-term predictions.

A multilayer human motion prediction perceptron by aggregating repetitive motion

A multilayer human motion prediction perceptron by aggregating repetitive motion

Spatiotemporal Consistency Learning From Momentum Cues for Human Motion Prediction

Extrapolating future human motion based on the historical human pose sequence is the foundation of various intelligent applications. Numerous deep learning-based algorithms have been designed to address this task, achieving state-of-the-art performance on different human motion benchmark datasets. However, most existing methods employ three-dimensional coordinates of joints to demonstrate dynamic motion contexts implicitly. Unfortunately, it remains challenging in capturing motion information from the pose sequence. In this paper, we advocate explicitly describing dynamic contexts via the momentum of human motion mechanic space, as the momentum of a joint is explicit, temporal consistent, and can provide abundant information to the model. In addition, the single-stream methods play a dominant role in the field of human motion prediction. They usually capture motion information via the strategy of continuous or sparse sampling, which might obviate global or detailed local information. Therefore, we present a simple yet effective dual-stream method that can consider both the detailed and global temporal information through a combination of continuous and sparse sampling. The proposed dual-stream paradigm enables the improvement of computational efficiency and the short-term prediction accuracy concurrently. Furthermore, we present a novel temporal attention-based graph convolutional network (TA-GCN) to derive a spatiotemporally consistent motion representation, which can adequately consider the rationality of human body topology. Extensive experiments on two large motion prediction benchmark datasets ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> ., Human 3.6M and CMU Mocap) show that our algorithm achieves state-of-the-art performance both qualitatively and quantitatively.

Predicting human poses via recurrent attention network

Human motion prediction is a challenging task due to the diversity and randomness of future poses. Due to the inherent topology of pose data, most recent work has used graph convolution networks (GCNs) to accomplish the task of human motion prediction. However, the GCN-based method has a major shortcoming in that the modeling of temporal information is insufficient. In this paper, we propose a simple approach that combines recurrent neural networks (RNNs) and attention mechanism for motion prediction, which considers both spatial relations between different joints and temporal correlation. The query, key and value of the attention mechanism allow us to select the information we need for the subsequent prediction. To solve the difficult problem of RNN training, we utilize uncertainty and strong short-term constraints to optimize the training process. We evaluate our method on several standard benchmark datasets for human motion prediction, i.e., the Human3.6M dataset and the CMU MoCap dataset. The experimental results show that our approach outperforms previous approaches.

Survey of maps of dynamics for mobile robots

Robotic mapping provides spatial information for autonomous agents. Depending on the tasks they seek to enable, the maps created range from simple 2D representations of the environment geometry to complex, multilayered semantic maps. This survey article is about maps of dynamics (MoDs), which store semantic information about typical motion patterns in a given environment. Some MoDs use trajectories as input, and some can be built from short, disconnected observations of motion. Robots can use MoDs, for example, for global motion planning, improved localization, or human motion prediction. Accounting for the increasing importance of maps of dynamics, we present a comprehensive survey that organizes the knowledge accumulated in the field and identifies promising directions for future work. Specifically, we introduce field-specific vocabulary, summarize existing work according to a novel taxonomy, and describe possible applications and open research problems. We conclude that the field is mature enough, and we expect that maps of dynamics will be increasingly used to improve robot performance in real-world use cases. At the same time, the field is still in a phase of rapid development where novel contributions could significantly impact this research area.

GGTr: An Innovative Framework for Accurate and Realistic Human Motion Prediction

Human motion prediction involves forecasting future movements based on past observations, which is a complex task due to the inherent spatial-temporal dynamics of human motion. In this paper, we introduced a novel framework, GGTr, which adeptly encapsulates these patterns by integrating positional graph convolutional network (GCN) layers, gated recurrent unit (GRU) network layers, and transformer layers. The proposed model utilizes an enhanced GCN layer equipped with a positional representation to aggregate information from body joints more effectively. To address temporal dependencies, we strategically combined GRU and transformer layers, enabling the model to capture both local and global temporal dependencies across body joints. Through extensive experiments conducted on Human3.6M and CMU-MoCap datasets, we demonstrated the superior performance of our proposed model. Notably, our framework shows significant improvements in predicting long-term movements, outperforming state-of-the-art methods substantially.

Task-Constrained Motion Planning Considering Uncertainty-Informed Human Motion Prediction for Human–Robot Collaborative Disassembly

While the disassembly of high-precision electronic devices is a predominantly labor-intensive process, collaborative robots provide a promising solution through human–robot collaboration. To ensure efficient yet safe collaboration, this article presents a new way to generate task-constrained and collision-free motion for a collaborative robot operating in a dynamic environment involving human movement, which is traditionally challenging due to the high degree of freedom of the corobot and the uncertainty nature of human motion. We first establish a neural human motion prediction model with quantified uncertainty, and then optimize the configuration of the robot online by taking the human motion and uncertainties into consideration. While such rationale is straightforward in nature, our method explicitly quantified the uncertainty of the neural human prediction model to further enhance the collaboration safety, and integrated the quantified uncertainty into the task-satisfied motion planning in real time to efficiently conduct tasks. Extensive experimental tests and comparison studies have been conducted to validate the efficiency and effectiveness of the proposed planning method.

Collaborative Multi-Dynamic Pattern Modeling for Human Motion Prediction

The dynamic information of the joints, such as the movement amplitude, is critical for forecasting precise human joint trajectories. Existing methods adopt global modeling in which all joints are treated as a whole to extract features for movement coordination. Though global modeling can exploit hidden relationships between joints, it also inevitably introduces undesired trajectory dependencies, which weakens the dynamic information effects and simplifies the constraints and kinetics model of joints. Therefore, we propose a dynamic pattern-based collaborative modeling framework (DPnet) that contains a keyframe enhanced module (KEM) and multi-channel feature extractor blocks (MFE-block). The KEM tackles the discontinuity between the last frame of observation and the first predicted one by duplicating the decisive frame. The MFE-block utilizes a multi-channel graph structure to enrich the dynamic information effects and recessive constraints of joints. To distinguish the dynamic information of each joint, we calculate the movement amplitude of the joints and propose three dynamic patterns, including active, inactive, and static patterns. We also propose a dynamic pattern-guided feature extractor (DP-FE) to alleviate the trajectory dependencies between joints with different dynamic patterns. We evaluate our approach on three standard benchmark datasets, including H3.6M [8], CMU-Mocap [44], and 3DPW [45]. Our approach achieves impressive results in both short-term and long-term predictions, confirming its effectiveness and efficiency.

KD-Former: Kinematic and dynamic coupled transformer network for 3D human motion prediction

Recent studies have made remarkable progress on 3D human motion prediction by describing motion with kinematic knowledge. However, kinematics only considers the 3D positions or rotations of human skeletons, failing to reveal the physical characteristics of human motion. Motion dynamics reflects the forces between joints, explicitly encoding the skeleton topology, whereas rarely exploited in motion prediction. In this paper, we propose the Kinematic and Dynamic coupled transFormer (KD-Former), which incorporates dynamics with kinematics, to learn powerful features for high-fidelity motion prediction. Specifically, We first formulate a reduced-order dynamic model of human body to calculate the forces of all joints. Then we construct a non-autoregressive encoder-decoder framework based on the transformer structure. The encoder involves a kinematic encoder and a dynamic encoder, which are respectively responsible for extracting the kinematic and dynamic features for given history sequences via a spatial transformer and a temporal transformer. Future query sequences are decoded in parallel in the decoder by leveraging the encoded kinematic and dynamic information of history sequences. Experiments on Human3.6M and CMU MoCap benchmarks verify the effectiveness and superiority of our method. Code will be available at: https://github.com/wslh852/KD-Former.git.

3-D Skeleton-Based Human Motion Prediction With Manifold-Aware GAN

In this work we propose a novel solution for 3D skeleton-based human motion prediction. The objective of this task consists in forecasting future human poses based on a prior skeleton pose sequence. This involves solving two main challenges still present in recent literature; (1) discontinuity of the predicted motion which results in unrealistic motions and (2) performance deterioration in long-term horizons resulting from error accumulation across time. We tackle these issues by using a compact manifold-valued representation of 3D human skeleton motion. Specifically, we model the temporal evolution of the 3D poses as trajectory, what allows us to map human motions to single points on a sphere manifold. Using such a compact representation avoids error accumulation and provides robust representation for long-term prediction while ensuring the smoothness and the coherence of the whole motion. To learn these non-Euclidean representations, we build a manifold-aware Wasserstein generative adversarial model that captures the temporal and spatial dependencies of human motion through different losses. Experiments have been conducted on CMU MoCap and Human 3.6M datasets and demonstrate the superiority of our approach over the state-of-the-art both in short and long term horizons. The smoothness of the generated motion is highlighted in the qualitative results.

Human Joint Kinematics Diffusion-Refinement for Stochastic Motion Prediction

Stochastic human motion prediction aims to forecast multiple plausible future motions given a single pose sequence from the past. Most previous works focus on designing elaborate losses to improve the accuracy, while the diversity is typically characterized by randomly sampling a set of latent variables from the latent prior, which is then decoded into possible motions. This joint training of sampling and decoding, however, suffers from posterior collapse as the learned latent variables tend to be ignored by a strong decoder, leading to limited diversity. Alternatively, inspired by the diffusion process in nonequilibrium thermodynamics, we propose MotionDiff, a diffusion probabilistic model to treat the kinematics of human joints as heated particles, which will diffuse from original states to a noise distribution. This process not only offers a natural way to obtain the "whitened'' latents without any trainable parameters, but also introduces a new noise in each diffusion step, both of which facilitate more diverse motions. Human motion prediction is then regarded as the reverse diffusion process that converts the noise distribution into realistic future motions conditioned on the observed sequence. Specifically, MotionDiff consists of two parts: a spatial-temporal transformer-based diffusion network to generate diverse yet plausible motions, and a flexible refinement network to further enable geometric losses and align with the ground truth. Experimental results on two datasets demonstrate that our model yields the competitive performance in terms of both diversity and accuracy.

Human Motion Prediction via Dual-Attention and Multi-Granularity Temporal Convolutional Networks

Intelligent devices, which significantly improve the quality of life and work efficiency, are now widely integrated into people’s daily lives and work. A precise understanding and analysis of human motion is essential for achieving harmonious coexistence and efficient interaction between intelligent devices and humans. However, existing human motion prediction methods often fail to fully exploit the dynamic spatial correlations and temporal dependencies inherent in motion sequence data, which leads to unsatisfactory prediction results. To address this issue, we proposed a novel human motion prediction method that utilizes dual-attention and multi-granularity temporal convolutional networks (DA-MgTCNs). Firstly, we designed a unique dual-attention (DA) model that combines joint attention and channel attention to extract spatial features from both joint and 3D coordinate dimensions. Next, we designed a multi-granularity temporal convolutional networks (MgTCNs) model with varying receptive fields to flexibly capture complex temporal dependencies. Finally, the experimental results from two benchmark datasets, Human3.6M and CMU-Mocap, demonstrated that our proposed method significantly outperformed other methods in both short-term and long-term prediction, thereby verifying the effectiveness of our algorithm.

Time Series Self-Attention Approach for Human Motion Forecasting: A Baseline 2D Pose Forecasting

Human motion forecasting is a necessary variable to analyze human motion concerning the safety system of the autonomous system that could be used in many applications, such as in auto-driving vehicles, auto-pilot logistics delivery, and gait analysis in the medical field. At the same time, many types of research have been conducted on 3D human motion prediction for short-term and long-term goals. This paper proposes human motion forecasting in the 2D plane as a reliable alternative in motion capture of the RGB camera attached to the devices. We proposed a method, the time series self-attention approach to generate the next future human motion in the short-term of 400 milliseconds and long-term of 1,000 milliseconds, resulting that the model could predict human motion with a slight error of 23.51 pixels for short-term prediction and 10.3 pixels for long-term prediction on average compared to the ground truth in the quantitative and qualitative evaluation. Our method outperformed the LSTM and GRU models on the Human3.6M dataset based on the MPJPE and MPJVE metrics. The average loss of correct key points varied based on the tolerance value. Our method performed better within the 50 pixels tolerance. In addition, our method is tested by images without key point annotations using OpenPose as the pose estimation method. Resulting, our method could predict well the position of the human but could not predict well for the human body pose. This research is a new baseline for the 2D human motion prediction using the Human3.6M dataset.

All-natural phyllosilicate-polysaccharide triboelectric sensor for machine learning-assisted human motion prediction

The rapid development of smart and carbon-neutral cities motivates the potential of natural materials for triboelectric electronics. However, the relatively deficient charge density makes it challenging to achieve high Maxwell’s displacement current. Here, we propose a methodology for improving the triboelectricity of marine polysaccharide by incorporating charged phyllosilicate nanosheets. As a proof-of-concept, a flexible, flame-retardant, and eco-friendly triboelectric sensor is developed based on all-natural composite paper from alginate fibers and vermiculite nanosheets. The interlaced fibers and nanosheets not only enable superior electrical output but also give rise to wear resistance and mechanical stability. The fabricated triboelectric sensor successfully monitors slight motion signals from various joints of human body. Moreover, an effective machine-learning model is developed for human motion identification and prediction with accuracy of 96.2% and 99.8%, respectively. This work offers a promising strategy for improving the triboelectricity of organo-substrates and enables implementation of self-powered and intelligent platform for emerging applications.

Bidirectional Transformer GAN for Long-term Human Motion Prediction

The mainstream motion prediction methods usually focus on short-term prediction, and their predicted long-term motions often fall into an average pose, i.e., the freezing forecasting problem [ 27 ]. To mitigate this problem, we propose a novel Bidirectional Transformer-based Generative Adversarial Network (BiTGAN) for long-term human motion prediction. The bidirectional setup leads to consistent and smooth generation in both forward and backward directions. Besides, to make full use of the history motions, we split them into two parts. The first part is fed to the Transformer encoder in our BiTGAN while the second part is used as the decoder input. This strategy can alleviate the exposure problem [ 37 ]. Additionally, to better maintain both the local (i.e., frame-level pose) and global (i.e., video-level semantic) similarities between the predicted motion sequence and the real one, the soft dynamic time warping (Soft-DTW) loss is introduced into the generator. Finally, we utilize a dual-discriminator to distinguish the predicted sequence at both frame and sequence levels. Extensive experiments on the public Human3.6M dataset demonstrate that our proposed BiTGAN achieves state-of-the-art performance on long-term (4 s ) human motion prediction, and reduces the average error of all actions by 4%.