Representation Of Human Motion Research Articles

Explainable AI in human motion: A comprehensive approach to analysis, modeling, and generation

Extensive research has been conducted on analyzing human movements, driven by its diverse practical applications such as human–robot interaction, human learning, and clinical diagnosis. However, the current state-of-the-art still encounters scientific challenges when it comes to modeling human movements. There are two key aspects that need to be addressed. Firstly, new models should consider the stochastic nature of human movement and the physical structure of the human body to accurately predict the patterns in full-body motion descriptors over time. Secondly, while deep learning algorithms have been utilized, they lack explainability in terms of predicting body posture sequences, making it essential to improve their comprehensible representation of human movement. This paper aims to tackle these challenges by presenting three innovative methods for creating explainable representations of human movement. The study formulates human body movement as a state-space model based on the Gesture Operational Model (GOM). Model parameters are estimated through either one-shot training employing Kalman Filters or data-intensive training utilizing artificial neural networks. The trained models are utilized for analyzing the dexterity of expert professionals in full-body movements, enabling the identification of dynamic associations between body joints and gesture recognition. Additionally, these models are employed to generate artificial professional movements.

Advancing Human Motion Recognition with SkeletonCLIP++: Weighted Video Feature Integration and Enhanced Contrastive Sample Discrimination.

This paper introduces 'SkeletonCLIP++', an extension of our prior work in human action recognition, emphasizing the use of semantic information beyond traditional label-based methods. The innovation, 'Weighted Frame Integration' (WFI), shifts video feature computation from averaging to a weighted frame approach, enabling a more nuanced representation of human movements in line with semantic relevance. Another key development, 'Contrastive Sample Identification' (CSI), introduces a novel discriminative task within the model. This task involves identifying the most similar negative sample among positive ones, enhancing the model's ability to distinguish between closely related actions. Incorporating the 'BERT Text Encoder Integration' (BTEI) leverages the pre-trained BERT model as our text encoder to refine the model's performance. Empirical evaluations on HMDB-51, UCF-101, and NTU RGB+D 60 datasets illustrate positive improvements, especially in smaller datasets. 'SkeletonCLIP++' thus offers a refined approach to human action recognition, ensuring semantic integrity and detailed differentiation in video data analysis.

Retracted: Human Motion Representation and Motion Pattern Recognition Based on Complex Fuzzy Theory

Retracted: Human Motion Representation and Motion Pattern Recognition Based on Complex Fuzzy Theory

Bodily expressed emotion understanding through integrating Laban movement analysis

Bodily expressed emotion understanding (BEEU) aims to automatically recognize human emotional expressions from body movements. Psychological research has demonstrated that people often move using specific motor elements to convey emotions. This work takes three steps to integrate human motor elements to study BEEU. First, we introduce BoME (body motor elements), a highly precise dataset for human motor elements. Second, we apply baseline models to estimate these elements on BoME, showing that deep learning methods are capable of learning effective representations of human movement. Finally, we propose a dual-source solution to enhance the BEEU model with the BoME dataset, which trains with both motor element and emotion labels and simultaneously produces predictions for both. Through experiments on the BoLD in-the-wild emotion understanding benchmark, we showcase the significant benefit of our approach. These results may inspire further research utilizing human motor elements for emotion understanding and mental health analysis.

Review for Optimal Human-Gesture Design Methodology and Motion representation of Medical Images using segmentation from depth data and gesture recognition.

Human gesture recognition and motion representation has become a vital base of current intelligent human-machine interfaces because of ubiquitous and more comfortable interaction. Human-Gesture recognition chiefly deals with recognizing meaningful, expressive body movements involving physical motions of face, head, arms, fingers, hands or body. This review article presents a concise overview of optimal human-gesture and motion representation of medical images. This paper surveys various works undertaken on human gesture design and discusses various design methodologies used for image segmentation and gesture recognition. It further provides a general idea of modeling techniques for analysing hand gesture images and even discusses the diverse techniques involved in motion recognition. This survey provides an insight into various efforts and developments made in the gesture/motion recognition domain through analyzing and reviewing the procedures, datasets, recognition rates and approaches employed for identifying diverse human motions and gestures for supporting better and devising improved applications in near future.

3-D Skeleton-Based Human Motion Prediction With Manifold-Aware GAN

In this work we propose a novel solution for 3D skeleton-based human motion prediction. The objective of this task consists in forecasting future human poses based on a prior skeleton pose sequence. This involves solving two main challenges still present in recent literature; (1) discontinuity of the predicted motion which results in unrealistic motions and (2) performance deterioration in long-term horizons resulting from error accumulation across time. We tackle these issues by using a compact manifold-valued representation of 3D human skeleton motion. Specifically, we model the temporal evolution of the 3D poses as trajectory, what allows us to map human motions to single points on a sphere manifold. Using such a compact representation avoids error accumulation and provides robust representation for long-term prediction while ensuring the smoothness and the coherence of the whole motion. To learn these non-Euclidean representations, we build a manifold-aware Wasserstein generative adversarial model that captures the temporal and spatial dependencies of human motion through different losses. Experiments have been conducted on CMU MoCap and Human 3.6M datasets and demonstrate the superiority of our approach over the state-of-the-art both in short and long term horizons. The smoothness of the generated motion is highlighted in the qualitative results.

Deep Learning-Based Motion Style Transfer Tools, Techniques and Future Challenges

In the fourth industrial revolution, the scale of execution for interactive applications increased substantially. These interactive and animated applications are human-centric, and the representation of human motion is unavoidable, making the representation of human motions ubiquitous. Animators strive to computationally process human motion in a way that the motions appear realistic in animated applications. Motion style transfer is an attractive technique that is widely used to create realistic motions in near real-time. motion style transfer approach employs existing captured motion data to generate realistic samples automatically and updates the motion data accordingly. This approach eliminates the need for handcrafted motions from scratch for every frame. The popularity of deep learning (DL) algorithms reshapes motion style transfer approaches, as such algorithms can predict subsequent motion styles. The majority of motion style transfer approaches use different variants of deep neural networks (DNNs) to accomplish motion style transfer approaches. This paper provides a comprehensive comparative analysis of existing state-of-the-art DL-based motion style transfer approaches. The enabling technologies that facilitate motion style transfer approaches are briefly presented in this paper. When employing DL-based methods for motion style transfer, the selection of the training dataset plays a key role in the performance. By anticipating this vital aspect, this paper provides a detailed summary of existing well-known motion datasets. As an outcome of the extensive overview of the domain, this paper highlights the contemporary challenges faced by motion style transfer approaches.

Structural feature representation and fusion of human spatial cooperative motion for action recognition

Structural feature representation and fusion of human spatial cooperative motion for action recognition

Skeleton-based motion prediction: A survey.

Human motion prediction based on 3D skeleton data is an active research topic in computer vision and multimedia analysis, which involves many disciplines, such as image processing, pattern recognition, and artificial intelligence. As an effective representation of human motion, human 3D skeleton data is favored by researchers because it provide resistant to light effects, scene changes, etc. earlier studies on human motion prediction focuses mainly on RBG data-based techniques. In recent years, researchers have proposed the fusion of human skeleton data and depth learning methods for human motion prediction and achieved good results. We first introduced human motion prediction research background and significance in this survey. We then summarized the latest deep learning-based techniques for predicting human motion in recent years. Finally, a detailed paper review and future development discussion are provided.

QMEDNet: A quaternion-based multi-order differential encoder–decoder model for 3D human motion prediction

In order to deal with the sequence information in the task of 3D human motion prediction effectively, many previous methods seek to predict the motion state of the next moment using the traditional recurrent neural network in Euclidean space. However human motion representation in Euclidean space has high distortion and shows a weak semantic expression when using deep learning models. In this work, we try to process human motion by mapping Euclidean space into a Hypercomplex vector space. We propose a novel model based on quaternion to predict the three-dimensional motion of a human body. The core idea of this study is to use the fusion information to understand and process the human motion state in quaternion space. The multi-order differential information is fused both in the encoder and decoder of feature extraction and mapped to the quaternion space, respectively. The encoder takes graph convolution as the basic unit and the decoder adopts gated recurrent units. Numerous experiments have been carried out to prove that the multi-order information in quaternion space can help build a more reasonable description for 3D human motion. The performance of the proposed QMEDNet is superior to most of the advanced short and long-term motion prediction methods in both public datasets, Human 3.6M and CMU Mocap.

MoCaNet: Motion Retargeting In-the-Wild via Canonicalization Networks

We present a novel framework that brings the 3D motion retargeting task from controlled environments to in-the-wild scenarios. In particular, our method is capable of retargeting body motion from a character in a 2D monocular video to a 3D character without using any motion capture system or 3D reconstruction procedure. It is designed to leverage massive online videos for unsupervised training, needless of 3D annotations or motion-body pairing information. The proposed method is built upon two novel canonicalization operations, structure canonicalization and view canonicalization. Trained with the canonicalization operations and the derived regularizations, our method learns to factorize a skeleton sequence into three independent semantic subspaces, i.e., motion, structure, and view angle. The disentangled representation enables motion retargeting from 2D to 3D with high precision. Our method achieves superior performance on motion transfer benchmarks with large body variations and challenging actions. Notably, the canonicalized skeleton sequence could serve as a disentangled and interpretable representation of human motion that benefits action analysis and motion retrieval.

Contextual spatio-temporal graph representation learning for reinforced human mobility mining

The rapid development of location-based services spurred a large number of user-centric applications. Particularly, an interesting topic has attracted the attention of researchers that is to link trajectories to users (TUL). Despite the significant progress made by recent deep learning-based human mobility learning models, tackling TUL problem is still challenging. In this paper, we propose a novel reinforced trajectory learning approach called GraphTUL that implements an adversarial network with the policy gradient to improve the identification ability and leverages both labeled and unlabeled trajectories to address the insufficient label issue in a semi-supervised manner. Besides, some critical factors related to personal context and indispensable elements in current mobility learning models are still missing. Thus, we propose a novel graph-based human motion representation model (CGE) to exploit the contextual information from users’ trajectories for alleviating data sparsity and contextual constraint issues. CGE builds a unified graph with historical check-ins to reflect users’ geographical preferences and visiting intentions. It allows us to sample synthetic but realistic trajectories for augmenting data and enhancing contextual check-in embedding. We also successfully apply it to next check-in prediction task. The experimental results conducted on several real-world datasets demonstrate that our proposed method achieves significantly better performance than the state-of-the-art baselines.

Human Motion Representation and Motion Pattern Recognition Based on Complex Fuzzy Theory

With the development of science and technology, the introduction of virtual reality technology has pushed the development of human-computer interaction technology to a new height. The combination of virtual reality and human-computer interaction technology has been applied more and more in military simulation, medical rehabilitation, game creation, and other fields. Action is the basis of human behavior. Among them, human behavior and action analysis is an important research direction. In human behavior and action, recognition research based on behavior and action has the characteristics of convenience, intuition, strong interaction, rich expression information, and so on. It has become the first choice of many researchers for human behavior analysis. However, human motion and motion pictures are complex objects with many ambiguous factors, which are difficult to express and process. Traditional motion recognition is usually based on two-dimensional color images, while two-dimensional RGB images are vulnerable to background disturbance, light, environment, and other factors that interfere with human target detection. In recent years, more and more researchers have begun to use fuzzy mathematics theory to identify human behaviors. The plantar pressure data under different motion modes were collected through experiments, and the current gait information was analyzed. The key gait events including toe-off and heel touch were identified by dynamic baseline monitoring. For the error monitoring of key gait events, the screen window is used to filter the repeated recognition events in a certain period of time, which greatly improves the recognition accuracy and provides important gait information for motion pattern recognition. The similarity matching is performed on each template, the correct rate of motion feature extraction is 90.2%, and the correct rate of motion pattern recognition is 96.3%, which verifies the feasibility and effectiveness of human motion recognition based on fuzzy theory. It is hoped to provide processing techniques and application examples for artificial intelligence recognition applications.

Erratum: Human Movement Representation on Multivariate Time Series for Recognition of Professional Gestures and Forecasting Their Trajectories

[This corrects the article DOI: 10.3389/frobt.2020.00080.].

Real-Time Prediction of Joint Forces by Motion Capture and Machine Learning.

Conventional biomechanical modelling approaches involve the solution of large systems of equations that encode the complex mathematical representation of human motion and skeletal structure. To improve stability and computational speed, being a common bottleneck in current approaches, we apply machine learning to train surrogate models and to predict in near real-time, previously calculated medial and lateral knee contact forces (KCFs) of 54 young and elderly participants during treadmill walking in a speed range of 3 to 7 km/h. Predictions are obtained by fusing optical motion capture and musculoskeletal modeling-derived kinematic and force variables, into regression models using artificial neural networks (ANNs) and support vector regression (SVR). Training schemes included either data from all subjects (LeaveTrialsOut) or only from a portion of them (LeaveSubjectsOut), in combination with inclusion of ground reaction forces (GRFs) in the dataset or not. Results identify ANNs as the best-performing predictor of KCFs, both in terms of Pearson R (0.89–0.98 for LeaveTrialsOut and 0.45–0.85 for LeaveSubjectsOut) and percentage normalized root mean square error (0.67–2.35 for LeaveTrialsOut and 1.6–5.39 for LeaveSubjectsOut). When GRFs were omitted from the dataset, no substantial decrease in prediction power of both models was observed. Our findings showcase the strength of ANNs to predict simultaneously multi-component KCF during walking at different speeds—even in the absence of GRFs—particularly applicable in real-time applications that make use of knee loading conditions to guide and treat patients.

Human Movement Representation on Multivariate Time Series for Recognition of Professional Gestures and Forecasting Their Trajectories

Human Movement Representation on Multivariate Time Series for Recognition of Professional Gestures and Forecasting Their Trajectories

Human Movement Representation on Multivariate Time Series for Recognition of Professional Gestures and Forecasting Their Trajectories.

Human-centered artificial intelligence is increasingly deployed in professional workplaces in Industry 4.0 to address various challenges related to the collaboration between the operators and the machines, the augmentation of their capabilities, or the improvement of the quality of their work and life in general. Intelligent systems and autonomous machines need to continuously recognize and follow the professional actions and gestures of the operators in order to collaborate with them and anticipate their trajectories for avoiding potential collisions and accidents. Nevertheless, the recognition of patterns of professional gestures is a very challenging task for both research and the industry. There are various types of human movements that the intelligent systems need to perceive, for example, gestural commands to machines and professional actions with or without the use of tools. Moreover, the interclass and intraclass spatiotemporal variances together with the very limited access to annotated human motion data constitute a major research challenge. In this paper, we introduce the Gesture Operational Model, which describes how gestures are performed based on assumptions that focus on the dynamic association of body entities, their synergies, and their serial and non-serial mediations, as well as their transitioning over time from one state to another. Then, the assumptions of the Gesture Operational Model are translated into a simultaneous equation system for each body entity through State-Space modeling. The coefficients of the equation are computed using the Maximum Likelihood Estimation method. The simulation of the model generates a confidence-bounding box for every entity that describes the tolerance of its spatial variance over time. The contribution of our approach is demonstrated for both recognizing gestures and forecasting human motion trajectories. In recognition, it is combined with continuous Hidden Markov Models to boost the recognition accuracy when the likelihoods are not confident. In forecasting, a motion trajectory can be estimated by taking as minimum input two observations only. The performance of the algorithm has been evaluated using four industrial datasets that contain gestures and actions from a TV assembly line, the glassblowing industry, the gestural commands to Automated Guided Vehicles as well as the Human–Robot Collaboration in the automotive assembly lines. The hybrid approach State-Space and HMMs outperforms standard continuous HMMs and a 3DCNN-based end-to-end deep architecture.

Exploiting upper-limb functional principal components for human-like motion generation of anthropomorphic robots

BackgroundHuman-likeliness of robot movements is a key component to enable a safe and effective human-robot interaction, since it contributes to increase acceptance and motion predictability of robots that have to closely interact with people, e.g. for assistance and rehabilitation purposes. Several parameters have been used to quantify how much a robot behaves like a human, which encompass aspects related to both the robot appearance and motion. The latter point is fundamental to allow the operator to interpret robotic actions, and plan a meaningful reactions. While different approaches have been presented in literature, which aim at devising bio-aware control guidelines, a direct implementation of human actions for robot planning is not straightforward, still representing an open issue in robotics.MethodsWe propose to embed a synergistic representation of human movements for robot motion generation. To do this, we recorded human upper-limb motions during daily living activities. We used functional Principal Component Analysis (fPCA) to extract principal motion patterns. We then formulated the planning problem by optimizing the weights of a reduced set of these components. For free-motions, our planning method results into a closed form solution which uses only one principal component. In case of obstacles, a numerical routine is proposed, incrementally enrolling principal components until the problem is solved with a suitable precision.ResultsResults of fPCA show that more than 80% of the observed variance can be explained by only three functional components. The application of our method to different meaningful movements, with and without obstacles, show that our approach is able to generate complex motions with a very reduced number of functional components. We show that the first synergy alone accounts for the 96% of cost reduction and that three components are able to achieve a satisfactory motion reconstruction in all the considered cases.ConclusionsIn this work we moved from the analysis of human movements via fPCA characterization to the design of a novel human-like motion generation algorithm able to generate, efficiently and with a reduced set of basis elements, several complex movements in free space, both in free motion and in case of obstacle avoidance tasks.

A review on multi-modal human motion representation recognition and its application in orthopedic rehabilitation training

Human motion recognition (HAR) is the technological base of intelligent medical treatment, sports training, video monitoring and many other fields, and it has been widely concerned by all walks of life. This paper summarized the progress and significance of HAR research, which includes two processes: action capture and action classification based on deep learning. Firstly, the paper introduced in detail three mainstream methods of action capture: video-based, depth camera-based and inertial sensor-based. The commonly used action data sets were also listed. Secondly, the realization of HAR based on deep learning was described in two aspects, including automatic feature extraction and multi-modal feature fusion. The realization of training monitoring and simulative training with HAR in orthopedic rehabilitation training was also introduced. Finally, it discussed precise motion capture and multi-modal feature fusion of HAR, as well as the key points and difficulties of HAR application in orthopedic rehabilitation training. This article summarized the above contents to quickly guide researchers to understand the current status of HAR research and its application in orthopedic rehabilitation training.

Predicting Perceived Naturalness of Human Animations Based on Generative Movement Primitive Models

We compared the perceptual validity of human avatar walking animations driven by six different representations of human movement using a graphics Turing test. All six representations are based on movement primitives (MPs), which are predictive models of full-body movement that differ in their complexity and prediction mechanism. Assuming that humans are experts at perceiving biological movement from noisy sensory signals, it follows that these percepts should be describable by a suitably constructed Bayesian ideal observer model. We build such models from MPs and investigate if the perceived naturalness of human animations are predictable from approximate Bayesian model scores of the MPs. We found that certain MP-based representations are capable of producing movements that are perceptually indistinguishable from natural movements. Furthermore, approximate Bayesian model scores of these representations can be used to predict perceived naturalness. In particular, we could show that movement dynamics are more important for perceived naturalness of human animations than single frame poses. This indicates that perception of human animations is highly sensitive to their temporal coherence. More generally, our results add evidence for a shared MP-representation of action and perception. Even though the motivation of our work is primarily drawn from neuroscience, we expect that our results will be applicable in virtual and augmented reality settings, when perceptually plausible human avatar movements are required.