Full-body Human Motion Research Articles

Object Motion Guided Human Motion Synthesis

Modeling human behaviors in contextual environments has a wide range of applications in character animation, embodied AI, VR/AR, and robotics. In real-world scenarios, humans frequently interact with the environment and manipulate various objects to complete daily tasks. In this work, we study the problem of full-body human motion synthesis for the manipulation of large-sized objects. We propose Object MOtion guided human MOtion synthesis (OMOMO), a conditional diffusion framework that can generate full-body manipulation behaviors from only the object motion. Since naively applying diffusion models fails to precisely enforce contact constraints between the hands and the object, OMOMO learns two separate denoising processes to first predict hand positions from object motion and subsequently synthesize full-body poses based on the predicted hand positions. By employing the hand positions as an intermediate representation between the two denoising processes, we can explicitly enforce contact constraints, resulting in more physically plausible manipulation motions. With the learned model, we develop a novel system that captures full-body human manipulation motions by simply attaching a smartphone to the object being manipulated. Through extensive experiments, we demonstrate the effectiveness of our proposed pipeline and its ability to generalize to unseen objects. Additionally, as high-quality human-object interaction datasets are scarce, we collect a large-scale dataset consisting of 3D object geometry, object motion, and human motion. Our dataset contains human-object interaction motion for 15 objects, with a total duration of approximately 10 hours.

Learning disentangled representations for controllable human motion prediction

Generative model-based motion prediction techniques have recently realized predicting controlled human motions, such as predicting multiple upper human body motions with similar lower-body motions. However, to achieve this, the state-of-the-art methods require either subsequently learning mapping functions to seek similar motions or training the model repetitively to enable control over the desired portion of body. In this paper, we propose a novel framework to learn disentangled representations for controllable human motion prediction. Our task is to predict multiple future human motions based on the past observed sequence, with the control of partial-body movements. Our network involves a conditional variational auto-encoder (CVAE) architecture to model full-body human motion, and an extra CVAE path to learn only the corresponding partial-body (e.g., lower-body) motion. Specifically, the inductive bias imposed by the extra CVAE path encourages two latent variables in two paths to respectively govern separate representations for each partial-body motion. With a single training, our model is able to provide two types of controls for the generated human motions: (i) strictly controlling one portion of human body and (ii) adaptively controlling the other portion, by sampling from a pair of latent spaces. Additionally, we extend and adapt a sampling strategy to our trained model to diversify the controllable predictions. Our framework also potentially allows new forms of control by flexibly customizing the input for the extra CVAE path. Extensive experimental results and ablation studies demonstrate that our approach is capable of predicting state-of-the-art controllable human motions both qualitatively and quantitatively.

Development of Smartphone Application for Markerless Three-Dimensional Motion Capture Based on Deep Learning Model.

To quantitatively assess pathological gait, we developed a novel smartphone application for full-body human motion tracking in real time from markerless video-based images using a smartphone monocular camera and deep learning. As training data for deep learning, the original three-dimensional (3D) dataset comprising more than 1 million captured images from the 3D motion of 90 humanoid characters and the two-dimensional dataset of COCO 2017 were prepared. The 3D heatmap offset data consisting of 28 × 28 × 28 blocks with three red–green–blue colors at the 24 key points of the entire body motion were learned using the convolutional neural network, modified ResNet34. At each key point, the hottest spot deviating from the center of the cell was learned using the tanh function. Our new iOS application could detect the relative tri-axial coordinates of the 24 whole-body key points centered on the navel in real time without any markers for motion capture. By using the relative coordinates, the 3D angles of the neck, lumbar, bilateral hip, knee, and ankle joints were estimated. Any human motion could be quantitatively and easily assessed using a new smartphone application named Three-Dimensional Pose Tracker for Gait Test (TDPT-GT) without any body markers or multipoint cameras.

Motion Inference Using Sparse Inertial Sensors, Self-Supervised Learning, and a New Dataset of Unscripted Human Motion.

In recent years, wearable sensors have become common, with possible applications in biomechanical monitoring, sports and fitness training, rehabilitation, assistive devices, or human-computer interaction. Our goal was to achieve accurate kinematics estimates using a small number of sensors. To accomplish this, we introduced a new dataset (the Virginia Tech Natural Motion Dataset) of full-body human motion capture using XSens MVN Link that contains more than 40 h of unscripted daily life motion in the open world. Using this dataset, we conducted self-supervised machine learning to do kinematics inference: we predicted the complete kinematics of the upper body or full body using a reduced set of sensors (3 or 4 for the upper body, 5 or 6 for the full body). We used several sequence-to-sequence (Seq2Seq) and Transformer models for motion inference. We compared the results using four different machine learning models and four different configurations of sensor placements. Our models produced mean angular errors of 10–15 degrees for both the upper body and full body, as well as worst-case errors of less than 30 degrees. The dataset and our machine learning code are freely available.

A novel approach to predicting human ingress motion using an artificial neural network

Due to the increased availability of digital human models, the need for knowing human movement is important in product design process. If the human motion is derived rapidly as design parameters change, a developer could determine the optimal parameters. For example, the optimal design of the door panel of an automobile can be obtained for a human operator to conduct the easiest ingress and egress motion. However, acquiring motion data from existing methods provides only unrealistic motion or requires a great amount of time. This not only leads to an increased time consumption for a product development, but also causes inefficiency of the overall design process. To solve such problems, this research proposes an algorithm to rapidly and accurately predict full-body human motion using an artificial neural network (ANN) and a motion database, as the design parameters are varied. To achieve this goal, this study refers to the processes behind human motor learning procedures. According to the previous research, human generate new motion based on past motion experience when they encounter new environments. Based on this principle, we constructed a motion capture database. To construct the database, motion capture experiments were performed in various environments using an optical motion capture system. To generate full-body human motion using this data, a generalized regression neural network (GRNN) was used. The proposed algorithm not only guarantees rapid and accurate results but also overcomes the ambiguity of the human motion objective function, which has been pointed out as a limitation of optimization-based research. Statistical criteria were utilized to confirm the similarity between the generated motion and actual human motion. Our research provides the basis for a rapid motion prediction algorithm that can include a variety of environmental variables. This research contributes to an increase in the usability of digital human models, and it can be applied to various research fields.

Human Pose Estimation from Video and IMUs.

In this work, we present an approach to fuse video with sparse orientation data obtained from inertial sensors to improve and stabilize full-body human motion capture. Even though video data is a strong cue for motion analysis, tracking artifacts occur frequently due to ambiguities in the images, rapid motions, occlusions or noise. As a complementary data source, inertial sensors allow for accurate estimation of limb orientations even under fast motions. However, accurate position information cannot be obtained in continuous operation. Therefore, we propose a hybrid tracker that combines video with a small number of inertial units to compensate for the drawbacks of each sensor type: on the one hand, we obtain drift-free and accurate position information from video data and, on the other hand, we obtain accurate limb orientations and good performance under fast motions from inertial sensors. In several experiments we demonstrate the increased performance and stability of our human motion tracker.

3D Human motion tracking by exemplar-based conditional particle filter

3D human motion tracking has received increasing attention in recent years due to its broad applications. Among various 3D human motion tracking methods, the particle filter is regarded as one of the most effective algorithms. However, there are still several limitations of current particle filter approaches such as low prediction accuracy and sensitivity to discontinuous motion caused by low frame rate or sudden change of human motion velocity. Targeting such problems, this paper presents a full-body human motion tracking system by proposing exemplar-based conditional particle filter (EC-PF) for monocular camera. By introducing a conditional term with respect to exemplars and image data, dynamic model is approximated and used to predict current states of particles in prediction phase. In update phase, weights of particles are refined by matching images with projected human model using a set of features.This method retains advantages of classic particle filters while increases prediction accuracy by replacing the smooth motion model with exemplars-based dynamic model which constrains evolved particles within an area closer to true state. Therefore, tracking robustness to discontinuous motion is improved such as under conditions of sudden change in motion velocity or using low-frame rate cameras. To verify the effectiveness and efficiency of the proposed algorithm, a variety of datasets are selected for testing and the results are also compared with the state-of-the-art methods in this domain.

Motion Tracking, Retrieval and 3D Reconstruction from Video

The work at hand presents a novel data-driven framework for 3D full body human motion reconstruction from uncalibrated monocular video data. To this end, we develop a knowledge base by taking 2D samples of the motion capture library from dierent viewing directions. This allows later steps to handle 2D query videos without any information on the viewing direction. We detect and track features from input video sequences by utilizing low-level image based feature detection techniques like MSER and SURF. This process is stabilized by back projection of high-level 3D prior information obtained from the motion capture library to the image plane. Extraction of suitable feature sets from both, input control signals and motion capture data, enables us to retrieve the best relevant prior poses from the motion capture library by employing fast motion retrieval techniques. Finally, 3D motion sequences are reconstructed by non-linear energy minimization, that takes into account multiple prior terms. Furthermore, we propose a method to estimate camera parameters from input video itself and sampling of motion capture library.

An Online Full-Body Motion Recognition Method Using Sparse and Deficient Signal Sequences

This paper presents a method to recognize continuous full-body human motion online by using sparse, low-cost sensors. The only input signals needed are linear accelerations without any rotation information, which are provided by four Wiimote sensors attached to the four human limbs. Based on the fused hidden Markov model (FHMM) and autoregressive process, a predictive fusion model (PFM) is put forward, which considers the different influences of the upper and lower limbs, establishes HMM for each part, and fuses them using a probabilistic fusion model. Then an autoregressive process is introduced in HMM to predict the gesture, which enables the model to deal with incomplete signal data. In order to reduce the number of alternatives in the online recognition process, a graph model is built that rejects parts of motion types based on the graph structure and previous recognition results. Finally, an online signal segmentation method based on semantics information and PFM is presented to finish the efficient recognition task. The results indicate that the method is robust with a high recognition rate of sparse and deficient signals and can be used in various interactive applications.

Continuous Full-Body Motion Control of Virtual Human Using Sparse Wiimotes

Sparse sensors that recognize full-body human motion and that control the motion of virtual humans have emerged as valuable research tools in the field of human–computer interactions. Here we propose a method for motion recognition and prolonged, continuous generation of motion data based on the recognition results. The only inputs required are the directional accelerations collected by four Wii remotes, which are attached on the four limbs of a human. The extended and continuous signal sequences are separated into small segments that can be described by particular motion content. Use of a fused hidden Markov model (FHMM) during the recognition process ensures the accuracy and efficiency with which independent motion segments are recognized. A graph model enhances the capacity of classification when dealing with a signal sequence associated with a prolonged motion. During the reconstruction and generation processes, an efficient state-based motion graph generates the extended and continuous virtual human motion data, which accurately reflects variation in the movement of the actors. Our method has a strong capacity to classify types of motion upon their recognition and the control process can be applied to a range of applications involving interaction.

The role of roles: Physical cooperation between humans and robots

Since the strict separation of working spaces of humans and robots has experienced a softening due to recent robotics research achievements, close interaction of humans and robots comes rapidly into reach. In this context, physical human–robot interaction raises a number of questions regarding a desired intuitive robot behavior. The continuous bilateral information and energy exchange requires an appropriate continuous robot feedback. Investigating a cooperative manipulation task, the desired behavior is a combination of an urge to fulfill the task, a smooth instant reactive behavior to human force inputs and an assignment of the task effort to the cooperating agents. In this paper, a formal analysis of human–robot cooperative load transport is presented. Three different possibilities for the assignment of task effort are proposed. Two proposed dynamic role exchange mechanisms adjust the robot’s urge to complete the task based on the human feedback. For comparison, a static role allocation strategy not relying on the human agreement feedback is investigated as well. All three role allocation mechanisms are evaluated in a user study that involves large-scale kinesthetic interaction and full-body human motion. Results show tradeoffs between subjective and objective performance measures stating a clear objective advantage of the proposed dynamic role allocation scheme.

Performance-based control interfaces using mixture of factor analyzers

This paper introduces an approach to performance animation that employs a small number of inertial measurement sensors to create an easy-to-use system for an interactive control of a full-body human character. Our key idea is to construct a global model from a prerecorded motion database and utilize them to construct full-body human motion in a maximum a posteriori framework (MAP). We have demonstrated the effectiveness of our system by controlling a variety of human actions, such as boxing, golf swinging, and table tennis, in real time. One unique property of our system is its ability to learn priors from a large and heterogeneous motion capture database and use them to generate a wide range of natural poses, a capacity that has not been demonstrated in previous data-driven character posing systems.

A Framework for Constructing Real-time Immersive Environments for Training Physical Activities

This paper describes a framework for constructing a three-dimensional immersive environment that can be used for training physical activities. The system is designed to capture three dimensional full-body human motion in real time and visualize the data either locally or remotely through three- dimensional display system so that the data can be viewed from arbitrary viewpoints. In the proposed system, an immersive environment is constructed through realistic reconstruction of a scene by applying a stereo algorithm on a set of images that is captured from multiple viewpoints. Specifically, twelve camera clusters that consist of four camera quadruples are used to capture the scene, where each camera cluster is processed by a pc independently and synchronously so that partial reconstructions from each viewpoint are merged to form a complete 3D description of the scene. This paper discusses in detail system architectures that enable synchronous operations across multiple computers while achieving parallel computations within each multi-processor system. A set of experiments is performed to learn tai-chi lessons in this environment where students are instructed to follow pre-recorded teacher’s movements while observing both their own motions and the teacher in real-time. The effect of learning in the virtual environment is discussed by comparing the performance of the trainee groups under different control environments.