Human Motion Reconstruction Research Articles

Calibrationless monocular vision musculoskeletal simulation during gait

With computer vision technology and prediction of ground reaction forces (GRF), a previous study performed markerless motion capture and musculoskeletal simulation with two smartphones (OpenCap). A recent approach can reconstruct 3D human motion from a single video without calibration and it may further simplify the motion capture process. However it has not been combined with musculoskeletal simulation and the validity is unclear. Therefore, the purpose of this study was to determine the validity of the musculoskeletal simulation using a monocular vision approach. An open-source dataset that contains motion capture and video data during gait from 10 healthy participants was used. Human motion reconstruction with the skinned human (SMPL) model was performed on each video. Virtual marker data was generated by extracting the position data from the SMPL skin vertices. Inverse kinematics, GRF prediction (only for monocular vision approach), inverse dynamics and static optimization were performed using a musculoskeletal model for experimental motion capture data and the generated virtual markers from videos. Mean absolute errors (MAE) between motion capture based and monocular vision based simulation outcomes were calculated. The MAE were 8.4° for joint angles, 5.0 % bodyweight for GRF, 1.1 % bodyweight*height for joint moments and 0.11 for estimated muscle activations from 16 muscles. The entire MAE was larger but some were comparable to OpenCap. Using the monocular vision approach, motion capture and musculoskeletal simulation can be done with no preparations and is beneficial for clinicians to quantify the daily gait assessment.

Human Motion Enhancement and Restoration via Unconstrained Human Structure Learning.

Human motion capture technology, which leverages sensors to track the movement trajectories of key skeleton points, has been progressively transitioning from industrial applications to broader civilian applications in recent years. It finds extensive use in fields such as game development, digital human modeling, and sport science. However, the affordability of these sensors often compromises the accuracy of motion data. Low-cost motion capture methods often lead to errors in the captured motion data. We introduce a novel approach for human motion reconstruction and enhancement using spatio-temporal attention-based graph convolutional networks (ST-ATGCNs), which efficiently learn the human skeleton structure and the motion logic without requiring prior human kinematic knowledge. This method enables unsupervised motion data restoration and significantly reduces the costs associated with obtaining precise motion capture data. Our experiments, conducted on two extensive motion datasets and with real motion capture sensors such as the SONY (Tokyo, Japan) mocopi, demonstrate the method's effectiveness in enhancing the quality of low-precision motion capture data. The experiments indicate the ST-ATGCN's potential to improve both the accessibility and accuracy of motion capture technology.

Spatial-Related Sensors Matters: 3D Human Motion Reconstruction Assisted with Textual Semantics

Leveraging wearable devices for motion reconstruction has emerged as an economical and viable technique. Certain methodologies employ sparse Inertial Measurement Units (IMUs) on the human body and harness data-driven strategies to model human poses. However, the reconstruction of motion based solely on sparse IMU data is inherently fraught with ambiguity, a consequence of numerous identical IMU readings corresponding to different poses. In this paper, we explore the spatial importance of sparse sensors, supervised by text that describes specific actions. Specifically, uncertainty is introduced to derive weighted features for each IMU. We also design a Hierarchical Temporal Transformer (HTT) and apply contrastive learning to achieve precise temporal and feature alignment of sensor data with textual semantics. Experimental results demonstrate our proposed approach achieves significant improvements in multiple metrics compared to existing methods. Notably, with textual supervision, our method not only differentiates between ambiguous actions such as sitting and standing but also produces more precise and natural motion.

Fast Human Motion reconstruction from sparse inertial measurement units considering the human shape

Inertial Measurement Unit-based methods have great potential in capturing motion in large-scale and complex environments with many people. Sparse Inertial Measurement Unit-based methods have more research value due to their simplicity and flexibility. However, improving the computational efficiency and reducing latency in such methods are challenging. In this paper, we propose Fast Inertial Poser, which is a full body motion estimation deep neural network based on 6 inertial measurement units considering body parameters. We design a network architecture based on recurrent neural networks according to the kinematics tree. This method introduces human body shape information by the causality of observations and eliminates the dependence on future frames. During the estimation of joint positions, the upper body and lower body are estimated using separate network modules independently. Then the joint rotation is obtained through a well-designed single-frame kinematics inverse solver. Experiments show that the method can greatly improve the inference speed and reduce the latency while ensuring the reconstruction accuracy compared with previous methods. Fast Inertial Poser runs at 65 fps with 15 ms latency on an embedded computer, demonstrating the efficiency of the model.

MDPose: Human Skeletal Motion Reconstruction Using WiFi Micro-Doppler Signatures

Motion tracking systems based on optical sensors typically suffer from poor lighting conditions, occlusion, limited coverage, and may raise privacy concerns. More recently, radiofrequency (RF) based approaches using commercial WiFi devices have emerged which offer low-cost ubiquitous sensing whilst preservin privacy. However, RF sensing systems typically output range-Doppler maps, time-frequency spectrograms, cross-range plots etc which cannot represent human motion intuitively and usually requires further processing. In this study, we propose MDPose, a novel framework for human skeletal motion reconstruction base on WiFi micro-Doppler signatures. MDPose provides an effective solution to represent human activity by reconstructing a skeleton model with 17 key points, which can assist with the interpretation of conventional RF sensing outputs in a more understandable way. Specifically, MDPose is implemented over three sequential stage to address a series of challenges: First, a denoising algorithm is employed to remove any unwanted noise that may affect feature extraction and enhance weak Doppler measurements. Secondly, a convolutional neural network (CNN)-recurrent neural network (RNN) architecture is applied to learn temporal spatial dependenc from clean micro-Doppler signatures and restore velocity information to key points under the supervision of the motion capture (Mocap) system. Finally, a pose optimisation mechanism based on learning optimisation vectors is employed to estimate the initial state of the skeleton and to limit additional errors. We hav conducted a comprehensive set of tests in a variety of environments using numerous subjects with a single receiver radar system to demonstrate the performance of MDPose, and report 29.4mm mean absolute error over all key points positions on several common daily activities, which has performance comparable to that of state-ofthe- art RF-based pose estimation systems.

SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data

Accurate and reliable human motion reconstruction is crucial for creating natural interactions of full-body avatars in Virtual Reality (VR) and entertainment applications. As the Metaverse and social applications gain popularity, users are seeking cost-effective solutions to create full-body animations that are comparable in quality to those produced by commercial motion capture systems. In order to provide affordable solutions though, it is important to minimize the number of sensors attached to the subject’s body. Unfortunately, reconstructing the full-body pose from sparse data is a heavily under-determined problem. Some studies that use IMU sensors face challenges in reconstructing the pose due to positional drift and ambiguity of the poses. In recent years, some mainstream VR systems have released 6-degree-of-freedom (6-DoF) tracking devices providing positional and rotational information. Nevertheless, most solutions for reconstructing full-body poses rely on traditional inverse kinematics (IK) solutions, which often produce non-continuous and unnatural poses. In this article, we introduce SparsePoser, a novel deep learning-based solution for reconstructing a full-body pose from a reduced set of six tracking devices. Our system incorporates a convolutional-based autoencoder that synthesizes high-quality continuous human poses by learning the human motion manifold from motion capture data. Then, we employ a learned IK component, made of multiple lightweight feed-forward neural networks, to adjust the hands and feet toward the corresponding trackers. We extensively evaluate our method on publicly available motion capture datasets and with real-time live demos. We show that our method outperforms state-of-the-art techniques using IMU sensors or 6-DoF tracking devices, and can be used for users with different body dimensions and proportions.

Human motion capture, reconstruction, and musculoskeletal analysis in real time

Optical motion capture is an essential tool for the study and analysis of human movement. Currently, most manufacturers of motion-capture systems provide software applications for reconstructing the movement in real time, thus allowing for on-the-fly visualization. The captured kinematics can be later used as input data for a further musculoskeletal analysis. However, in advanced biofeedback applications, the results of said analysis, such as joint torques, ground-reaction forces, muscle efforts, and joint-reaction forces, are also required in real time.In this work, an extended Kalman filter (EKF) previously developed by the authors for real-time, whole-body motion capture and reconstruction is augmented with inverse dynamics and muscle-efforts optimization, enabling the calculation and visualization of the latter, along with joint-reaction forces, while capturing the motion.A modified version of the existing motion-capture algorithm provides the positions, velocities, and accelerations at every time step. Then, the joint torques are calculated by solving the inverse-dynamics problem, using force-plate measurements along with previously estimated body-segment parameters. Once the joint torques are obtained, an optimization problem is solved, in order to obtain the muscle forces that provide said torques while minimizing an objective function. This is achieved by a very efficient quadratic programming algorithm, thoroughly tuned for this specific problem.With this procedure, it is possible to capture and label the optical markers, reconstruct the motion of the model, solve the inverse dynamics, and estimate the individual muscle forces, all while providing real-time visualization of the results.

An attention-based deep learning approach for inertial motion recognition and estimation in human-robot collaboration

In line with a human-centric smart manufacturing vision, human-robot collaboration is striving to combine robots’ high efficiency and quality with humans’ rapid adaptability and high flexibility. In particular, perception, recognition and estimation of human motion determine when and what robot to collaborate with humans. This work presents an attention-based deep learning approach for inertial motion recognition and estimation in order to infer when robotic assistance will be requested by the human and to allow the robot to perform partial human tasks. First, in the stage of motion perception and recognition, quaternion-based calibration and forward kinematic analysis methods enable the reconstruction of human motion based on data streaming from an inertial motion capture device. Then, in the stage of motion estimation, residual module and Bidirectional Long Short-Term Memory module are integrated with proposed attention mechanism for estimating arm motion trajectories further. Experimental results show the effectiveness of the proposed approach in achieving better recognition and estimation in comparison with traditional approaches and existing deep learning approaches. It is experimentally verified in a laboratory environment involving a collaborative robot employed in a small part assembly task.

Optimal Reconstruction of Human Motion From Scarce Multimodal Data

Wearable sensing has emerged as a promising solution for enabling unobtrusive and ergonomic measurements of the human motion. However, the reconstruction performance of these devices strongly depends on the quality and the number of sensors, which are typically limited by wearability and economic constraints. A promising approach to minimize the number of sensors is to exploit dimensionality reduction approaches that fuse <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">prior</i> information with insufficient sensing signals, through minimum variance estimation. These methods were successfully used for static hand pose reconstruction, but their translation to motion reconstruction has not been attempted yet. In this work, we propose the usage of functional principal component analysis to decompose multimodal, time-varying motion profiles in terms of linear combinations of basis functions. Functional decomposition enables the estimation of the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a priori</i> covariance matrix, and hence the fusion of scarce and noisy measured data with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a priori</i> information. We also consider the problem of identifying which elemental variables to measure as the most informative for a given class of tasks. We applied our method to two different datasets of upper limb motion D1 (joint trajectories) and D2 (joint trajectories + EMG data) considering an optimal set of measures (four joints for D1 out of seven, three joints, and eight EMGs for D2 out of seven and twelve, respectively). We found that our approach enables the reconstruction of upper limb motion with a median error of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$0.013 \pm 0.006$</tex-math></inline-formula> rad for D1 (relative median error 0.9%), and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$0.038 \pm 0.023$</tex-math></inline-formula> rad and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$0.003 \pm 0.002$</tex-math></inline-formula> mV for D2 (relative median error 2.9% and 5.1%, respectively).

Dynamic Monitoring Method of Physical Training Intensity Based on Wearable Sensors

Physical training data are greatly affected by the human movement state, and the current monitoring method has the problem of low accuracy. A dynamic monitoring method of physical training intensity is designed based on the wearable sensor. The wearable sensor network is established, the quaternion is transformed into the corresponding Euler angle, and the measurement of the sensor is transformed into the human body coordinate system. In the process of human motion reconstruction, the human motion state parameters are estimated based on the wearable sensor data, the parameter eigenvalues are extracted, and the decision tree classification algorithm is used to identify the physical training state. According to the classification results, the dynamic monitoring model of three-dimensional energy training intensity is established, and the numerical index of training intensity is obtained. In terms of various physical training items, the average monitoring accuracy of the dynamic monitoring method of physical training intensity based on the wearable sensor is 97.12%, which is 3.70%, 4.59%, and 5.04% higher than the methods based on reflected interframe image, median average filtering algorithm, and Internet of Things technology, respectively, so as to realize the accurate monitoring of sports state.

A Review: Deep Learning for 3D Reconstruction of Human Motion Detection

3D reconstruction of human motion is an important research topic in VR/AR content creation, virtual fitting, human-computer interaction and other fields. Deep learning theory has made important achievements in human motion detection, recognition, tracking and other aspects, and human motion detection and recognition is an important link in 3D reconstruction. In this paper, the deep learning algorithms in recent years, mainly used for human motion detection and recognition, are reviewed, and the existing methods are divided into three types: CNN-based, RNN-based and GNN-based. At the same time, the main stream data sets and frameworks adopted in the references are summarized. The content of this paper provides some references for the research of 3D reconstruction of human motion.

Human motion reconstruction using deep transformer networks

• An attention-based deep neural network for natural human motion reconstruction. • An effective model for producing realistic human motion from very few constraints. • A deep neural network outperforming previous RNN-based methods and the Transformer. • The best feature combination for faithful human motion reconstruction. Establishing a human motion reconstruction system from very few constraints imposed on the body has been an interesting and important research topic because it significantly reduces the degrees of freedom to be managed. However, it is a well-known mathematically ill-posed problem as the dimension of constraints is much lower than that of the human pose to be determined. Therefore, it is challenging to directly reconstruct the whole body joint information from very few constraints due to many possible solutions. To address this issue, we present a novel deep learning framework with an attention mechanism using large-scale motion capture (mocap) data for mapping very few user-defined constraints into the human motion as realistically as possible. Our system is built upon the attention networks for looking back further to achieve better results. Experimental results show that our network model is capable of producing more accurate results compared with previous approaches. We also conducted several experiments to test all possible combinations of the features extracted from the mocap data, and found the best feature combination to generate high-quality poses.

Feature Recognition of Human Motion Behavior Based on Depth Sequence Analysis

The current research on still image recognition has been very successful, but the study of action recognition for video classes is still a challenging topic. In this work, we propose a random projection-based human action recognition algorithm to address the lack of depth information in color information (RGB video frames) that is not easily affected by environmental factors such as illumination and the lack of ability to recognize actions along the direction of view. A network structure is designed to take the obvious advantage of long- and short-term memory networks for controlling and remembering long sequences of historical information. The network structure in this paper is constituted by multiple memory units. At the same time, this paper constructs the spatial features, temporal features, and depth features of the three recognition stream outputs into a feature matrix, whose feature matrix is divided into multiple temporal segments according to the temporal dimension, then inputs them into the network layer in order, and achieves the fusion of the feature matrix in this paper according to their correlation characteristics on the temporal axis. Here, we proposed the concept of random batch projection operators. This basically uses as much sublimitation information as possible to improve projection accuracy by randomly selecting several subdependencies as projections defined during projection. A compressed sensing design of human motion acceleration data for low-power body area networks is proposed, and the basic idea and implementation process of compressed sensing theory for human motion data compression and reconstruction in wireless body area networks are introduced in detail.

A pilot study on locomotion training via biomechanical models and a wearable haptic feedback system

Locomotion is a fundamental human skill. Real-time sensing and feedback is a promising strategy for motion training to reconstruct healthy locomotion patterns lost due to aging or disease, and to prevent injuries. In this paper, we present a pilot study on locomotion training via biomechanical modeling and a wearable haptic feedback system. In addition, a novel simulation framework for motion tracking and analysis is introduced. This unified framework, implemented within the Unity environment, is used to analyze subject’s baseline and performance characteristics, and to provide real-time haptic feedback during locomotion. The framework incorporates accurate musculoskeletal models derived from OpenSim, closed-form calculations of muscle routing kinematics and kinematic Jacobian matrices, dynamic performance metrics (i.e., muscular effort), human motion reconstruction via inertial measurement unit (IMU) sensors, and real-time visualization of the motion and its dynamics. A pilot study was conducted in which 6 healthy subjects learned to alter running patterns to lower the knee flexion moment (KFM) through haptic feedback. We targeted three gait parameters (trunk lean, cadence, and foot strike) that previous studies had identified as having an influence on reducing the knee flexion moment and associated with increased risk of running injuries. All subjects were able to adopt altered running patterns requiring simultaneous changes to these kinematic parameters and reduced their KFM to 30–85% of their baseline values. The muscular effort during motion training stayed comparable to subjects’ baseline. This study shows that biomechanical modeling, together with real-time sensing and wearable haptic feedback can greatly increase the efficiency of motion training.

A Review on Human Motion Capture

The evolution in technology leads to current situation of detection and recognition of human movements which is more prevalent due to many applications in various fields. This paper provides with some recent trends in human motion capture (HMC) technology. Animation is a dynamic factor in different fields like entertainment, medicine etc. about which objects or images are manipulated to resemble as running pictures. For the animation process to speed up more, motion capture was invented, a means by which we are able to capture real-time motions and then recorded motion data into a three-dimensional model in virtual environment. Human motion capture is the method of capturing human motion from real life using various sensing objects into the computer. This paper discusses about some human motion capture methods, their processing, tracking and reconstruction of motion. A brief review of literature with different methodologies is provided. Mainly focusing on recent motion capture technologies, the utilization of easier, less time consuming and accurate motion capture methods.

A Novel Adaptive Kalman Filtering Approach to Human Motion Tracking With Magnetic-Inertial Sensors

This article presents an adaptive Kalman filtering approach to estimate the orientation of human body segments by using magnetic-inertial measurement units (MIMUs) with three-axis gyroscope, accelerometer, and magnetometer. In order to mitigate the negative impact of the external accelerations and ferromagnetic disturbances, the disturbances are modeled by first-order Gauss–Markov processes, and two parallel adaptive Kalman filters containing disturbance models are implemented to eliminate the disturbances. An adaptive factor is introduced to adjust the process noise covariance matrix to compensate for modeling errors induced by unmodeled gyroscope biases and the erroneous process noise covariance matrices. Furthermore, the outliers are checked and discarded by hypothesis tests on the innovation, and the measurement is rejected when the innovation exceeds a given confidence interval. Then, the estimated gravity acceleration and geomagnetic field are used to calculate the orientation quaternion using triaxial attitude determination algorithm. Finally, experiments under different motion scenarios are carried out to evaluate the effectiveness of the proposed method, and the performance of the proposed method is discussed in comparison with existing ones. A forward kinematics three-dimensional (3-D) human motion reconstruction method is proposed to drive the human skeleton model to reproduce human motion with a visual interface based on ROS platform.

Single-view and Multi-view Methods in Marker-less 3D Human Motion Capture

Human motion capture has now played a pivotal role in more and more applications, including biomechanics, sports, image segment, animation, robotics, etc. Although commercial marker-based human motion capture models have matured, the shortcomings, such as obtrusion, expense, errors due to damage to the marker trajectories, long set-up times and etc. exposed by this approach are becoming more and more apparent. Marker-less human motion capture analysis is likely to provide inexpensive and efficient solutions to solving these problems for the reconstruction of human motion in the future. In this paper, we discuss and compare the background and characteristics of marker-based and marker-less human motion capture models. Then we divide the marker-less human motion capture into single view and multi view and display some popular models. These methods are also categorized according to their internal logic and algorithms. Finally, we present some of the major shortcomings of the current marker-less human motion capture models and the future direction of development.

Real‐time 3D human pose and motion reconstruction from monocular RGB videos

AbstractReal‐time three‐dimensional (3D) pose estimation is of high interest in interactive applications, virtual reality, activity recognition, and most importantly, in the growing gaming industry. In this work, we present a method that captures and reconstructs the 3D skeletal pose and motion articulation of multiple characters using a monocular RGB camera. Our method deals with this challenging, but useful, task by taking advantage of the recent development in deep learning that allows two‐dimensional (2D) pose estimation of multiple characters and the increasing availability of motion capture data. We fit 2D estimated poses, extracted from a single camera via OpenPose, with a 2D multiview joint projections database that is associated with their 3D motion representations. We then retrieve the 3D body pose of the tracked character, ensuring throughout that the reconstructed movements are natural, satisfy the model constraints, are within a feasible set, and are temporally smooth without jitters. We demonstrate the performance of our method in several examples, including human locomotion, simultaneously capturing of multiple characters, and motion reconstruction from different camera views.

Reconstruction of 3D human motion in real-time using particle swarm optimization with GPU-accelerated fitness function

In this paper, a novel framework for acceleration of 3D model-based, markerless visual tracking in multi-camera videos is proposed. The objective function being the most computationally demanding part of model-based 3D motion reconstruction is calculated on a GPU. The proposed framework effectively utilizes the rendering power of OpenGL to render the 3D models in the predicted poses, whereas the CUDA threads are used to match such rendered models with the image observations and to perform particle swarm optimization-based tracking. We demonstrate effective parallelization of the particle swarm optimization on GPU. Execution of time-consuming parts of the algorithm on GPU using CUDA-OpenGL significantly accelerates the 3D motion reconstruction, making our method capable of tracking full-body movements with a maximum speed of 15 fps. Qualitative and quantitative experimental results on various four-camera benchmark datasets demonstrate the efficiency and accuracy of our method for real-time motion tracking.

Reconstruction of human swing leg motion with passive biarticular muscle models

Template models, which are utilized to demonstrate general aspects in human locomotion, mostly investigate stance leg operation. The goal of this paper is presenting a new conceptual walking model benefiting from swing leg dynamics. Considering a double pendulum equipped with combinations of biarticular springs for the swing leg beside spring-mass (SLIP) model for the stance leg, a novel SLIP-based model, is proposed to explain human-like leg behavior in walking. The action of biarticular muscles in swing leg motion helps represent human walking features, like leg retraction, ground reaction force and generating symmetric walking patterns, in simulations. In order to stabilize the motion by the proposed passive structure, swing leg biarticular muscle parameters such as lever arm ratios, stiffnesses and rest lengths need to be properly adjusted. Comparison of simulation results with human experiments shows the ability of the proposed model in replicating kinematic and kinetic behavior of both stance and swing legs as well as biarticular thigh muscle force of the swing leg. This substantiates the important functional role of biarticular muscles in leg swing.