Motion Capture Sequences Research Articles

Functional mobility assessment in Parkinson’s: A pilot feasibility study and a new methodological approach

Background
 Parkinson’s disease (PD) is associated with motor impairments that negatively impact functional mobility (Armstrong & Okun, 2020). Functional mobility is “a person’s ability to move to accomplish activities of daily living” (Bouça-Machado et al., 2020). Whilst several assessment scales exist to evaluate PD, many focus on symptom severity or use speed as a proxy for quality of movement (e.g. Timed Up and Go; Buckley et al., 2019; Mirelman et al., 2019). Here we share the development of an integrated motion capture and pressure-sensitive gait mat sequence to assess functional mobility.
 Design/Methods
 To capture aspects of functional mobility accurately, we developed a new assessment protocol (FMA-P) to measure which aspects of movement differed between people with (n = 12) and without (n = 12) PD. The sequence involved rising from a chair, locomotion, turning, bending, picking up and placing an object (keys), and returning to sit down on the chair (mean of three trials).
 Results
 The mean age of our participants was 65.25 ± 7.8 years, 46% females. We found significant differences in locomotion such as a lower step ratio (p = 0.023), stride length (p < 0.001), and stride velocity (p = 0.019), as well as a higher double support time (p = 0.004), and stride length variability (p = 0.004) in people with PD (PwP) compared to aged-matched healthy participants. PwP showed a lower toe-off set angle (p = 0.042), lower foot height (p = 0.007), and a lower heel strike angle (p = 0.006) during locomotion. Further, a lower shoulder-elbow arm swing angular velocity (p=0.042), and a higher elbow-wrist arm swing angular velocity (p = 0.054) were observed in PwP compared to controls. Getting up from a chair was associated with a lower inclination amplitude in PwP compared to healthy controls (p = 0.047). In the turning section, we found a later onset of pelvis rotation in PwP (p = 0.034). More time was required for PwP to pick the keys up from the floor compared to controls (p = 0.002).
 Discussion
 The new measurement protocol could contribute to early diagnosis of PD, offering a quantitative evaluation of functional mobility. For example, reduced toe and heel offset angles may be early determinants of developing shuffling in later disease stages. Moreover, our findings provide objective and precise information about a variety of qualitative aspects of functional mobility. This approach can therefore be used to evaluate change over time thereby improving the evidence base of outcome variables for intervention studies. The FMA-P is of high clinical relevance to aid in the quantitative evaluation of clinical measures of motor impairment.
 References
 Armstrong, M. J., & Okun, M. S. (2019). Diagnosis and treatment of Parkinson disease. A review. JAMA, 232(6), 548-560. https://doi.org/10.1001/jama.2019.22360
 Bouça-Machado, R., Duarte, G. S., Patriarca, M., Castro Caldas, A., Alarcão, J., Fernandes, R. M., Mestre, T. A.,, Matias, R., & Ferreira, J. J. (2020). Measurement instruments to assess functional mobility in Parkinson’s disease: A systematic review. Movement Disorders Clinical Practice, 7(2), 129-139. https://doi.org/10.1002/mdc3.12874
 Buckely, C., Alcock, L., McArdle, R., Rana Zia Ur Rehman, R. Z. U., Del Din, S., Mazzà, C., Yarnall, A.J., & Rochester, L. (2019). The role of movement analysis in diagnosing and monitoring neurodegenerative conditions: Insights from gait and postural control. Brain Sciences, 9(2), Article 34. https://doi.org/10.3390/brainsci9020034
 Mirelman, A., Bonato, P., Cmicioli, R., Ellis, T. D., Giladi, N., Hamilton, J. L., Hass, C. J., Hausdorff, J. M., Pelosin, E., & Almeida Q. J. (2019). Gait impairments in Parkinson’s disease. The Lancet Neurology, 18(7), 697-708. https://doi.org/10.1016/s1474-4422(19)30044-4

Skeleton-Based Action Segmentation With Multi-Stage Spatial-Temporal Graph Convolutional Neural Networks

The ability to identify and temporally segment fine-grained actions in motion capture sequences is crucial for applications in human movement analysis. Motion capture is typically performed with optical or inertial measurement systems, which encode human movement as a time series of human joint locations and orientations or their higher-order representations. State-of-the-art action segmentation approaches use multiple stages of temporal convolutions. The main idea is to generate an initial prediction with several layers of temporal convolutions and refine these predictions over multiple stages, also with temporal convolutions. Although these approaches capture long-term temporal patterns, the initial predictions do not adequately consider the spatial hierarchy among the human joints. To address this limitation, we recently introduced multi-stage spatial-temporal graph convolutional neural networks (MS-GCN). Our framework replaces the initial stage of temporal convolutions with spatial graph convolutions and dilated temporal convolutions, which better exploit the spatial configuration of the joints and their long-term temporal dynamics. Our framework was compared to four strong baselines on five tasks. Experimental results demonstrate that our framework is a strong baseline for skeleton-based action segmentation.

Tree based regression methods for gap reconstruction of motion capture sequences

In the article, we study application of tree based regression methods for reconstruction of gaps in marker trajectories in optical motion capture systems. Having perfect ground truth sequences, we artificially introduced gaps of lengths between 0.1 and 2 s, then reconstructed them using various tree based regression methods – CART, LS boost, bagging, random forests, M5P (model tree). We also tested several methods known from the literature – neural networks (NN), interpolation, and matrix completion. We found out that in general, tree based methods perform poorly, but M5P model tree performs very well, especially for longer gaps, with results on average on par or better than neural networks. Finally, we compared the outcomes of M5P and NN, they appear to be uncorrelated, so they offer the possibility to create an ensemble predictor based on both methods.

Anatomically Detailed Simulation of Human Torso

Many existing digital human models approximate the human skeletal system using rigid bodies connected by rotational joints. While the simplification is considered acceptable for legs and arms, it significantly lacks fidelity to model rich torso movements in common activities such as dancing, Yoga, and various sports. Research from biomechanics provides more detailed modeling for parts of the torso, but their models often operate in isolation and are not fast and robust enough to support computationally heavy applications and large-scale data generation for full-body digital humans. This paper proposes a new torso model that aims to achieve high fidelity both in perception and in functionality, while being computationally feasible for simulation and optimal control tasks. We build a detailed human torso model consisting of various anatomical components, including facets, ligaments, and intervertebral discs, by coupling efficient finite-element and rigid-body simulations. Given an existing motion capture sequence without dense markers placed on the torso, our new model is able to recover the underlying torso bone movements. Our method is remarkably robust that it can be used to automatically "retrofit" the entire Mixamo motion database of highly diverse human motions without user intervention. We also show that our model is computationally efficient for solving trajectory optimization of highly dynamic full-body movements, without relying any reference motion. Physiological validity of the model is validated against established literature.

Detection and Classification of Artifact Distortions in Optical Motion Capture Sequences

Optical motion capture systems are prone to errors connected to marker recognition (e.g., occlusion, leaving the scene, or mislabeling). These errors are then corrected in the software, but the process is not perfect, resulting in artifact distortions. In this article, we examine four existing types of artifacts and propose a method for detection and classification of the distortions. The algorithm is based on the derivative analysis, low-pass filtering, mathematical morphology, and loose predictor. The tests involved multiple simulations using synthetically-distorted sequences, performance comparisons to human operators (concerning real life data), and an applicability analysis for the distortion removal.

Gap Reconstruction in Optical Motion Capture Sequences Using Neural Networks.

Optical motion capture is a mature contemporary technique for the acquisition of motion data; alas, it is non-error-free. Due to technical limitations and occlusions of markers, gaps might occur in such recordings. The article reviews various neural network architectures applied to the gap-filling problem in motion capture sequences within the FBM framework providing a representation of body kinematic structure. The results are compared with interpolation and matrix completion methods. We found out that, for longer sequences, simple linear feedforward neural networks can outperform the other, sophisticated architectures, but these outcomes might be affected by the small amount of data availabe for training. We were also able to identify that the acceleration and monotonicity of input sequence are the parameters that have a notable impact on the obtained results.

Primitive Object Grasping for Finger Motion Synthesis

AbstractWe developed a new framework to generate hand and finger grasping motions. The proposed framework provides online adaptation to the position and orientation of objects and can generate grasping motions even when the object shape differs from that used during motion capture. This is achieved by using a mesh model, which we call primitive object grasping (POG), to represent the object grasping motion. The POG model uses a mesh deformation algorithm that keeps the original shape of the mesh while adapting to varying constraints. These characteristics are beneficial for finger grasping motion synthesis that satisfies constraints for mimicking the motion capture sequence and the grasping points reflecting the shape of the object. We verify the adaptability of the proposed motion synthesizer according to its position/orientation and shape variations of different objects by using motion capture sequences for grasping primitive objects, namely, a sphere, a cylinder, and a box. In addition, a different grasp strategy called a three‐finger grasp is synthesized to validate the generality of the POG‐based synthesis framework.

Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical Models

Deep video action recognition models have been highly successful in recent years but require large quantities of manually-annotated data, which are expensive and laborious to obtain. In this work, we investigate the generation of synthetic training data for video action recognition, as synthetic data have been successfully used to supervise models for a variety of other computer vision tasks. We propose an interpretable parametric generative model of human action videos that relies on procedural generation, physics models and other components of modern game engines. With this model we generate a diverse, realistic, and physically plausible dataset of human action videos, called PHAV for “Procedural Human Action Videos”. PHAV contains a total of 39,982 videos, with more than 1000 examples for each of 35 action categories. Our video generation approach is not limited to existing motion capture sequences: 14 of these 35 categories are procedurally-defined synthetic actions. In addition, each video is represented with 6 different data modalities, including RGB, optical flow and pixel-level semantic labels. These modalities are generated almost simultaneously using the Multiple Render Targets feature of modern GPUs. In order to leverage PHAV, we introduce a deep multi-task (i.e. that considers action classes from multiple datasets) representation learning architecture that is able to simultaneously learn from synthetic and real video datasets, even when their action categories differ. Our experiments on the UCF-101 and HMDB-51 benchmarks suggest that combining our large set of synthetic videos with small real-world datasets can boost recognition performance. Our approach also significantly outperforms video representations produced by fine-tuning state-of-the-art unsupervised generative models of videos.

Robust and automatic motion-capture data recovery using soft skeleton constraints and model averaging.

Motion capture allows accurate recording of human motion, with applications in many fields, including entertainment, medicine, sports science and human computer interaction. A common difficulty with this technology is the occurrence of missing data, due to occlusions, or recording conditions. Various models have been proposed to estimate missing data. Some are based on interpolation, low-rank properties or inter-correlations. Others involve dataset matching or skeleton constraints. While the latter have the advantage of promoting a realistic motion estimation, they require prior knowledge of skeleton constraints, or the availability of a prerecorded dataset. In this article, we propose a probabilistic averaging method of several recovery models (referred to as Probabilistic Model Averaging (PMA) in this paper), based on the likelihoods of the distances between body points. This method has the advantage of being automatic, while allowing an efficient gap data recovery. To support and validate the proposed method, we use a set of four individual recovery models, based on linear/nonlinear regression in local coordinate systems. Finally, we propose two heuristic algorithms to enforce skeleton constraints in the reconstructed motion, which can be used on any individual recovery model. For validation purposes, random gaps were introduced into motion-capture sequences, and the effects of factors such as the number of simultaneous gaps, gap length and sequence duration were analyzed. Results show that the proposed probabilistic averaging method yields better recovery than (i) each of the four individual models and (ii) two recent state-of-the-art models, regardless of gap length, sequence duration and number of simultaneous gaps. Moreover, both of our heuristic skeleton-constraint algorithms significantly improve the recovery for 7 out of 8 tested motion-capture sequences (p < 0.05), for 10 simultaneous gaps of 5 seconds. The code is available for free download at: https://github.com/numediart/MocapRecovery.

A physics-based approach to motion capture data processing for virtual plant modeling and simulation

Dynamic virtual plant simulation is an attractive research issue in both botany and computer graphics. Data-driven method is an efficient way for motion analysis and animation synthesis. As a widely used tool, motion capture has been used in plant motion data acquisition and analysis. The most prominent and important problem in motion capture for plants is primary data processing such as missing markers reconstruction. This paper presents a novel physics-based approach to motion capture data processing of plants. Firstly, a physics-based mechanics model is found by Lagrangian mechanics for a motion captured plant organ such as a leaf, and then its dynamic mechanical properties are analyzed and relevant model parameters are evaluated. Further, by using the physical model with evaluated parameters, we can calculate the next positions of a maker to reconstruct the missing makers in motion capture sequence. We take an example of a maize leaf and pachira leaf to examine the proposed approach, and the results show that the physics-based method is feasible and effective for plant motion data processing.

Self‐similarity Analysis for Motion Capture Cleaning

AbstractMotion capture sequences may contain erroneous data, especially when the motion is complex or performers are interacting closely and occlusions are frequent. Common practice is to have specialists visually detect the abnormalities and fix them manually. In this paper, we present a method to automatically analyze and fix motion capture sequences by using self‐similarity analysis. The premise of this work is that human motion data has a high‐degree of self‐similarity. Therefore, given enough motion data, erroneous motions are distinct when compared to other motions. We utilize motion‐words that consist of short sequences of transformations of groups of joints around a given motion frame. We search for the K‐nearest neighbors (KNN) set of each word using dynamic time warping and use it to detect and fix erroneous motions automatically. We demonstrate the effectiveness of our method in various examples, and evaluate by comparing to alternative methods and to manual cleaning.

Automatic Segmentation of Data Sequences

Segmenting temporal data sequences is an important problem which helps in understanding data dynamics in multiple applications such as epidemic surveillance, motion capture sequences, etc. In this paper, we give DASSA, the first self-guided and efficient algorithm to automatically find a segmentation that best detects the change of pattern in data sequences. To avoid introducing tuning parameters, we design DASSA to be a multi-level method which examines segments at each level of granularity via a compact data structure called the segment-graph. We build this data structure by carefully leveraging the information bottleneck method with the MDL principle to effectively represent each segment.Next, DASSA efficiently finds the optimal segmentation via a novel average-longest-path optimization on the segment-graph. Finally we show how the outputs from DASSA can be naturally interpreted to reveal meaningful patterns. We ran DASSA on multiple real datasets of varying sizes and it is very effective in finding the time-cut points of the segmentations (in some cases recovering the cut points perfectly) as well as in finding the corresponding changing patterns.

Generic Content-Based Retrieval of Marker-Based Motion Capture Data.

In this work, we propose an original scheme for generic content-based retrieval of marker-based motion capture data. It works on motion capture data of arbitrary subject types and arbitrary marker attachment and labelling conventions. Specifically, we propose a novel motion signature to statistically describe both the high-level and the low-level morphological and kinematic characteristics of a motion capture sequence, and conduct the content-based retrieval by computing and ordering the motion signature distance between the query and every item in the database. The distance between two motion signatures is computed by a weighted sum of differences in separate features contained in them. For maximum retrieval performance, we propose a method to pre-learn an optimal set of weights for each type of motion in the database through biased discriminant analysis, and adaptively choose a good set of weights for any given query at the run time. Excellence of the proposed scheme is experimentally demonstrated on various data sets and performance metrics.

3D skeleton‐based action recognition by representing motion capture sequences as 2D‐RGB images

AbstractIn recent years, 3D skeleton‐based action recognition has become a popular technique of action classification, thanks to development and availability of cheaper depth sensors. State‐of‐the‐art methods generally represent motion sequences as high dimensional trajectories followed by a time‐warping technique. These trajectories are used to train a classification model to predict the classes of new sequences. Despite the success of these techniques in some fields, particularly when the data used are captured by a high‐precision motion capture system, action classification is still less successful than the field of image classification, especially with the advance of deep learning. In this paper, we present a new representation of motion sequences (Seq2Im—for sequence to image), which projects motion sequences onto the RGB domain. The 3D coordinates of joints are mapped to red, green, and blue values, and therefore, action classification becomes an image classification problem and algorithms for this field can be applied. This representation was tested with basic image classification algorithms (namely, support vector machine, k‐nearest neighbor, and random forests) in addition to convolutional neural networks. Evaluation of the proposed method on standard 3D human action recognition datasets shows its potential for action recognition and outperforms most of the state‐of‐the‐art results.

Person identity recognition on motion capture data using multiple actions

In this paper, we introduce a novel method for person identity recognition (identification) on skeleton animation/motion capture data representing persons performing various actions. The joints positions or orientation angles and the forward differences of these quantities are used to represent a motion capture sequence. First K-means clustering is applied on training data to discover the most representative patterns on joints positions or orientation angles (dynemes) and their forward differences (F-dynemes). Each frame is then assigned to one of these patterns and the frequency of occurrence histograms for each movement are constructed in a bag-of-words fashion. Person identity recognition is done through a nearest neighbor classifier. The proposed method is experimentally tested on a number of datasets of motion capture data, with very good results.

Cohomological learning of periodic motion

This work develops a novel framework which can automatically detect, parameterize and interpolate periodic motion patterns obtained from a motion capture sequence. Using our framework, periodic motions such as walking and running gaits or any motion sequence with periodic structure such as cleaning, dancing etc. can be detected automatically and without manual marking of the period start and end points. Our approach constructs an intrinsic parameterization of the motion and is computationally fast. Using this parameterization, we are able generate prototypical periodic motions. Additionally, we are able to interpolate between various motions, yielding a rich class of ‘mixed’ periodic actions. Our approach is based on ideas from applied algebraic topology. In particular, we apply a novel persistent cohomology based method for the first time in a graphics application which enables us to recover circular coordinates of motions. We also develop a suitable notion of homotopy which can be used to interpolate between periodic motion patterns. Our framework is directly applicable to the construction of walk cycles for animating character motions with motion graphs or state machine driven animation engines and processed our examples at an average speed of 11.78 frames per second

3D Trajectory Reconstruction under Perspective Projection

We present an algorithm to reconstruct the 3D trajectory of a moving point from its correspondence in a collection of temporally non-coincidental 2D perspective images, given the time of capture that produced each image and the relative camera poses at each time instant. Triangulation-based solutions do not apply, as multiple views of the point may not exist at each time instant. We represent a 3D trajectory using a linear combination of compact trajectory basis vectors, such as the discrete cosine transform basis, that have been shown to approximate object independence. We note that such basis vectors are also coordinate independent, which allows us to directly use camera poses estimated from stationary areas in the scene (in contrast to nonrigid structure from motion techniques where cameras are simultaneously estimated). This reduces the reconstruction optimization to a linear least squares problem, allowing us to robustly handle missing data that often occur due to motion blur, texture deformation, and self occlusion. We present an algorithm to determine the number of trajectory basis vectors, individually for each trajectory via a cross validation scheme and refine the solution by minimizing the geometric error. The relationship between point and camera motion can cause degeneracies to occur. We geometrically analyze the problem by studying the relationship of the camera motion, point motion, and trajectory basis vectors. We define the reconstructability of a 3D trajectory under projection, and show that the estimate approaches the ground truth when reconstructability approaches infinity. This analysis enables us to precisely characterize cases when accurate reconstruction is achievable. We present qualitative results for the reconstruction of several real-world scenes from a series of 2D projections where high reconstructability can be guaranteed, and report quantitative results on motion capture sequences.

Skeleton-based action recognition with extreme learning machines

Action and gesture recognition from motion capture and RGB-D camera sequences has recently emerged as a renowned and challenging research topic. The current methods can usually be applied only to small datasets with a dozen or so different actions, and the systems often require large amounts of time to train the models and to classify new sequences. In this paper, we first extract simple but effective frame-level features from the skeletal data and build a recognition system based on the extreme learning machine. We then propose three modeling methods for post-processing the classification outputs to obtain the recognition results on the action sequence level. We test the proposed method on three public datasets ranging from 11 to 40 action classes. For all datasets, the method can classify the sequences with accuracies reaching 96–99% and with the average classification time for one sequence on a single computer core around 4 ms. Fast training and testing and the high accuracy make the proposed method readily applicable for online recognition applications.

Activity-based methods for person recognition in motion capture sequences

In this paper we present two algorithms for efficient person recognition operating upon motion capture data, depicting persons performing various everyday activities. The first approach is driven from the assumption that, if two motion sequences depict a certain activity performed by the same person, then, consecutive frames (poses) of one sequence are expected to be similar to consecutive frames of the other. The proposed method constructs a pose correspondence matrix to represent the similarity between poses and utilizes an intuitive method for estimating a similarity score between two motion capture sequences, based on the structure of the correspondence matrix. The second algorithm is based on a Bag of Words model (BoW), where histograms are extracted from motion sequences, based on the frequency of occurrences of characteristic poses. This method is combined with the application of Locality Preserving Projections (LPP) on the data, in order to reduce their dimensionality. Our methods achieved more than 98% correct person recognition rate, in three different datasets.

Action recognition on motion capture data using a dynemes and forward differences representation

In this paper we introduce a novel method for action/movement recognition in motion capture data. The joints orientation angles and the forward differences of these angles in different temporal scales are used to represent a motion capture sequence. Initially K-means is applied on training data to discover the most representative patterns on orientation angles and their forward differences. A novel K-means variant that takes into account the periodic nature of angular data is applied on the former. Each frame is then assigned to one or more of these patterns and histograms that describe the frequency of occurrence of these patterns for each movement are constructed. Nearest neighbour and SVM classification are used for action recognition on the test data. The effectiveness and robustness of this method is shown through extensive experimental results on four standard databases of motion capture data and various experimental setups.