Kinematic Skeleton Research Articles

TechnoSapiens: merging humans with technology in augmented reality

We present a marker-less AR/DR system that can replace the arm of the user with a virtual bionic prosthesis in real time including finger tracking. For this, we use a mixed reality HMD that provides the user with a stereo image based on video-see-through (VST). We apply chroma-keying to remove the user’s arm from each captured image and input reconstructed background information into the removed pixels. Before rendering the prosthesis model into the image, we re-target motion capture data of the user’s hand to the kinematic skeleton of the prosthesis to match the current hand pose. This system opens new research possibilities on self- and other-perception of bionic bodies. In a first evaluation study of the system, we propose that users perceive the virtual prosthesis model as a part of their body (i.e., that they experience a sense of ownership). We tested this assumption in a laboratory study with 27 individuals who used the system to perform a series of simple tasks in AR with their prosthesis. We measured body ownership and other measures with self-reports. In support of the hypothesis, users experienced a sense of body ownership. Also, a feeling of self-presence is induced during the task, and participants rated the overall experience as positive.

Biofidelic image registration for head and neck region utilizing an in-silico articulated skeleton as a transformation model

Objective. We propose an integration scheme for a biomechanical motion model into a deformable image registration. We demonstrate its accuracy and reproducibility for adaptive radiation therapy in the head and neck region. Approach. The novel registration scheme for the bony structures in the head and neck regions is based on a previously developed articulated kinematic skeleton model. The realized iterative single-bone optimization process directly triggers posture changes of the articulated skeleton, exchanging the transformation model within the deformable image registration process. Accuracy in terms of target registration errors in the bones is evaluated for 18 vector fields of three patients between each planning CT and six fraction CT scans distributed along the treatment course. Main results. The median of target registration error distribution of the landmark pairs is 1.4 ± 0.3 mm. This is sufficient accuracy for adaptive radiation therapy. The registration performs equally well for all three patients and no degradation of the registration accuracy can be observed throughout the treatment. Significance. Deformable image registration, despite its known residual uncertainties, is until now the tool of choice towards online re-planning automation. By introducing a biofidelic motion model into the optimization, we provide a viable way towards an in-build quality assurance.

Flextegrity simple cubic lattices

Flextegrity lattices are spatial grids composed of stiff segments kept in contact by compliant pre-tensioned tendons. The kinematic skeleton is sensible to the orientation of the segments, since their relative rotation produces the straining of the tendons to an amount that depends upon the angle of rotation and the shape of the pitch surfaces of the contact joints: these dictate the constitutive properties of the lattice in response to external actions. Two- and three-dimensional lattices are investigated, in which the contact pitch surfaces, obtained with axial-symmetric toothed conjugate profiles, mimic the kinematics of spheres, centred at the nodes of a simple cubic lattice, in pure rolling motion. The allowed mechanisms are discussed under infinitesimal deformation, to recognize possible eigenstress states in the lattice. The response under finite deformations is worked out for two-dimensional lattices under symmetric and asymmetric loading. The theoretical predictions are compared with experimental results on 3D-printed physical models. Possible extensions are discussed for lattices with segments of varying size, different arrangements and multi-stable contact joints. The flextegrity microstructure can represent a mesoscopic model for homogeneous crystals composed of non-pointwise molecules, but it could actually be manufactured in metamaterials with peculiar properties.

Modeling Physical Activity in Children by Combining Raw Hip-Worn Accelerometry, 2D Pose Estimation, and Direct Observation

This work presents an inductive search of physical activity intensity features for developing computer vision data sets used in automatic physical activity observation systems. An online survey was conducted to calibrate the ground truth definitions in a data set of video synchronized with accelerometers. Experts from the research field were asked to classify 24 short video samples of children&#x2019;s physical activity relative to three metabolic equivalence of task units &#x2013; the presumed threshold of moderate to vigorous physical activity. Leave-one-out disagreement analysis was applied until moderate agreement was achieved (12 respondents remain, Light&#x2019;s <inline-formula> <tex-math notation="LaTeX">$\kappa $ </tex-math></inline-formula> &#x003D; 0.62). The predictive power of features from hip-worn ActiGraph wGT3X-BT 30Hz raw accelerations and several freely available 2D pose estimation models are explored by cut-point analyses and logistic regression while using several approaches to account for uncertainty. Features from the acceleration- and video pose estimation domains are combined in correlation analyses in a twelve-hour data set representing relatively unstructured behavior from 24 children ages 8&#x2013;13 filmed in four different locations. Results indicate that changes in the hip angles of pose-estimated kinematic skeletons across 10 fps video frames can supplement or possibly substitute accelerometers for estimating physical activity intensity in uncrowded indoor scenes. When taking such an approach to labeling physical activity recognition data sets, good joint tracking capacity of the pose estimation method should increase the robustness of the hip angle features.

Symmetry-Aware Semantic Segmentation of a 3D Human Body Model

<p indent="0mm">An automatic segmentation method for a 3D human model is proposed for the symmetry-aware semantic segmentation. Firstly, a symmetry-aware kinematic skeleton is extracted from the input human body model via template skeleton embedding. Secondly, the number of the segments is determined by the kinematic skeleton; some key points corresponding to the segments are extracted meanwhile. Thirdly, while taking the key points as the boundary constraints, the harmonic fields are constructed to obtain a set of isolines as the potential cutting boundaries. Finally, the key points are taken as the clustering centers to conduct a spectral clustering operation, which guides the selections of the cutting boundaries. In this way, the semantic information in the kinematic skeleton can be directly transferred to the segments, and the number of segments can be controlled by the structure of the kinematic skeleton. Results on the SCAPE dataset, the MPI-FAUST dataset and the Princeton Segmentation Benchmark show that the proposed method can automatically achieve symmetry-aware semantic segmentation, and is robust to the shape and the pose of the human body model.

Simple action for depression detection: using kinect-recorded human kinematic skeletal data

BackgroundDepression, a common worldwide mental disorder, which brings huge challenges to family and social burden around the world is different from fluctuant emotion and psychological pressure in their daily life. Although body signs have been shown to present manifestations of depression in general, few researches focus on whole body kinematic cues with the help of machine learning methods to aid depression recognition. Using the Kinect V2 device to record participants’ simple kinematic skeleton data of the participant’s body joints, the presented spatial features and low-level features is directly extracted from the record original Kinect-3D coordinates. This research aimed to constructed machine learning model with the preprocessed data importing, which could be used for depression automatic classification.MethodsConsidering some patients’ conditions and current status and refer to psychiatrists’ advices, simple and significant designed stimulus task will lead human skeleton data collection job. With original Kinect skeleton data extracting and preprocessing, the proposed experiment demonstrated four strong machine learning tools: Support Vector Machine, Logistic Regression, Random Forest and Gradient Boosting. Using the precision, recall, sensitivity, specificity, roc-curve, confusion matrix et.al, indicators were calculated as the measurement of methods, which were commonly used to evaluate classification methodologies.ResultsAcross screened 64 pairs with age and gender totally matching in depression and control group, and Gradient Boosting achieved the best performance with the prediction accuracy of 76.92%. Sorted by female (54.69%) and male for the gender-based depression recognition, we applied best performance classifier Gradient Boosting got prediction accuracy of 66.67% in the male group, and 71.73% in the female group. Utilizing the best model Gradient Boosting for age-based classification, prediction accuracy got 76.92% in the older group (age >40, 50% of total) and 53.85% accuracy in the younger group (age <= 40).ConclusionThe depression and non-depression individuals can be well classified by computational models using Kinect captured skeletal data. The Gradient Boosting, an excellent machine learning tool, get the performance in the four methods we demonstrated. Meanwhile, in the gender-based depression classification also gets reasonable accuracy. In particular, the recognition results of the old group are significantly better than that of the young group. All these findings suggest that kinematic skeletal data based depression recognition can be applied as an effective tool for assisting in depression analysis.

MotioNet

We introduce MotioNet , a deep neural network that directly reconstructs the motion of a 3D human skeleton from a monocular video. While previous methods rely on either rigging or inverse kinematics (IK) to associate a consistent skeleton with temporally coherent joint rotations, our method is the first data-driven approach that directly outputs a kinematic skeleton, which is a complete, commonly used motion representation. At the crux of our approach lies a deep neural network with embedded kinematic priors, which decomposes sequences of 2D joint positions into two separate attributes: a single, symmetric skeleton encoded by bone lengths, and a sequence of 3D joint rotations associated with global root positions and foot contact labels. These attributes are fed into an integrated forward kinematics (FK) layer that outputs 3D positions, which are compared to a ground truth. In addition, an adversarial loss is applied to the velocities of the recovered rotations to ensure that they lie on the manifold of natural joint rotations. The key advantage of our approach is that it learns to infer natural joint rotations directly from the training data rather than assuming an underlying model, or inferring them from joint positions using a data-agnostic IK solver. We show that enforcing a single consistent skeleton along with temporally coherent joint rotations constrains the solution space, leading to a more robust handling of self-occlusions and depth ambiguities.

Kinematic skeleton graph augmented network for human parsing

Human parsing, which is a task of labeling pixels in human images into different fine-grained semantic parts, has achieved significant progress during the past decade. However, there are still several challenges in human parsing, due to occlusions, varying poses and similar appearance between the left/right parts. To tackle these problems, a Human Kinematic Skeleton Graph Layer (HKSGL) is proposed to augment regular neural networks with human kinematic skeleton information. The HKSGL has two major components: kinematic skeleton graph and interconnected modular neural layer. The kinematic skeleton graph is a user pre-defined skeleton graph, which models the interconnections between different semantic parts. Then the skeleton graph is passed to the interconnected modular neural layer which is composed of a set of modular blocks corresponding to different semantic parts. The HKSGL is a lightweight, low costs layer which can be easily attached to any existing neural networks. To demonstrate the power of the HKSGL, a new dataset on human parsing in occlusions is also collected, termed the RAP-Occ. Extensive experiments have been performed on four datasets on human parsing, including the LIP, the CIHP, the ATR and the RAP-Occ. And two popular baselines, i.e., the Deeplab V3+ and the CE2P, are agumented by the proposed HKSGL. Competitive performance of the augmented models has been achieved in comparison with state-of-the-art methods.

Symmetry-aware kinematic skeleton generation of a 3D human body model

In this paper, an automatic method is proposed to generate a symmetry-aware kinematic skeleton for a human body model with an arbitrary pose and orientation. First, a template kinematic skeleton with semantics is embedded into the input human body model. Then, the joints of the embedded kinematic skeleton are refined according to the geometry of the human body model and some prior knowledge. Finally, a specific local coordinate system is defined on the kinematic skeleton and is used to distinguish the symmetry of the kinematic skeleton. In this way, the symmetric joints of the kinematic skeleton, e.g., the left knee joint and the right knee joint, can be distinguished. Quantitative and qualitative analysis and comparison show that the proposed method can generate a symmetry-aware kinematic skeleton with accurate joints and has no restrictions on the pose and orientation of the input human body model. Moreover, this paper presents validation of the proposed method in many applications, such as shape alignment, shape deformation, shape co-segmentation and shape correspondence.

A Hybrid Material Point Method for Frictional Contact with Diverse Materials

We present a new hybrid Lagrangian Material Point Method for simulating elastic objects like hair, rubber, and soft tissues that utilizes a Lagrangian mesh for internal force computation and an Eulerian mesh for self collision as well as coupling with external materials. While recent Material Point Method (MPM) techniques allow for natural simulation of hyperelastic materials represented with Lagrangian meshes, they utilize an updated Lagrangian discretization where the Eulerian grid degrees of freedom are used to take variations of the potential energy. This often coarsens the degrees of freedom of the Lagrangian mesh and can lead to artifacts. We develop a hybrid approach that retains Lagrangian degrees of freedom while still allowing for natural coupling with other materials simulated with traditional MPM, e.g. sand, snow, etc. Furthermore, while recent MPM advances allow for resolution of frictional contact with codimensional simulation of hyperelasticity, they do not generalize to the case of volumetric materials. We show that our hybrid approach resolves these issues. We demonstrate the efficacy of our technique with examples that involve elastic soft tissues coupled with kinematic skeletons, extreme deformation, and coupling with multiple elastoplastic materials. Our approach also naturally allows for two-way rigid body coupling.

Markerless Multiview Motion Capture with 3D Shape Model Adaptation

AbstractIn this paper, we address simultaneous markerless motion and shape capture from 3D input meshes of partial views onto a moving subject. We exploit a computer graphics model based on kinematic skinning as template tracking model. This template model consists of vertices, joints and skinning weights learned a priori from registered full‐body scans, representing true human shape and kinematics‐based shape deformations. Two data‐driven priors are used together with a set of constraints and cues for setting up sufficient correspondences. A Gaussian mixture model‐based pose prior of successive joint configurations is learned to soft‐constrain the attainable pose space to plausible human poses. To make the shape adaptation robust to outliers and non‐visible surface regions and to guide the shape adaptation towards realistically appearing human shapes, we use a mesh‐Laplacian‐based shape prior. Both priors are learned/extracted from the training set of the template model learning phase. The output is a model adapted to the captured subject with respect to shape and kinematic skeleton as well as the animation parameters to resemble the observed movements. With example applications, we demonstrate the benefit of such footage. Experimental evaluations on publicly available datasets show the achieved natural appearance and accuracy.

VNect

We present the first real-time method to capture the full global 3D skeletal pose of a human in a stable, temporally consistent manner using a single RGB camera. Our method combines a new convolutional neural network (CNN) based pose regressor with kinematic skeleton fitting. Our novel fully-convolutional pose formulation regresses 2D and 3D joint positions jointly in real time and does not require tightly cropped input frames. A real-time kinematic skeleton fitting method uses the CNN output to yield temporally stable 3D global pose reconstructions on the basis of a coherent kinematic skeleton. This makes our approach the first monocular RGB method usable in real-time applications such as 3D character control---thus far, the only monocular methods for such applications employed specialized RGB-D cameras. Our method's accuracy is quantitatively on par with the best offline 3D monocular RGB pose estimation methods. Our results are qualitatively comparable to, and sometimes better than, results from monocular RGB-D approaches, such as the Kinect. However, we show that our approach is more broadly applicable than RGB-D solutions, i.e., it works for outdoor scenes, community videos, and low quality commodity RGB cameras.

Multi‐Variate Gaussian‐Based Inverse Kinematics

AbstractInverse kinematics (IK) equations are usually solved through approximated linearizations or heuristics. These methods lead to character animations that are unnatural looking or unstable because they do not consider both the motion coherence and limits of human joints. In this paper, we present a method based on the formulation of multi‐variate Gaussian distribution models (MGDMs), which precisely specify the soft joint constraints of a kinematic skeleton. Each distribution model is described by a covariance matrix and a mean vector representing both the joint limits and the coherence of motion of different limbs. The MGDMs are automatically learned from the motion capture data in a fast and unsupervised process. When the character is animated or posed, a Gaussian process synthesizes a new MGDM for each different vector of target positions, and the corresponding objective function is solved with Jacobian‐based IK. This makes our method practical to use and easy to insert into pre‐existing animation pipelines. Compared with previous works, our method is more stable and more precise, while also satisfying the anatomical constraints of human limbs. Our method leads to natural and realistic results without sacrificing real‐time performance.

MARCOnI-ConvNet-Based MARker-Less Motion Capture in Outdoor and Indoor Scenes.

Marker-less motion capture has seen great progress, but most state-of-the-art approaches fail to reliably track articulated human body motion with a very low number of cameras, let alone when applied in outdoor scenes with general background. In this paper, we propose a method for accurate marker-less capture of articulated skeleton motion of several subjects in general scenes, indoors and outdoors, even from input filmed with as few as two cameras. The new algorithm combines the strengths of a discriminative image-based joint detection method with a model-based generative motion tracking algorithm through an unified pose optimization energy. The discriminative part-based pose detection method is implemented using Convolutional Networks (ConvNet) and estimates unary potentials for each joint of a kinematic skeleton model. These unary potentials serve as the basis of a probabilistic extraction of pose constraints for tracking by using weighted sampling from a pose posterior that is guided by the model. In the final energy, we combine these constraints with an appearance-based model-to-image similarity term. Poses can be computed very efficiently using iterative local optimization, since joint detection with a trained ConvNet is fast, and since our formulation yields a combined pose estimation energy with analytic derivatives. In combination, this enables to track full articulated joint angles at state-of-the-art accuracy and temporal stability with a very low number of cameras. Our method is efficient and lends itself to implementation on parallel computing hardware, such as GPUs. We test our method extensively and show its advantages over related work on many indoor and outdoor data sets captured by ourselves, as well as data sets made available to the community by other research labs. The availability of good evaluation data sets is paramount for scientific progress, and many existing test data sets focus on controlled indoor settings, do not feature much variety in the scenes, and often lack a large corpus of data with ground truth annotation. We therefore further contribute with a new extensive test data set called MPI-MARCOnI for indoor and outdoor marker-less motion capture that features 12 scenes of varying complexity and varying camera count, and that features ground truth reference data from different modalities, ranging from manual joint annotations to marker-based motion capture results. Our new method is tested on these data, and the data set will be made available to the community.

Articulated and Generalized Gaussian Kernel Correlation for Human Pose Estimation.

In this paper, we propose an articulated and generalized Gaussian kernel correlation (GKC)-based framework for human pose estimation. We first derive a unified GKC representation that generalizes the previous sum of Gaussians (SoG)-based methods for the similarity measure between a template and an observation both of which are represented by various SoG variants. Then, we develop an articulated GKC (AGKC) by integrating a kinematic skeleton in a multivariate SoG template that supports subject-specific shape modeling and articulated pose estimation for both the full body and the hands. We further propose a sequential (body/hand) pose tracking algorithm by incorporating three regularization terms in the AGKC function, including visibility, intersection penalty, and pose continuity. Our tracking algorithm is simple yet effective and computationally efficient. We evaluate our algorithm on two benchmark depth data sets. The experimental results are promising and competitive when compared with the state-of-the-art algorithms.

Kinematic Skeleton Based Control of a Virtual Simulator for Military Training

Virtual simulation technology has been considered as a highly efficient and cost-effective solution for a soldier training system, and evolved into diverse combinations of hardware and software. To maximize the virtual reality effect within a restricted space, a locomotion interface such as an omni-directional treadmill is introduced as a major component of a virtual simulator, therefore real time interaction between human and the virtual simulator becomes very important. Displacement and heading changes of the trainee are crucial information to control the virtual simulator when we implement highly reactive motion control for the omni-directional treadmill and interaction control of the virtual contents. This paper proposes a control parameter estimation algorithm for the virtual training simulator by using two types of motion capture sensors and presents the experimental results. Kinematic joint positions are analyzed to estimate the trainee’s location and velocity for feedback and feedforward control of the omni-directional treadmill. The accuracy of two approaches is evaluated by comparing with the reference system, which gives a ground truth value.

Designing an Adaptive 3D Body Model Suitable for People with Limited Body Abilities

The purpose of this research was to develop a generalized adaptive 3D body model the posture of which could be adapted to different positions in order to perform a virtual prototyping of garments for people with limited body abilities. The digital data of a tested person’s body was acquired using the GOM Atos scanning system. Careful surface reconstruction was performed in order to provide an adequate mesh for further posture adaptation. The experimental part of this article presents the usages of a variety of graphic programs in order to provide an adaptive 3D body model through kinematic skeleton construction. The mesh-deformation during posture adaptation was improved using advanced tools of applied software packages. The usefulness of the gained 3D body model was determined by calculating the differences between the real and digital 3D body model measurements.

Kinematic skeleton extraction from 3D articulated models

We propose a method to extract a kinematic skeleton from an articulated 3D model. Unlike the curve skeleton which concentrates on the abstract structure of given model, we aim to obtain the skeleton with accurate joint positions, which can be directly used for character rigging and animation. Restricting our focus on the articulated body, we construct its skeleton in a bottom-up manner starting from a set of automatically obtained feature points. Our method is a hybrid approach combining the advantages of topology-based and geometry-based methods using the Morse theory and the shape descriptor. It does not require manually-chosen feature points or markers, and is independent of both postures and the number of branches of the given model. Experiments show that our method can efficiently extract kinematic skeletons of various kinds of articulated bodies, and also that the results can be directly applied to the character rigging.

Representing mesh-based character animations

We propose a new approach to represent and manipulate a mesh-based character animation preserving its time-varying details. Our method first decomposes the input mesh animation into coarse and fine deformation components. A model for the coarse deformations is constructed by an underlying kinematic skeleton structure and blending skinning weights. Thereafter, a non-linear probabilistic model is used to encode the fine time-varying details of the input animation. The user can manipulate the corresponding skeleton-based component of the input, which can be done by any standard animation package, and the final result is generated including its important time-varying details. By converting an input sample animation into our new hybrid representation, we are able to maintain the flexibility of mesh-based methods during animation creation while allowing for practical manipulations using the standard skeleton-based paradigm. We demonstrate the performance of our method by converting and manipulating several mesh animations generated by different performance capture approaches and apply it to represent and manipulate cloth simulation data.

A novel template‐based automatic rigging algorithm for articulated‐character animation

ABSTRACTRigging is a process for creating skeletons used to animate articulated characters. In conventional computer‐animation software, this process must be performed manually. Although several automatic rigging algorithms have been proposed, these methods still require user intervention. This paper proposes an automatic algorithm that generates an inverse kinematic skeleton for a character by locating an appropriate template skeleton on the extracted curve skeleton of the input 3D character model. After the curve skeleton is extracted, it is analyzed and classified into an appropriate category. The classification conditions are developed from the characteristics of each kind of real animal. We also develop an algorithm to extract the anatomical meaning of each skeleton segment. On the basis of the classification result, a suitable template skeleton is retrieved from the database. Each bone of the template skeleton can then be located on the appropriate skeleton segment of the input skeleton graph by using the extracted anatomical meanings. In contrast to previous methods, the algorithm does not require the input 3D character models to have certain poses or orientations. Moreover, all processes can be completed without user intervention. Copyright © 2012 John Wiley &amp; Sons, Ltd.