Skeleton-based Approaches Research Articles

Multidimensional Refinement Graph Convolutional Network With Robust Decouple Loss for Fine-Grained Skeleton-Based Action Recognition.

Graph convolutional networks (GCNs) have been widely used in skeleton-based action recognition. However, existing approaches are limited in fine-grained action recognition due to the similarity of interclass data. Moreover, the noisy data from pose extraction increase the challenge of fine-grained recognition. In this work, we propose a flexible attention block called channel-variable spatial-temporal attention (CVSTA) to enhance the discriminative power of spatial-temporal joints and obtain a more compact intraclass feature distribution. Based on CVSTA, we construct a multidimensional refinement GCN (MDR-GCN) that can improve the discrimination among channel-, joint-, and frame-level features for fine-grained actions. Furthermore, we propose a robust decouple loss (RDL) that significantly boosts the effect of the CVSTA and reduces the impact of noise. The proposed method combining MDR-GCN with RDL outperforms the known state-of-the-art skeleton-based approaches on fine-grained datasets, FineGym99 and FSD-10, and also on the coarse NTU-RGB + D 120 dataset and NTU-RGB + D X-view version. Our code is publicly available at https://github.com/dingyn-Reno/MDR-GCN.

Human Body Pose Estimation for Gait Identification: A Comprehensive Survey of Datasets and Models

Person identification is a problem that has received substantial attention, particularly in security domains. Gait recognition is one of the most convenient approaches enabling person identification at a distance without the need for high-quality images. There are several review studies addressing person identification such as the utilization of facial images, silhouette images, and wearable sensor. Despite skeleton-based person identification gaining popularity while overcoming the challenges of traditional approaches, existing survey studies lack the comprehensive review of skeleton-based approaches to gait identification. We present a detailed review of the human pose estimation and gait analysis that makes the skeleton-based approaches possible. The study covers various types of related datasets, tools, methodologies, and evaluation metrics with associated challenges, limitations, and application domains. Detailed comparisons are presented for each of these aspects with recommendations for potential research and alternatives. A common trend throughout this paper is the positive impact that deep learning techniques are beginning to have on topics such as human pose estimation and gait identification. The survey outcomes might be useful for the related research community and other stakeholders in terms of performance analysis of existing methodologies, potential research gaps, application domains, and possible contributions in the future.

Diffusion Tensor Based White Matter Tract Atlases for Pediatric Populations.

Diffusion Tensor Imaging (DTI) is a non-invasive neuroimaging method that has become the most widely employed MRI modality for investigations of white matter fiber pathways. DTI has proven especially valuable for improving our understanding of normative white matter maturation across the life span and has also been used to index clinical pathology and cognitive function. Despite its increasing popularity, especially in pediatric research, the majority of existing studies examining infant white matter maturation depend on regional or white matter skeleton-based approaches. These methods generally lack the sensitivity and spatial specificity of more advanced functional analysis options that provide information about microstructural properties of white matter along fiber bundles. DTI studies of early postnatal brain development show that profound microstructural and maturational changes take place during the first two years of life. The pattern and rate of these changes vary greatly throughout the brain during this time compared to the rest of the life span. For this reason, appropriate image processing of infant MR imaging requires the use of age-specific reference atlases. This article provides an overview of the pre-processing, atlas building, and the fiber tractography procedures used to generate two atlas resources, one for neonates and one for 1- to 2-year-old populations. Via the UNC-NAMIC DTI Fiber Analysis Framework, our pediatric atlases provide the computational templates necessary for the fully automatic analysis of infant DTI data. To the best of our knowledge, these atlases are the first comprehensive population diffusion fiber atlases in early pediatric ages that are publicly available.

3DMesh-GAR: 3D Human Body Mesh-Based Method for Group Activity Recognition.

Group activity recognition is a prime research topic in video understanding and has many practical applications, such as crowd behavior monitoring, video surveillance, etc. To understand the multi-person/group action, the model should not only identify the individual person’s action in the context but also describe their collective activity. A lot of previous works adopt skeleton-based approaches with graph convolutional networks for group activity recognition. However, these approaches are subject to limitation in scalability, robustness, and interoperability. In this paper, we propose 3DMesh-GAR, a novel approach to 3D human body Mesh-based Group Activity Recognition, which relies on a body center heatmap, camera map, and mesh parameter map instead of the complex and noisy 3D skeleton of each person of the input frames. We adopt a 3D mesh creation method, which is conceptually simple, single-stage, and bounding box free, and is able to handle highly occluded and multi-person scenes without any additional computational cost. We implement 3DMesh-GAR on a standard group activity dataset: the Collective Activity Dataset, and achieve state-of-the-art performance for group activity recognition.

Skeleton-based relational reasoning for group activity analysis

Research on group activity recognition mostly leans on the standard two-stream approach (RGB and Optical Flow) as their input features. Few have explored explicit pose information, with none using it directly to reason about the persons interactions. In this paper, we leverage the skeleton information to learn the interactions between the individuals straight from it. With our proposed method GIRN, multiple relationship types are inferred from independent modules, that describe the relations between the body joints pair-by-pair. Additionally to the joints relations, we also experiment with the previously unexplored relationship between individuals and relevant objects (e.g. volleyball). The individuals distinct relations are then merged through an attention mechanism, that gives more importance to those individuals more relevant for distinguishing the group activity. We evaluate our method in the Volleyball dataset, obtaining competitive results to the state-of-the-art. Our experiments demonstrate the potential of skeleton-based approaches for modeling multi-person interactions.

An investigation on skeleton-based top-down modelling approaches of complex industrial product

In industry, today’s approach to assembly design is still largely based on a bottom-up approach which, in contrast with the most advanced top-down techniques, is unfit to deal with very large and complex products. The reason for this lies in the high number of relationships to be established between parts and in the lack of a high-level control of the assembly design. This makes the management of design changes a labor-intensive process and the capture of design intent difficult to achieve. The paper, referring to the most advanced research fields of Concurrent Engineering and Knowledge-Based Engineering, focuses on a top-down modelling approach based on skeleton, which constitutes the most natural but still scarcely exploited way to attain a high reactivity to design modifications. Through the application of suitable methodologies, such as that one for a SKeLeton geometry–based Assembly Context Definition (SKL-ACD), the skeleton is also able to capture and codify assembly process engineering information since the early phases of the product development process. With the purpose of promoting the knowledge of these skeleton-based modelling techniques, that have a great relevance for training professional, technical and mechanical engineers, this paper implements the SKL-ACD methodology to an industrial case study in order to identify, with a unique and repeatable workflow, the reference geometrical entities and the mutual relationships to embed into the product skeleton. The skeleton types and the related fields of use are also described, placing particular emphasis on problems or shortcomings still not resolved, especially in consideration of the need to assist the designer in defining the impact of a parameter on assembly modification and in avoiding loops while defining formulas. A new tool, in the form of a multilayer graph, is finally proposed that is able to display and differentiate clearly the formulas, the design parameters and the impact of their modification on skeleton entities and members of the assembly.

Spatio Temporal Joint Distance Maps for Skeleton-Based Action Recognition Using Convolutional Neural Networks

Skeleton-based action recognition has become popular with the recent developments in sensor technology and fast pose estimation algorithms. The existing research works have attempted to address the action recognition problem by considering either spatial or temporal dynamics of the actions. But, both the features (spatial and temporal) would contribute to solve the problem. In this paper, we address the action recognition problem using 3D skeleton data by introducing eight Joint Distance Maps, referred to as Spatio Temporal Joint Distance Maps (ST-JDMs), to capture spatio temporal variations from skeleton data for action recognition. Among these, four maps are defined in spatial domain and remaining four are in temporal domain. After construction of ST-JDMs from an action sequence, they are encoded into color images. This representation enables us to fine-tune the Convolutional Neural Network (CNN) for action classification. The empirical results on the two datasets, UTD MHAD and NTU RGB+D, show that ST-JDMs outperforms the other state-of-the-art skeleton-based approaches by achieving recognition accuracies 91.63% and 80.16%, respectively.

A Cuboid CNN Model With an Attention Mechanism for Skeleton-Based Action Recognition

The introduction of depth sensors such as Microsoft Kinect have driven research in human action recognition. Human skeletal data collected from depth sensors convey a significant amount of information for action recognition. While there has been considerable progress in action recognition, most existing skeleton-based approaches neglect the fact that not all human body parts move during many actions, and they fail to consider the ordinal positions of body joints. Here, and motivated by the fact that an action's category is determined by local joint movements, we propose a cuboid model for skeleton-based action recognition. Specifically, a cuboid arranging strategy is developed to organize the pairwise displacements between all body joints to obtain a cuboid action representation. Such a representation is well structured and allows deep CNN models to focus analyses on actions. Moreover, an attention mechanism is exploited in the deep model, such that the most relevant features are extracted. Extensive experiments on our new Yunnan University-Chinese Academy of Sciences-Multimodal Human Action Dataset (CAS-YNU MHAD), the NTU RGB+D dataset, the UTD-MHAD dataset, and the UTKinect-Action3D dataset demonstrate the effectiveness of our method compared to the current state-of-the-art.

A real-time gesture recognition system using near-infrared imagery.

Visual hand gesture recognition systems are promising technologies for Human Computer Interaction, as they allow a more immersive and intuitive interaction. Most of these systems are based on the analysis of skeleton information, which is in turn inferred from color, depth, or near-infrared imagery. However, the robust extraction of skeleton information from images is only possible for a subset of hand poses, which restricts the range of gestures that can be recognized. In this paper, a real-time hand gesture recognition system based on a near-infrared device is presented, which directly analyzes the infrared imagery to infer static and dynamic gestures, without using skeleton information. Thus, a much wider range of hand gestures can be recognized in comparison with skeleton-based approaches. To validate the proposed system, a new dataset of near-infrared imagery has been created, from which good results that outperform other state-of-the-art strategies have been obtained.

R3DG features: Relative 3D geometry-based skeletal representations for human action recognition

Recently introduced cost-effective depth sensors coupled with real-time skeleton extraction algorithms have generated a renewed interest in skeleton-based human action recognition. Most of the existing skeleton-based approaches use either the joint locations or the joint angles to represent the human skeleton. In this paper, we introduce and evaluate a new family of skeletal representations for human action recognition, which we refer to as R3DG features. The proposed representations explicitly model the 3D geometric relationships between various body parts using rigid body transformations, i.e., rotations and translations in 3D space. Using the proposed skeletal representations, human actions are modeled as curves in R3DG feature spaces. Finally, we perform action recognition by classifying these curves using a combination of dynamic time warping, Fourier temporal pyramid representation and support vector machines. Experimental results on five benchmark action datasets show that the proposed representations perform better than many existing skeletal representations. The proposed approach also outperforms various state-of-the-art skeleton-based human action recognition approaches.

Dental radiographs and photographs in human forensic identification

Dentistry can contribute for the identification of human remains after any disasters or crimes in assistance to other medical specialties. The algorithm can be developed by comparing post mortem and ante mortem dental radiographs. This work aims to introduce photographic images in addition to radiographs. In this research a contour and skeleton-based shape extraction as well as matching algorithm for dental images is proposed. An active contour model with selective binary and Gaussian filtering regularised level set method is used for contour extraction. Shape matching is done by both contour and skeleton-based approaches. The experimental results are obtained from a database of dental images include both radiographs and photographs. This algorithm provides better matching decision about the person than the existing algorithms since it includes skeleton measures also. The performance measures obtained and the hit-rate indicates that the better matching is observed with radiographic than the photographic images.

New skeleton-based approaches for Bayesian structure learning of Bayesian networks

Automatically learning the graph structure of a single Bayesian network (BN) which accurately represents the underlying multivariate probability distribution of a collection of random variables is a challenging task. But obtaining a Bayesian solution to this problem based on computing the posterior probability of the presence of any edge or any directed path between two variables or any other structural feature is a much more involved problem, since it requires averaging over all the possible graph structures. For the former problem, recent advances have shown that search+score approaches find much more accurate structures if the search is constrained by a previously inferred skeleton (i.e. a relaxed structure with undirected edges which can be inferred using local search based methods). Based on similar ideas, we propose two novel skeleton-based approaches to approximate a Bayesian solution to the BN learning problem: a new stochastic search which tries to find directed acyclic graph (DAG) structures with a non-negligible score; and a new Markov chain Monte Carlo method over the DAG space. These two approaches are based on the same idea. In a first step, both employ a previously given skeleton and build a Bayesian solution constrained by this skeleton. In a second step, using the preliminary solution, they try to obtain a new Bayesian approximation but this time in an unconstrained graph space, which is the final outcome of the methods. As shown in the experimental evaluation, this new approach strongly boosts the performance of these two standard techniques proving that the idea of employing a skeleton to constrain the model space is also a successful strategy for performing Bayesian structure learning of BNs.

ProcDef: Local-to-global Deformation for Skeleton-free Character Animation

Animations of characters with flexible bodies such as jellyfish, snails, and, hearts are difficult to design using traditional skeleton-based approaches. A standard approach is keyframing, but adjusting the shape of the flexible body for each key frame is tedious. In addition, the character cannot dynamically adjust its motion to respond to the environment or user input. This paper introduces a new procedural deformation framework (ProcDef) for designing and driving animations of such flexible objects. Our approach is to synthesize global motions procedurally by integrating local deformations. ProcDef provides an efficient design scheme for local deformation patterns; the user can control the orientation and magnitude of local deformations as well as the propagation of deformation signals by specifying line charts and volumetric fields. We also present a fast and robust deformation algorithm based on shape-matching dynamics and show some example animations to illustrate the feasibility of our framework.