Skeleton Data Research Articles

Adaptive Attention Memory Graph Convolutional Networks for Skeleton-Based Action Recognition.

Graph Convolutional Networks (GCNs) have attracted a lot of attention and shown remarkable performance for action recognition in recent years. For improving the recognition accuracy, how to build graph structure adaptively, select key frames and extract discriminative features are the key problems of this kind of method. In this work, we propose a novel Adaptive Attention Memory Graph Convolutional Networks (AAM-GCN) for human action recognition using skeleton data. We adopt GCN to adaptively model the spatial configuration of skeletons and employ Gated Recurrent Unit (GRU) to construct an attention-enhanced memory for capturing the temporal feature. With the memory module, our model can not only remember what happened in the past but also employ the information in the future using multi-bidirectional GRU layers. Furthermore, in order to extract discriminative temporal features, the attention mechanism is also employed to select key frames from the skeleton sequence. Extensive experiments on Kinetics, NTU RGB+D and HDM05 datasets show that the proposed network achieves better performance than some state-of-the-art methods.

Human Activity Recognition Using The Human Skeleton Provided by Kinect

In this paper, a new method is proposed for people tracking using the human skeleton provided by the Kinect sensor, Our method is based on skeleton data, which includes the coordinate value of each joint in the human body. For data classification, the Support Vector Machine (SVM) and Random Forest techniques are used. To achieve this goal, 14 classes of movements are defined, using the Kinect Sensor to extract data containing 46 features and then using them to train the classification models. The system was tested on 12 subjects, each of whom performed 14 movements in each experiment. Experiment results show that the best average accuracy is 90.2 % for the SVM model and 99 % for the Random forest model. From the experiments, we concluded that the best distance between the Kinect sensor and the human body is one meter.

SKELETON-BASED ACTION RECOGNITION USING FEATURE FUSION FOR SPATIAL-TEMPORAL GRAPH CONVOLUTIONAL NETWORKS

Human action recognition (HAR) has been used in a variety of applications such as gaming, healthcare, surveillance, and robotics. Research on utilizing data such as color, depth, and skeletal data has been extensively conducted to achieve high-performance HAR. Compared with color and depth data, skeletal data are more compact, therefore, they are more efficient for computation and storage. Moreover, skeletal data are invariant to clothing textures, background, and lighting conditions. With the booming of deep learning, HAR has received a lot of attention. Spatial-Temporal Graph Convolution Networks (ST-GCN) have proved to be state-of-the-art architecture for HAR using skeleton data. However, this does not hold when working with challenging datasets that contain incomplete and noisy skeletal data. In this paper, a new method is proposed for HAR by adding a Feature Fusion module and applying hyperparameter optimization. The performance of the proposed method is evaluated on the challenging dataset CMDFALL and the newly-built MICA-Action3D dataset. Experimental results show that the proposed method significantly improves the performance of ST-GCN on these challenging datasets.

Combining Supervised and Unsupervised Learning Algorithms for Human Activity Recognition.

Human activity recognition is an extensively researched topic in the last decade. Recent methods employ supervised and unsupervised deep learning techniques in which spatial and temporal dependency is modeled. This paper proposes a novel approach for human activity recognition using skeleton data. The method combines supervised and unsupervised learning algorithms in order to provide qualitative results and performance in real time. The proposed method involves a two-stage framework: the first stage applies an unsupervised clustering technique to group up activities based on their similarity, while the second stage classifies data assigned to each group using graph convolutional networks. Different clustering techniques and data augmentation strategies are explored for improving the training process. The results were compared against the state of the art methods and the proposed model achieved 90.22% Top-1 accuracy performance for NTU-RGB+D dataset (the performance was increased by approximately 9% compared with the baseline graph convolutional method). Moreover, inference time and total number of parameters stay within the same magnitude order. Extending the initial set of activities with additional classes is fast and robust, since there is no required retraining of the entire architecture but only to retrain the cluster to which the activity is assigned.

Integrating vertex and edge features with Graph Convolutional Networks for skeleton-based action recognition

Methods based on Graph Convolutional Networks(GCN) for skeleton-based action recognition have achieved great success due to their ability to exploit graph structural information from skeleton data. Recently, the bone information has attracted considerable attention as an effective modality which complements the more conventional joint information for action recognition. However, most existing GCN-based methods extract the joint and bone features with two separate GCN networks, ignoring the dependencies between them. In this paper, a novel GCN model is proposed to exploit the information across joints, bones and their relationship collaboratively on a single undirected graph instead of two separate networks. We call the proposed model Vertex-Edge Graph Convolutional Network (VE-GCN) since it conducts the graph convolution operation on the sampling area containing the designated vertexes from joints and edges from bones, respectively. In addition to conducting the Vertex-Edge graph convolution based on the physical connections of the skeleton, we further apply the Vertex-Edge graph convolution to the non-physical joint-joint and joint-bone connections to capture the distal dependencies, and then the convolution results on the non-physical connections are incorporated into the VE-GCN. Moreover, the Conditional Random Field (CRF) is adopted as the loss function to achieve the task of action recognition. Experimental results on four challenging benchmarks (NTU RGB+D, NTU RGB+D 120, N-UCLA, SYSU) show that the proposed model achieves state-of-the-art performance.

Experimental Study of a Deep-Learning RGB-D Tracker for Virtual Remote Human Model Reconstruction.

Tracking movements of the body in a natural living environment of a person is a challenging undertaking. Such tracking information can be used as a part of detecting any onsets of anomalies in movement patterns or as a part of a remote monitoring environment. The tracking information can be mapped and visualized using a virtual avatar model of the tracked person. This paper presents an initial novel experimental study of using a commercially available deep-learning body tracking system based on an RGB-D sensor for virtual human model reconstruction. We carried out our study in an indoor environment under natural conditions. To study the performance of the tracker, we experimentally study the output of the tracker which is in the form of a skeleton (stick-figure) data structure under several conditions in order to observe its robustness and identify its drawbacks. In addition, we show and study how the generic model can be mapped for virtual human model reconstruction. It was found that the deep-learning tracking approach using an RGB-D sensor is susceptible to various environmental factors which result in the absence and presence of noise in estimating the resulting locations of skeleton joints. This as a result introduces challenges for further virtual model reconstruction. We present an initial approach for compensating for such noise resulting in a better temporal variation of the joint coordinates in the captured skeleton data. We explored how the extracted joint position information of the skeleton data can be used as a part of the virtual human model reconstruction.

Fusion of Skeleton and RGB Features for RGB-D Human Action Recognition

The output of Microsoft Kinect is a multimodal signal, which provides RGB videos, depth sequences and skeleton information at the same time, opening up a new opportunity for the research of human action recognition. However, for different single modalities of the signals, how to exploit and fuse useful features of these various sources remains a very challenging problem. Most of the methods based on RGB-D action recognition simply fuse the multimodal features, ignoring the potential semantic relationship between different models. In this paper, we propose a multi-modal action recognition model based on Bilinear Pooling and Attention Network (BPAN), which could effectively fuse multi-modal for RGB-D action recognition. Firstly, we adopt the efficient data preprocessing methods for RGB and skeleton data. Then, we propose a multimodal fusion network combining RGB video and skeleton sequences. The proposed BPAN module could effectively compress the features of RGB and skeleton, and project them into latent subspace to get the fusion features. In the end, a fully connected three-layer perceptron is adopted to obtain the final classification decision. Experimental results on three public datasets demonstrate that our proposed method leads to a more favorable performance compared with the state-of-the-art methods.

Skeleton-based Action Recognition with Two-Branch Graph Convolutional Networks

Graph convolution is a popular technique for action recognition based on human skeleton data. Due to the fact that human skeleton data can be treated as a graph in three dimensions, Graph Convolutional Networks (GCN) represent the input data as a graph structure to perform the recognition task, and thus numerous approaches based on GCN to recognize actions have achieved great results. In spite of the input data being structured as a four-dimensional tensor in GCN, it still can not fully exploit the contained rich action-related information. Therefore, we proposed a new model that the three-dimensional skeleton data is put into both the Convolutional Neural Network (CNN) and the Graph Convolutional Neural Network branches to perform feature extraction in the spatiotemporal dimension separately, then the output information is fused for prediction. Given the richness of time-domain information characteristics, feature extraction is enhanced by increasing the model’s depth. After the pooling layer and the fully connected layer, we concatenate the outputs at the ends of the graph data stream and the convolution data stream to obtain the network’s final output. Finally, the prediction results can be obtained via the SoftMax layer. On the Kinetic 400 dataset, our suggested model outperforms Benchmark STGCN in terms of accuracy. The experiment results indicate that the proposed novel model successfully increases the generalization ability and classification performance for action recognition.

Student Behavior Recognition System for the Classroom Environment Based on Skeleton Pose Estimation and Person Detection.

Human action recognition has attracted considerable research attention in the field of computer vision, especially for classroom environments. However, most relevant studies have focused on one specific behavior of students. Therefore, this paper proposes a student behavior recognition system based on skeleton pose estimation and person detection. First, consecutive frames captured with a classroom camera were used as the input images of the proposed system. Then, skeleton data were collected using the OpenPose framework. An error correction scheme was proposed based on the pose estimation and person detection techniques to decrease incorrect connections in the skeleton data. The preprocessed skeleton data were subsequently used to eliminate several joints that had a weak effect on behavior classification. Second, feature extraction was performed to generate feature vectors that represent human postures. The adopted features included normalized joint locations, joint distances, and bone angles. Finally, behavior classification was conducted to recognize student behaviors. A deep neural network was constructed to classify actions, and the proposed system was able to identify the number of students in a classroom. Moreover, a system prototype was implemented to verify the feasibility of the proposed system. The experimental results indicated that the proposed scheme outperformed the skeleton-based scheme in complex situations. The proposed system had a 15.15% higher average precision and 12.15% higher average recall than the skeleton-based scheme did.

Part-wise Spatio-temporal Attention Driven CNN-based 3D Human Action Recognition

Recently, human activity recognition using skeleton data is increasing due to its ease of acquisition and finer shape details. Still, it suffers from a wide range of intra-class variation, inter-class similarity among the actions and view variation due to which extraction of discriminative spatial and temporal features is still a challenging problem. In this regard, we present a novel Residual Inception Attention Driven CNN (RIAC-Net) Network, which visualizes the dynamics of the action in a part-wise manner. The complete skeletonis partitioned into five key parts: Head to Spine, Left Leg, Right Leg, Left Hand, Right Hand. For each part, a Compact Action Skeleton Sequence (CASS) is defined. Part-wise skeleton-based motion dynamics highlights discriminative local features of the skeleton that helps to overcome the challenges of inter-class similarity and intra-class variation with improved recognition performance. The RIAC-Net architecture is inspired by the concept of inception-residual representation that unifies the Attention Driven Residues (ADR) with inception-based Spatio-Temporal Convolution Features (STCF) to learn efficient salient action features. An ablation study is also carried out to analyze the effect of ADR over simple residue-based action representation. The robustness of the proposed framework is evaluated by performing an extensive experiment on four challenging datasets: UT Kinect Action 3D, Florence 3D action, MSR Daily Action3D, and NTU RGB-D datasets, which consistently demonstrate the superiority of the proposed method over other state-of-the-art methods.

Modeling spatiotemporal patterns of gait anomaly with a CNN-LSTM deep neural network

In this work, we propose an end-to-end deep learning model that uses the skeleton data recorded by Kinect to capture spatiotemporal patterns for gait anomaly recognition. Via considering the entire skeleton, the proposed model captures the relationship between different body joints in locomotion. Unlike the common two-class or one-class approaches in skeleton-based methods, the proposed model considers a multi-class framework. Such a multi-class technique can be easily adapted for a more frequent and less expensive gait assessment outside of motion capture facilities. The proposed deep learning model is trained and evaluated on the publicly available Walking gait dataset and achieves an average accuracy of 90.57% in identifying nine different walking patterns. Through transfer learning, we also evaluate our model on two other publicly available datasets, acquiring an average accuracy of 83.64% on a dataset of three classes and 90.83% on a dataset with six classes of normal/pathological gait patterns. The results of this work indicate the potential of markerless modalities such as Kinect for designing less costly and more convenient health infrastructures for assisted living. Additionally, an automatic and non-invasive gait assessment can further augment the clinical diagnosis for an extensive list of ailments that cause different types of gait disorders.

Towards a deep human activity recognition approach based on video to image transformation with skeleton data

Towards a deep human activity recognition approach based on video to image transformation with skeleton data

Research on Character Behavior Recognition Based on Local Spatio-temporal Relationship in Surveillance Video

Because of its wide application value, behavior recognition has long been one of the research hots pots in the field of computer vision and pattern recognition. At present, the method based on local features and word packet model has been widely used in the field of behavior recognition. However, this method does not consider the temporal and spatial relationship between features, and the local temporal and spatial relationship between features is very important for behavior representation and behavior recognition. In view of the above problems, this paper proposes a modeling method of character behavior recognition based on local spatio-temporal relationship in surveillance video. Firstly, each part of the proposed network model is introduced in detail, and then the proposed model is compared with the advanced skeleton action recognition methods in recent years on several skeleton data sets. Finally, the effectiveness of the proposed method is verified. The experimental results show that, compared with the recognition results of related literatures, the features extracted by choosing the starting point of trajectory have better recognition performance under the fusion framework.

Fusion of Multi-Sensor-Based Biomechanical Gait Analysis Using Vision and Wearable Sensor

The purpose of this research work is to investigate the biomechanics of pelvis, hip, knee, and ankle joint motion using a Kinect sensor & inertial measurement unit (IMU) sensor during the normal walk. In this paper, a very cost-effective gait analysis system based on Microsoft Kinect v2 and Inertial Measurement Unit (IMU) device is presented. Kinect sensor is used for acquiring 3D skeleton data (camera (x, y, z), depth (x, y) orientation (x, y, z, w), color (x, y)) with 25 human body joints. For this analysis, the lower extremities joints i.e. spinal cord joint, hip, knee, and ankle joints of both left and right legs are being considered. The main contribution of this research work is the joint angle calculation of lower extremities of human gait based on Microsoft Kinect sensor V2 and IMU sensor. From the law of cosine, the joint angle is calculated between the two joints and plotted for a single subject. We came with the observation that the characteristics of the human knee joint and ankle joint are inversely related to each other. There are two sharp humps for knee and ankle joints during the normal walk. During the swing phase, the knee joint is highly activated while during toe-off and heel strike it is least activated. This analysis of clinical data is very useful for prosthesis limb and exoskeleton design. The stability of calculated joints trajectories is validated using the limit cycle curve. A system is designed for real-time analysis of biomechanics of different lower limbs joints using gait.

Combining skeleton and accelerometer data for human fine-grained activity recognition and abnormal behaviour detection with deep temporal convolutional networks

Combining skeleton and accelerometer data for human fine-grained activity recognition and abnormal behaviour detection with deep temporal convolutional networks

Frailty Level Classification of the Community Elderly Using Microsoft Kinect-Based Skeleton Pose: A Machine Learning Approach

Frailty is one of the most important geriatric syndromes, which can be associated with increased risk for incident disability and hospitalization. Developing a real-time classification model of elderly frailty level could be beneficial for designing a clinical predictive assessment tool. Hence, the objective of this study was to predict the elderly frailty level utilizing the machine learning approach on skeleton data acquired from a Kinect sensor. Seven hundred and eighty-seven community elderly were recruited in this study. The Kinect data were acquired from the elderly performing different functional assessment exercises including: (1) 30-s arm curl; (2) 30-s chair sit-to-stand; (3) 2-min step; and (4) gait analysis tests. The proposed methodology was successfully validated by gender classification with accuracies up to 84 percent. Regarding frailty level evaluation and prediction, the results indicated that support vector classifier (SVC) and multi-layer perceptron (MLP) are the most successful estimators in prediction of the Fried’s frailty level with median accuracies up to 97.5 percent. The high level of accuracy achieved with the proposed methodology indicates that ML modeling can identify the risk of frailty in elderly individuals based on evaluating the real-time skeletal movements using the Kinect sensor.

Skeleton-based human action evaluation using graph convolutional network for monitoring Alzheimer’s progression

Human action evaluation (HAE) involves judgments about the abnormality and quality of human actions. If performed effectively, HAE based on skeleton data can be used to monitor the outcomes of behavioral therapies for Alzheimer’s disease (AD). In this paper, we propose a two-task graph convolutional network (2T-GCN) to represent skeleton data for HAE tasks involving abnormality detection and quality evaluation. The network is first evaluated using the UI-PRMD dataset and demonstrates accurate abnormality detection. Regarding quality evaluation, in addition to laboratory-collected UI-PRMD data, we test the network on a set of real exercise data collected from patients with AD. A numerical score indicating the degree to which actions deviate from normal is taken to reflect the severity of AD; thus, we apply 2T-GCN to determine such scores. Experimental results show that numerical scores for certain exercises performed by patients with AD are consistent with their AD severity level as identified by clinical staff. This corroboration highlights the potential of our approach for monitoring AD and other neurodegenerative diseases.

Zero-Shot Action Recognition with Three-Stream Graph Convolutional Networks.

Large datasets are often used to improve the accuracy of action recognition. However, very large datasets are problematic as, for example, the annotation of large datasets is labor-intensive. This has encouraged research in zero-shot action recognition (ZSAR). Presently, most ZSAR methods recognize actions according to each video frame. These methods are affected by light, camera angle, and background, and most methods are unable to process time series data. The accuracy of the model is reduced owing to these reasons. In this paper, in order to solve these problems, we propose a three-stream graph convolutional network that processes both types of data. Our model has two parts. One part can process RGB data, which contains extensive useful information. The other part can process skeleton data, which is not affected by light and background. By combining these two outputs with a weighted sum, our model predicts the final results for ZSAR. Experiments conducted on three datasets demonstrate that our model has greater accuracy than a baseline model. Moreover, we also prove that our model can learn from human experience, which can make the model more accurate.

Adaptive most joint selection and covariance descriptions for a robust skeleton-based human action recognition

In this paper, we propose two effective manners of utilizing skeleton data for human action recognition (HAR). The proposed method on one hand takes advantage of the skeleton data thanks to their robustness to human appearance change as well as the real-time performance. On the other hand, it avoids inherent drawbacks of the skeleton data such as noises, incorrect human skeleton estimation due to self-occlusion of human pose. To this end, in terms of feature designing, we propose to extract covariance descriptors from joint velocity and combine them with those of joint position. In terms of 3-D skeleton-based activity representation, we propose two schemes to select the most informative joints. The proposed method is evaluated on two benchmark datasets. On the MSRAction-3D dataset, the proposed method outperformed different hand-designed features-based methods. On the challenging dataset CMDFall, the proposed method significantly improves accuracy when compared with techniques based on recent neuronal networks. Finally, we investigate the robustness of the proposed method via a cross-dataset evaluation.

3D mesh model segmentation based on skeleton extraction

ABSTRACT In order to solve the problem of low segmentation accuracy of 3D mesh models, a non-deep learning method based on skeleton extraction is proposed. First, the skeleton of the model is extracted. Then, the skeleton feature points are extracted by analysing the relationship between the skeleton data. The feature point with the smallest distance from the centroid in the same neighbourhood is selected as the segmentation point. Finally, based on the improved region-based algorithm, the segmented skeleton region is obtained with the segmentation points as the seeds, and the corresponding original data is segmented through inverse mapping. The experimental results show that compared with the existing non-deep learning methods, the proposed method can obtain better semantic separation results and segmentation boundaries.