3D Skeleton Data Research Articles

A Comprehensive Survey of Vision-Based Human Action Recognition Methods.

Although widely used in many applications, accurate and efficient human action recognition remains a challenging area of research in the field of computer vision. Most recent surveys have focused on narrow problems such as human action recognition methods using depth data, 3D-skeleton data, still image data, spatiotemporal interest point-based methods, and human walking motion recognition. However, there has been no systematic survey of human action recognition. To this end, we present a thorough review of human action recognition methods and provide a comprehensive overview of recent approaches in human action recognition research, including progress in hand-designed action features in RGB and depth data, current deep learning-based action feature representation methods, advances in human–object interaction recognition methods, and the current prominent research topic of action detection methods. Finally, we present several analysis recommendations for researchers. This survey paper provides an essential reference for those interested in further research on human action recognition.

Skeleton-Based Human Action Recognition through Third-Order Tensor Representation and Spatio-Temporal Analysis

Given the broad range of applications from video surveillance to human–computer interaction, human action learning and recognition analysis based on 3D skeleton data are currently a popular area of research. In this paper, we propose a method for action recognition using depth sensors and representing the skeleton time series sequences as higher-order sparse structure tensors to exploit the dependencies among skeleton joints and to overcome the limitations of methods that use joint coordinates as input signals. To this end, we estimate their decompositions based on randomized subspace iteration that enables the computation of singular values and vectors of large sparse matrices with high accuracy. Specifically, we attempt to extract different feature representations containing spatio-temporal complementary information and extracting the mode-n singular values with regards to the correlations of skeleton joints. Then, the extracted features are combined using discriminant correlation analysis, and a neural network is used to recognize the action patterns. The experimental results presented use three widely used action datasets and confirm the great potential of the proposed action learning and recognition method.

View Adaptive Neural Networks for High Performance Skeleton-Based Human Action Recognition.

Skeleton-based human action recognition has recently attracted increasing attention thanks to the accessibility and the popularity of 3D skeleton data. One of the key challenges in action recognition lies in the large variations of action representations when they are captured from different viewpoints. In order to alleviate the effects of view variations, this paper introduces a novel view adaptation scheme, which automatically determines the virtual observation viewpoints over the course of an action in a learning based data driven manner. Instead of re-positioning the skeletons using a fixed human-defined prior criterion, we design two view adaptive neural networks, i.e., VA-RNN and VA-CNN, which are respectively built based on the recurrent neural network (RNN) with the Long Short-term Memory (LSTM) and the convolutional neural network (CNN). For each network, a novel view adaptation module learns and determines the most suitable observation viewpoints, and transforms the skeletons to those viewpoints for the end-to-end recognition with a main classification network. Ablation studies find that the proposed view adaptive models are capable of transforming the skeletons of various views to much more consistent virtual viewpoints. Therefore, the models largely eliminate the influence of the viewpoints, enabling the networks to focus on the learning of action-specific features and thus resulting in superior performance. In addition, we design a two-stream scheme (referred to as VA-fusion) that fuses the scores of the two networks to provide the final prediction, obtaining enhanced performance. Moreover, random rotation of skeleton sequences is employed to improve the robustness of view adaptation models and alleviate overfitting during training. Extensive experimental evaluations on five challenging benchmarks demonstrate the effectiveness of the proposed view-adaptive networks and superior performance over state-of-the-art approaches.

Human interaction recognition using spatial-temporal salient feature

Depth sensor is widely used today and has great impact in object pose estimation, camera tracking, human actions, and scene reconstruction. This paper presents a novel method for human interaction recognition based on 3D skeleton data captured by Kinect sensor using hierarchical spatial-temporal saliency-based representation method. Hierarchical saliency can be conceptualized as Salient Actions at the highest level, determined by the initial movement in an interaction; Salient Points at middle level, determined by a single time point uniquely identified for all instances of Salient Action; Salient Joints at the lowest level, determined by the greatest positional changes of human joints in a Salient Action sequence. Given the interaction saliency at different levels, several types of features, such as spatial displacement, direction relations, and etc., are introduced based on action characteristics. Since there are few publicly accessible test datasets, we created a new dataset with eight types of interactions named K3HI, using the Microsoft Kinect. The method was tested based on Support Vector Machine (SVM) multi-class classifier. In the experiment, the results demonstrate that the average recognition accuracy of hierarchical saliency-based representation is 90.29%, outperforming methods using other features.

Recognizing Involuntary Actions from 3D Skeleton Data Using Body States

Human action recognition has been one of the most active fields of research in computer vision for last years. Two dimensional action recognition methods are facing serious challenges such as occlusion and missing the third dimension of data. Development of depth sensors has made it feasible to track positions of human body joints over time. This paper proposes a novel method of action recognition which uses temporal 3D skeletal Kinect data. This method introduces the definition of body states and then every action is modeled as a sequence of these states. The learning stage uses Fisher Linear Discriminant Analysis (LDA) to construct discriminant feature space for discriminating the body states. Moreover, this paper suggests the use of the Mahalonobis distance as an appropriate distance metric for the classification of the states of involuntary actions. Hidden Markov Model (HMM) is then used to model the temporal transition between the body states in each action. According to the results, this method significantly outperforms other popular methods, with recognition rate of 88.64% for eight different actions and up to 96.18% for classifying fall actions.

Elders’ fall detection based on biomechanical features using depth camera

An accidental fall poses a serious threat to the health of the elderly. With the advances of technology, an increased number of surveillance systems have been installed in the elderly home to help medical staffs find the elderly at risk. Based on the study of human biomechanical equilibrium, we proposed a fall detection method based on 3D skeleton data obtained from the Microsoft Kinect. This method leverages the accelerated velocity of Center of Mass (COM) of different body segments and the skeleton data as main biomechanical features, and adopts Long Short-Term Memory networks (LSTM) for fall detection. Compared with other fall detection methods, it does not require older people to wear any other sensors and can protect the privacy of the elderly. According to the experiment to validate our method using the existing database, we found that it could efficiently detect the fall behaviors. Our method provides a feasible solution for the fall detection that can be applied at homes of the elderly.

Hierarchical topic modeling with pose-transition feature for action recognition using 3D skeleton data

Despite impressive achievements in image processing and artificial intelligence in the past decade, understanding video-based action remains a challenge. However, the intensive development of 3D computer vision in recent years has brought more potential research opportunities in pose-based action detection and recognition. Thanks to the advantages of depth camera devices like the Microsoft Kinect sensor, we developed an effective approach to in-depth analysis of indoor actions using skeleton information, in which skeleton-based feature extraction and topic model-based learning are two major contributions. Geometric features, i.e. joint distance, joint angle, and joint-plane distance are calculated in the spatio-temporal dimension. These features are merged into two types, called pose and transition features, and then are provided to codebook construction to convert sparse features into visual words by k-means clustering. An efficient hierarchical model is developed to describe the full correlation of feature - poselet - action based on Pachinko Allocation Model. This model has the potential to uncover more hidden poselets, which have been recognized as the valuable information and help to differentiate pose-sharing actions. The experimental results on several well-known datasets, such as MSR Action 3D, MSR Daily Activity 3D, Florence 3D Action, UTKinect-Action 3D, and NTU RGB+D Action Recognition, demonstrate the high recognition accuracy of the proposed method. Our method outperforms state-of-the-art methods in the field in most dataset benchmarks.

Improving bag-of-poses with semi-temporal pose descriptors for skeleton-based action recognition

Over the last few decades, human action recognition has become one of the most challenging tasks in the field of computer vision. Effortless and accurate extraction of 3D skeleton information has been recently achieved by means of economical depth sensors and state-of-the-art deep learning approaches. In this study, we introduce a novel bag-of-poses framework for action recognition using 3D skeleton data. Our assumption is that any action can be represented by a set of predefined spatiotemporal poses. The pose descriptor is composed of three parts. The first part is concatenation of the normalized coordinate of the skeleton joints. The second part is consisted of temporal displacement of the joints constructed with predefined temporal offset, and the third part is temporal displacement with the previous frame in the sequence. In order to generate the key poses, we apply K-means clustering over all the training pose descriptors of the dataset. SVM classifier is trained with the generated key poses to classify an action pose. Accordingly, every action in the dataset is encoded with key pose histograms. ELM classifier is used for action recognition due to its fast, accurate and reliable performance compared to the other classifiers. The proposed framework is validated with five publicly available benchmark 3D action datasets and achieved state-of-the-art results on three of the datasets and competitive results on the other two datasets compared to the other methods.

Real-time action detection and temporal segmentation in continuous video

ABSTRACTTemporal segmentation of actions has been under intensive focus in the field of computer vision for a prolonged period. The present study proposed a template-based framework to resolve the issues concerning timeliness and real-time performance in the temporal segmentation in a continuous video. A complete action can be detected, based on the previous frames, and the action can be segmented immediately without waiting for the follow-up frames. Herein, characteristic frames are selected by a martingale-based method, followed by the formation of the corresponding motion history through backtracking along the characteristic frames, and the final segmentation is determined according to the recognition model trained by the extreme learning machine. In the experiment on the IXMAS database, the average rate of the detection of action reached 91%, and the accuracy in the frame level reached 83.5%. In the experiment on the 3D skeleton data based on Kinect, the detection rate reached 94%.

Space-time representation of people based on 3D skeletal data: A review

Spatiotemporal human representation based on 3D visual perception data is a rapidly growing research area. Representations can be broadly categorized into two groups, depending on whether they use RGB-D information or 3D skeleton data. Recently, skeleton-based human representations have been intensively studied and kept attracting an increasing attention, due to their robustness to variations of viewpoint, human body scale and motion speed as well as the realtime, online performance. This paper presents a comprehensive survey of existing space-time representations of people based on 3D skeletal data, and provides an informative categorization and analysis of these methods from the perspectives, including information modality, representation encoding, structure and transition, and feature engineering. We also provide a brief overview of skeleton acquisition devices and construction methods, enlist a number of benchmark datasets with skeleton data, and discuss potential future research directions.

Human Activity Recognition in Real-Times Environments using Skeleton Joints

In this research work, we proposed a most effective noble approach for Human activity recognition in real-time environments. We recognize several distinct dynamic human activity actions using kinect. A 3D skeleton data is processed from real-time video gesture to sequence of frames and getter skeleton joints (Energy Joints, orientation, rotations of joint angles) from selected setof frames. We are using joint angle and orientations, rotations information from Kinect therefore less computation required. However, after extracting the set of frames we implemented several classification techniques Principal Component Analysis (PCA) with several distance based classifiers and Artificial Neural Network (ANN) respectively with some variants for classify our all different gesture models. However, we conclude that use very less number of frame (10-15%) for train our system efficiently from the entire set of gesture frames. Moreover, after successfully completion of our classification methods we clinch an excellent overall accuracy 94%, 96% and 98% respectively. We finally observe that our proposed system is more useful than comparing to other existing system, therefore our model is best suitable for real-time application such as in video games for player action/gesture recognition.

A web based remote visualization architecture for big neuroimaging data

Event Abstract Back to Event A web based remote visualization architecture for big neuroimaging data Chao-Chun Chuang1* 1 National center for high-performance computing, Taiwan As recent neuroimaging research indicates, 3D visualization is helpful for understanding complex neuron structure. It also can help scientists share their visualization neuroimaging data and make their representation easier to understand [1][2]. In most cases, scientists downloaded these big neuroimaging data and then visualized on a local desktop environments with 3D visualization tools. However, in recent years, higher resolution datasets in the neuroimaging research continue massive growth with higher experimental resolving power. It becomes inconvenient to download these big neuroimaging data without enough network bandwidth and difficult to display these images in user client without enough computer resources. Thus, we demo a web-based remote visualization architecture for these big neuroimaging data, and take FlyCircuit 3D viewer as an example. FlyCircuit 3D viewer uses advanced 3-layer web technologies, web server (WebSockets, Canvas), visualization server (VNC and VirtualGL) and user client (HTML5) for interactive online 3D surface, skeleton and volumetric neuroimaging data in any web browser environments without requiring any browser plugin[2][3][4][5]. The first layer, web server communicates with the second layer visualization server by WebSockets and Canvas. Then third layer user client can view 3D neuroimaging data from web page constructed by HTML5. Thus, with powerful visualization server, users can easily interactive the 3D visualization neuron imaging by web browser in a lightweight computer resource. Finally, this visualization architecture can handle massive 3D neuronal image data collected in experiments from different research groups as well as manage bio-images with deeper neurological insight. Figure 1

Human Action Recognition Based on Kinect and MCRF Model

In view of the low efficiency and angle problem of human action recognition, a algorithm with Kinect-3D skeleton and MCRF model was proposed. Its 3D skeleton data has less and key information, and MCRF model was able to fusion many features and advantage of context information. First, human action was divided into global action, arm action, and leg action, extracted features through several feature subsets. Then, CRF model was used for every subset to generate, each CRF sets was merged together into MRCF model which was utilized to recognize human action. The experimental results show that the method has higher detection rate.

A Low Cost System for 3D Motion Analysis Using Microsoft Kinect

Many motion analysis systems which have been introduced in the past few years are currently receiving interests from researchers and developers due to their usefulness and wide application capability in the future. However, many of those systems meet with difficulties for the real applications because of high cost for the implementation and less accuracy. This paper introduces a new 3D motion analysis system which can be implemented at a lower cost and acceptable accuracy for various applications. The key component of our new system is the use of the MSK (Microsoft Kinect) sensor system which is equipped with both visual camera and infrared camera. It can provide the color image, the 3D depth image and the 3D skeleton data without wearing any marker device on the human body while it can provide acceptable accuracy in 3D motion trace at low cost. Our system can be exploited for a base framework for various 3D motion-based applications such as physical rehabilitation support, sport motion analysis and biomechanical applications.