Skeleton Tracking Research Articles

Augmented reality applications for cultural heritage using Kinect

AbstractThis paper explores the use of data from the Kinect sensor for performing augmented reality, with emphasis on cultural heritage applications. It is shown that the combination of depth and image correspondences from the Kinect can yield a reliable estimate of the location and pose of the camera, though noise from the depth sensor introduces an unpleasant jittering of the rendered view. Kalman filtering of the camera position was found to yield a much more stable view. Results show that the system is accurate enough for in situ augmented reality applications. Skeleton tracking using Kinect data allows the appearance of participants to be augmented, and together these facilitate the development of cultural heritage applications.

Human Activity-Understanding: A Multilayer Approach Combining Body Movements and Contextual Descriptors Analysis

A deep understanding of human activity is key to successful human-robot interaction (HRI). The translation of sensed human behavioural signals/cues and context descriptors into an encoded human activity remains a challenge because of the complex nature of human actions. In this paper, we propose a multilayer framework for the understanding of human activity to be implemented in a mobile robot. It consists of a perception layer which exploits a D-RGB-based skeleton tracking output used to simulate a physical model of virtual human dynamics in order to compensate for the inaccuracy and inconsistency of the raw data. A multi-support vector machine (MSVM) model trained with features describing the human motor coordination through temporal segments in combination with environment descriptors (object affordance) is used to recognize each sub-activity (classification layer). The interpretation of sequences of classified elementary actions is based on discrete hidden Markov models (DHMMs) (interpretation layer). The framework assessment was performed on the Cornell Activity Dataset (CAD-120) [ 1 ]. The performances of our method are comparable with those presented in [ 2 ] and clearly show the relevance of this model-based approach.

Bayesian classification and analysis of gait disorders using image and depth sensors of Microsoft Kinect

This paper presents a novel method of Bayesian gait recognition using Microsoft (MS) Kinect image and depth sensors and skeleton tracking in three-dimensional space. Although video sequences acquired by a complex camera system enable a very precise data analysis, it is possible to use much simpler technical devices to analyze video frames with sufficient accuracy for many applications. The use of the MS Kinect allows a simple 3-D modeling using its image and depth sensors for data acquisition, resulting in a matrix of 640×480 elements used for spatial modeling of a moving body. The experimental part of the paper is devoted to the study of three data sets: (i) 18 individuals with Parkinson's disease, (ii) 18 healthy age-matched controls, and (iii) 15 trained young individuals forming the second reference set. The proposed algorithm involves methods for the estimation of the average stride length and gait speed of individuals in these sets. Digital signal processing methods and Bayesian probability classification algorithms are then used for gait feature analysis to recognize individuals suspected of having Parkinson's disease. The results include the estimation of the characteristics of selected gait features for patients with Parkinson's disease and for individuals from the reference sets, presentation of decision boundaries, and comparison of classification efficiency for different features. The achieved accuracy of the probabilistic classification was 94.1%.

Tracking systems for virtual rehabilitation: objective performance vs. subjective experience. A practical scenario.

Motion tracking systems are commonly used in virtual reality-based interventions to detect movements in the real world and transfer them to the virtual environment. There are different tracking solutions based on different physical principles, which mainly define their performance parameters. However, special requirements have to be considered for rehabilitation purposes. This paper studies and compares the accuracy and jitter of three tracking solutions (optical, electromagnetic, and skeleton tracking) in a practical scenario and analyzes the subjective perceptions of 19 healthy subjects, 22 stroke survivors, and 14 physical therapists. The optical tracking system provided the best accuracy (1.074 ± 0.417 cm) while the electromagnetic device provided the most inaccurate results (11.027 ± 2.364 cm). However, this tracking solution provided the best jitter values (0.324 ± 0.093 cm), in contrast to the skeleton tracking, which had the worst results (1.522 ± 0.858 cm). Healthy individuals and professionals preferred the skeleton tracking solution rather than the optical and electromagnetic solution (in that order). Individuals with stroke chose the optical solution over the other options. Our results show that subjective perceptions and preferences are far from being constant among different populations, thus suggesting that these considerations, together with the performance parameters, should be also taken into account when designing a rehabilitation system.

Informative joints based human action recognition using skeleton contexts

The launching of Microsoft Kinect with skeleton tracking technique opens up new potentials for skeleton based human action recognition. However, the 3D human skeletons, generated via skeleton tracking from the depth map sequences, are generally very noisy and unreliable. In this paper, we introduce a robust informative joints based human action recognition method. Inspired by the instinct of the human vision system, we analyze the mean contributions of human joints for each action class via differential entropy of the joint locations. There is significant difference between most of the actions, and the contribution ratio is highly in accordance with common sense. We present a novel approach named skeleton context to measure similarity between postures and exploit it for action recognition. The similarity is calculated by extracting the multi-scale pairwise position distribution for each informative joint. Then feature sets are evaluated in a bag-of-words scheme using a linear CRFs. We report experimental results and validate the method on two public action dataset. Experiments results have shown that the proposed approach is discriminative for similar human action recognition and well adapted to the intra-class variation.

Toward Real-Time and Efficient Compression of Human Time-Varying Meshes

In this paper, a novel skeleton-based approach to human time-varying mesh (H-TVM) compression is presented. The topic of TVM compression is new and has many challenges, such as handling the lack of obvious mapping of vertices across frames and handling the variable connectivity across frames, while maintaining efficiency, which are the most important ones. Very few works exist in the literature, while not all of the challenges have been addressed yet. In addition, developing an efficient and real-time solution, handling the above, obviously is a difficult task. We attempt to address the H-TVM compression problem inspired from video coding using different types of frames and trying to efficiently remove inter-frame geometric redundancy utilizing the recent advances in human skeleton tracking. The overall approach focuses on compression efficiency, low distortion, and low computation time enabling for real-time transmission of H-TVMs. It efficiently compresses geometry and vertex attributes of TVMs. In addition, this paper is the first to provide an efficient method for connectivity coding of TVMs, by introducing a modification to the state-of-the-art MPEG-4 TFAN algorithm. Experiments are conducted in the MPEG-3DGC TVM database. The method outperforms the state-of-the-art standardized static mesh coder MPEG-4 TFAN at low bit-rates, while remaining competent at high bit-rates. It gives a practical proof of concept that in the combined problem of geometry, connectivity, and vertex attribute coding of TVMs, efficient inter-frame redundancy removal is possible, establishing ground for further improvements. Finally, this paper proposes a method for motion-based coding of H-TVMs that can further enhance the overall experience when H-TVM compression is used in a tele-immersion scenario.

Dark Environment Motion Analysis Using Scalable Model and Vector Angle Technique

Detecting human existence in video streams is a fundamental task in many video processing applications. In this paper, a novel procedure is produced to model, analyze and recognize human motions (jogging and walking in dark environment) in video streams. There are four major areas that are related in this project for human motion analysis: (1) developing human body structure based on human skeleton model, (2) tracking and data collecting human motion with side view, (3) recognizing human activities from image sequences, and (4) image processing technique using edge detection and vectors angle calculation. All algorithms are developed using MATLAB software. Segmentation is developed to reduce the amount of data and filters out the useless information. Two methods are proposed for angle calculation and activities classification. Results showed that angle between 153.76°-180° for method 1 and 49.64°-92.86° for method 2 is classified as walking while jogging is 95.17°-138.72° for method 1 and 22.62°-56.31° for method 2.

Quaternionic Signal Processing Techniques for <newline/>Automatic Evaluation of Dance Performances <newline/>From MoCap Data

In this paper, the problem of automatic dance performance evaluation from human Motion Capture (MoCap) data is addressed. A novel framework is presented, using data captured by Kinect-based human skeleton tracking, where the evaluation of user's performance is achieved against a gold-standard performance of a teacher. The framework addresses several technical challenges, including global and local temporal synchronization, spatial alignment and comparison of two “dance motion signals.” Towards the solution of these technical challenges, a set of appropriate quaternionic vector-signal processing methodologies is proposed, where the 4D (spatiotemporal) human motion data are represented as sequences of pure quaternions. Such a quaternionic representation offers several advantages, including the facts that joint angles and rotations are inherently encoded in the phase of quaternions and the three coordinates variables ( X,Y,Z) are treated jointly, with their intra-correlations being taken into account. Based on the theory of quaternions, a number of advantageous algorithms are formulated. Initially, global temporal synchronization of dance MoCap data is achieved by the use of quaternionic cross-correlations, which are invariant to rigid spatial transformations between the users. Secondly, a quaternions-based algorithm is proposed for the fast spatial alignment of dance MoCap data. Thirdly, the MoCap data can be temporally synchronized in a local fashion, using Dynamic Time Warping techniques adapted to the specific problem. Finally, a set of quaternionic correlation-based measures (scores) are proposed for evaluating and ranking the performance of a dancer. These quaternions-based scores are invariant to rigid transformations, as proved and demonstrated. A total score metric, through a weighted combination of three different metrics is proposed, where the weights are optimized using Particle Swarm Optimization (PSO). The presented experimental results using the Huawei/3DLife/EMC 2 dataset are promising and verify the effectiveness of the proposed methods.

Self-organizing maps for hand and full body tracking

Abstract Touch-free gesture technology opens new avenues for human–machine interaction. We show how self-organizing maps (SOM) can be used for hand and full body tracking. We use a range camera for data acquisition and apply a SOM-learning process for each frame in order to capture the pose. In a next step we introduce an extension of the SOM to 1D and 2D segments for an improved representation and skeleton tracking of body and hand. The proposed SOM based algorithms are very efficient and robust, and produce good tracking results. Their efficiency allows to implement these algorithms on embedded systems, which we demonstrate on an ARM-based embedded platform.

Reliability and Validity of Kinect RGB-D Sensor for Assessing Standing Balance

Microsoft Kinect uses a built-in RGB-D sensor and the skeleton tracking algorithm to capture 3-D movements of the human body. It also has the potential for assessing postural stability, which is fundamental for most motor activities. The aim of this paper was to investigate whether standing balance can be evaluated reliably and validly by this low-cost device. Nine healthy subjects were required to maintain balance during three standing positions (double limb stance with feet apart, double limb stance with feet together and single limb stance). The center of mass (COM) was calculated from the body's kinematic data acquired by the Kinect system and Optotrak Certus motion capture system. The position variability and average velocity of the COM in the horizontal plane were calculated and used to evaluate the subject's balance. These COM parameters from the two systems showed excellent and comparable test-retest reliability (intraclass correlation coefficient ). In addition, although the average velocity of the COM calculated from Kinect was significantly lower, each COM parameter showed excellent concurrent validity and a significant linear relationship existed between the two systems, which meant that biases may be corrected using linear calibration equations. Therefore, Kinect may be a valid, reliable, and convenient device for assessing standing balance when its measured COM parameters are properly calibrated.

Designing and Evaluating a Social Gaze-Control System for a Humanoid Robot

This paper describes a context-dependent social gaze-control system implemented as part of a humanoid social robot. The system enables the robot to direct its gaze at multiple humans who are interacting with each other and with the robot. The attention mechanism of the gaze-control system is based on features that have been proven to guide human attention: nonverbal and verbal cues, proxemics, the visual field of view, and the habituation effect. Our gaze-control system uses Kinect skeleton tracking together with speech recognition and SHORE-based facial expression recognition to implement the same features. As part of a pilot evaluation, we collected the gaze behavior of 11 participants in an eye-tracking study. We showed participants videos of two-person interactions and tracked their gaze behavior. A comparison of the human gaze behavior with the behavior of our gaze-control system running on the same videos shows that it replicated human gaze behavior 89% of the time.

Touch-Less Java Based Hand Tracking System to Control an Artificial Arm

Modern sensing technologies create new possibilities to control mobile robots without any dedicated manipulators. In this article authors present a novel method that enables driving of the Mindstorms NXT artificial arm with Microsoft Kinect, using gesture recognition and hand tracking. To imitate the movement of an artificial robotic arm, an algorithm of the human-computer interaction is employed using skeleton tracking and gesture control in 3D space. The solution is implemented using free, open source software, Java based.

Gesture Acquisition and Tracking with Kinect under Complex Background

With the rapid development of economy and society, the demand for the man-machine interactive experience is in high enhancement. Hand gesture tracking and recognition is a key technology. This paper studies the obtaining and following about dynamic hand gesture based on Kinect, which is under complex background. Not only has important theoretical significance and tracking, but also it has a broad application prospect. This paper first introduces the human skeleton tracking technology which was based on Kinect. The gesture movement orbit was acquired by tracking the hand gesture and separating the gesture from the complex background so as to lay the foundation for gesture recognition. Software was developed which can recognize human action language, control the game to respond, achieve human-computer interaction more friendly and make the game more fun by the beforehand action.

Animation of 3D Human Model Using Markerless Motion Capture Applied To Sports

Markerless motion capture is an active research in 3D virtualization. In proposed work we presented a system for markerless motion capture for 3D human character animation, paper presents a survey on motion and skeleton tracking techniques which are developed or are under development. The paper proposed a method to transform the motion of a performer to a 3D human character (model), the 3D human character performs similar movements as that of a performer in real time. In the proposed work, human model data will be captured by Kinect camera, processed data will be applied on 3D human model for animation. 3D human model is created using open source software (MakeHuman). Anticipated dataset for sport activity is considered as input which can be applied to any HCI application.

3P2-V03 Kinectを利用したロボットアームの制御(非接触センシング)

The purpose of this study is the realizing of natural industrial robot arm control method by tracking the operator's arm motion. In this study, we detect subject's motion of the arm using skeleton tracking of the Kinect. Using the results of this motion tracking motion of the industrial robot arm was controlled. In this study, we suggest two methods for the robot arm controlling. One is the gesture control method which controls a robot arm according to pre-decided hand or arm motions, and the other is the motion tracking method which controls a robot arm as same as hand or arm motions. Each method were tested using our developed control system. In this paper we report the results of evaluation for the two methods.

Method for User Interface of Large Displays Using Arm Pointing and Finger Counting Gesture Recognition

Although many three-dimensional pointing gesture recognition methods have been proposed, the problem of self-occlusion has not been considered. Furthermore, because almost all pointing gesture recognition methods use a wide-angle camera, additional sensors or cameras are required to concurrently perform finger gesture recognition. In this paper, we propose a method for performing both pointing gesture and finger gesture recognition for large display environments, using a single Kinect device and a skeleton tracking model. By considering self-occlusion, a compensation technique can be performed on the user's detected shoulder position when a hand occludes the shoulder. In addition, we propose a technique to facilitate finger counting gesture recognition, based on the depth image of the hand position. In this technique, the depth image is extracted from the end of the pointing vector. By using exception handling for self-occlusions, experimental results indicate that the pointing accuracy of a specific reference position was significantly improved. The average root mean square error was approximately 13 pixels for a 1920 × 1080 pixels screen resolution. Moreover, the finger counting gesture recognition accuracy was 98.3%.

Fall Risk Assessment and Early-Warning for Toddler Behaviors at Home

Accidental falls are the major cause of serious injuries in toddlers, with most of these falls happening at home. Instead of providing immediate fall detection based on short-term observations, this paper proposes an early-warning childcare system to monitor fall-prone behaviors of toddlers at home. Using 3D human skeleton tracking and floor plane detection based on depth images captured by a Kinect system, eight fall-prone behavioral modules of toddlers are developed and organized according to four essential criteria: posture, motion, balance, and altitude. The final fall risk assessment is generated by a multi-modal fusion using either a weighted mean thresholding or a support vector machine (SVM) classification. Optimizations are performed to determine local parameter in each module and global parameters of the multi-modal fusion. Experimental results show that the proposed system can assess fall risks and trigger alarms with an accuracy rate of 92% at a speed of 20 frames per second.

Improved Skeleton Tracking by Duplex Kinects: A Practical Approach for Real-Time Applications

Recent development of per-frame motion extraction method can generate the skeleton of human motion in real-time with the help of RGB-D cameras such as Kinect. This leads to an economic device to provide human motion as input for real-time applications. As generated by a single-view image plus depth information, the extracted skeleton usually has problems of unwanted vibration, bone-length variation, self-occlusion, etc. This paper presents an approach to overcome these problems by synthesizing the skeletons generated by duplex Kinects, which capture the human motion in different views. The major technical difficulty of this synthesis comes from the inconsistency of two skeletons. Our algorithm is formulated under the constrained optimization framework by using the bone-lengths as hard constraints and the tradeoff between inconsistent joint positions as soft constraints. Schemes are developed to detect and re-position the problematic joints generated by per-frame method from duplex Kinects. As a result, we develop an easy, cheap and fast approach that can improve the skeleton of human motion at an average speed of 5 ms per frame.

Concurrent validity of the Microsoft Kinect for assessment of spatiotemporal gait variables

Spatiotemporal characteristics of gait such as step time and length are often associated with overall physical function in clinical populations, but can be difficult, time consuming and obtrusive to measure. This study assessed the concurrent validity of overground walking spatiotemporal data recorded using a criterion reference – a marker-based three-dimensional motion analysis (3DMA) system – and a low-cost, markerless alternative, the automated skeleton tracking output from the Microsoft Kinect™ (Kinect). Twenty-one healthy adults performed normal walking trials while being monitored using both systems. The outcome measures of gait speed, step length and time, stride length and time and peak foot swing velocity were derived using supervised automated analysis. To assess the agreement between the Kinect and 3DMA devices, Bland–Altman 95% bias and limits of agreement, percentage error, relative agreement (Pearson's correlation coefficients: r) overall agreement (concordance correlation coefficients: rc) and landmark location linearity as a function of distance from the sensor were determined. Gait speed, step length and stride length from the two devices possessed excellent agreement (r and rc values >0.90). Foot swing velocity possessed excellent relative (r=0.93) but only modest overall (rc=0.54) agreement. Step time (r=0.82 and rc=0.23) and stride time (r=0.69 and rc=0.14) possessed excellent and modest relative agreement respectively but poor overall agreement. Landmark location linearity was excellent (R2=0.991). This widely available, low-cost and portable system could provide clinicians with significant advantages for assessing some spatiotemporal gait parameters. However, caution must be taken when choosing outcome variables as some commonly reported variables cannot be accurately measured.

RGB-Depth feature for 3D human activity recognition

We study the problem of human activity recognition from RGB-Depth (RGBD) sensors when the skeletons are not available. The skeleton tracking in Kinect SDK works well when the human subject is facing the camera and there are no occlusions. In surveillance or nursing home monitoring scenarios, however, the camera is usually mounted higher than human subjects, and there may be occlusions. The interest-point based approach is widely used in RGB based activity recognition, it can be used in both RGB and depth channels. Whether we should extract interest points independently of each channel or extract interest points from only one of the channels is discussed in this paper. The goal of this paper is to compare the performances of different methods of extracting interest points. In addition, we have developed a depth map-based descriptor and built an RGBD dataset, called RGBD-SAR, for senior activity recognition. We show that the best performance is achieved when we extract interest points solely from RGB channels, and combine the RGB-based descriptors with the depth map-based descriptors. We also present a baseline performance of the RGBD-SAR dataset.