Microsoft Kinect Sensor Research Articles

Development and Stability Analysis of an Imitation Learning-Based Pose Planning Approach for Multi-Section Continuum Robot

Recently, continuum flexible robots have been designed for the use in diverse applications; including the exploration of confined static and dynamic environments. One of the challenging tasks for those robots is planning optimal trajectories due to, not only the redundant Degrees of Freedom (DOF) they own but also their compliant behaviour. In this paper, an Imitation-based Pose Planning (IbPP) approach is proposed to teach a two-section continuum robot the motion primitives that will facilitate achieving and generalizing for spatial point-to-point motion which involves both position and orientation goals encoded in a dual quaternion form. Two novel approaches are proposed in this research to intuitively generate the motion demonstrations that will be used in the proposed IbPP. Namely, a flexible input interface, acting as a twin robot, is designed to allow a human to demonstrate different motions for the robot end-effector. Alternatively, as a second approach, the Microsoft Kinect sensor is used to provide motion demonstrations faster via human arm movements. Based on the kinematic model of the two-section continuum robot, a Model Reference Adaptive Control (MRAC) algorithm is developed to achieve tracking the generated trajectory from the IbPP and to guarantee the robustness against the model uncertainties and external disturbances. Moreover, controller stability analysis is developed based on Lyapunov criteria. Finally, simulations are conducted for the two-section continuum robot to prove the ability of the proposed IbPP with the two proposed inputs to learn and generalize for spatial motions, which in future could be easily accommodated for robots with multiple sections. In addition, the proposed MRAC shows a significant performance towards following the required trajectory and rejecting the external disturbance.

Spatiotemporal spectral histogramming analysis in hand gesture signature recognition

Dynamic signature recognition emerges to perfectly solve the hygiene concern due to its no-contact characteristic. Nevertheless, the recognition of dynamic texture is challenging compared with the static signature image due to their unknown spatial and temporal nature. In this work, we present a multi-view spatiotemporal approach based on spectral histogramming for hand gesture signature recognition. A Microsoft Kinect sensor is adopted to capture the motion of signing in a sequence of depth frames. The depth frame sequence is viewed from three directional sights to retrieve rich information, such as temporal changes at each spatial location, the signing motion flow of each vertical and horizontal spatial space in a temporal manner. Furthermore, the proposed approach performs feature description on different levels of locality. This function enables a multi-resolution analysis on this dynamic signature. The robustness of the proposed approach is reflected with the promising result by striking the state-of-the-art performance, as substantiated in the empirical results.

Using New Camera-Based Technologies for Gait Analysis in Older Adults in Comparison to the Established GAITRite System.

Various gait parameters can be used to assess the risk of falling in older adults. However, the state-of-the-art systems used to quantify gait parameters often come with high costs as well as training and space requirements. Gait analysis systems, which use mobile and commercially available cameras, can be an easily available, marker-free alternative. In a study with 44 participants (age ≥ 65 years), gait patterns were analyzed with three different systems: a pressure sensitive walkway system (GAITRite-System, GS) as gold standard, Motognosis Labs Software using a Microsoft Kinect Sensor (MKS), and a smartphone camera-based application (SCA). Intertrial repeatability showed moderate to excellent results for MKS (ICC(1,1) 0.574 to 0.962) for almost all measured gait parameters and moderate reliability in SCA measures for gait speed (ICC(1,1) 0.526 to 0.535). All gait parameters of MKS showed a high level of agreement with GS (ICC(2,k) 0.811 to 0.981). Gait parameters extracted with SCA showed poor reliability. The tested gait analysis systems based on different camera systems are currently only partially able to capture valid gait parameters. If the underlying algorithms are adapted and camera technology is advancing, it is conceivable that these comparatively simple methods could be used for gait analysis.

Random forest-based classsification and analysis of hemiplegia gait using low-cost depth cameras.

Hemiplegia is a form of paralysis that typically has the symptom of dysbasia. In current clinical rehabilitations, to measure the level of hemiplegia gaits, clinicians often conduct subject evaluations through observations, which is unreliable and inaccurate. The Microsoft Kinect sensor (MS Kinect) is a widely used, low-cost depth sensor that can be used to detect human behaviors in real time. The purpose of this study is to investigate the usage of the Kinect data for the classification and analysis of hemiplegia gait. We first acquire the gait data by using a MS Kinect and extract a set of gait features including the stride length, gait speed, left/right moving distances, and up/down moving distances. With the gait data of 60 subjects including 20 hemiplegia patients and 40 healthy subjects, we employ a random forest-based classification approach to analyze the importances of different gait features for hemiplegia classification. Thanks to the over-fitting avoidance nature of the random forest approach, we do not need to have a careful control over the percentage of patients in the training data. In our experiments, our approach obtained the averaged classification accuracy of 90.65% among all the combinations of the gait features, which substantially outperformed state-of-the-art methods. The best classification accuracy of our approach is 95.45%, which is superior than all existing methods. Additionally, our approach also correctly reveals the importance of different gait features for hemiplegia classification. Our random forest-based approach outperforms support vector machine-based method and the Bayesian-based method, and can effectively extract gait features of subjects with hemiplegia for the classification and analysis of hemiplegia. Graphical Abstract Random Forest based Classsification and Analysis of Hemiplegia Gait using Low-cost Depth Cameras. Left: Motion capture with MS Kinect; Top-right: Random Forest Classsification based on the extracted gait features; Bottom-right: Sensitivity and specificity evaluation of the proposed classification approach.

Human gait assessment using a 3D marker-less multimodal motion capture system

Gait analysis is the measurement, processing and systematic interpretation of biomechanical parameters that characterize human locomotion. It supports the identification of movement limitations and development of rehabilitation procedures. Accurate Gait analysis is important in sports analysis, medical field, and rehabilitation. Although Gait analysis is performed in several laboratories in many countries, there are many issues such as: (i) the high cost of precise Motion Capture systems; (ii) the scarcity of qualified personnel to operate them; (iii) expertise required to interpret their results; (iv) space requirements to install and store these systems; as well as difficulties related to the measurement protocols of each system; (vi) limited availability (vii) and the use of markers can be a barrier for some clinical use cases (e.g. patients recovering from orthopedics surgeries). In this work, we present a low cost and more accessible system based on the integration of a Multiple Microsoft Kinect sensors and multiple Shimmer inertial sensors to capture human Gait. The novel multimodal system combines data from inertial and 3D depth cameras and outputs spatiotemporal Gait variables. A comparison of this system with the VICON system (the gold standard in Motion Capture) was performed. Our relatively low-cost marker-less multimodal motion generates a complete 360-degree skeleton view. We compare our system with the VICON via gait spatiotemporal variables: Gait cycle time, stride time, Gait length (distance between two strides), stride length, and velocity. The system was also evaluated with knee and hip joint angles measurement accuracy. The results show high correlation for spatiotemporal variables and joint angles inside the 95% bootstrap prediction when compared with VICON.

Kinect-based human finger tracking method for natural haptic rendering

In the multi-modal natural human-computer interaction (HCI), real-time finger position detection with high accuracy becomes an important basis for interactive modeling of the virtual environment. In this paper, a novel fingertip auto-positioning (FAP) detection method is proposed for tracking finger position with a Microsoft Kinect sensor. Based on the finger skeleton point obtained through the Kinect SDK (Software Development Kit), skeleton point correction and fingertip auto-positioning are realized with the pixels circle which covers the fingertip area within a certain threshold. The experimental results show that the average AMPE (Absolute Mean Percentage Error) of the proposed finger position detection method are 1.47%, 1.62% and 0.80% respectively in the direction of X, Y and Z axes while its execution rate is 23 Hz. The proposed method can be applied to haptic based virtual reality applications and provide required position information for haptic modeling.

Mobile Robot Navigation Using an Object Recognition Software with RGBD Images and the YOLO Algorithm

ABSTRACTThis work presents a vision system based on the YOLO algorithm to identify static objects that could be obstacles in the path of a mobile robot. In order to identify the objects and its distances, a Microsoft Kinect sensor was used. In addition, a Nvidia Jetson TX2 GPU was used to increase the image processing algorithm performance. Our experimental results indicate that the YOLO network has detected all the predefined obstacles for which it has been trained with good reliability and the calculus of the distance using the depth information returned by the Microsoft Kinect camera had an error below of 3,64%.

Semi-Supervised Cross-Modality Action Recognition by Latent Tensor Transfer Learning

Microsoft's Kinect sensors are receiving an increasing amount of interests by security researchers since they are cost-effective and can provide both visual and depth modality data at the same time. Unfortunately, depth or RGB modalities are unavailable in training or testing procedures in some realistic scenarios. Therefore, we explore a new problem focusing on the arbitrary absence of modality, which is completely different from the conventional action recognition. The new problem in action recognition aims to deal with cross modality data (e.g., RGB training and depth testing data), “missing” modality data (e.g., RGB training and RGB-D test data), and single-modality data (e.g., RGB/depth in both phases). Accordingly, our method aims to borrow some information (e.g., correlation between two modalities) from the well-established RGB-D dataset and apply it to the existing dataset to recover some latent information to improve the performance of recognition. For instance, a cross-modality regularizer is used to preserve the correlation of RGB and depth modalities. The “missing” knowledge is considered as latent information, which is recovered by low-rank learning in our model. In the real world, the target data are usually sparsely labeled or completely unlabeled; however, we could exploit the pseudolabels of the target as prior knowledge for “supervised” learning in the target domain. Accordingly, we propose a semi-supervised model for transfer learning. The experiments on three widely used RGB-D action datasets show that our method performs better than that of the state-of-the-art transfer learning methods in most cases in terms of accuracy and time efficiency.

The effect of changing training data on a fixed deep learning detection model

Within the lack of accurate data, for some computer vision applications, researchers usually use other pictures collected from different sources for the training. To know the effect of these added data, we compare the detection results of a customized dataset of objects, using the same detection model, while changing the training data fed into the network. For our work, we run the detection on images captured by the Microsoft Kinect sensor after training the network on different combinations of training data. The first part of the training data is captured by the Kinect itself, and the second is collected from several sources from the internet, referred to as collected images. We then change the distribution of these images between training and validation to feed them into the fixed training model. The results prove that this distribution of data can considerably affect training and detection results under the same model parameters. In addition, mixing the captured images with other collected ones can improve these results.

Skeleton Based View Invariant Human Action Recognition using Convolutional Neural Networks

The skeletal based human action recognition has its significant applications in the field of human computer interaction and human recognition from surveillance videos. However, the tasks suffers from the major challenges like view variance and noise in the data. These problems are limiting the performance of human action recognition. This paper focuses to solve these problems by adopting sequence based view invariant transform to effectively represent the spatio-temporal information of the skeletal data. The task of human action recognition in this paper is performed in three stages. Firstly, the raw 3D skeletal joint data obtained from the Microsoft Kinect sensor is transformed to eliminate the problem of view variations on a spatio-temporal data by implementing sequence based view invariant transform. In the second stage, the transformed joint locations of the skeletal data will be converted to RGB images by a color coding technique and forms a transformed joint location maps (TJLMs) . As a third stage, the discriminating features were extracted by the novel CNN architecture to performs the human action recognition task by means of class scores. Noticeable amount of recognition scores are achieved. Extensive experiments in four difficult 3D action datasets constantly show our method's superiority. The performance of the proposed method is compared with the other state-of-the-art methods.

A Smart Home Environment to Support Risk Monitoring for the Elderly Living Independently

Elderly people prefer to live independently despite being vulnerable to age-related challenges. Constant monitoring is required in cases where the elderly are living alone. The home environment can be a dangerous environment for the elderly due to adverse events that can occur at any time. The potential risks for the elderly living independently can be categorised as injury in the home, home environmental risks, and inactivity due to unconsciousness.
 
 The aim of this paper is to discuss the development of a low-cost Smart Home Environment (SHE) that can support risk and safety monitoring for the elderly living independently. An unobtrusive and low cost SHE prototype that uses a Raspberry Pi 3 model B, a Microsoft Kinect Sensor and an Aeotec 4-in-1 Multisensor was designed and implemented. An experimental evaluation was conducted to determine the accuracy with which the prototype SHE detected abnormal events. The results show that the prototype has a mean accuracy, sensitivity and specificity of 94%, 96.92% and 88.93% respectively. The sensitivity shows that the chance of the prototype missing a risk situation is 3.08%, and the specificity shows that the chance of incorrectly classifying a non-risk situation is 11.07%.

The Consequence of Repetitive Heavy Object Lifting on the Normal Standing Posture of Factory Workers.

Low back pain is one of the most common physical symptom and is frequently related with an abnormal body posture. It may be caused by poor upper body and limb coordination; repetitive lifting of heavy objects or poor working are ergonomics. This study analysis the consequence of repetitive heavy lifting on the normal standing posture of factory workers. To asses the posture malformations the Microsoft Kinect sensor was used to obtain postural data from 88 factory workers. The study has shown that more than 90% of the study group has some sort of postural malformation and lower back pain.

A Robust Framework for Abnormal Human Action Recognition Using $\boldsymbol{\mathcal{R}}$ -Transform and Zernike Moments in Depth Videos

The aim of the algorithm is to detect the abnormal actions that are more prone to elderly people in order to make them more independent and improve their quality of life. The framework is structured to construct a robust feature vector by computing R-transform and Zernike moments on average energy silhouette images (AESIs). The AESIs are generated by the integral sum of the segmented silhouettes obtained from the Microsoft's Kinect sensor v1. The proposed feature descriptor possesses scale-, translation-, and rotation-invariant properties that are also less sensitive to noise and minimizes data redundancy. It enhances the proposed algorithm's robustness and makes the classification process more efficient. The proposed work is validated on a novel abnormal human action (AbHA) dataset and three publically available 3D datasets-UR fall detection dataset, Kinect Activity Recognition dataset, and multiview NUCLA dataset. The proposed framework exhibits superior results from other state-of-the-art methods in terms of average recognition accuracy (ARA). The experimental results demonstrate 96.5%, 96.64%, 95.9%, and 86.4% ARA on the UR fall detection dataset, the KARD dataset, the AbHA dataset, and the multi-view NUCLA dataset, respectively.

Recognition of static hand gesture with using ANN and SVM

Hand gesture recognition is a relevant study topic for a reason that sometimes we may not be in position to communicate verbally. There is need to design Hand gesture recognition systems in order to help people adopt to nonverbal communication mainly sign language. However, there is no clue to understand the meaning of gesture through the computers directly. So this calls for definitions that generalize models in a computer. That is why the machine-learning approaches are implemented in recognition systems. There are generally two types of hand gestures recognition systems which researches have concentrated on. These include static and dynamic Hand gesture recognition systems. However, in building Hand gesture recognition systems, various machine learning approaches have been used. For implementing the proposed system, MS Kinect depth sensor was used as a hardware. The Kinect depth sensor is composed of an infrared camera. This is an advantage to the systems that are designed basing on the depth sensing because factors like color, clothing and background have less effect on the performance. So Kinect based depth sensor systems have a high accuracy and performance making them relevant and applicable in our daily lives. In this paper, we propose a static hand gesture recognition system in real time using two machine learning methods namely Support Vector Machine and Artificial Neural Networks. We use of the newly launched Microsoft Kinect sensor for image extraction. The sensor helps us to extract the hand images. We implement the system on a Matlab platform for reasons that Matlab is widely used by researchers in different fields and that can handle complex computations. In the training of the model, we collect a hundred depth-based Histogram of Oriented Gradient features per alphabet from the hand gesture images which we trained, tested and validated using Artificial Neural Networks (ANN) and Support Vector Machine (SVM). From this dataset, we can generate the generalized gesture model for each alphabet image. For the proposed system, the classification with ANN proves a higher performance then SVM.

Fast and automatic assessment of fall risk by coupling machine learning algorithms with a depth camera to monitor simple balance tasks

BackgroundFalls in the elderly constitute a major health issue associated to population ageing. Current clinical tests evaluating fall risk mostly consist in assessing balance abilities. The devices used for these tests can be expensive or inconvenient to set up. We investigated whether, how and to which extent fall risk could be assessed using a low cost ambient sensor to monitor balance tasks.MethodEighty four participants, forty of which were 65 or older, performed eight simple balance tasks in front of a Microsoft Kinect sensor. Custom-made algorithms coupled to the Kinect sensor were used to automatically extract body configuration parameters such as body centroid and dispersion. Participants were then classified in two groups using a clustering method. The clusters were formed based on the parameters measured by the sensor for each balance task. For each participant, fall risk was independently assessed using known risk factors as age and average physical activity, as well as the participant’s performance on the Timed Up and Go clinical test.ResultsStanding with a normal stance and the eyes closed on a foam pad, and standing with a narrow stance and the eyes closed on regular ground were the two balance tasks for which the classification’s outcome best matched fall risk as assessed by the three known risk factors. Standing on a foam pad with eyes closed was the task driving to the most robust results.ConclusionOur method constitutes a simple, fast, and reliable way to assess fall risk more often with elderly people. Importantly, this method requires very little space, time and equipment, so that it could be easily and frequently used by a large number of health professionals, and in particular by family physicians. Therefore, we believe that the use of this method would substantially contribute to improve fall prevention.Trial registration: CER-VD 2015-00035. Registered 7 December 2015.

Article Users Activity Gesture Recognition on Kinect Sensor Using Convolutional Neural Networks and FastDTW for Controlling Movements of a Mobile Robot

In this paper, we use data from the Microsoft Kinect sensor that processes the captured imageof a person using and extracting the joints information on every frame. Then, we propose the creation ofan image derived from all the sequential frames of a gesture the movement, which facilitates training in aconvolutional neural network. We trained a CNN using two strategies: combined training and individualtraining. The strategies were experimented in the convolutional neural network (CNN) using theMSRC-12 dataset, obtaining an accuracy rate of 86.67% in combined training and 90.78% of accuracyrate in the individual training.. Then, the trained neural network was used to classify data obtained fromKinect with a person, obtaining an accuracy rate of 72.08% in combined training and 81.25% inindividualized training. Finally, we use the system to send commands to a mobile robot in order to controlit.

A Novel Human Activity Recognition Scheme for Smart Health Using Multilayer Extreme Learning Machine

In recent years, more and more wearable sensors have been employed in smart health applications. Wearable sensors not only can be used to collect valuable health-related data of their users, they can be also used in conjunction with other infrastructure-bound sensors, such as Microsoft Kinect sensor, to facilitate privacy-aware fine-grained activity tracking. This fusion of multimodal data promises a new type of smart health applications that coach a user to live a healthier life style by monitoring the user in realtime and reminding him or her when he or she engages in an unhealthy activity. In this paper, we investigate how to achieve fine-grained activity recognition in the context of such an application. In our scheme, the identification accuracy is improved by incorporating a nonlinear and local similarity measure, namely kernel risk-sensitive loss, into a novel multilayer neural network learning algorithm, called as stacked extreme learning machine. Furthermore, to achieve a good generalization performance with minimal human intervention, Jaya as a popular optimization algorithm, is also used to adjust key parameters in our proposed approach. The experiments are conducted to verify the effectiveness of the proposed scheme.

Upbeat: Augmented Reality-Guided Dancing for Prosthetic Rehabilitation of Upper Limb Amputees.

Unsuccessful rehabilitation therapy is a widespread issue amongst modern day amputees. Of the estimated 10 million amputees worldwide, 3 million of whom are upper limb amputees, a large majority are discontent and experience rejection with their current prosthesis during activities of daily living (ADL). Here we introduce Upbeat, an augmented reality (AR) dance game designed to improve rehabilitation therapies in upper limb amputees. In Upbeat, the patient is instructed to follow a virtual dance instructor, performing choreographed dance movements containing hand gestures involved in upper limb rehabilitation therapy. The patient's position is then tracked using a Microsoft Kinect sensor while the hand gestures are analyzed using EMG data collected from a Myo Armband. Additionally, a gamified score is calculated based on how many gestures and movements were correctly performed. Upon completion of the game, a diagnostic summary of the results is shown in the form of a graph summarizing the collected EMG data, as well as with a video displaying an augmented visualization of the patient's upper arm muscle activity during gameplay. By gamifying the rehabilitation process, Upbeat has the potential to improve therapy on upper limb amputees by enabling the start of rehabilitation immediately after trauma, providing personalized feedback which professionals can utilize to accurately assess patient's progress, and increasing patient excitement, therefore increasing patient willingness to complete rehabilitation. This paper is concerned with the description and evaluation of our prototypic implementation of Upbeat that will serve as the basis for conducting clinical studies to evaluate its impact on rehabilitation.

Can Kinect aid motor learning in sportsmen? A study for three standing techniques in judo.

Our objective was to examine how exercises with the second generation of the Microsoft Kinect sensor may aid in the process of motor learning in young judo practitioners. We addressed improvements in spatio-temporal accuracy during execution of three standing techniques in judo, in a simple paradigm designed to study short-term practice effects. Two groups of judokas, 12 athletes each—one aided with Kinect and our dedicated software vs a group of controls—were asked to mimic previously recorded master-level performances of the three techniques, established as benchmarks by a two times world champion in judo. In five training sessions, athletes of the aided group used a virtual-reality setup in which they trained with a virtual representation of the master displayed on a large screen with a simultaneous real-time visualisation of their own movements in the form of an avatar based on body joint localisation, as determined by Kinect, which also measured their performance. The control group used Kinect in the 1st and 5th session, which was necessary for the measurements that constituted the basis for subsequent statistical comparisons, whereas the 2nd, 3rd, and 4th session in this group was guided by a coach, without the use of the Kinect setup. In addition, athletes of the two groups had unrestricted access to a video recording of the master performing the three throws. We found statistically significant improvements (p < 0.05) in the accuracy of executing the three techniques between the 1st and the 5th training session for the aided group but not for the control group. We conclude that incorporating Kinect based exercises into a judo training programme may be a useful means to supporting motor learning, therefore enhancing training efficiency, and thus improving performance.

Real time sign language recognition using depth sensor

Communication via gestures is a visual dialect utilised by deaf and hard-of-hearing (HoH) people group. This paper proposed a system for sign language recognition utilising human skeleton data provided from Microsoft's Kinect sensor to recognising sign gestures. The Kinect sensor generates the skeleton of a human body and distinguishes 20 joints in it. The proposed method utilises 11 out of 20 joints and extracts 35 novel features per frame, based on distances, angles and velocity involving upper body joints. Multi-class support vector machine classified the 35 Indian sign gestures in real time with accuracy of 87.6%. The proposed method is robust in cluttered environment and viewpoint variation.