Kinect Sensor Research Articles

A Novel Human Activity Recognition Scheme for Smart Health Using Multilayer Extreme Learning Machine

In recent years, more and more wearable sensors have been employed in smart health applications. Wearable sensors not only can be used to collect valuable health-related data of their users, they can be also used in conjunction with other infrastructure-bound sensors, such as Microsoft Kinect sensor, to facilitate privacy-aware fine-grained activity tracking. This fusion of multimodal data promises a new type of smart health applications that coach a user to live a healthier life style by monitoring the user in realtime and reminding him or her when he or she engages in an unhealthy activity. In this paper, we investigate how to achieve fine-grained activity recognition in the context of such an application. In our scheme, the identification accuracy is improved by incorporating a nonlinear and local similarity measure, namely kernel risk-sensitive loss, into a novel multilayer neural network learning algorithm, called as stacked extreme learning machine. Furthermore, to achieve a good generalization performance with minimal human intervention, Jaya as a popular optimization algorithm, is also used to adjust key parameters in our proposed approach. The experiments are conducted to verify the effectiveness of the proposed scheme.

재활 및 운동보조를 위한 착용형 소프트로봇 개발

This paper presents a wearable soft robot to reduce the wearer"s inconvenience, which is one of the major problems of conventional exoskeleton robots for exercise aids for the elderly and rehabilitation. Lightweight pneumatic artificial muscles were used to help the rotational motions of the wearer’s body, such as elbow joints, to assist human activities in accordance with the coordinated reference-motion. Additionally, a kinect sensor was implemented to measure the motions of a human in 3D based on the skeleton mapping technique with sufficient accuracy instead of attaching heavy sensors on a robotic suit. The assisting torques produced by the soft robot with the wearer’s joints were simulated using dynamic behaviors of artificial muscles, which were measured from actual tests. The control characteristics of the proposed soft robot were verified in arm lifting with the payload and walking motions.

Research of “Silk Road” Knowledge Teaching Mode Based on Human-Computer Virtual Interaction

With the implementation of “One Belt One Road” strategy, cultural relics on Silk Road have been introduced to classes. Aiming at the disadvantages of traditional history education such as monotonous mode and insufficient intuitiveness, this paper combines Kinect sensor with cultural relics on Silk Road and explores a multilingual virtual teaching mode of cultural relics which supports somatosensory interaction. It changes the past teaching mode which is dull and abstract and injects new, economical and practical vitality into Silk Road culture, which can help with the inheritance and development of Silk Road culture in the new era.

Cardiopulmonary resuscitation quality parameters from motion capture data using Differential Evolution fitting of sinusoids

Cardiopulmonary resuscitation (CPR) is alongside electrical defibrillation the most crucial countermeasure for sudden cardiac arrest, which affects thousands of individuals every year. In this paper, we present a novel approach including sinusoid models that use skeletal motion data from an RGB-D (Kinect) sensor and the Differential Evolution (DE) optimization algorithm to dynamically fit sinusoidal curves to derive frequency and depth parameters for cardiopulmonary resuscitation training. It is intended to be part of a robust and easy-to-use feedback system for CPR training, allowing its use for unsupervised training. The accuracy of this DE-based approach is evaluated in comparison with data of 28 participants recorded by a state-of-the-art training mannequin. We optimized the DE algorithm hyperparameters and showed that with these optimized parameters the frequency of the CPR is recognized with a median error of ±2.9 compressions per minute compared to the reference training mannequin.

Upbeat: Augmented Reality-Guided Dancing for Prosthetic Rehabilitation of Upper Limb Amputees.

Unsuccessful rehabilitation therapy is a widespread issue amongst modern day amputees. Of the estimated 10 million amputees worldwide, 3 million of whom are upper limb amputees, a large majority are discontent and experience rejection with their current prosthesis during activities of daily living (ADL). Here we introduce Upbeat, an augmented reality (AR) dance game designed to improve rehabilitation therapies in upper limb amputees. In Upbeat, the patient is instructed to follow a virtual dance instructor, performing choreographed dance movements containing hand gestures involved in upper limb rehabilitation therapy. The patient's position is then tracked using a Microsoft Kinect sensor while the hand gestures are analyzed using EMG data collected from a Myo Armband. Additionally, a gamified score is calculated based on how many gestures and movements were correctly performed. Upon completion of the game, a diagnostic summary of the results is shown in the form of a graph summarizing the collected EMG data, as well as with a video displaying an augmented visualization of the patient's upper arm muscle activity during gameplay. By gamifying the rehabilitation process, Upbeat has the potential to improve therapy on upper limb amputees by enabling the start of rehabilitation immediately after trauma, providing personalized feedback which professionals can utilize to accurately assess patient's progress, and increasing patient excitement, therefore increasing patient willingness to complete rehabilitation. This paper is concerned with the description and evaluation of our prototypic implementation of Upbeat that will serve as the basis for conducting clinical studies to evaluate its impact on rehabilitation.

Suitability of the Kinect Sensor and Leap Motion Controller-A Literature Review.

As the need for sensors increases with the inception of virtual reality, augmented reality and mixed reality, the purpose of this paper is to evaluate the suitability of the two Kinect devices and the Leap Motion Controller. When evaluating the suitability, the authors’ focus was on the state of the art, device comparison, accuracy, precision, existing gesture recognition algorithms and on the price of the devices. The aim of this study is to give an insight whether these devices could substitute more expensive sensors in the industry or on the market. While in general the answer is yes, it is not as easy as it seems: There are significant differences between the devices, even between the two Kinects, such as different measurement ranges, error distributions on each axis and changing depth precision relative to distance.

A Hybrid Path-Planning Strategy for Mobile Robots with Limited Sensor Capabilities.

This paper introduces a strategy for the path planning problem for platforms with limited sensor and processing capabilities. The proposed algorithm does not require any prior information but assumes that a mapping algorithm is used. If enough information is available, a global path planner finds sub-optimal collision-free paths within the known map. For the real time obstacle avoidance task, a simple and cost-efficient local planner is used, making the algorithm a hybrid global and local planning solution. The strategy was tested in a real, cluttered environment experiment using the Pioneer P3-DX and the Xbox 360 kinect sensor, to validate and evaluate the algorithm efficiency.

A Differential Evolution Approach to Optimize Weights of Dynamic Time Warping for Multi-Sensor Based Gesture Recognition.

In this research, we present a differential evolution approach to optimize the weights of dynamic time warping for multi-sensory based gesture recognition. Mainly, we aimed to develop a robust gesture recognition method that can be used in various environments. Both a wearable inertial sensor and a depth camera (Kinect Sensor) were used as heterogeneous sensors to verify and collect the data. The proposed approach was used for the calculation of optimal weight values and different characteristic features of heterogeneous sensor data, while having different effects during gesture recognition. In this research, we studied 27 different actions to analyze the data. As finding the optimal value of the data from numerous sensors became more complex, a differential evolution approach was used during the fusion and optimization of the data. To verify the performance accuracy of the presented method in this study, a University of Texas at Dallas Multimodal Human Action Datasets (UTD-MHAD) from previous research was used. However, the average recognition rates presented by previous research using respective methods were still low, due to the complexity in the calculation of the optimal values of the acquired data from sensors, as well as the installation environment. Our contribution was based on a method that enabled us to adjust the number of depth cameras and combine this data with inertial sensors (multi-sensors in this study). We applied a differential evolution approach to calculate the optimal values of the added weights. The proposed method achieved an accuracy 10% higher than the previous research results using the same database, indicating a much improved accuracy rate of motion recognition.

Can Kinect aid motor learning in sportsmen? A study for three standing techniques in judo.

Our objective was to examine how exercises with the second generation of the Microsoft Kinect sensor may aid in the process of motor learning in young judo practitioners. We addressed improvements in spatio-temporal accuracy during execution of three standing techniques in judo, in a simple paradigm designed to study short-term practice effects. Two groups of judokas, 12 athletes each—one aided with Kinect and our dedicated software vs a group of controls—were asked to mimic previously recorded master-level performances of the three techniques, established as benchmarks by a two times world champion in judo. In five training sessions, athletes of the aided group used a virtual-reality setup in which they trained with a virtual representation of the master displayed on a large screen with a simultaneous real-time visualisation of their own movements in the form of an avatar based on body joint localisation, as determined by Kinect, which also measured their performance. The control group used Kinect in the 1st and 5th session, which was necessary for the measurements that constituted the basis for subsequent statistical comparisons, whereas the 2nd, 3rd, and 4th session in this group was guided by a coach, without the use of the Kinect setup. In addition, athletes of the two groups had unrestricted access to a video recording of the master performing the three throws. We found statistically significant improvements (p < 0.05) in the accuracy of executing the three techniques between the 1st and the 5th training session for the aided group but not for the control group. We conclude that incorporating Kinect based exercises into a judo training programme may be a useful means to supporting motor learning, therefore enhancing training efficiency, and thus improving performance.

Development of a 3D User Interface based on Kinect Sensor and Bend-Sensing Data Glove for Controlling Software in the Mining Industry

Development of a 3D User Interface based on Kinect Sensor and Bend-Sensing Data Glove for Controlling Software in the Mining Industry

Improvements of direct current motor control and motion trajectory algorithm development for automated guided vehicle

The precise control of the two-wheel independent drive motor speed of automated guided vehicle is the prerequisite for high-precision operation of its actuator. The optimization of motor control performance has been a topic of concern to researchers. First, the model identification of direct current motor is done. Second, there are two control strategies, including parallel proportional–integral–differential control strategy and the coupling proportional–integral–differential control strategy, which are researched and compared for the direct current motor control system of automated guided vehicle in dispatching the goods from one location to another with greater ease. Finally, the motion trajectory algorithm of automated guided vehicle is also proposed on the basis of kinematics theory. Besides, the above control strategies and algorithm improved for automated guided vehicle are simulated by the MATLAB tool and the experimental simulations by Kinect sensor, and the simulation results show that the coupling proportional–integral–differential control strategy is slightly better in controlling performance because of some smaller cumulative errors and the better start performance. Meanwhile, the motion trajectory algorithm of automated guided vehicle under both the control strategies can achieve different effects in different control cases.

An Agile Approach to Improve the Usability of a Physical Telerehabilitation Platform

The goal of a telerehabilitation platform is to safely and securely facilitate the rehabilitation of patients through the use of telecommunication technologies complemented with the use of biomedical smart sensors. The purpose of this study was to perform a usability evaluation of a telerehabilitation platform. To improve the level of usability, the researchers developed and proposed an iterative process. The platform uses a digital representation of the patient which duplicates the therapeutic exercise being executed by the patient; this is detected by a Kinect camera and sensors in real time. This study used inspection methods to perform a usability evaluation of an exploratory prototype of a telerehabilitation platform. In addition, a cognitive workload assessment was performed to complement the usability evaluation. Users were involved through all the stages of the iterative refinement process. Usability issues were progressively reduced from the first iteration to the fourth iteration according to improvements which were developed and applied by the experts. Usability issues originally cataloged as catastrophic were reduced to zero, major usability problems were reduced to ten (2.75%) and minor usability problems were decreased to 141 (38.74%). This study also intends to serve as a guide to improve the usability of e-Health systems in alignment with the software development cycle.

Research on gesture recognition of smart data fusion features in the IoT

With the rapid development of Internet of things technology, the interaction between people and things has become increasingly frequent. Using simple gestures instead of complex operations to interact with the machine, the fusion of smart data feature information and so on has gradually become a research hotspot. Considering that the depth image of the Kinect sensor lacks color information and is susceptible to depth thresholds, this paper proposes a gesture segmentation method based on the fusion of color information and depth information; in order to ensure the complete information of the segmentation image, a gesture feature extraction method based on Hu invariant moment and HOG feature fusion is proposed; and by determining the optimal weight parameters, the global and local features are effectively fused. Finally, the SVM classifier is used to classify and identify gestures. The experimental results show that the proposed fusion features method has a higher gesture recognition rate and better robustness than the traditional method.

Graph Cut-Based Human Body Segmentation in Color Images Using Skeleton Information from the Depth Sensor.

Segmentation of human bodies in images is useful for a variety of applications, including background substitution, human activity recognition, security, and video surveillance applications. However, human body segmentation has been a challenging problem, due to the complicated shape and motion of a non-rigid human body. Meanwhile, depth sensors with advanced pattern recognition algorithms provide human body skeletons in real time with reasonable accuracy. In this study, we propose an algorithm that projects the human body skeleton from a depth image to a color image, where the human body region is segmented in the color image by using the projected skeleton as a segmentation cue. Experimental results using the Kinect sensor demonstrate that the proposed method provides high quality segmentation results and outperforms the conventional methods.

Omnidirectional 3D Point Clouds Using Dual Kinect Sensors

This paper proposes a registration method for two sets of point clouds obtained from dual Kinect V2 sensors, which are facing each other to capture omnidirectional 3D data of the objects located in between the two sensors. Our approach aims at achieving a handy registration without the calibration-assisting devices such as the checker board. Therefore, it is suitable in portable camera setting environments with frequent relocations. The basic idea of the proposed registration method is to exploit the skeleton information of the human body provided by the two Kinect V2 sensors. That is, a set of correspondence pairs in skeleton joints of human body detected by Kinect V2 sensors is used to determine the calibration matrices, then Iterative Closest Point (ICP) algorithm is adopted for finely tuning the calibration parameters. The performance of the proposed method is evaluated by constructing 3D point clouds for human bodies and by making geometric measurements for cylindrical testing objects.

An improved technique for increasing the accuracy of joint-to-ground distance tracking in kinect v2 for foot-off and foot contact detection

The Kinect sensor has been widely used in different applications such as rehabilitation and gait analysis. Whilst Kinect v2 was released with several improvements over its predecessor, it still incorporates depth-map intrinsic inaccuracies. This results in inconsistencies in skeletal-data acquisition, especially in joint localisation and distance-to-ground tracking with respect to the Kinect’s 3-D Cartesian coordinate reference point. This research proposes a correction technique based on the two-point linear equation derived from the information gathered from different subjects’ skeletal data and data regression analysis to compensate the inaccuracies in joint-to-ground data collection. The research also proposes a new footsteps detection method based on skeletal data and plane detection techniques that calculates a footstep by using the ankle’s Euclidean distance from the floor, regardless of the subject’s distance from the camera. The results show that after the correction technique was applied, data acquisition proved to be consistent and more accurate within a distance range of 1.6–2.9 m from the Kinect camera, regardless of the subject’s location to the camera’s reference point. Moreover, the inconsistency of joint data read by the Kinect was reduced from 25.69% to 5.25% and the footsteps detection accuracy increased from 42.85% to 79.76% on average for both legs.

Automatic detection of stereotyped movements in autistic children using the Kinect sensor

Autism Spectrum Disorders (ASD) is a developmental disorder that affects communications, social skills or behaviours that can occur in some people. Children or adults with ASD often have repetitive motor movements or unusual behaviours. The objective of this work is to automatically detect stereotypical motor movements in real time using Kinect sensor. The approach is based on the $P Point-Cloud Recogniser to identify multi-stroke gestures as point clouds. This paper presents new methodology to automatically detect five stereotypical motor movements: body rocking, hand flapping, fingers flapping, hand on the face and hands behind back. With many ASD-children, our proposed system gives us satisfactory results. This can help to implement a smart video surveillance system and then helps clinicians in the diagnosing ASD.

Automatic detection of stereotyped movements in autistic children using the Kinect sensor

Autism Spectrum Disorders (ASD) is a developmental disorder that affects communications, social skills or behaviours that can occur in some people. Children or adults with ASD often have repetitive motor movements or unusual behaviours. The objective of this work is to automatically detect stereotypical motor movements in real time using Kinect sensor. The approach is based on the $P Point-Cloud Recogniser to identify multi-stroke gestures as point clouds. This paper presents new methodology to automatically detect five stereotypical motor movements: body rocking, hand flapping, fingers flapping, hand on the face and hands behind back. With many ASD-children, our proposed system gives us satisfactory results. This can help to implement a smart video surveillance system and then helps clinicians in the diagnosing ASD.

Development of a Novel Home Based Multi-Scene Upper Limb Rehabilitation Training and Evaluation System for Post-Stroke Patients

Upper limb rehabilitation requires long-term, repetitive rehabilitation training and assessment. However, many patients cannot afford for heavy medical fees. It is necessary to design an effective, low-cost, and reasonable home rehabilitation and evaluation system. In this paper, we developed a novel home-based multi-scene upper limb rehabilitation training and evaluation system for post-stroke patients. Based on the Kinect sensor and the posture sensor, the multi-sensors fusion method was used to track the motion of the patients. Multiple virtual scenes were designed to encourage rehabilitation training of upper limbs and trunk. A rehabilitation evaluation method was proposed integrating Fugl-Meyer assessment (FMA) scale and upper limb reachable workspace relative surface area (RSA). Furthermore, an FMA-RSA assessment model was established to assess an upper limb motor function. Correlation-based dynamic time warping was used to solve the problem of inconsistent upper limb movement path in different patients. Two clinical trials were conducted. The experimental results show that the system is very friendly to the subjects. The rehabilitation assessment results by this system are highly correlated with the therapist's (the highest forecast accuracy was 92.7% in the 13th item). It also reveals that long-term rehabilitation training can improve the upper limb motor function of the patients statistically significant (p=0.02 <; 0.05). The system has the potential to become an effective home rehabilitation training and evaluation system.

Human activity recognition from histogram of spatiotemporal depth features

The recent evolution in sensor-based depth information has been developed a sounding scope to work for human activity recognition using depth image sequences. Activity recognition is having a key importance due to its advantages in several domain like surveillance systems at airport, patient monitoring system, care of elderly people, etc. This work proposed the objective and evaluations to recognise daily life human activities by spatiotemporal depth information. For this purpose, Kinect sensor is used to collect the data pertaining to a single activity performed by multiple person at a time. The spatiotemporal depth features are computed for activity recognition and support vector machine is used in classification phase. We have nine class of human actions in the database for RGB-D human activity recognition. This dataset is reconfigured from Cornell human activity and Berkeley multimodal human action databases. For multiple human action recognition, 91.38% accuracy is achieved on the synthetic dataset.