Microsoft Kinect Sensor Research Articles

Design and development of a feedback system for automatic treadmill speed adaptation

Treadmills with automatic speed adjustment offer a unique advantage: they can mimic outdoor conditions for specific training or testing protocols. This versatility makes them highly applicable in both sports and rehabilitation settings. This study presents a novel framework based on a feedback control loop system, conceived to control a treadmill belt speed using a non-invasive and low-cost sensor (Microsoft Kinect) to detect the users’ position and avoid object obstruction issues. The speed of the treadmill belt is regulated according to the user’s speed, by means of a proportional-integrative-derivative (PID) controller and a parabolic gain function. By tuning the gain function parameters, the user can customize the response of the treadmill. Position data collected during exercise using the Microsoft Kinect sensor was compared with that collected with a stereophotogrammetric motion capture system, showing promising results in terms of accuracy in position assessment. The comparison highlighted a 0.9 degree of correlation between the two systems during the running and cross-country indoor skiing tests performed. In addition, upon considering the relationship between the differences and averages of the two measures, no systematic bias was identified. The system proved to be functional for running and cross-country skiing, and it can therefore re-create similar typical characteristics of outdoor environments (e.g., speed and slope) thanks to the non-invasive user’s position detection and the customizable gain function.

Motion Capture in Mixed-Reality Applications: A Deep Denoising Approach

Motion capture is a fundamental technique in the development of video games and in film production to animate a virtual character based on the movements of an actor, creating more realistic animations in a short amount of time. One of the ways to obtain this movement from an actor is to capture the motion of the player through an optical sensor to interact with the virtual world. However, during movement some parts of the human body can be occluded by others and there can be noise caused by difficulties in sensor capture, reducing the user experience. This work presents a solution to correct the motion capture errors from the Microsoft Kinect sensor or similar through a deep neural network (DNN) trained with a pre-processed dataset of poses offered by Carnegie Mellon University (CMU) Graphics Lab. A temporal filter is implemented to smooth the movement, given by a set of poses returned by the deep neural network. This system is implemented in Python with the TensorFlow application programming interface (API), which supports the machine learning techniques and the Unity game engine to visualize and interact with the obtained skeletons. The results are evaluated using the mean absolute error (MAE) metric where ground truth is available and with the feedback of 12 participants through a questionnaire for the Kinect data.

In-air Hand Gesture Signature Recognition: An iHGS Database Acquisition Protocol.

Background:With the advances in current technology, hand gesture recognition has gained considerable attention. It has been extended to recognize more distinctive movements, such as a signature, in human-computer interaction (HCI) which enables the computer to identify a person in a non-contact acquisition environment. This application is known as in-air hand gesture signature recognition. To our knowledge, there are no publicly accessible databases and no detailed descriptions of the acquisitional protocol in this domain. Methods:This paper aims to demonstrate the procedure for collecting the in-air hand gesture signature's database. This database is disseminated as a reference database in the relevant field for evaluation purposes. The database is constructed from the signatures of 100 volunteer participants, who contributed their signatures in two different sessions. Each session provided 10 genuine samples enrolled using a Microsoft Kinect sensor camera to generate a genuine dataset. In addition, a forgery dataset was also collected by imitating the genuine samples. For evaluation, each sample was preprocessed with hand localization and predictive hand segmentation algorithms to extract the hand region. Then, several vector-based features were extracted. Results:In this work, classification performance analysis and system robustness analysis were carried out. In the classification analysis, a multiclass Support Vector Machine (SVM) was employed to classify the samples and 97.43% accuracy was achieved; while the system robustness analysis demonstrated low error rates of 2.41% and 5.07% in random forgery and skilled forgery attacks, respectively. Conclusions:These findings indicate that hand gesture signature is not only feasible for human classification, but its properties are also robust against forgery attacks.

Examination of overhead and single-leg squat kinematics and lower extremity injury in female collegiate athletes.

Overhead squat (OHS) and single-leg squat (SLS) tests have been used as screening tools to identify injury risk in athletic populations. Yet, no study has examined the association between specific trunk and lower extremity (LE) kinematic measures acquired during OHS and SLS performance and LE injury incidence in female collegiate athletes. The purpose of this study was to examine the association between LE and trunk kinematics captured during OHS and SLS performance and LE injury in female collegiate athletes. This was a prospective cohort study. One hundred eight Division I female collegiate athletes underwent OHS and SLS testing before the start of their competitive season. A Microsoft Kinect sensor using Athletic Movement Assessment software (PhysiMax®) was used to capture kinematic variables: hip flexion, knee flexion, knee frontal plane projection angle (FPPA), and trunk flexion. Participants were tracked during one competitive season for incidence of LE injury. Logistic regression models were used to examine the association between OHS and SLS measures and injury. Twenty-three (21.3%) participants suffered a LE injury. No significant associations were found between any OHS (odds ratio [OR] range: 0.91-1.08) or SLS (OR range: 0.81-1.22) kinematic measure and incidence of LE injury. Sagittal plane LE and trunk kinematics and knee FPPA during OHS and SLS performance were not associated with LE injury in this cohort of athletes. Our findings do not support the use of these select kinematic measures captured during OHS and SLS testing as stand-alone injury risk assessments in female collegiate athletes.

Investigation into the suitability of kinect sensor for automated body measurement

Due to the low cost and wide availability of the Kinect sensor, researchers and experts in the field of anthropometry, sizing and clothing fiting are leveraging on its inbuilt 3D camera to develop systems for automated body measurement. This study focuses on the evaluation of the Microsoft Kinect (V1) sensor to determine its suitability for automated body measurement. The study was conducted by data collection of various body dimensions of test subjects using a measuring tape as a reference. Furthermore, a statistical approach known as the measurement system analysis was used to investigate the sensor's capability to produce accurate, reliable and consistent body measurements. The results obtained shows indicates that there exists very little variation when the measurement is repeated. Also, the instrument is relatively stable, with minimal bias which can be corrected by calibration. The outcome of the study proves the effectiveness of the Microsoft Kinect sensor as a means of conducting body measurement.

Machine learning based text neck syndrome detection using Microsoft Kinect sensor

Text Neck Syndrome is repeated stress injury to the neck, or it is a pain encountered in the neck due to excessive usage of handheld devices for a prolonged period. In this paper, we have used the Microsoft Kinect sensor to create the dataset for normal and abnormal sitting postures by obtaining 3D skeletal coordinates of people. Features are extracted from the data collected from the Kinect sensor data based on the various parameters like Minkowski distance, Euclidean distance, Manhattan distance, etc. Then we have applied different machine learning algorithms, namely, K-nearest neighbours, Random forest, Support vector machine, Logistic and Linear Regression, to classify them. The performance of various algorithms is analysed. The best testing accuracy achieved is 92.85% by using Random Forest and Support Vector Machine algorithms.

ART – BASED TRAINING METHODS FOR EMPOWERING ADULTS IN THE DIGITAL ERA

Abstract. Since the dawn of humanity, dancing has been central to life occasions, for celebration, mourning, entertaining or even communicating. Still, it remains a vital practice around the world. With the development of technology, many efforts have been made to adapt specific choreographies and dancing rituals to the modern era, via digital platforms. However, these sequences cannot be construed as realistic when they are viewed through computers, because the actual expressions are lost in the translation. A lot of attempts have been made to provide a solution to this issue, but the most successful are the motion capturing devices and equipment. In this research, we focus on the recording of choreographies, which represent important social concepts, using a Microsoft Kinect sensor as a motion capturing device. Microsoft Kinect is chosen because it is a low-cost device that can deliver adequate results. The outcome of the research is a “handbook” in virtual reality environment, which can be used as an educational tool by anyone. The project aims at being a response to the global need of today’s world for creation of inclusive environments, where socially aware people can peacefully co – exist. The digital handbook can enhance the skills and competences of adult trainers in arts and culture and every participant gets in touch with sensitive concepts and develops social awareness. Inspiration for the project is drawn via social and humanistic values stemming from philosophical, sociological and psychological texts and it is innovative not only for its content but also for the way this is developed.

Classification of human movements by using Kinect sensor

In recent years, studies have been carried out to classify human movements in many areas such as health and safety. To classify human movements, image processing methods have also started to be used in recent years. With the help of learning-based algorithms, human posture can be defined in the images obtained by various imaging methods. The predecessor methods of these classification algorithms are machine learning and deep learning. In addition, in recent years, the use of sensors that can detect human joints in perceiving human posture has also increased. The Kinect sensor, developed by Microsoft, is one of the most frequently used sensors because it is not wearable and can detect joints with infrared rays and transfer this information directly to the computer via USB connection. This study used a dataset called CAD60 that included real-time human posture information and images obtained using a Microsoft Kinect sensor, which is available in the literature. This dataset contains data that includes different movements/postures of different people. Within the scope of this study, the performances of these algorithms were obtained by using classification algorithms with the MATLAB program and these performances were compared. The classification algorithms have been used to try to improve the results by using different architectures. When raw data is used, classification accuracy is obtained as 72.60% with one of the machine learning methods, the Cosine K-Nearest Neighbor method. With the feature selection method, this success value has been increased to 74.18%. In addition, when classified by the Support Vector Machines method after the feature extraction process using the Long Short Term Memory method from the deep network architectures, which is the method proposed in this study, the accuracy rate was increased to 98.95%. The best method of classifying human posture was investigated by using different methods and a method was proposed by comparing it with the literature.

Gesture Phase Segmentation Dataset: An Extension for Development of Gesture Analysis Models

In recent years, experts in theory of gesture have been showing some interest in automating the discovery of gesture information. Such an automation can help them in reducing the inherent subjectivity of gesture studies. Usually, to produce information for linguistic and psycholinguistic studies, the researchers analyze a video of people speaking and gesturing. This annotation task is costly and it is the goal of automation. Such videos compose the datasets that allow the development of automated models capable to carry out part of the analysis of gestures. In this paper, we present a detailed documentation about the Gesture Phase Segmentation Dataset, publicized in UCI Machine Learning Repository, and an extension of such dataset. Such dataset is especially prepared to be used in the development of models capable to carry out the segmentation of gestures in their phases. The extended dataset is composed by nine videos of three people gesturing and telling stories. The data was captured with Microsoft Kinect Sensor and they are represented by spatial coordinates and temporal information (velocity and acceleration). The data are labeled following four phase of gesture (preparation, stroke, hold and retraction) and rest positions.

Hand Gesture Recognition with Symmetric Pattern under Diverse Illuminated Conditions Using Artificial Neural Network

This paper investigated the effects of variant lighting conditions on the recognition process. A framework is proposed to improve the performance of gesture recognition under variant illumination using the luminosity method. To prove the concept, a workable testbed has been developed in the laboratory by using a Microsoft Kinect sensor to capture the depth images for the purpose of acquiring diverse resolution data. For this, a case study was formulated to achieve an improved accuracy rate in gesture recognition under diverse illuminated conditions. For data preparation, American Sign Language (ASL) was used to create a dataset of all twenty-six signs, evaluated in real-time under diverse lighting conditions. The proposed method uses a set of symmetric patterns as a feature set in order to identify human hands and recognize gestures extracted through hand perimeter feature-extraction methods. A Scale-Invariant Feature Transform (SIFT) is used in the identification of significant key points of ASL-based images with their relevant features. Finally, an Artificial Neural Network (ANN) trained on symmetric patterns under different lighting environments was used to classify hand gestures utilizing selected features for validation. The experimental results showed that the proposed system performed well in diverse lighting effects with multiple pixel sizes. A total aggregate 97.3% recognition accuracy rate is achieved across 26 alphabet datasets with only a 2.7% error rate, which shows the overall efficiency of the ANN architecture in terms of processing time.

3D Vision Using Multiple Structured Light-Based Kinect Depth Cameras

Real-time 3D scanning of a scene or object using multiple depth cameras is often required in many applications but is still a challenging task for the computer vision community, especially when the object or scene is partially occluded and dynamic. If active depth sensors are used in this case, their resulting depth map quality gets degraded due to interference between active radiations from each depth sensor. Passive 3D sensors like stereo cameras can avoid the issue of interference as they do not emit any active radiation, but they face correspondence problems. Since releasing the commodity depth sensor Microsoft Kinect, researchers are getting more interested in active depth-sensing. However, Kinect sensors have some easily noticeable limitations concerning 3D reconstruction such as: they can provide depth maps for a limited range, their field of view is restricted and holes are observed in the depth map due to occlusion. The above-mentioned limitations can be overcome if multiple Kinect sensors are used simultaneously instead of a single Kinect sensor. Still, the challenge here is to avoid interference between these sensors. We present a comprehensive review of possible solutions to avoid interference between multiple Kinect sensors. Furthermore, we introduce the Kinect technology in detail along with applications where multiple Kinect sensors are used in the literature. We expect that this paper will be helpful to the researchers who want to use multiple Kinect sensors in sharing the workplace in their research.

Serious Game for VR Road Crossing in Special Needs Education

Autism spectrum disorder (ASD) is a developmental disability that can impair communication, social skills, living skills, and learning capabilities. Learning approaches usually differ between mainstream schools and special needs schools, to cater for the different learning processes of children with ASD. Besides the traditional classroom-based education, alternative technology and methods are explored for special needs education. One method is to train children with ASD using Virtual Reality (VR) technologies. Many prior works show the effectiveness of VR-based learning with varying degrees of success. Some children with ASD may face challenges to gain independent living skills. Their parents or guardians have to expend a significant amount of effort in taking care of children with ASD. It will be very helpful if they can have a learning opportunity to gain such living skills. In this research, we develop a VR serious game to train children with ASD one of the basic living skills for road crossing safely. The VR serious game operates on multiple types of platforms, with various user interaction inputs including the Microsoft Kinect sensor, keyboard, mouse, and touch screen. The game design and methodology will be described in this paper. Experiments have been conducted to evaluate the learning effectiveness of the road crossing game, with very positive results achieved in the quiz and survey questionnaire after the gameplay.

Automatic Landmark Detection of Human Back Surface from Depth Images via Deep Learning

Studying human postural structure is one of the challenging issues among scholars and physicians. The spine is known as the central axis of the body, and due to various genetic and environmental reasons, it could suffer from deformities leading to physical dysfunction and correspondingly affecting people's quality of life. Radiography is the most common method for detecting these deformities; however, it frequently exposes the patient to X-rays and ionization and consequently increases cancer risk in patients particularly children and pregnant women.The purpose of this research is to provide an entirely safe and non-invasive method to examine the spiral structure and its deformities. Hence, it is attempted to find the exact location of anatomical landmarks on the human back surface, which provides useful and functional information about the status of the human postural structure to the physician. In this study, using Microsoft Kinect sensor, the depth images from the human back surface of 105 people were recorded and, our proposed approach – a deep convolution neural network- was used as a model to estimate the locations of anatomical landmarks. In network architecture, two learning processes, including landmark position and affinity between the two associated landmarks, are successively performed in two separate branches. This is a bottom-up approach; thus, the runtime complexity is considerably reduced, and then the resulting anatomical points are evaluated concerning manual landmarks marked by the operator as the benchmark. Our results showed 86.9% of PDJ and 80% of PCK that demonstrate more effectiveness compared to other methods.

Automatic Landmark Detection of Human Back Surface from Depth Images via Deep Learning

Studying human postural structure is one of the challenging issues among scholars and physicians. The spine is known as the central axis of the body, and due to various genetic and environmental reasons, it could suffer from deformities leading to physical dysfunction and correspondingly affecting people's quality of life. Radiography is the most common method for detecting these deformities; however, it frequently exposes the patient to X-rays and ionization and consequently increases cancer risk in patients particularly children and pregnant women.The purpose of this research is to provide an entirely safe and non-invasive method to examine the spiral structure and its deformities. Hence, it is attempted to find the exact location of anatomical landmarks on the human back surface, which provides useful and functional information about the status of the human postural structure to the physician. In this study, using Microsoft Kinect sensor, the depth images from the human back surface of 105 people were recorded and, our proposed approach – a deep convolution neural network- was used as a model to estimate the locations of anatomical landmarks. In network architecture, two learning processes, including landmark position and affinity between the two associated landmarks, are successively performed in two separate branches. This is a bottom-up approach; thus, the runtime complexity is considerably reduced, and then the resulting anatomical points are evaluated concerning manual landmarks marked by the operator as the benchmark. Our results showed 86.9% of PDJ and 80% of PCK that demonstrate more effectiveness compared to other methods.

Edge preserving range image smoothing using hybrid locally kernel-based weighted least square

In this paper, we propose the hybrid locally kernel-based weighted least square (HKLS) method to reduce the noise of the range images captured from the Microsoft Kinect sensor. Although removing the noise of Kinect sensor is inevitable in mobile robot applications, preserving the main structures of an indoor scene is a key point of the smoothing methods for range data. Our method uses a situation estimation module to apply an appropriate method for denoising the input data. In this paper, situations are classified into two categories including flat regions and edge regions. The locally kernel-based weighted least square method is used for the edge regions that have significant changes in the surface normals and the weight of each point is assigned with the Gaussian mixture model. The Bilateral filtering is applied for the flat regions with no sharp change in surface normals. We provided a solution for the noise data issue and proof of uncertainty reduction in presence of noise data. There is a fusion module for assigning the proportion of each method according to the different situations. We evaluated our method on the NYUv2 dataset that contains different RGB-D indoor scenes with other state-of-the-art methods. We achieved the value of 49.48 for the Z-MAE and 0.502 for the N-MAE error metrics.

Comparison of Azure Kinect overground gait spatiotemporal parameters to marker based optical motion capture

BackgroundInstrumented measurement of spatiotemporal parameters during walking can provide valuable information on an individual’s overall function and health. Efficient, inexpensive, and accurate measurement of overground walking spatiotemporal parameters would be a critical component of providing point-of-care assessments of gait function, concussion recovery, fall-risk, and cognitive decline. Depth cameras combined with skeleton pose tracking algorithms, such as the Microsoft Kinect with body tracking software, have been used to measure walking spatiotemporal parameters. However, the ability of the latest generation Microsoft Kinect sensor, the Azure Kinect, to accurately measure overground walking spatiotemporal parameters has not been evaluated in the literature. Research questionThe purpose of this work was to compare overground walking spatiotemporal parameters measurements from a 12 camera Vicon optical motion capture system to measurements of a single Azure Kinect with body tracking SDK (software development kit). MethodsSpatiotemporal parameters of overground walking were simultaneously collected on twenty young healthy participants. Stride length, stride time, step length and step width were derived from ankle joint center locations and measurements from the two instruments were compared using descriptive statistics, scatter plots, Pearson correlation analyses, and Bland-Altman analyses. ResultsPearson correlation coefficients were greater than 0.87 for all spatiotemporal parameters with most parameters demonstrating very strong (> 0.9) agreement. The mean of the differences for stride length between measurements was 35.6 mm for the left limb and 39.1 mm for the right limb, both of which are less than 3% of average stride length. Mean of the differences for step width and stride time were less than 2% and 1% of their averages respectively. SignificanceA single Microsoft Azure Kinect with body tracking SDK can provide clinically relevant measurement of walking spatiotemporal parameters, providing accessible and objective measurements that can improve clinical decision making across a variety of patient populations.

Concurrent validity of a markerless motion capture system for the assessment of shoulder functional movement

As a markerless motion capture system (MLS), the Microsoft Kinect sensor may be a low-cost and portable option for measuring shoulder joint angles. However, the system's concurrent validity to capture shoulder functional movement is unclear. The purpose of this study was to investigate the concurrent validity of MLS for capturing shoulder kinematics during functional movement by comparing the MLS measures to those of the marker-based motion capture system (MBS). Twenty-five healthy participants were included in this study. Using the Microsoft Kinect sensor as the MLS and Vicon as the MBS, six shoulder functional movements were measured in all three anatomical planes concurrently. The six movements included flexion to max, maximum extension, abduction to 90°, internal rotation at 90° abduction, external rotation at 0° abduction, and maximum horizontal adduction. The shoulder joint kinematics was measured with both systems. The MLS showed a good to excellent correlation (r ​> ​0.75) with MBS for abduction to 90°, maximum external rotation at 0° abduction and maximum internal rotation at 90° abduction. The results for maximum extension (r ​= ​0.727) and maximum horizontal adduction (r ​= ​0.619) showed a moderate to good correlation. For maximum flexion, a poor correlation was observed (r ​= ​0.479). The validity of the MLS to measure shoulder kinematics would be acceptable. The system demonstrates potential as a measurement tool of joint motions for a function assessment and rehabilitation purpose.

Contactless Haptic Display Through Magnetic Field Control.

Haptic rendering enables people to touch, perceive, and manipulate virtual objects in a virtual environment. Using six cascaded identical hollow disk electromagnets and a small permanent magnet attached to an operator's finger, this paper proposes and develops an untethered haptic interface through magnetic field control. The concentric hole inside the six cascaded electromagnets provides the workspace, where the 3D position of the permanent magnet is tracked with a Microsoft Kinect sensor. The driving currents of six cascaded electromagnets are calculated in real-time for generating the desired magnetic force. Offline data from an FEA (finite element analysis) based simulation, determines the relationship between the magnetic force, the driving currents, and the position of the permanent magnet. A set of experiments including the virtual object recognition experiment, the virtual surface identification experiment, and the user perception evaluation experiment were conducted to demonstrate the proposed system, where Microsoft HoloLens holographic glasses are used for visual rendering. The proposed magnetic haptic display leads to an untethered and non-contact interface for natural haptic rendering applications, which overcomes the constraints of mechanical linkages in tool-based traditional haptic devices.

In-air Hand Gesture Signature Recognition: An iHGS Database Acquisition Protocol

Background With the advances in current technology, hand gesture recognition has gained considerable attention. It has been extended to recognize more distinctive movements, such as a signature, in human-computer interaction (HCI) which enables the computer to identify a person in a non-contact acquisition environment. This application is known as in-air hand gesture signature recognition. To our knowledge, there are no publicly accessible databases and no detailed descriptions of the acquisitional protocol in this domain. Methods This paper aims to demonstrate the procedure for collecting the in-air hand gesture signature’s database. This database is disseminated as a reference database in the relevant field for evaluation purposes. The database is constructed from the signatures of 100 volunteer participants, who contributed their signatures in two different sessions. Each session provided 10 genuine samples enrolled using a Microsoft Kinect sensor camera to generate a genuine dataset. In addition, a forgery dataset was also collected by imitating the genuine samples. For evaluation, each sample was preprocessed with hand localization and predictive hand segmentation algorithms to extract the hand region. Then, several vector-based features were extracted. Results In this work, classification performance analysis and system robustness analysis were carried out. In the classification analysis, a multiclass Support Vector Machine (SVM) was employed to classify the samples and 97.43% accuracy was achieved; while the system robustness analysis demonstrated low error rates of 2.41% and 5.07% in random forgery and skilled forgery attacks, respectively. Conclusions These findings indicate that hand gesture signature is not only feasible for human classification, but its properties are also robust against forgery attacks.

Application of Fuzzy and Rough Logic to Posture Recognition in Fall Detection System.

Considering that the population is aging rapidly, the demand for technology for aging-at-home, which can provide reliable, unobtrusive monitoring of human activity, is expected to expand. This research focuses on improving the solution of the posture detection problem, which is a part of fall detection system. Fall detection, using depth maps obtained by the Microsoft Kinect sensor, is a two-stage method. We concentrate on the first stage of the system, that is, pose recognition from a depth map. For lying pose detection, a new hybrid FRSystem is proposed. In the system, two rule sets are investigated, the first one created based on a domain knowledge and the second induced based on the rough set theory. Additionally, two inference aggregation approaches are considered with and without the knowledge measure. The results indicate that the new axiomatic definition of knowledge measures, which we propose has a positive impact on the effectiveness of inference and the rule induction method reducing the number of rules in a set maintains it.