Kinect V2 Sensor Research Articles

Automated classification of movement quality using the Microsoft Kinect V2 sensor

Practitioners commonly perform movement quality assessment through qualitative assessment protocols, which can be time-intensive and prone to inter-rater measurement bias. The advent of portable and inexpensive marker-less motion capture systems can improve assessment through objective joint kinematic analysis. The current study aimed to evaluate various machine learning models that used kinematic features from Kinect position data to classify a performer's Movement Competency Screen (MCS) score. A Kinect V2 sensor collected position data from 31 physically active males as they performed bilateral squat, forward lunge, and single-leg squat; and the movement quality was rated according to the MCS criteria. Features were extracted and selected from domain knowledge-based kinematic variables as model input. Multiclass logistic regression (MLR) was then performed to translate joint kinematics into MCS score. Performance indicators were calculated after a 10-fold cross validation of each model developed from Kinect-based kinematic variables. The analyses revealed that the models' sensitivity, specificity, and accuracy ranged from 0.66 to 0.89, 0.58 to 0.86, and 0.74 to 0.85, respectively. In conclusion, the Kinect-based automated movement quality assessment is a suitable, novel, and practical approach to movement quality assessment.

The Accuracy of the Microsoft Kinect V2 Sensor for Human Gait Analysis. A Different Approach for Comparison with the Ground Truth.

Several studies have examined the accuracy of the Kinect V2 sensor during gait analysis. Usually the data retrieved by the Kinect V2 sensor are compared with the ground truth of certified systems using a Euclidean comparison. Due to the Kinect V2 sensor latency, the application of a uniform temporal alignment is not adequate to compare the signals. On that basis, the purpose of this study was to explore the abilities of the dynamic time warping (DTW) algorithm to compensate for sensor latency (3 samples or 90 ms) and develop a proper accuracy estimation. During the experimental stage, six iterations were performed using the a dual Kinect V2 system. The walking tests were developed at a self-selected speed. The sensor accuracy for Euclidean matching was consistent with that reported in previous studies. After latency compensation, the sensor accuracy demonstrated considerably lower error rates for all joints. This demonstrated that the accuracy was underestimated due to the use of inappropriate comparison techniques. On the contrary, DTW is a potential method that compensates for the sensor latency, and works sufficiently in comparison with certified systems.

Kinect Controlled NAO Robot for Telerehabilitation

Abstract In this paper, we focus on the human upper limb rehabilitation scheme that utilizes the concept ofteleoperation. Teleoperation can help the therapist demonstrate different rehab exercises to a different group of people at the same time remotely. Different groups of people from a different place connected to the same network can get therapy from the same therapist at the same time using the telerehabilitation scheme. Here, we presented a humanoid robot NAO that can be operated remotely by a therapist to demonstrate the exercise to a patient. To mimic the movement demonstrated by the therapist, Kinect V2 sensor which is a markerless vision-based motion-tracking device, was used. Modified Denavit-Hartenberg (DH) convention was used for the kinematic modeling of the human upper arm. From the Kinect data, a geometric solution was developed to find a unique inverse kinematic solution of human upper-extremity. Experimental results revealed that NAO could be teleoperated successfully to instruct and demonstrate patients to perform different arm movement exercises in real-time.

Faster R–CNN–based apple detection in dense-foliage fruiting-wall trees using RGB and depth features for robotic harvesting

Apples in modern orchards with vertical-fruiting-wall trees are comparatively easier to harvest and specifically suitable for robotic picking, where accurate apple detection and obstacle-free access are fundamentally important. However, field images have complex backgrounds because of the presence of nontarget trees and fruit in adjacent rows. An outdoor machine vision system was developed with a low-cost Kinect V2 sensor to improve the accuracy of apple detection by filtering the background objects using depth features. A total of 800 set images were acquired in a commercial fruiting-wall Scifresh apple orchard with dense-foliage canopy. Images were collected in both daytime and nighttime with artificial light. The sensor was kept at 0.5 m to the tree canopies. A depth threshold of 1.2 m was used to remove background. Two Faster R–CNN based architectures ZFNet and VGG16 were employed to detect the Original-RGB and the Foreground-RGB images. Results showed that the highest average precision (AP) of 0.893 was achieved for the Foreground-RGB images with VGG16, which cost 0.181 s on average to process a 1920 × 1080 image. AP values for the Foreground-RGB images with ZFNet and VGG16 were both higher than those of the Original-RGB images. The results indicated that the use of a depth filter to remove background trees improved fruit detection accuracy by 2.5% and that only a minimal difference was found in processing speed between two image datasets. The proposed technique and results are expected to be applicable for robotic harvesting on fruiting-wall apple orchards.

Automatic measurement of the hand dimensions using consumer 3D cameras

<p class="Abstract">This article describes the metrological characterisation of two prototypes that use the point clouds acquired by consumer 3D cameras for the measurement of the human hand geometrical parameters. The initial part of the work is focused on the general description of algorithms that allow for the derivation of dimensional parameters of the hand. Algorithms were tested on data acquired using Microsoft Kinect v2 and Intel RealSense D400 series sensors. The accuracy of the proposed measurement methods has been evaluated in different tests aiming to identify bias errors deriving from point-cloud inaccuracy and at the identification of the effect of the hand pressure and the wrist flexion/extension. Results evidenced an accuracy better than 1 mm in the identification of the hand’s linear dimension and better than 20 cm<sup>3 </sup>for hand volume measurements. The relative uncertainty of linear dimensions, areas, and volumes was in the range of 1-10 %. Measurements performed with the Intel RealSense D400 were, on average, more repeatable than those performed with Microsoft Kinect. The uncertainty values limit the use of these devices to applications where the requested accuracy is larger than 5 % (volume measurements), 3 % (area measurements), and 1 mm (hands’ linear dimensions and thickness).</p>

Kinect v2 based system for gait assessment of children with cerebral palsy in rehabilitation settings

Cerebral palsy (CP) describes a group of disorders of movement, posture and balance caused by a non-progressive brain injury in the immature brain. It is the most prevalent cause of chronic motor disability in childhood, and although two thirds of CP children are able to walk, they show gait limitations. In this context, rehabilitation therapy can improve muscle balance and gait. Previous studies showed the importance of gait analysis as part of multidisciplinary tools for effective rehabilitation treatment. However, the high cost and the infrastructure required for the implementation of commercial gait analysis systems as well as the time-consuming preparation procedures, limit the access to this service. A low cost, non-restrictive, portable and of simple operation and installation system was developed based on Kinect v2 sensor. This study aims to validate it for capturing and analysing gait parameters in children with cerebral palsy. Several gait parameters from eleven CP patients with different levels of disability classified as a function of the Gross Motor Function Classification System (GMFCS) from II to III were recorded while they walked on a flat surface. The Kinect-based gait analysis system was compared with video-recording that yielded the same results. These results show the potential of Kinect to analyse gait in frail patient populations unobtrusively and with very low cost. More importantly, regarding to spatial parameters, the Kinect system was useful even for the worst case of GMFCS III that show a large gait variability with abnormal patterns.

Applicability of a Single Depth Sensor in Real-Time 3D Clothes Simulation: Augmented Reality Virtual Dressing Room Using Kinect Sensor

A busy lifestyle led people to buy readymade clothes from retail stores with or without fit-on, expecting a perfect match. The existing online cloth shopping systems are capable of providing only 2D images of the clothes, which does not lead to a perfect match for the individual user. To overcome this problem, the apparel industry conducts many studies to reduce the time gap between cloth selection and final purchase by introducing “virtual dressing rooms.” This paper discusses the design and implementation of augmented reality “virtual dressing room” for real-time simulation of 3D clothes. The system is developed using a single Microsoft Kinect V2 sensor as the depth sensor, to obtain user body parameter measurements, including 3D measurements such as the circumferences of chest, waist, hip, thigh, and knee to develop a unique model for each user. The size category of the clothes is chosen based on the measurements of each customer. The Unity3D game engine was incorporated for overlaying 3D clothes virtually on the user in real time. The system is also equipped with gender identification and gesture controllers to select the cloth. The developed application successfully augmented the selected dress model with physics motions according to the physical movements made by the user, which provides a realistic fitting experience. The performance evaluation reveals that a single depth sensor can be applied in the real-time simulation of 3D cloth with less than 10% of the average measurement error.

Robotic Mapping Using Autonomous Vehicle

Mapping the interior of the building is the first step in order to obtain the description of that building. In this work, an autonomous robot is designed and built. The robot is able to navigate through an unknown environment, avoiding obstacles and simultaneously build a 3D map on its own. This autonomous feature is implemented using minimal and economically viable hardware such as Kinect v2 sensor and Arduino. Also, an open-source platform Robotic Operating System is used, on top of which the mapping and navigation algorithms are implemented. The 3D map built can be used with a lot of existing devices which involves localization. Post-processing procedures are used to obtain point clouds for building the 3D map. Emphasis is put on the efficient use of the available resources to generate the 3D maps of unknown indoor environment.

Object Extraction Using Probabilistic Maps of Color, Depth, and Near-Infrared Information

Object extraction is one of the important and chal-lenging tasks in the computer vision and/or robotics ? elds.This task is to extract the object from the scene using anypossible cues. The scenario discussed in this study was the objectextraction which considering the Space of Interest (SOI), i.e.,the three dimensional area where the object probably existed.To complete such task, the object extraction method based onthe probabilistic maps of multiple cues was proposed. Thanksto the Kinect V2 sensor, multiple cues such as color, depth, andnear-infrared information can be acquired simultaneously. TheSOI was modeled by a simple probabilistic model by consideringthe geometry of the possible objects and the reachability of thesystem acquired from depth information. To model the color andnear-infrared information, a Gaussian mixture models (GMM)was used. All of the models were combined to generate theprobabilistic maps that were used to extract the object fromthe scene. To validate the proposed object extraction, severalexperiments were conducted to investigate the best combinationof the cues used in this study.

A pilot study on automated quantitative grading of facial functions

Quantitative grading of facial paralysis (FP) and the associated loss of facial function are essential to evaluate the severity and to track deterioration or improvement of the condition following treatment. To date, several computer-assisted grading systems have been proposed but none have gained widespread clinical acceptance. There is still a need for an accurate quantitative assessment tool that is automatic, inexpensive, easy to use, and has low inter-observer variability. The aim of the authors is to develop such a comprehensive Automated Facial Grading (AFG) system. One of this system’s modules: the resting symmetry module has already been presented. The present study describes the implementation of the second module for grading voluntary movements. The system utilizes the Kinect v2 sensor to detect and capture facial landmarks in real time. The functions of three regions, the eyebrows, eyes and mouth, are evaluated by quantitatively grading four voluntary movements. Preliminary results on normal subjects and patients are promising. The AFG system is a novel system that is suitable for clinical use because it is fast, objective, easy to use, and inexpensive. With further enhancement, it can be extended to become a virtual facial rehabilitation tool.

IKarate: Karate Kata Aiding System

Karate is a martial art that can be performed using hands and feet to deliver and block strikes. Karate Kata moves must be executed in a unique way, many moves are performed incorrectly during training. In this paper, we offer a system that Karate performers, coaches, judges and sporting clubs could use. The system aids the Karate performers by capturing their moves using Kinect v2 sensor, pre-processing these moves and then analyzing these moves using F-DTW. A report is displayed to the performers that is easily understandable, to learn how the mistakes were made, and how to fix it/learn from them. F-DTW was used for proving the concept, and an average accuracy of 93.65% was achieved. This Paper is mainly concerned about the first Karate Kata (Heian Shodan).

Deep Learning-Based Upper Limb Functional Assessment Using a Single Kinect v2 Sensor.

We develop a deep learning refined kinematic model for accurately assessing upper limb joint angles using a single Kinect v2 sensor. We train a long short-term memory recurrent neural network using a supervised machine learning architecture to compensate for the systematic error of the Kinect kinematic model, taking a marker-based three-dimensional motion capture system (3DMC) as the golden standard. A series of upper limb functional task experiments were conducted, namely hand to the contralateral shoulder, hand to mouth or drinking, combing hair, and hand to back pocket. Our deep learning-based model significantly improves the performance of a single Kinect v2 sensor for all investigated upper limb joint angles across all functional tasks. Using a single Kinect v2 sensor, our deep learning-based model could measure shoulder and elbow flexion/extension waveforms with mean CMCs >0.93 for all tasks, shoulder adduction/abduction, and internal/external rotation waveforms with mean CMCs >0.8 for most of the tasks. The mean deviations of angles at the point of target achieved and range of motion are under 5° for all investigated joint angles during all functional tasks. Compared with the 3DMC, our presented system is easier to operate and needs less laboratory space.

Assessing walking adaptability in stroke patients

Purpose The ability to adapt walking is important for safe ambulation. Assessments of impairments in walking adaptability with the Interactive Walkway may aid in the development of individualized therapy strategies of stroke patients. The Interactive Walkway is an overground walkway with Kinect v2 sensors for a markerless registration of full-body kinematics, which can be augmented with (gait-dependent) visual context to assess walking adaptability. This study aims to evaluate the potential of the Interactive Walkway as a new technology for assessing walking adaptability in stroke patients. Materials and methods: 30 stroke patients and 30 controls performed clinical tests, quantitative gait assessments and various walking-adaptability tasks on the Interactive Walkway. Outcome measures were compared between stroke patients and controls to examine known-groups validity. Pearson’s correlation coefficients were calculated to assess the relationship between walking-adaptability outcomes and commonly used clinical test scores of walking ability and spatiotemporal gait parameters of unconstrained walking. Results: Good known-groups validity for walking-adaptability outcomes was demonstrated. In addition, the vast majority of walking-adaptability outcomes did not or only moderately correlate with clinical test scores of walking ability and unconstrained walking parameters. Conclusion: Interactive Walkway walking-adaptability outcomes have good known-groups validity and complement standard clinical tests and spatiotemporal gait parameters. IMPLICATIONS FOR REHABILITATION The Interactive Walkway allows for a comprehensive walking-adaptability assessment. Good known-groups validity for walking-adaptability tasks was demonstrated and walking-adaptability tasks complemented clinical tests and gait parameters. The Interactive Walkway has potential for monitoring recovery of walking after stroke. Assessments of walking adaptability may contribute to individualized interventions.

Accuracy of image data stream of a markerless motion capture system in determining the local dynamic stability and joint kinematics of human gait

Assessment of gait parameters is commonly performed through the high-end motion tracking systems, which limits the measurement to sophisticated laboratory settings due to its excessive cost. Recently, Microsoft Kinect (v2) sensor has become popular in clinical gait analysis due to its low-cost. But, determining the accuracy of its RGB-D image data stream in measuring the joint kinematics and local dynamic stability remains an unsolved problem. This study examined the suitability of Kinect(v2) RGB-D image data stream in assessing those gait parameters. Fifteen healthy participants walked on a treadmill during which lower body kinematics were measured by a Kinect(v2) sensor and a optophotogrametric tracking system, simultaneously. Extended Kalman filter was used to extract the lower extremity joint angles from Kinect, while inverse kinematics was used for the gold standard system. For both systems, local dynamic stability was assessed using maximal Lyapunov exponent. Sprague’s validation metrics, root mean square error (RMSE) and normalized RMSE were computed to confirm the difference between the joint angles time series of the two systems while relative agreement between them was investigated through Pearson’s correlation coefficient (pr). Fisher’s Exact Test was performed on maximal Lyapunov exponent to investigate the data independence while reliability was assessed using intraclass correlation coefficients. This study concludes that the RGB-D data stream of Kinect sensor is efficient in estimating joint kinematics, but not suitable for measuring the local dynamic stability.

Avoiding 3D Obstacles in Mixed Reality: Does It Differ from Negotiating Real Obstacles?

Mixed-reality technologies are evolving rapidly, allowing for gradually more realistic interaction with digital content while moving freely in real-world environments. In this study, we examined the suitability of the Microsoft HoloLens mixed-reality headset for creating locomotor interactions in real-world environments enriched with 3D holographic obstacles. In Experiment 1, we compared the obstacle-avoidance maneuvers of 12 participants stepping over either real or holographic obstacles of different heights and depths. Participants’ avoidance maneuvers were recorded with three spatially and temporally integrated Kinect v2 sensors. Similar to real obstacles, holographic obstacles elicited obstacle-avoidance maneuvers that scaled with obstacle dimensions. However, with holographic obstacles, some participants showed dissimilar trail or lead foot obstacle-avoidance maneuvers compared to real obstacles: they either consistently failed to raise their trail foot or crossed the obstacle with extreme lead-foot margins. In Experiment 2, we examined the efficacy of mixed-reality video feedback in altering such dissimilar avoidance maneuvers. Participants quickly adjusted their trail-foot crossing height and gradually lowered extreme lead-foot crossing heights in the course of mixed-reality video feedback trials, and these improvements were largely retained in subsequent trials without feedback. Participant-specific differences in real and holographic obstacle avoidance notwithstanding, the present results suggest that 3D holographic obstacles supplemented with mixed-reality video feedback may be used for studying and perhaps also training 3D obstacle avoidance.

Deep learning based segmentation for automated training of apple trees on trellis wires

Trellised fruiting-wall training systems are becoming standard for modern apple orchards due to their high fruit yield and quality, and their suitability to robotic operations in pruning and harvesting. In a common practice of training young apple trees to a trellis-trained canopy system in PNW region of US, trees branches are manually selected and then tied to horizontal trellis wires in 6 or 7 tiers. As manual training of apple trees to these modern orchard architectures is becoming challenging due to less availability of skilled human labor with quickly increased labor cost, automated training using sensing and robotic techniques could be an alternative solution. Segmenting trunks, branches, and trellis wires is a critical step in automating a tree training operation. In this study, a deep learning-based semantic segmentation method was developed for automatically performing this segmentation task. A Kinect V2 sensor was used to obtain the RGB and point cloud data of target trees. Then both Simple- and Foreground-RGB images were used for training a convolutional neural network (CNN)-based segmentation network (SegNet) to segment the trunk, branch, and trellis wire. Trunks and branches, which share some common features, were segmented from each other with accuracies of 0.82 and 0.89 for Simple-RGB images and 0.91 and 0.92 for Foreground-RGB images, respectively. Similarly, trellis wires, which have distinct features from both the trunk and branches, were segmented with accuracies of 0.92 and 0.97 for the Simple- and Foreground-RGB images, respectively. Obtained results showed that the performance of the developed semantic segmentation technique was better with Foreground-RGB images compared to the same with Simple-RGB images. Accuracy in identifying the segmented region boundaries in Foreground-RGB images, represented by the Boundary-F1 score, was 0.93, 0.89, and 0.91 for the trunk, branch, and trellis wire, respectively. These results showed a promising potential for adopting deep learning-based semantic segmentation for automating apple tree training in orchard environment.

Design of Authoring Tool for Static and Dynamic Projection Mapping

This study introduces the design details of a tool to create interactive projection-mapping content in a convenient manner. For the proposed tool design, a homography-based camera–projector calibration method was applied with the use of red–green–blue-depth images from a Kinect V2 sensor that did not require accurate camera calibration prerequisites. In addition, the proposed tool simultaneously achieved static projection mapping that projected the image content onto a fixed object, and dynamic projection mapping that projected the image content onto a user’s body, by tracing the moving user. To verify the effectiveness of the proposed content-creation tool, users with no programming capabilities were employed to create contents that were projected onto various objects in fixed positions and a user’s body in various poses, thereby analyzing the tool’s completeness. Moreover, the projection accuracy was analyzed at different depth positions, and the projection-mapping accuracy was verified with the use of the proposed method.

IKarate: Improving Karate Kata

Abstract Karate is a martial art that can be practiced using hands and feet to deliver and block strikes. Karate moves must be done in a certain way, many moves are practiced incorrectly during training. In this paper, we present a fully functional system which Karate players, coaches, judges and clubs could use. The system helps in capturing Karate moves using Kinect v2 sensor and analyzing these moves using F-DTW. Real-time feedback and a report are displayed to the users using a simple GUI (Graphical User Interface) that the users can easily use, to learn how a mistake was made, and how to fix it/learn from it. F-DTW was used for proving the concept, and an average accuracy of 90% was achieved. This Paper is mainly concerned about the moves named: Age-Uke, Mae-Geri, Gedan-Barai and Soto-Uke.

A low-cost vision system based on the analysis of motor features for recognition and severity rating of Parkinson\u2019s Disease

BackgroundAssessment and rating of Parkinson’s Disease (PD) are commonly based on the medical observation of several clinical manifestations, including the analysis of motor activities. In particular, medical specialists refer to the MDS-UPDRS (Movement Disorder Society – sponsored revision of Unified Parkinson’s Disease Rating Scale) that is the most widely used clinical scale for PD rating. However, clinical scales rely on the observation of some subtle motor phenomena that are either difficult to capture with human eyes or could be misclassified. This limitation motivated several researchers to develop intelligent systems based on machine learning algorithms able to automatically recognize the PD. Nevertheless, most of the previous studies investigated the classification between healthy subjects and PD patients without considering the automatic rating of different levels of severity.MethodsIn this context, we implemented a simple and low-cost clinical tool that can extract postural and kinematic features with the Microsoft Kinect v2 sensor in order to classify and rate PD. Thirty participants were enrolled for the purpose of the present study: sixteen PD patients rated according to MDS-UPDRS and fourteen healthy paired subjects. In order to investigate the motor abilities of the upper and lower body, we acquired and analyzed three main motor tasks: (1) gait, (2) finger tapping, and (3) foot tapping. After preliminary feature selection, different classifiers based on Support Vector Machine (SVM) and Artificial Neural Networks (ANN) were trained and evaluated for the best solution.ResultsConcerning the gait analysis, results showed that the ANN classifier performed the best by reaching 89.4% of accuracy with only nine features in diagnosis PD and 95.0% of accuracy with only six features in rating PD severity. Regarding the finger and foot tapping analysis, results showed that an SVM using the extracted features was able to classify healthy subjects versus PD patients with great performances by reaching 87.1% of accuracy. The results of the classification between mild and moderate PD patients indicated that the foot tapping features were the most representative ones to discriminate (81.0% of accuracy).ConclusionsThe results of this study have shown how a low-cost vision-based system can automatically detect subtle phenomena featuring the PD. Our findings suggest that the proposed tool can support medical specialists in the assessment and rating of PD patients in a real clinical scenario.

Comparison across vehicles of passenger head kinematics in abrupt vehicle maneuvers

Objective: Studies of vehicle occupant motions in response to abrupt vehicle maneuvers have demonstrated movements that may result in changes in the level of protection for the occupant if a crash subsequently occurs. The previous studies have typically used a single vehicle. The current study assesses whether the patterns of occupant head movement are different across passenger vehicle types.Method: Data collection was conducted on a closed test track with the same driver for all trials. A passenger sedan, a minivan, and a pickup truck were equipped with inertial measurement units to quantify vehicle dynamics. Head location was tracked using Microsoft Kinect v2 sensor and a novel methodology that fits 3 D head scan data to the depth data acquired in the vehicle. Twelve men and women with a wide range of body size and age were recruited. The primary purpose of the study was obfuscated by telling the participants that the focus was on vehicle ride motion. Participants sat in the right front seat and wore the vehicle belt. The first event during the test track route was a hard brake (approximately 1 g) to a stop from 35 mph (56 kph). Within the space of approximately 5 min the participants also experienced two aggressive, right-going lane changes, a sharp right turn with simultaneous hard braking, and a second hard braking event. The vehicles were presented in random order for each participant. This paper presents comparison across vehicles of head motions in the braking and lane-change maneuvers.Results: Accelerations were similar across the vehicles for both braking and lane-change events. The means (standard deviations) of forward head-CG excursion in the first braking event were 162 (54), 112 (39), and 176 (46) mm for the minivan, passenger car, and truck, respectively. The forward head excursion in the passenger car was found to be significantly smaller than in the other two vehicles using a paired t-test (p < 0.01). Across vehicles, the mean excursion in the second braking exposure was smaller than in the first (p < 0.01). In the first lane change event, the mean (SD) inboard head excursions were 126 (51), 110 (49), and 140 (68) mm; the values were not significantly different across vehicles or in the second lane-change event. A detailed investigation did not reveal an explanation for the smaller head excursions in the passenger car.Discussion: This is the first quantitative occupant kinematics study to compare responses across vehicles. Although a significant difference was found between vehicles, the overall responses are similar to those observed in a previous study.Conclusions: The results confirm previous studies showing large variance in excursions across occupants. Further study is needed to understand the factors that affect responses across vehicles.