Microsoft Kinect Sensor Research Articles

Design and Implementation of Home Training System based on Personal Fitness Using Kinect Sensor

In this paper, we implemented and developed a home training program using sensors. The implemented program grasps the user's athletic ability and suggests personalized exercise program. In modern society, the aging population is growing rapidly, and most of them are working and living in the interior. For this reason, health management of the aging population is becoming an important social issue, and it is recognized that muscle strength is an important factor. In this paper, we designed and developed a home training program for aged people to exercise in a comfortable environment like natural environment. In the developed program, we extract the skeletal feature points of the user using Microsoft's Kinect sensor and generate feature vectors to check the user's motion state. The proposed algorithm measures the user 's athletic ability and suggests a suitable exercise based on the information. As a result of applying kinematic sensor measurements to the proposed algorithm, it showed up to 97% accuracy. We measured the user's athletic ability through the program using the proposed algorithm and suggested the exercise suitable for the individual ability based on this information.

A multicenter randomized controlled trial comparing gamification with remote monitoring against standard rehabilitation for patients after arthroscopic shoulder surgery.

Gamification has become increasingly popular in rehabilitation and is viewed as a tool to improve patient activation, motivation, and engagement. The aim of this study was to compare the efficacy of validated exergames played through a system using "depth sensor" and bespoke software against standard physiotherapy in patients treated with arthroscopic shoulder surgery. This included the following common conditions: subacromial impingement syndrome, calcific tendinopathy, and rotator cuff tear. Following arthroscopic shoulder surgery, patients were randomized into 1 of 2 groups: In the standard rehabilitation group, patients were followed up for 12 weeks after surgery with standard postoperative physiotherapy and underwent electronic measurements of their active range of movement (ROM). In the exergame group, patients followed a postoperative regimen of exergames using the principles of gamification with physiotherapy support. Patients were given an exergame schedule prescribed by their therapist on Medical Interactive Recovery Assistant (MIRA) software (MIRA Rehab, London, UK) paired with a Microsoft Kinect sensor (Microsoft, Redmond, WA, USA). The primary outcome was active ROM objectively measured by MIRA and Kinect. Secondary outcome measures included the Oxford Shoulder Score, the Disabilities of the Arm, Shoulder and Hand score, and the EQ-VAS score at 12 weeks after surgery. A total of 71 patients were recruited to the study. We excluded 7 patients based on intraoperative findings. Thirty-three patients were treated with exergames, and 31 patients underwent conventional physiotherapy. There was no significant difference between the 2 groups in baseline ROM. Postoperatively, there was no significant difference in any of the cardinal planes of movement (forward flexion, P = .64; abduction, P = .33; and external rotation, P = .75). The mean Oxford Shoulder Score improved from 29.25 to 38.2 in the control group (P = .001) and from 27.1 to 35.1 in the trial group (P = .01); there was no significant difference between the groups at 12 weeks (P = .246). The mean Disabilities of the Arm, Shoulder and Hand score improved from 38.13 to 16.98 in the control group (P = .001) and from 42.3 to 22.54 in the trial group (P = .007); there was no significant difference between the 2 groups (P = .328). There was no significant difference in the EQ-VAS score in either group at any time point (P = .5866). This randomized controlled trial demonstrates that exergames can be used effectively in the rehabilitation of patients following arthroscopic shoulder surgery. Outcomes, judged by ROM and patient-reported outcome measures, are equivalent to conventional physiotherapy rehabilitation protocols. This health care innovation has the potential to relieve some of the heavy burden placed on physiotherapy departments for "routine" postoperative care in shoulder surgery.

Assessing the Feasibility of Using Kinect 3D Images to Predict Light Lamb Carcasses Composition from Leg Volume.

Simple SummaryThe present study aimed to evaluate the accuracy of the leg volume obtained by the Microsoft Kinect sensor to predict the carcass composition of twenty-two male light lambs. The carcasses were divided into eight cuts, which were grouped according to their commercial value into high-value, medium value, and low-value. Several linear, area, and volume leg measurements were performed to predict cut and carcass composition. The leg volume determined by 3D image reconstruction using Microsoft Kinect sensor and Archimedes principle shows the higher correlations values with cuts and carcass. Additionally, it was observed that the models, which include the leg volume obtained by the Kinect sensor, are very good in predicting the weight of the medium value and leg cuts (R2 of 0.763 and 0.829, respectively). Thus, the results of this study confirm the good ability to estimate cuts and body traits from light lambs with volume measurements, particularly those obtained with the Kinect 3D sensor.This study aimed to evaluate the accuracy of the leg volume obtained by the Microsoft Kinect sensor to predict the composition of light lamb carcasses. The trial was performed on carcasses of twenty-two male lambs (17.6 ± 1.8 kg, body weight). The carcasses were split into eight cuts, divided into three groups according to their commercial value: high-value, medium value, and low-value group. Linear, area, and volume of leg measurements were obtained to predict carcass and cuts composition. The leg volume was acquired by two different methodologies: 3D image reconstruction using a Microsoft Kinect sensor and Archimedes principle. The correlation between these two leg measurements was significant (r = 0.815, p < 0.01). The models to predict cuts and carcass traits that include leg Kinect 3D sensor volume are very good in predicting the weight of the medium value and leg cuts (R2 of 0.763 and 0.829, respectively). Furthermore, the model, which includes the Kinect leg volume, explained 85% of its variation for the carcass muscle. The results of this study confirm the good ability to estimate cuts and carcass traits of light lamb carcasses with leg volume obtained with the Kinect 3D sensor.

The Relationship between Landing Error Scoring System Performance and Injury in Female Collegiate Athletes.

BackgroundThe Landing Error Scoring System (LESS) is a standardized tool used to identify aberrant biomechanical movement patterns during a jump-landing task. Prior authors have examined the value of the LESS in identifying ACL injury risk in athletic populations. Yet, no study has evaluated the association between LESS performance and incidence of any type of lower extremity injury in female collegiate athletes across multiple sports.PurposeThe purpose of this study was to examine the association between LESS performance as measured with a markerless motion-capture system and lower extremity injury in female collegiate athletes.Study DesignProspective cohort study.MethodsOne hundred and ten DI female collegiate athletes (basketball, n=12; field hockey, n=17; gymnastics, n=14; lacrosse, n=27; softball, n=23; volleyball, n=17) completed a jump-landing test prior to the start of their sport seasons. The LESS was automatically scored using a Microsoft Kinect sensor and Athletic Movement Assessment software (PhysiMax®). Participants were tracked throughout one competitive season for incidence of time-loss lower extremity injury. A Receiver Operating Characteristic curve determined the optimal cutpoint for the total LESS score for predicting injury. Pearson’s Chi squared statistics examined the association between injury and LESS total scores >5. The Fisher exact test evaluated group differences for the frequency of receiving an error on individual LESS test items.ResultsFemale collegiate athletes with LESS scores >5 were not more likely to be injured than those with scores ≤5 (χ2=2.53, p=0.111). The relative risk of injury to this group was 1.78 (95% CI=0.86, 3.68) while the odds ratio was 2.10 (95% CI=0.83, 5.27). The uninjured group was more likely to receive an error on lateral trunk flexion at initial contact than the injured group (p=0.023).ConclusionThe LESS total score was not associated with an increased odds of lower extremity injury in this cohort of female collegiate athletes. Future studies to examine the association between individual LESS item scores and injury are warranted.Level of Evidence1b.

Towards rehabilitation at home after total knee replacement

In this paper, we present the design and implementation of an avatar-based interactive system that facilitates rehabilitation for people who have received total knee replacement surgeries. The system empowers patients to carry out exercises prescribed by a clinician at the home settings more effectively. Our system helps improve accountability for both patients and clinicians. The primary sensing modality is the Microsoft Kinect sensor, which is a depth camera that comes with a Software Development Kit (SDK). The SDK provides access to 3-dimensional skeleton joint positions to software developers, which significantly reduces the challenges in developing accurate motion tracking systems, especially for use at home. However, the Kinect sensor is not well-equipped to track foot orientation and its subtle movements. To overcome this issue, we augment the system with a commercial off-the-shelf Inertial Measurement Unit (IMU). The two sensing modalities are integrated where the Kinect serves as the primary sensing modality and the IMU is used for exercises where Kinect fails to produce accurate measurement. In this pilot study, we experiment with four rehabilitation exercises, namely, quad set, side-lying hip abduction, straight raise leg, and ankle pump. The Kinect is used to assess the first three exercises, and the IMU is used to assess the ankle pump exercise.

A Review on the Use of Microsoft Kinect for Gait Abnormality and Postural Disorder Assessment.

Gait and posture studies have gained much prominence among researchers and have attracted the interest of clinicians. The ability to detect gait abnormality and posture disorder plays a crucial role in the diagnosis and treatment of some diseases. Microsoft Kinect is presented as a noninvasive sensor essential for medical diagnostic and therapeutic purposes. There are currently no relevant studies that attempt to summarise the existing literature on gait and posture abnormalities using Kinect technology. The purpose of this study is to critically evaluate the existing research on gait and posture abnormalities using the Kinect sensor as the main diagnostic tool. Our studies search identified 458 for gait abnormality, 283 for posture disorder of which 26 studies were included for gait abnormality, and 13 for posture. The results indicate that Kinect sensor is a useful tool for the assessment of kinematic features. In conclusion, Microsoft Kinect sensor is presented as a useful tool for gait abnormality, postural disorder analysis, and physiotherapy. It can also help track the progress of patients who are undergoing rehabilitation.

Fusion of Multi-Sensor-Based Biomechanical Gait Analysis Using Vision and Wearable Sensor

The purpose of this research work is to investigate the biomechanics of pelvis, hip, knee, and ankle joint motion using a Kinect sensor & inertial measurement unit (IMU) sensor during the normal walk. In this paper, a very cost-effective gait analysis system based on Microsoft Kinect v2 and Inertial Measurement Unit (IMU) device is presented. Kinect sensor is used for acquiring 3D skeleton data (camera (x, y, z), depth (x, y) orientation (x, y, z, w), color (x, y)) with 25 human body joints. For this analysis, the lower extremities joints i.e. spinal cord joint, hip, knee, and ankle joints of both left and right legs are being considered. The main contribution of this research work is the joint angle calculation of lower extremities of human gait based on Microsoft Kinect sensor V2 and IMU sensor. From the law of cosine, the joint angle is calculated between the two joints and plotted for a single subject. We came with the observation that the characteristics of the human knee joint and ankle joint are inversely related to each other. There are two sharp humps for knee and ankle joints during the normal walk. During the swing phase, the knee joint is highly activated while during toe-off and heel strike it is least activated. This analysis of clinical data is very useful for prosthesis limb and exoskeleton design. The stability of calculated joints trajectories is validated using the limit cycle curve. A system is designed for real-time analysis of biomechanics of different lower limbs joints using gait.

Evaluating Lower Limb Kinematics Using Microsoft's Kinect: A Simple, Novel Method.

Purpose: Our aim was to evaluate the Microsoft Kinect sensor (MKS) as a markerless system for motion capture and analysis of lower limb motion, compare it with a state-of-the-art marker-based system (MBS), and investigate its accuracy in simultaneously capturing several lower limb joint movements on several planes while participants walked freely. Method: Participants were asked to walk while gait data were simultaneously recorded by both the MKS and the MBS. Software for analysing the Kinect data stream was developed using Microsoft Visual Studio and Kinect for Windows software development kits. Visual three-dimensional (3D) C-Motion software was used to calculate 3D joint angles of the MBS. Deviation of the joint angles calculated by the two systems was calculated using root-mean-square error (RMSE) on the basis of a designated formula. Results: The calculated RMSE average was <5° between the two systems, a level of difference that has practically no clinical significance. Conclusions: Quantitative measurements of the joint angles of the knee and hip can be acquired using one MKS with some accuracy. The system can be advantageous for clinical use, at the pre- and post-treatment stages of rehabilitation, at significantly lower costs. Further evaluation of the MKS should be performed with larger study populations.

Computer Vision for Elderly Care Based on Deep Learning CNN and SVM

Computer vision has wide application in medical sciences such as health care and home automation. This study on computer vision for elderly care is based on a Microsoft Kinect sensor considers an inexpensive, three dimensional, non-contact technique, that is comfortable for patients while being highly reliable and suitable for long term monitoring. This paper proposes a hand gesture system for elderly health care based on deep learning convolutional neural network (CNN) that is used to extract features and to classify five gestures according to five categories using a support vector machine (SVM). The proposed system is beneficial for elderly patients who are voiceless or deaf-mute and unable to communicate with others. Each gesture indicates a specific request such as “Water”, “Meal”, “Toilet”, “Help” and “Medicine” and translates as a command sending to a Microcontroller circuit that sends the request to the caregiver’s mobile phone via the global system for mobile communication (GSM). The system was tested in an indoor environment and provides reliable outcomes and a useful interface for older people with disabilities in their limbs to communicate with their families and caregivers.

Motor Indicators for the Assessment of Frozen Shoulder Rehabilitation via a Virtual Reality Training System

Adhesive capsulitis (also known as frozen shoulder) is a common clinical shoulder disorder and can be effectively improved through physical rehabilitation. With advancements in technology, virtual reality (VR) has been increasingly employed in rehabilitation treatments. However, most relevant studies have merely employed traditional assessment tools to assess the therapeutic effects rather than the substantial amount of motor trajectory data or task performance collected by motor training systems. In this research, an innovative frozen shoulder rehabilitation system using a Microsoft Kinect sensor and VR was successfully developed and five task-oriented motor indices and task performance were proposed to assess motor performance. A clinical experiment involving twenty patients was conducted. Objective clinical assessment outcomes verified the effectiveness of the developed system for frozen shoulder rehabilitation. The improvements assessed according to motor indices and task performance were consistent with the objective clinical assessment results. Furthermore, correlation analysis showed that several items in the task performance and motor indices were significantly correlated to clinical assessment items. Moreover, numerous items in the task performance and motor indices capable of predicting the clinical assessment results were identified through stepwise regression analysis. The results of this research can facilitate the subsequent development of new assessment methods.

A Low-Cost Telerehabilitation Paradigm for Bimanual Training

The COVID-19 pandemic has transformed daily life, as individuals must reduce contacts among each other to prevent the spread of the disease. Consequently, patients&#x2019; access to outpatient rehabilitation care was curtailed and their prospect for recovery has been compromised. Telerehabilitation has the potential to provide these patients with equally efficacious therapy in their homes. Using commercial gaming devices with embedded motion sensors, data on movement can be collected toward objective assessment of motor performance, followed by training and documentation of progress. Herein, we present a low-cost telerehabilitation system dedicated to bimanual exercise, wherein the healthy arm drives movements of the affected arm. In the proposed setting, a patient manipulates a dowel embedded with a sensor in front of a Microsoft Kinect sensor. In order to provide an engaging environment for the exercise, the dowel is interfaced with a personal computer, to serve as a controller. The patient&#x2019;s gestures are translated into actions in a custom-made citizen-science project. Along with the system, we introduce an algorithm for classification of the bimanual movements, whose inner workings are detailed in terms of the procedures performed for dimensionality reduction, feature extraction, and movement classification. We demonstrate the feasibility of our system on eight healthy subjects, offering support to the validity of the algorithm. These preliminary findings set forth the development of precise motion analysis algorithms in affordable home-based rehabilitation.

Identifying Fall Risk Predictors by Monitoring Daily Activities at Home Using a Depth Sensor Coupled to Machine Learning Algorithms.

Because of population ageing, fall prevention represents a human, economic, and social issue. Currently, fall-risk is assessed infrequently, and usually only after the first fall occurrence. Home monitoring could improve fall prevention. Our aim was to monitor daily activities at home in order to identify the behavioral parameters that best discriminate high fall risk from low fall risk individuals. Microsoft Kinect sensors were placed in the room of 30 patients temporarily residing in a rehabilitation center. The sensors captured the patients’ movements while they were going about their daily activities. Different behavioral parameters, such as speed to sit down, gait speed or total sitting time were extracted and analyzed combining statistical and machine learning algorithms. Our algorithms classified the patients according to their estimated fall risk. The automatic fall risk assessment performed by the algorithms was then benchmarked against fall risk assessments performed by clinicians using the Tinetti test and the Timed Up and Go test. Step length, sit-stand transition and total sitting time were the most discriminant parameters to classify patients according to their fall risk. Coupling step length to the speed required to stand up or the total sitting time gave rise to an error-less classification of the patients, i.e., to the same classification as that of the clinicians. A monitoring system extracting step length and sit-stand transitions at home could complement the clinicians’ assessment toolkit and improve fall prevention.

A Review on Arabic Sign Language Translator Systems

Deaf and dumb peoples are suffering difficulties most of the time in communicating with society. They use sign language to communicate with each other and with normal people. But Normal people find it more difficult to understand the sign language and gestures made by deaf and dumb people. Therefore, many techniques have been employed to tackle this problem by converting the sign language to a text or a voice and vice versa. In recent years, research has progressed steadily in regard to the use of computers to recognize and translate the sign language. This paper reviews significant projects in the field beginning with important steps of sign language translation. These projects can be classified according to the use of an input device into image-based and device-based. Image-based is used the traditional methods of image processing and features extraction, by using a digital camera such as a webcam. Device-based uses different devices like (Microsoft Kinect sensor, electronic glove and leap motion controller). These devices are used to reduce the time of both image processing and extraction features. Then the accuracy rates of using device-based are ranged between in 90%-99% where the accuracy rates of using image-based are ranged between 85%-93%.

Novel data fusion strategy for human gait analysis using multiple kinect sensors

Human gait analysis using microsoft Kinect sensor is an intriguing area of research. Fusing multiple sensors’ data for clinical gait analysis using calibrated instruments helps in producing more accurate results. A novel technique for spatial calibration is proposed and implemented in this paper. This paper emphasizes on a novel technique of using five Kinect V2 sensors (four clients and one server) and data fusion methods to create unified skeletons for subject-wise gait analysis. These eradicate the problems of inaccurate skeleton poses caused due to occlusions or tracking failures from a single Kinect which otherwise remains a problem in most vision-based sensing systems. Two methods are compared for estimating states of discrete time linear systems. The first one is classical Kalman filtering, which gives accurate results when state disturbances are assumed to be Gaussian white noises and measurement noises and the statistical properties are procurable. Secondly, a Set-Membership filter is used which relies upon the principle of prediction and correction as well. A novel Set-Membership filtering approach is proposed where the measurement noise is modelled by multivariate Gaussian probability density function bounded by −1 to +1. Based on our observations of the linear frameworks joined with interval investigation, the two phases of the estimator are done in a productive way. Both the fusion techniques are tested on overground and treadmill data and the results are validated and compared with ground truth obtained using Qualisys motion capture systems. The proposed approach is also compared quantitatively with state-of-the-art methods.

Sex differences in lower extremity kinematics during overhead and single leg squat tests

ABSTRACT The Overhead Squat (OHS) and Single-Leg Squat (SLS) are two clinical tests used by practitioners to identify high-risk biomechanical movement patterns. Several published studies have reported sex differences in SLS performance; however, few have investigated variations during the OHS and none has measured kinematics with a marker-less motion capture system. Therefore, this study aimed to compare biomechanical movement patterns between male and female collegiate athletes during OHS and SLS testing. Seventy-five females and 58 male athletes completed OHS and SLS . A Microsoft Kinect sensor using Athletic Movement Assessment software (PhysiMax®) was used to measure kinematics. For the OHS, males displayed greater peak knee frontal plane projection angles (FPPA) (M:26°±10°, F:20°±8°; P < 0.05), peak hip flexion (M:-94°±14°, F:-87°±15°; P < 0.05), and peak trunk flexion angles (M:11°±11°, F:6°±9°; P < 0.05). For the SLS (dominant-limb), males displayed greater peak trunk flexion (M:32°±6°, F:27°±7°; P < 0.05). For the non-dominant limb, females displayed greater peak knee FPPA (F:-12°±9°, M:-8°±9°; P < 0.05) whereas males displayed greater peak trunk flexion angles (M:32°±5°, F:27°±7°; P < 0.05). These findings suggest the need for practitioners to develop sex-specific corrective exercise programmes in effort to improve lower extremity kinematics in athletes.

GPU-based parallel Shadow Features generation at neural system for improving gait human activity recognition

In this paper, we propose a new method for improving human activity recognition (HAR) datasets in order to increase their classification accuracy when trained with a certain classifier like a Neural Network. In this paper a novel training/testing process for building/testing a classification model for human activity recognition (HAR) is proposed. Traditionally, HAR is done by a classifier that learns what activities a person is doing by training with skeletal data obtained from a motion sensor such as Microsoft Kinect or accelerometer sensors. These skeletal data are the spatial coordinates (x, y, z) of different parts of the human body. In addition to the spatial features that describe current positions in the skeletal data, new features called Shadow Features are used to improve the supervised learning efficiency and accuracy of Neural Network classifiers. Shadow Features are inferred from the dynamics of body movements, thereby modelling the underlying momentum of the performed activities. They provide extra dimensions of information for characterizing activities in the classification process and thus significantly improving the accuracy. These Shadow Features are generated based on the existing features obtained from sensor datasets. In this paper we show that the accuracy of a neural network classifier can be significantly improved by the addition of Shadow Features and we also show that the generation of Shadow Features can be achieved with little time cost, on the fly, with the NVIDIA GPU technology and the CUDA programming model, this way we can improve the Neural Network accuracy at almost no time cost. GPUs are particularly suitable for generating Shadow Features, since they possess multiple cores which can be taken advantage of, in order to generate Shadow Features for multiple data columns in parallel, therefore reducing a lot of processing time, especially when dealing with huge datasets.

Exploring Movement Impairments in Patients With Parkinson's Disease Using the Microsoft Kinect Sensor: A Feasibility Study.

Background: Current assessments of motor symptoms in Parkinson's disease are often limited to clinical rating scales.Objectives: To develop a computer application using the Microsoft Kinect sensor to assess performance-related bradykinesia.Methods: The developed application (Motorgame) was tested in patients with Parkinson's disease and healthy controls. Participants were assessed with the Movement Disorder Society Unified Parkinson's disease Rating Scale (MDS-UPDRS) and standardized clinical side effect rating scales, i.e., UKU Side Effect Rating Scale and Simpson-Angus Scale. Additionally, tests of information processing (Symbol Coding Task) and motor speed (Token Motor Task), together with a questionnaire, were applied.Results: Thirty patients with Parkinson's disease and 33 healthy controls were assessed. In the patient group, there was a statistically significant (p < 0.05) association between prolonged time of motor performance in the Motorgame and upper body rigidity and bradykinesia (MDS-UPDRS) with the strongest effects in the right hand (p < 0.001). In the entire group, prolonged time of motor performance was significantly associated with higher Simson-Angus scale rigidity score and higher UKU hypokinesia scores (p < 0.05). A shortened time of motor performance was significantly associated with higher scores on information processing (p < 0.05). Time of motor performance was not significantly associated with Token Motor Task, duration of illness, or hours of daily physical activity. The Motorgame was well-accepted.Conclusions: In the present feasibility study the Motorgame was able to detect common motor symptoms in Parkinson's disease in a statistically significant and clinically meaningful way, making it applicable for further testing in larger samples.

WITHDRAWN: Machine learning based gait abnormality detection using Microsoft Kinect sensor

Abstract In this paper, the investigation of human gait for normal and abnormal persons is done. It involves investigating different body joints, especially the foot while walking, on heel off and heel strike stage. The heel off and heel strike comes under phases of the gait cycle of a person. In a gait cycle, several features were calculated for different persons. The designed system is continuously monitoring the person’s features and can derive comprehensive body joints information of the gait cycle. In this study, a data set is constructed. Twenty-four persons (male and female) participated, including those who have a normal and abnormal gait. The persons are instructed to walk along a straight-line path of 8 ft in length. Data is recorded using Kinect Sensor v1.0, and several features are calculated, and the analysis of features is done using MATLAB. This investigation includes creating a GUI tool for feature extraction from the Microsoft Kinect Sensor and classification of feature set for the normal and abnormal gait of a person using the basic feature extraction model. 83.33% of accuracy is achieved for identifying that the person is a normal or an abnormal during walking.

Particle Physics Simulator for Scientific Education using Augmented Reality

In this era of fourth industrial revolution, young learners need to be equipped with 21st century skills, such as critical thinking, creativity, communication, collaboration, innovation and problem solving. Augmented Reality (AR) based learning systems are an effective tool to embed these skills. This paper presents a detailed review of latest research on an AR-based learning systems. Furthermore, an AR-based learning system is proposed to demonstrate the particle physics experiments i.e. proton-proton collision and Higgs field. The proposed learning system algorithms are developed using particle system of unity 3D software. Then, Microsoft Kinect sensor is interfaced with unity 3D to create an immersive experience. Then, the qualitative analysis of the proposed system and latest AR-based learning systems is presented. Finally, the quantitative analysis of the proposed system is conducted. Overall, the results suggest that 85% of the participants recommended the proposed learning system.

The 3D-DTW Custom IP based FPGA Hardware Acceleration for Action Recognition

Abstract This article proposes an implementation of an action recognition system, which allows the user to perform operations in real time. The Microsoft Kinect (RGB-D) sensor plays a central role in this system, which provides the skeletal joint information of humans directly. Computationally efficient skeletal joint position features are considered for describing each action. The dynamic time warping algorithm (DTW) is a widely used algorithm in many applications such as similarity sequence search, classification, and speech recognition. It provides the highest accuracy compared to all other algorithms. However, the computational time of the DTW algorithm is a major drawback in real world applications. To speed up the basic DTW algorithm, a novel three-dimensional dynamic time warping (3D-DTW) classification algorithm is proposed in this work. The proposed 3D-DTW algorithm is implemented in both software and field programmable gate array (FPGA) hardware modeling techniques. The performance of the 3D-DTW algorithm is evaluated for 12 actions in which each action is described with the feature vector size of 576 over 32 frames. From our software modeling results, it has been shown that the proposed algorithm performs the action classification accurately. However, the computation time of the 3D-DTW algorithm increases linearly when we increase either the number of actions or the feature vector size of each action. For further speedup, an efficient custom 3D-DTW intellectual property (IP) core is developed using the Xilinx Vivado high-level synthesis (HLS) tool to accelerate the 3D-DTW algorithm in FPGA hardware. The CPU centric software modeling of the 3D-DTW algorithm is compared with its hardware accelerated custom IP core. It has been shown that the developed 3D-DTW Custom IP core computation time is 40 times faster than its software counterpart. As the hardware results are promising, a parallel hardware software co-design architecture is proposed for the Xilinx Zynq-7020 System on Chip (SoC) FPGA for action recognition. The HLS simulation and synthesis results are provided to support the practical implementation of the proposed architecture. Our proposed approach outperforms many of the existing state-of-the-art DTW based action recognition techniques by providing the highest accuracy of 97.77%.