MS Kinect Research Articles

A Three-Dimensional Real-Time Gait-Based Age Detection System Using Machine Learning

Human biometric analysis has gotten much attention due to its widespread use in different research areas, such as security, surveillance, health, human identification, and classification. Human gait is one of the key human traits that can identify and classify humans based on their age, gender, and ethnicity. Different approaches have been proposed for the estimation of human age based on gait so far. However, challenges are there, for which an efficient, low-cost technique or algorithm is needed. In this paper, we propose a three-dimensional real-time gait-based age detection system using a machine learning approach. The proposed system consists of training and testing phases. The proposed training phase consists of gait features extraction using the Microsoft Kinect (MS Kinect) controller, dataset generation based on joints’ position, pre-processing of gait features, feature selection by calculating the Standard error and Standard deviation of the arithmetic mean and best model selection using R2 and adjusted R2 techniques. T-test and ANOVA techniques show that nine joints (right shoulder, right elbow, right hand, left knee, right knee, right ankle, left ankle, left, and right foot) are statistically significant at a 5% level of significance for age estimation. The proposed testing phase correctly predicts the age of a walking person using the results obtained from the training phase. The proposed approach is evaluated on the data that is experimentally recorded from the user in a real-time scenario. Fifty (50) volunteers of different ages participated in the experimental study. Using the limited features, the proposed method estimates the age with 98.0% accuracy on experimental images acquired in real-time via a classical general linear regression model.

Development of a Robust, Simple, and Affordable Human Gait Analysis System Using Bottom-Up Pose Estimation With a Smartphone Camera.

Gait analysis is used in many fields such as Medical Diagnostics, Osteopathic medicine, Comparative and Sports-related biomechanics, etc. The most commonly used system for capturing gait is the advanced video camera-based passive marker system such as VICON. However, such systems are expensive, and reflective markers on subjects can be intrusive and time-consuming. Moreover, the setup of markers for certain rehabilitation patients, such as people with stroke or spinal cord injuries, could be difficult. Recently, some markerless systems were introduced to overcome the challenges of marker-based systems. However, current markerless systems have low accuracy and pose other challenges in gait analysis with people in long clothing, hiding the gait kinematics. The present work attempts to make an affordable, easy-to-use, accurate gait analysis system while addressing all the mentioned issues. The system in this study uses images from a video taken with a smartphone camera (800 × 600 pixels at an average rate of 30 frames per second). The system uses OpenPose, a 2D real-time multi-person keypoint detection technique. The system learns to associate body parts with individuals in the image using Convolutional Neural Networks (CNNs). This bottom-up system achieves high accuracy and real-time performance, regardless of the number of people in the image. The proposed system is called the “OpenPose based Markerless Gait Analysis System” (OMGait). Ankle, knee, and hip flexion/extension angle values were measured using OMGait in 16 healthy volunteers under different lighting and clothing conditions. The measured kinematic values were compared with a standard video camera based normative dataset and data from a markerless MS Kinect system. The mean absolute error value of the joint angles from the proposed system was less than 90 for different lighting conditions and less than 110 for different clothing conditions compared to the normative dataset. The proposed system is adequate in measuring the kinematic values of the ankle, knee, and hip. It also performs better than the markerless systems like MS Kinect that fail to measure the kinematics of ankle, knee, and hip joints under dark and bright light conditions and in subjects with long robe clothing.

A Prototype for Mexican Sign Language Recognition and Synthesis in Support of a Primary Care Physician

Few hearing people know and use Mexican Sign Language (MSL). Consequently, this is the main barrier between deaf and hearing people. This study proposes a system that recognizes and animates in real time a set of signs belonging to the semantic field of a general medicine consultation service. Therefore, a linkage between a hearing doctor and a deaf patient can be established in a non-intrusive way and with an easy dynamic interaction. Our main contribution is a bidirectional translator system for Mexican sign language in the context of a general medical consultation. The recognition module uses a MS Kinect sensor to obtain the sign trajectories, and images to feed hidden Markov Models (HMMs) for processing signs samples in real time. The experiments show recognition of 82 different signs from 22 participants. As a result, accuracy and F1 scores average rates of 99% and 88% were obtained.

‘Address and command’: Two-handed mid-air interactions with multiple home devices

Mid-air gestural interaction with multiple targets (or devices) in the physical space presents several challenges: gesture design gets more complex in the attempt to avoid gesture conflicts and accidental activations; the potential commands are many but the gesture set must be kept small to enable memorability; and, from a design-process perspective, there are not readily available technologies for implementation and prototyping. In this paper, we present the Address and Command (A&C) interaction model that enables two-handed mid-air interactions with multiple remote devices, in a research approach that includes gesture elicitation, prototyping, and evaluation of the user experience. The A&C model requires that users employ the non-dominant hand to address a device (address gestures) and the dominant hand to provide a command to it (command gestures). This approach affords interactions that simultaneously address and command multiple devices in ubiquitous environments that respond to mid-air input. A&C interactions also afford the design of a single gesture for the same command when applicable for multiple devices, thus decreasing the total number of gestures to be memorized and promoting end-user learning of the gesture set. We have (a) conducted an elicitation study (n = 18) to define A&C gestures for seven smart home devices and twelve commands; (b) developed a spatial augmented reality prototype that responds to mid-air gesture commands exploiting MS Kinect sensor and Visual Gesture Builder, Unity 3D and projection-mapping technologies onto foam mockups; (c) tested the approach and the gesture vocabulary in terms of usability, memorability and user experience at a scenario of 36 tasks among devices (n = 17). We found that A&C interactions are feasible, fast, error-free, easy to learn and remember, and are highly valued in terms of user experience.

Detection of Virus Using Artificial Intelligence and Machine Learning Concepts

This paper is devoted, to study of recent viruses, detection and prediction of viruses using AI and Machine Learning Techniques. Recently in December 2019 a new virus was discovered named coronavirus. So, to detect this type of viruses in the future and present, we can use AI and Machine Learning concepts for the detection of viruses. Artificial intelligence (AI) research is growing rapidly within the medicine research. In 2016, Artificial Intelligence projects on health care attracted more investment than Artificial Intelligence projects within any other sector of the global economy. It is well, known that in feature we may overcome the different types of new viruses across the city. The detection of the viruses is done with the help of the MS Kinect sensor. This sensor analyses human breathing, and if that person is not able to breathe properly then that person must be suffering from coughing, sneezing, etc. This particular sensor will be fixed at the public places if it identifies the person who is suffering from breathing problems, the person will be sanitized or will be provided with the mask or if it is a serious condition that person will be admitted to the hospital for the treatment. So that the spreading of virus decreases at public places. This sensor can also be used for checking the heartbeat pluses of the humans and animals also. Animals should also be monitoring with their health conditions because there are chances of spreading viruses from animals also.

Toward Kinecting cognition by behaviour recognition-based deep learning and big data

The majority of older people wish to live independently at home as long as possible despite having a range of age-related conditions including cognitive impairment. To facilitate this, there has been an extensive focus on exploring the capability of new technologies with limited success. This paper investigates whether MS Kinect (a motion-based sensing 3-D scanner device) within the MiiHome (My Intelligent Home) project in conjunction with other sensory data, machine learning and big data techniques can assist in the diagnosis and prognosis of cognitive impairment and hence prolong independent living. A pool of Kinect devices and various sensors powered by minicomputers providing internet connectivity are being installed in up to 200 homes. This enables continuous remote monitoring of elderly residents living alone. Passive and off-the-shelf sensor technologies were chosen to implement data acquisition specifically from sources that are part of the fabric of the homes, so that no extra effort is required from the participants. Various constraints including environmental, geometrical and big data were identified and appropriately dealt with. A visualization tool (MAGID) was developed for validation and verification of numerous behavioural activities. Then, a subset of data, from twelve pensioners aged over 65 with age-related cognitive decline and frailty, were collected over a period of 6 months. These data were subjected to several machine learning algorithms (multilayer perceptron neural network, neuro-fuzzy and deep learning) for classification and to extract routine behavioural patterns. These patterns were then analysed further to ascertain any health-related information and their attributes. For the first time, important routine behaviour related to Activities of Daily Living (ADL) of elderly people with cognitive and physical decline has been learnt by machine learning techniques from selected sample data obtained by MS Kinect. Medically important behaviour, e.g. eating, walking, sitting, was best learnt by deep learning with accuracy of 99.30% during training stage and average error rate of 1.83% with maximum of 12.98% during the implementation phase. Observations obtained from the application of the above learnt behaviours are presented as trends over a period of time. These trends, supplemented by other sensory signals, have provided a clearer picture of physical (in)activities (including falls) of the pensioners. The calculated behavioural attributes related to key indicators of health events can be used to model the trajectory of health status related to cognitive decline in a home setting. These results, based on a small number of elderly residents over a short period of time, imply that within the results obtained from the MiiHome project, it is possible to find indicators of cognitive decline. However, further studies are needed for full clinical validation of these indications in conjunction with assessment of cognitive decline of the participants.

Mid-Air Gesture Control of Multiple Home Devices in Spatial Augmented Reality Prototype

Touchless, mid-air gesture-based interactions with remote devices have been investigated as alternative or complementary to interactions based on remote controls and smartphones. Related studies focus on user elicitation of a gesture vocabulary for one or a few home devices and explore recommendations of respective gesture vocabularies without validating them by empirical testing with interactive prototypes. We have developed an interactive prototype based on spatial Augmented Reality (AR) of seven home devices. Each device responds to touchless gestures (identified from a previous elicitation study) via the MS Kinect sensor. Nineteen users participated in a two-phase test (with and without help provided by a virtual assistant) according to a scenario that required from each user to apply 41 gestural commands (19 unique). We report on main usability indicators: task success, task time, errors (false negative/positives), memorability, perceived usability, and user experience. The main conclusion is that mid-air interaction with multiple home devices is feasible, fairly easy to learn and apply, and enjoyable. The contributions of this paper are (a) validation of a previously elicited gesture set; (b) development of a spatial AR prototype for testing of mid-air gestures, and (c) extensive assessment of gestures and evidence in favor of mid-air interaction in smart environments.

Sleep Apnea Detection with Polysomnography and Depth Sensors.

This paper is devoted to proving two goals, to show that various depth sensors can be used to record breathing rate with the same accuracy as contact sensors used in polysomnography (PSG), in addition to proving that breathing signals from depth sensors have the same sensitivity to breathing changes as in PSG records. The breathing signal from depth sensors can be used for classification of sleep apnea events with the same success rate as with PSG data. The recent development of computational technologies has led to a big leap in the usability of range imaging sensors. New depth sensors are smaller, have a higher sampling rate, with better resolution, and have bigger precision. They are widely used for computer vision in robotics, but they can be used as non-contact and non-invasive systems for monitoring breathing and its features. The breathing rate can be easily represented as the frequency of a recorded signal. All tested depth sensors (MS Kinect v2, RealSense SR300, R200, D415 and D435) are capable of recording depth data with enough precision in depth sensing and sampling frequency in time (20–35 frames per second (FPS)) to capture breathing rate. The spectral analysis shows a breathing rate between 0.2 Hz and 0.33 Hz, which corresponds to the breathing rate of an adult person during sleep. To test the quality of breathing signal processed by the proposed workflow, a neural network classifier (simple competitive NN) was trained on a set of 57 whole night polysomnographic records with a classification of sleep apneas by a sleep specialist. The resulting classifier can mark all apnea events with 100% accuracy when compared to the classification of a sleep specialist, which is useful to estimate the number of events per hour. When compared to the classification of polysomnographic breathing signal segments by a sleep specialist, which is used for calculating length of the event, the classifier has an score of 92.2% Accuracy of 96.8% (sensitivity 89.1% and specificity 98.8%). The classifier also proves successful when tested on breathing signals from MS Kinect v2 and RealSense R200 with simulated sleep apnea events. The whole process can be fully automatic after implementation of automatic chest area segmentation of depth data.

Random forest-based classsification and analysis of hemiplegia gait using low-cost depth cameras.

Hemiplegia is a form of paralysis that typically has the symptom of dysbasia. In current clinical rehabilitations, to measure the level of hemiplegia gaits, clinicians often conduct subject evaluations through observations, which is unreliable and inaccurate. The Microsoft Kinect sensor (MS Kinect) is a widely used, low-cost depth sensor that can be used to detect human behaviors in real time. The purpose of this study is to investigate the usage of the Kinect data for the classification and analysis of hemiplegia gait. We first acquire the gait data by using a MS Kinect and extract a set of gait features including the stride length, gait speed, left/right moving distances, and up/down moving distances. With the gait data of 60 subjects including 20 hemiplegia patients and 40 healthy subjects, we employ a random forest-based classification approach to analyze the importances of different gait features for hemiplegia classification. Thanks to the over-fitting avoidance nature of the random forest approach, we do not need to have a careful control over the percentage of patients in the training data. In our experiments, our approach obtained the averaged classification accuracy of 90.65% among all the combinations of the gait features, which substantially outperformed state-of-the-art methods. The best classification accuracy of our approach is 95.45%, which is superior than all existing methods. Additionally, our approach also correctly reveals the importance of different gait features for hemiplegia classification. Our random forest-based approach outperforms support vector machine-based method and the Bayesian-based method, and can effectively extract gait features of subjects with hemiplegia for the classification and analysis of hemiplegia. Graphical Abstract Random Forest based Classsification and Analysis of Hemiplegia Gait using Low-cost Depth Cameras. Left: Motion capture with MS Kinect; Top-right: Random Forest Classsification based on the extracted gait features; Bottom-right: Sensitivity and specificity evaluation of the proposed classification approach.

Unsupervised GEI-Based Gait Disorders Detection From Different Views.

Early detection of gait disorders may provide a safer living for elderly people. In this paper, we propose an automatic method for detecting gait disorders using RGB or RGBD camera (e.g., MS Kinect, Asus Xtion PRO). We use Gait Energy Image (GEI) as our main feature that can be computed from different views. Our method depends on computing GEI, learning the representative features from the GEI using convolutional autoencoder, and using anomaly detection method for detecting abnormal gait. We applied the proposed method on two different public datasets that include normal and abnormal gait from different views. Experimental results show that our method achieves high accuracy in detecting different gait disorders from different views, which makes it general to be applied to home environment and adds a step towards convenient in-home automatic health care services.

Recognition of static hand gesture with using ANN and SVM

Hand gesture recognition is a relevant study topic for a reason that sometimes we may not be in position to communicate verbally. There is need to design Hand gesture recognition systems in order to help people adopt to nonverbal communication mainly sign language. However, there is no clue to understand the meaning of gesture through the computers directly. So this calls for definitions that generalize models in a computer. That is why the machine-learning approaches are implemented in recognition systems. There are generally two types of hand gestures recognition systems which researches have concentrated on. These include static and dynamic Hand gesture recognition systems. However, in building Hand gesture recognition systems, various machine learning approaches have been used. For implementing the proposed system, MS Kinect depth sensor was used as a hardware. The Kinect depth sensor is composed of an infrared camera. This is an advantage to the systems that are designed basing on the depth sensing because factors like color, clothing and background have less effect on the performance. So Kinect based depth sensor systems have a high accuracy and performance making them relevant and applicable in our daily lives. In this paper, we propose a static hand gesture recognition system in real time using two machine learning methods namely Support Vector Machine and Artificial Neural Networks. We use of the newly launched Microsoft Kinect sensor for image extraction. The sensor helps us to extract the hand images. We implement the system on a Matlab platform for reasons that Matlab is widely used by researchers in different fields and that can handle complex computations. In the training of the model, we collect a hundred depth-based Histogram of Oriented Gradient features per alphabet from the hand gesture images which we trained, tested and validated using Artificial Neural Networks (ANN) and Support Vector Machine (SVM). From this dataset, we can generate the generalized gesture model for each alphabet image. For the proposed system, the classification with ANN proves a higher performance then SVM.

Efficient and Robust Skeleton-Based Quality Assessment and Abnormality Detection in Human Action Performance.

Elderly people can be provided with safer and more independent living by the early detection of abnormalities in their performing actions and the frequent assessment of the quality of their motion. Low-cost depth sensing is one of the emerging technologies that can be used for unobtrusive and inexpensive motion abnormality detection and quality assessment. In this study, we develop and evaluate vision-based methods to detect and assess neuromusculoskeletal disorders manifested in common daily activities using three-dimensional skeletal data provided by the SDK of a depth camera (e.g., MS Kinect and Asus Xtion PRO). The proposed methods are based on extracting medically -justified features to compose a simple descriptor. Thereafter, a probabilistic normalcy model is trained on normal motion patterns. For abnormality detection, a test sequence is classified as either normal or abnormal based on its likelihood, which is calculated from the trained normalcy model. For motion quality assessment, a linear regression model is built using the proposed descriptor in order to quantitatively assess the motion quality. The proposed methods were evaluated on four common daily actions-sit to stand, stand to sit, flat walk, and gait on stairs-from two datasets, a publicly released dataset and our dataset that was collected in a clinic from 32 patients suffering from different neuromusculoskeletal disorders and 11 healthy individuals. Experimental results demonstrate promising results, which is a step toward having convenient in-home automatic health care services.

INTEGRATION OF THE PROCESS OF COMPUTER GAME DEVELOPMENTWITH AUGMENTED REALITY IN STREAM-EDUCATION COMPONENTS

This article describes the links between STREAM-education components (Science, Technology, Reading/Writing, En-gineering, Arts, Mathematics) and the stages of computer game development (StoryTelling/Scripts, GameDesign,2D/3D-Computer graphics, Programming, Testing, and GameLevel) on the example of using sensors of human movement control. In order to effec-tively integrate the computer game development process with the augmented reality into the component of STREAM-education, the method for creating a real-complemented virtual training exercises using an example of training with a ball on the basis of the MS Kinect motion sensor is proposed. As a result of the analysis of the virtual environment characteristics created bythe sensor of the MS Kinect controller and taking into account the limits of the Scratch program, it is proposed to formalize the characteristics of the virtual environment. The results of the analysis of games offered by LumoPlay and MotionMagix softwaremanufacturers in logic, sports, entertaining classes in various game genres and relaxation with special effects are given. Examples of experiments are shown during the organization of two Meet and Code 2018master-classes for schoolchildren of virtual programming, complemented by the reality of football training, which has confirmed the interest of students in participating in many STREAM-related learning process-es. The results of the work also identified the directions of further research and development;in the research process the estimation of the visualization error of virtual objects determined the need for the development of a special algorithm for automatic correction of the coordinates of the location of virtual sprite objects that are part of a single group of objects player's body; Many examples of commercial game systems are based on new templates that allow you to quickly change the graphic part of the game, so it is planned to create similar scratch templates for developing games.

Analysing Balance Loss in VR Interaction with HMDs

Immersive virtual reality (VR) has been used in different fields such as training, educational programs, entertainment, psychological treatments, and rehabilitation. Despite its broad utilization, some issues limit its application such as the loss of balance. Balance is disturbed because visual stimuli received from the virtual scenario are not in harmony with perceived stimuli by the proprioception and vestibular systems that remain in contact with the real environment. With the increasing popularity and accessibility of high-quality VR systems, concerns have been raised about the propensity of VR to induce balance loss. Balance is essential for safe use of VR experience and its loss can result in severe injury. In this work, we present a methodology and the necessary tools to quantify the influence of VR on the user’s balance and assess risk of falls during VR interaction. By means of an experiment making use of an Oculus Rift and a MS Kinect Sensor, we observe, quantify and compare the effect of VR scenes with different levels of danger on the balance of users, as well as the effect of visual and auditory warnings of balance loss. Results suggest that auditory signs were not effective in warning users about risk of fall, and that the order which the scenes are presented to users affects their behavior. Users who were first presented to a more challenging scene proceeded more carefully and most of the time carried this behavior to the other less challenging scenes.

Robot-Based Image Analysis for Evaluating Rehabilitation after Brain Surgery

After certain types of brain surgery, patients are often affected by changes in both their dynamic balance and facial disorder. Because rehabilitation takes several months, it is important that both doctors and patients are able to monitor progress quantitatively. At present, such quantification is subjective and highly dependent on the doctor’s opinion. Thus, we here investigate the use of robot-based image analysis for measuring rehabilitation. To evaluate a patient’s dynamic balance, we developed a mobile robotic platform that uses a stereovision camera (MS Kinect) to capture a video of the subject walking along a hospital corridor. To evaluate a patient’s facial disorders, the same camera is used in a static mode to detect and capture precise facial movements that the subject is asked to perform. From these videos, specific patterns can be extracted for rehabilitation process description.

Breathing Analysis Using Thermal and Depth Imaging Camera Video Records.

The paper is devoted to the study of facial region temperature changes using a simple thermal imaging camera and to the comparison of their time evolution with the pectoral area motion recorded by the MS Kinect depth sensor. The goal of this research is to propose the use of video records as alternative diagnostics of breathing disorders allowing their analysis in the home environment as well. The methods proposed include (i) specific image processing algorithms for detecting facial parts with periodic temperature changes; (ii) computational intelligence tools for analysing the associated videosequences; and (iii) digital filters and spectral estimation tools for processing the depth matrices. Machine learning applied to thermal imaging camera calibration allowed the recognition of its digital information with an accuracy close to 100% for the classification of individual temperature values. The proposed detection of breathing features was used for monitoring of physical activities by the home exercise bike. The results include a decrease of breathing temperature and its frequency after a load, with mean values −0.16 °C/min and −0.72 bpm respectively, for the given set of experiments. The proposed methods verify that thermal and depth cameras can be used as additional tools for multimodal detection of breathing patterns.

Extraction of breathing features using MS Kinect for sleep stage detection

This paper presents the contactless measuring of breathing using the MS Kinect depth sensor and compares the results obtained with records of breathing taken by polysomnography (PSG). We explore the methods of signal denoising, resampling, and spectral analysis of acquired data as well as feature extraction and their Bayesian classification. The proposed methodology was applied for analysis of the long-term monitoring of individuals who were observed simultaneously by PSG and MS Kinect in the sleep laboratory. After time synchronization of polysomnographic and MS Kinect video data, features were extracted from both signals and compared. The average error of the frequency while being evaluated by MS Kinect that was related to that obtained by PSG was 3.75 %. The mean accuracy of the Bayesian classification of features into two classes (i.e. wake or sleep) was 88.90 and 88.95 % for the PSG and MS Kinect measurements, respectively. The strong likeness of features supports the hypothesis that contactless techniques may represent a valid alternative to the present approach of sleep monitoring, thereby allowing data acquisition in the home environment as well.

Moving Real Exergaming Engines on the Web: The webFitForAll Case Study in an Active and Healthy Ageing Living Lab Environment.

Exergames have been the subject of research and technology innovations for a number of years. Different devices and technologies have been utilized to train the body and the mind of senior people or different patient groups. In the past, we presented FitForAll, the protocol efficacy of which was proven through widely taken (controlled) pilots with more than 116 seniors for a period of two months. The current piece of work expands this and presents the first truly web exergaming platform, which is solely based on HTML5 and JavaScript without any browser plugin requirements. The adopted architecture (controller application communication framework) combines a unified solution for input devices such as MS Kinect and Wii Balance Βoard which may seamlessly be exploited through standard physical exercise protocols (American College of Sports Medicine guidelines) and accommodate high detail logging; this allows for proper pilot testing and usability evaluations in ecologically valid Living Lab environments. The latter type of setups is also used herein for evaluating the web application with more than a dozen of real elderly users following quantitative approaches.

Tracking people in RGBD videos using deep learning and motion clues

Tracking people in videos is an important topic in surveillance. We consider the problem of human tracking in RGBD videos filmed by sensors such as MS Kinect and Primesense. Our goal is to track persons where the crowd of people is known in advance or all persons in the video have appeared in the very beginning. Thus we can train a classifier to help classify and track persons across the video. A deep learning model trained with big data has been proved to be an effective classifier for various kinds of objects. We propose to train a deep convolutional neural network, which improves tracking performance, to classify people. And a motion model based on spatial and kinetic clues is combined with the network to track people in the scene. We demonstrate the effectiveness of our method by evaluating it on several datasets and comparing with traditional methods like SVM.

Indian sign language recognition using SVM

Needs and new technologies always inspire people to make new ways to interact with machines. This interaction can be for a specific purpose or a framework which can be applied to many applications. Sign language recognition is a very important area where an easiness in interaction with human or machine will help a lot of people. At this time, India has 2.8M people who can't speak or can't hear properly. This paper targets Indian sign recognition area based on dynamic hand gesture recognition techniques in real-time scenario. The captured video was converted to HSV color space for pre-processing and then segmentation was done based on skin pixels. Also Depth information was used in parallel to get more accurate results. Hu-Moments and motion trajectory were extracted from the image frames and the classification of gestures was done by Support Vector Machine. The system was tested with webcam as well as with MS Kinect. This type of system would be helpful in teaching and communication of hearing impaired persons.