Kinect V2 Sensor Research Articles

Detecting airport luggage dimensions through low-cost depth sensors

A factor that impacts airlines' resources is the verification of luggage’s dimensions during the boarding process. Companies often rely on a human operator to perform this check using a manual template, which can cause delays. As an alternative, companies are investing in self bag drop systems. This process introduces new technological challenges since, in this scenario, checking the conformity of luggage dimensions is delegated to the passenger, which can lead to errors. In addition, current solutions use specific computational devices, such as laser scanners, that are expressive in size and cost, which may require interventions in the airport infrastructure. To overcome this, isolated initiatives are observed with alternative technologies, such as low-cost depth sensors, but they usually come without a scientific investigation. In this sense, this work investigates the technical viability of using such low-cost devices to obtain the dimensions of airport baggage. To do so, we developed a model that obtains a 3D point cloud of the luggage surface through a Microsoft Kinect V2 sensor. This cloud is treated and processed to extract the dimensions of the luggage. In order to validate this approach, a full-scale physical prototype was built and tested. The results indicate that the mean absolute error of the obtained dimension by the proposed model is 2.86 cm. Such data suggest that this technology has the potential to become an alternative to detect the dimensions of airport luggage.

Analyzing lower body movements using machine learning to classify autistic children

Autism spectrum disorder is a neurological disorder that affects children at an early age and its symptoms appear in varying degrees. Early detection of ASD and follow-up of treatment by specialists help improve autistic people's lives and reduce the symptoms they suffer from. In the past few years, a group of studies conducted gait analysis to diagnose children with autism, and although the results of these studies showed weak motor skills of Autism Spectrum Disorder children (ASD) compared to their peers of Typically Developing children (TD), there are some challenges facing researchers in this field. The first is the ability to create A safe, effective, friendly environment that reduces tension in ASD children and helps classify children into ASD/TD effectively. And the second is the ability to create an automated ASD diagnosis system. In this work, a system based on machine learning and augmented reality technology has been proposed to create an effective and friendly interactive environment that helps diagnose ASD and classify children to ASD / TD. The movement of the lower limbs of 15 ASD children and 15 TD children are tracked via a marker-less technique through a Kinect v2 sensor when playing a specific game based on kicking the ball task, that is using augmented reality technology. Then the kinematics features are extracted through Linear Discriminant Analysis technology. To determine these features' ability in the classification process, three machine learning algorithms are used, namely Support Vector Machine, Linear Discriminant Analysis, and Decision Tree. The results showed that the obtained kinematics features can classify children into ASD/TD, where the Support Vector Machine classifier obtained a high accuracy of 96.7% with sensitivity equal to 93.3% and specificity equal to 100%. In addition, the Linear Discriminant Analysis classifier obtained less accuracy of 93.3% with sensitivity equal to 93.3% and specificity equal to 93.3% compared with the Support Vector Machine classifier. Where the Decision Tree classifier obtained the lowest accuracy of 80.0% with sensitivity equal to 73.3% and specificity equal to 86.7% compared with the Support Vector Machine and Linear Discriminant Analysis classifiers. In addition to that, when using Linear Discriminant Analysis to reduce dimensions and extract features, the accuracy, sensitivity, and specificity of the Support Vector Machine classifier were increased and achieved by 100%. Based on these results, it has been proven that the proposed method is effective in classifying children to ASD/TD.

Hybrid deep learning model-based human action recognition in indoor environment

AbstractHuman activity recognition (HAR) is an emerging challenge among researchers. HAR has many possible uses in various fields, including healthcare, sports, and security. Furthermore, there are only a few publicly accessible datasets for classifying and recognizing physical activity in the literature, and these datasets comprise fewer activities. We created and compared our dataset with available datasets, that is, NTU-RGBD, UP-FALL, UR-Fall, WISDM, and UCI HAR. The proposed dataset consists of seven activities: eating, exercise, handshake, situps, vomiting, headache, and walking. The activities were collected from 20 people between the ages of 25 and 40 years using Kinect V2 sensor at 30 FPS. For classification, we use deep learning architectures based on convolutional neural network (CNN) and long short-term memory (LSTM). Additionally, we developed a novel hybrid deep learning model by combining a CNN, a bidirectional LSTM unit, and a fully connected layer for activity identification. The suggested model builds unique guided features using the preprocessed skeleton coordinates and their distinctive geometrical and kinematic aspects. Results from the experiment are contrasted with the performance of stand-alone CNNs, LSTMs, and ConvLSTM. The proposed model’s accuracy of 99.5% surpasses that of CNN, LSTM, and ConvLSTM, which have accuracy rates of 95.76%, 97%, and 98.89%, respectively. Our proposed technique is invariant of stance, speed, individual, clothes, etc. The proposed dataset sample is accessible to the general public.

System for Estimation of Human Anthropometric Parameters Based on Data from Kinect v2 Depth Camera.

Anthropometric measurements of the human body are an important problem that affects many aspects of human life. However, anthropometric measurement often requires the application of an appropriate measurement procedure and the use of specialized, sometimes expensive measurement tools. Sometimes the measurement procedure is complicated, time-consuming, and requires properly trained personnel. This study aimed to develop a system for estimating human anthropometric parameters based on a three-dimensional scan of the complete body made with an inexpensive depth camera in the form of the Kinect v2 sensor. The research included 129 men aged 18 to 28. The developed system consists of a rotating platform, a depth sensor (Kinect v2), and a PC computer that was used to record 3D data, and to estimate individual anthropometric parameters. Experimental studies have shown that the precision of the proposed system for a significant part of the parameters is satisfactory. The largest error was found in the waist circumference parameter. The results obtained confirm that this method can be used in anthropometric measurements.

A Novel Multi-Modal Teleoperation of a Humanoid Assistive Robot with Real-Time Motion Mimic

This research shows the development of a teleoperation system with an assistive robot (NAO) through a Kinect V2 sensor, a set of Meta Quest virtual reality glasses, and Nintendo Switch controllers (Joycons), with the use of the Robot Operating System (ROS) framework to implement the communication between devices. In this paper, two interchangeable operating models are proposed. An exclusive controller is used to control the robot's movement to perform assignments that require long-distance travel. Another teleoperation protocol uses the skeleton joints information readings by the Kinect sensor, the orientation of the Meta Quest, and the button press and thumbstick movements of the Joycons to control the arm joints and head of the assistive robot, and its movement in a limited area. They give image feedback to the operator in the VR glasses in a first-person perspective and retrieve the user's voice to be spoken by the assistive robot. Results are promising and can be used for educational and therapeutic purposes.

Quantitative Upper Limb Assessment With Natural User Interface in Children With Hemiparesis

Quantitative, efficient, and accurate measurement of upper limb motor performance is relevant for monitoring upper limb progress in daily activities of people with cerebral palsy, which is helpful to define an appropriate rehabilitation procedure. Currently, motion capture sensors using a Natural User Interface (NUI) are the most feasible solutions due to the low-cost implementation, portability, and effectiveness for upper limb motion analysis. A quantitatively assessment study of the upper limb motor performance for children with spastic hemiparesis using the NUI Kinect v2 sensor is described in this paper. The study participants were eighteen children with an average age of 8.28 ± 2.32 years having a Manual Ability Classification System level I and II. The assessment was done for each participant before and after the application of a Modified Constrained-Induced Movement Therapy (mCIMT) applied five days a week along twelve weeks. According to obtained results significant differences (p<0.05) in the Fugl-Meyer Assessment (FMA) score for motor performance in abduction movement of upper limb between the first and last sessions were quantitatively detected. The described NUI assessment method helped to quantitatively show that the applied therapy sessions were effective to improve upper extremity motor performance. Additionally, the upper limb movements motorically limited after the rehabilitation therapy were also identified. The main contribution of the described NUI assessment method is its potential use as a quantitative measurement tool, which might be used by specialists to objectively diagnose and consequently define an appropriate rehabilitation therapy for patients with hemiparesis.

Validation of Angle Estimation Based on Body Tracking Data from RGB-D and RGB Cameras for Biomechanical Assessment.

Motion analysis is an area with several applications for health, sports, and entertainment. The high cost of state-of-the-art equipment in the health field makes it unfeasible to apply this technique in the clinics' routines. In this vein, RGB-D and RGB equipment, which have joint tracking tools, are tested with portable and low-cost solutions to enable computational motion analysis. The recent release of Google MediaPipe, a joint inference tracking technique that uses conventional RGB cameras, can be considered a milestone due to its ability to estimate depth coordinates in planar images. In light of this, this work aims to evaluate the measurement of angular variation from RGB-D and RGB sensor data against the Qualisys Tracking Manager gold standard. A total of 60 recordings were performed for each upper and lower limb movement in two different position configurations concerning the sensors. Google's MediaPipe usage obtained close results compared to Kinect V2 sensor in the inherent aspects of absolute error, RMS, and correlation to the gold standard, presenting lower dispersion values and error metrics, which is more positive. In the comparison with equipment commonly used in physical evaluations, MediaPipe had an error within the error range of short- and long-arm goniometers.

DATaR: Depth Augmented Target Redetection using Kernelized Correlation Filter.

Unlike deep learning which requires large training datasets, correlation filter-based trackers like Kernelized Correlation Filter (KCF) use implicit properties of tracked images (circulant structure) for training in real time. Despite their popularity in tracking applications, there exists significant drawbacks of the tracker in cases like occlusions and out-of-view scenarios. This paper attempts to address some of these drawbacks with a novel RGB-D Kernel Correlation tracker in target re-detection. Our target re-detection framework not only re-detects the target in challenging scenarios but also intelligently adapts to avoid any boundary issues. Our results are experimentally evaluated using (a) standard dataset and (b) real time using the Microsoft Kinect V2 sensor. We believe this work will set the basis for improvement in the effectiveness of kernel-based correlation filter trackers and will further the development of a more robust tracker.

Logarithmic Observation of Feature Depth for Image-Based Visual Servoing

Due to the robustness to robot modeling and camera calibration errors and avoidance of complete target geometry, image-based visual servoing has always been an important topic in the fields such as robotics, computer vision and so forth. When the image information obtained by the camera is mapped to the robotic task space to design the servoing control law, the resulting interaction matrix, which links the spatial velocity of the camera to the temporal variation of the selected image features, depends on the unknown feature depths. The use of inaccurate feature depths may influence the stability and robustness of the controller, and even cause the failure of the task. In this article, based on the perspective camera model, by employing the principle of reduced order observer, a novel logarithmic observer is presented for on-line recovery of feature depth. Compared with the typical observers now available, the presented observer offers several advantages: global convergence, a faster convergence rate of error structure than exponential error structure, a less restricted observability condition and greater robustness against measurements with noise. The comparison results of numerical simulations indicate the superiority of the presented observer, and real experiments with Kinect v2 sensor further validate the effectiveness of the presented observer in practical situation. Note to Practitioners—This article was motivated by the depth problem in the image-based visual servoing scheme, but it can also be used in other situations where the image depth information is needed, such as 3D reconstruction, robot navigation, etc. The existing depth acquisition methods include TOF sensors, stereo vision, depth observers and so on. However, TOF sensors are sensitive to light conditions, and the mounting space of stereo vision is slightly large, and there is contradiction between observation performance and computational complexity in most existing observers. In this article, a novel structure of logarithmic reduced order observer is described in detail, which can be utilized to estimate the depth information of images easily. The simulations and experiments verify the good performance of the observer. The limitations of the given observer are that the estimation accuracy is not very good under weak excitation, and the camera needs to be calibrated in advance. Future work will focus on overcoming these two limitations.

Banana Pseudostem Width Detection Based on Kinect V2 Depth Sensor.

This study used Kinect V2 sensor to collect the three-dimensional point cloud data of banana pseudostem and developed an automatic measurement method of banana pseudostem width. The banana plant was selected as the research object in a banana plantation in Fusui, Guangxi. The mobile measurement of banana pseudostem was carried out at a distance of 1 m from the banana plant using the field operation platform with Kinect V2 as the collection equipment. To eliminate the background data and improve the processing speed, a cascade classifier was used to recognize banana pseudostems from the depth image, extract the region of interest (ROI), and transform the ROI into a color point cloud combined with the color image; secondly, the point cloud was sparse by down-sampling; then, the point cloud noise was removed according to the classification of large-scale and small-scale noise; finally, the stem point cloud was segmented along the y-axis, and the difference between the maximum and minimum values in the x-axis direction of each segment was calculated as its horizontal width. The center point of each segment point cloud was used to fit the slope of the stem centerline, and the average horizontal width was corrected to the stem diameter. The test results show that the average measurement error is only 2.7 mm, the average relative error was 1.34%, and the measurement time is only about 300 ms. It could provide an effective solution for the automatic and rapid measurement of stem width of banana plants and other similar plants.

Human activity recognition using CNN and long term short term memory

Human activity recognition aims to work out the activities performed by someone in a picture or video. Examples of actions are running, sitting, sleeping, and standing. Complex movement patterns and harmful occurrences like falling may be a part of these activities. The suggested ConvLSTM network can be created by successively combining fully connected layers, long immediate memory (LSTM) networks, and convolutional neural networks (CNN). The acquisition system will pre calculate skeleton coordinates using human detection and pose estimation from the image/video sequence. The ConvLSTM model builds new controlled features from the raw skeleton coordinates and their distinctive geometric and kinematic properties. Raw skeleton coordinates are utilized to generate geometric and kinematic properties supported by relative joint position values, joint differences, and their angular velocities. By utilizing a multi-player trained CNN-LSTM combination, novel spatiotemporal directed features can be obtained. The classification head with completely connected layers is then utilized. The suggested model was tested using the KinectHAR dataset, which consists of 130,000 samples with 81 attribute variables and was compiled using the Kinect (v2) sensor Experimental data is used to compare the performance of independent CNN and LSTM networks.

Classification of facial paralysis based on machine learning techniques

Facial paralysis (FP) is an inability to move facial muscles voluntarily, affecting daily activities. There is a need for quantitative assessment and severity level classification of FP to evaluate the condition. None of the available tools are widely accepted. A comprehensive FP evaluation system has been developed by the authors. The system extracts real-time facial animation units (FAUs) using the Kinect V2 sensor and includes both FP assessment and classification. This paper describes the development and testing of the FP classification phase. A dataset of 375 records from 13 unilateral FP patients and 1650 records from 50 control subjects was compiled. Artificial Intelligence and Machine Learning methods are used to classify seven FP categories: the normal case and three severity levels: mild, moderate, and severe for the left and right sides. For better prediction results (Accuracy = 96.8%, Sensitivity = 88.9% and Specificity = 99%), an ensemble learning classifier was developed rather than one weak classifier. The ensemble approach based on SVMs was proposed for the high-dimensional data to gather the advantages of stacking and bagging. To address the problem of an imbalanced dataset, a hybrid strategy combining three separate techniques was used. Model robustness and stability was evaluated using fivefold cross-validation. The results showed that the classifier is robust, stable and performs well for different train and test samples. The study demonstrates that FAUs acquired by the Kinect sensor can be used in classifying FP. The developed FP assessment and classification system provides a detailed quantitative report and has significant advantages over existing grading scales.

Towards a Low-Cost Solution for Gait Analysis Using Millimeter Wave Sensor and Machine Learning.

Human Activity Recognition (HAR) that includes gait analysis may be useful for various rehabilitation and telemonitoring applications. Current gait analysis methods, such as wearables or cameras, have privacy and operational constraints, especially when used with older adults. Millimeter-Wave (MMW) radar is a promising solution for gait applications because of its low-cost, better privacy, and resilience to ambient light and climate conditions. This paper presents a novel human gait analysis method that combines the micro-Doppler spectrogram and skeletal pose estimation using MMW radar for HAR. In our approach, we used the Texas Instruments IWR6843ISK-ODS MMW radar to obtain the micro-Doppler spectrogram and point clouds for 19 human joints. We developed a multilayer Convolutional Neural Network (CNN) to recognize and classify five different gait patterns with an accuracy of 95.7 to 98.8% using MMW radar data. During training of the CNN algorithm, we used the extracted 3D coordinates of 25 joints using the Kinect V2 sensor and compared them with the point clouds data to improve the estimation. Finally, we performed a real-time simulation to observe the point cloud behavior for different activities and validated our system against the ground truth values. The proposed method demonstrates the ability to distinguish between different human activities to obtain clinically relevant gait information.

Correction to: depth analysis of kinect v2 sensor in different mediums

Correction to: depth analysis of kinect v2 sensor in different mediums

Monitoring Volcanic and Tectonic Sandbox Analogue Models Using the Kinect v2 Sensor

AbstractThe measurement of surface deformation in analogue models of volcanic and tectonic processes is an area in continuous development. Properly quantifying topography change in analogue models is key for a useful comparison between experiment results and nature. The aim of this work is to evaluate the capabilities of the simple and cheap Microsoft® Kinect v2 sensor for monitoring analogue models made of granular materials. Microsoft® Kinect v2 is a video‐gaming RedGreenBlue‐Depth device combining an optical camera and an infrared distance measurement sensor. The precision of the device for model topography measurements has been quantified using 64 experiments, with variable granular materials materials and distance to the model. Additionally, we tested the capabilities of averaging several distance images to increase the precision. We have developed a specific software to facilitate the acquisition and processing of the Kinect v2 data in experiment monitoring. Our results show that measurement precision is material dependent: with clear‐colored and fine‐grained materials, a precision ∼1.0 mm for digital elevation models with a 1.6 mm pixel size can be obtained. We show that by averaging ≥5 consecutive images the distance precision can reach values as low as 0.5 mm. To show the Kinect v2 capabilities, we present monitoring results from case study experiments modeling tectonics and volcano deformation. The Kinect v2 achieves lower spatial resolutions and precision than more sophisticated techniques such as photogrammetry. However, Kinect v2 provides a cheap, straightforward and powerful tool for monitoring the topography changes in sandbox analogue models.

Automatic Evaluation of Motor Rehabilitation Exercises Based on Deep Mixture Density Neural Networks

An automatic assessment system for physical telerehabilitation could reduce the time and cost of treatments. But such assessment involves stochastic uncertainties, nonlinearities, and complexities of human movement. Probabilistic models and deep structures are two categories that could, respectively, address the stochastic uncertainty and complexity of motion data. In the proposed Deep Mixture Density Network (DMDN), probabilistic models were concurrently processed along with deep neural networks. More specifically, a multi-branch convolutional layer extracted the deep features of motion data, a Long Short Term Memory (LSTM) learned its temporal dependencies, and a Gaussian Mixture Model (GMM) handled the stochastic interaction of its preceding layers in reaching a more valid assessment and improved generalization to new movements. Finally, the weighted mean of the GMM components was used as the performance score for exercises. Input data were the time series related to the joints' position and orientation extracted from the Kinect v2 sensor video. A clinical reference score for each movement was also included for training the DMDN. In addition to comparisons with the state-of-the-art algorithms, an ablation study of the various elements comprising the DMDN was performed. Three configurations of convolutional filter window transitions across input data were also investigated. Results indicate that the proposed DMDN with one-dimensional parallel window transitions outperforms the other competing strategies in the ablation study. It also achieves higher reliability in terms of a lower RMSE standard deviation against a DMDN without GMM strategy while ranking competitively according to the Spearman correlation coefficient and Root Mean Square Error.

Sarawak Traditional Dance Motion Analysis and Comparison using Microsoft Kinect V2

This research project aimed to develop a software program or an interactive dance motion analysis application that utilizes modern technology to preserve and maintain the Sarawak traditional dance culture. The software program employs the Microsoft Kinect V2 to collect the digital dance data. The proposed method analyses the collected dance data for comparison purposes only. The comparison process was executed by displaying a traditional dance on the screen where the user who wants to learn the traditional dance can follow it and obtain results on how similar the dance is compared to the recorded dance data. The comparison of the performed and recorded dance data was visualized in graph form. The comparison graph showed that the Microsoft Kinect V2 sensors were capable of comparing the dance motion but with minor glitches in detecting the joint orientation. Using better depth sensors would make the comparison more accurate and less likely to have problems with figuring out how the joints move.

Kinect v2-Assisted Semi-Automated Method to Assess Upper Limb Motor Performance in Children.

The interruption of rehabilitation activities caused by the COVID-19 lockdown has significant health negative consequences for the population with physical disabilities. Thus, measuring the range of motion (ROM) using remotely taken photographs, which are then sent to specialists for formal assessment, has been recommended. Currently, low-cost Kinect motion capture sensors with a natural user interface are the most feasible implementations for upper limb motion analysis. An active range of motion (AROM) measuring system based on a Kinect v2 sensor for upper limb motion analysis using Fugl-Meyer Assessment (FMA) scoring is described in this paper. Two test groups of children, each having eighteen participants, were analyzed in the experimental stage, where upper limbs’ AROM and motor performance were assessed using FMA. Participants in the control group (mean age of 7.83 ± 2.54 years) had no cognitive impairment or upper limb musculoskeletal problems. The study test group comprised children aged 8.28 ± 2.32 years with spastic hemiparesis. A total of 30 samples of elbow flexion and 30 samples of shoulder abduction of both limbs for each participant were analyzed using the Kinect v2 sensor at 30 Hz. In both upper limbs, no significant differences (p < 0.05) in the measured angles and FMA assessments were observed between those obtained using the described Kinect v2-based system and those obtained directly using a universal goniometer. The measurement error achieved by the proposed system was less than ±1° compared to the specialist’s measurements. According to the obtained results, the developed measuring system is a good alternative and an effective tool for FMA assessment of AROM and motor performance of upper limbs, while avoiding direct contact in both healthy children and children with spastic hemiparesis.

Comparative analysis on path planning of ATR using RRT*, PSO, and modified APF in CG-Space

Path planning of All-Terrain Rover over 3D terrains is one of the most challenging robotics problems. Prior research has been carried out over 2D static and dynamic configuration space. Currently, all available algorithms for 3D path planning of articulated rover require optimization for wheel-terrain interaction. In this comparative analysis, the path planning is carried out over the 10 degrees of freedom Rover CG-Space instead of ground. CG-Space is the collection of all possible center of gravity locations of the Rover while traversing on terrain. The introduction of preobtained CG-Space eliminates the need for optimization during the planning stage. In this work, a comprehensive path planning comparison has been obtained among modified artificial potential field (APF) method, sampling-based technique (RRT*), and particle swarm optimization (PSO) algorithm based on path length, path smoothness, and computational time. All algorithms are applied over CG-Space associated with two categories of terrains: even and uneven. Even terrains have a well-defined surface equation, while uneven terrains are generated using Kinect V2 sensor mounted over the sand arena. A continuous surface from CG-Space is generated using cubic spline interpolation. The simulation results show that all three algorithms generate obstacle-free paths in a different time span. Although sampling-based and APF-based techniques take about five times lesser time, the obtained paths have moderate smoothness. PSO performs better than RRT* and modified APF for the requirements of optimal path length and smoothness. Thus, PSO is the optimal way to plan the path if the computational cost associated is acceptable. On the other hand, if time is the major planning concern, then RRT* is the best because the path smoothening takes very little time when using separate cubic spline techniques after obtaining the path nodes. This comparative analysis is useful in selecting an appropriate algorithm depending on the terrain complexity and obstacle conditions.

Human Activity Recognition via Hybrid Deep Learning Based Model

In recent years, Human Activity Recognition (HAR) has become one of the most important research topics in the domains of health and human-machine interaction. Many Artificial intelligence-based models are developed for activity recognition; however, these algorithms fail to extract spatial and temporal features due to which they show poor performance on real-world long-term HAR. Furthermore, in literature, a limited number of datasets are publicly available for physical activities recognition that contains less number of activities. Considering these limitations, we develop a hybrid model by incorporating Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) for activity recognition where CNN is used for spatial features extraction and LSTM network is utilized for learning temporal information. Additionally, a new challenging dataset is generated that is collected from 20 participants using the Kinect V2 sensor and contains 12 different classes of human physical activities. An extensive ablation study is performed over different traditional machine learning and deep learning models to obtain the optimum solution for HAR. The accuracy of 90.89% is achieved via the CNN-LSTM technique, which shows that the proposed model is suitable for HAR applications.