Human Gesture Recognition Research Articles

Gesture Tracking and Recognition Algorithm for Dynamic Human Motion Using Multimodal Deep Learning

To address the problems of the traditional human motion gesture tracking and recognition methods, such as poor tracking effect, low recognition accuracy, high frame loss rate, and long-time cost, a dynamic human motion gesture tracking and recognition algorithm using multimode deep learning was proposed. Firstly, the collected human motion images are repaired in the three-dimensional (3D) environment, and the multimodal 3D human motion model is reconstructed using the processed images. Secondly, according to the results of model reconstruction, the camera gesture and other parameters of the keyframe are used to construct the target tracking optimization function so as to achieve the purpose of accurate tracking of human motion. Finally, for multimodal human motion gesture learning, a convolutional neural network (CNN) is developed. The trained CNN is utilized to complete dynamic human motion recognition after convolutional and pooling calculations. The results demonstrate that the proposed algorithm is effective in tracking human motion gestures. The average recognition accuracy is 96%, the average frame loss rate is 8.8%, the time cost is low, and the proposed algorithm has a high F-measure and much lower power consumption than other algorithms, indicating that the proposed algorithm is effective.

Advances in human action, activity and gesture recognition

A set of advanced approaches and models on human action, activity, gesture, and behavior recognition related aspects along with associated challenges are summarized in this note. Notably, Video-based human activity recognition, sensor-based activity analysis, skeleton-based activity recognition, assisted daily living, anomaly detection, facial expression and emotion analysis, gesture and sign language are highlighted in the works. This special issue also introduced six new datasets, while exploring a total of 55 different datasets in its 21 research articles. Apart from the the areas covered in this issue, research on multi-modal analysis, action quality assessment, real-time applications, and activity and behavior computing on edge devices are some of the dominant future challenges to deal with. We firmly believe that the original works and ideas presented in this special issue will serve as helpful references for the relevant research communities in the journey towards a brighter future.

A Hand Gesture Recognition Algorithm Based on Multi-scale Hybrid Features

A human hand gesture recognition algorithm based on multi-scale hybrid features is pro-posed. A large number of gesture data images are organized and annotated by automatic im-age annotation algorithm. Based on dense sampling, the multi-scale Gaussian pyramid is taken into account for enhancing the image details; it can realize the recognition of both large and small target gestures. Meanwhile, according to the aggregation and distribution of gestures, statistical analysis is used to optimize the size and aspect ratio of anchors, which can further improve the accuracy of recognition. The experiment results show that the mean average precisions of this algorithm can be improved to 99.7% and 93.3% for the simple gesture and complex gesture, respectively, which are higher than that obtained by these traditional methods, such as faster region-convolution neural network (Faster R-CNN), You Only Look Once (YOLO), and SSD.

Real-Time Hand Gesture Recognition for Media Players

In this modern age of advancement in computing which has made the use of natural user interface much needed. Human gesture recognition is a mode of non-verbal interaction medium and can provide the most intuitive and natural way to interact with computer. The main aim is to make the interaction between human and computer as natural as the interaction between humans. The objective of this paper is to recognize the dynamic hand gesture images based on shapes and relative positions or direction of hand which is extracted from input video stream recording. These hand gestures can be used to recognition system to control multimedia applications running on computer using different gesture commands

CNN Deep Learning with Wavelet Image Fusion of CCD RGB-IR and Depth-Grayscale Sensor Data for Hand Gesture Intention Recognition.

Pixel-based images captured by a charge-coupled device (CCD) with infrared (IR) LEDs around the image sensor are the well-known CCD Red–Green–Blue IR (the so-called CCD RGB-IR) data. The CCD RGB-IR data are generally acquired for video surveillance applications. Currently, CCD RGB-IR information has been further used to perform human gesture recognition on surveillance. Gesture recognition, including hand gesture intention recognition, is attracting great attention in the field of deep neural network (DNN) calculations. For further enhancing conventional CCD RGB-IR gesture recognition by DNN, this work proposes a deep learning framework for gesture recognition where a convolution neural network (CNN) incorporated with wavelet image fusion of CCD RGB-IR and additional depth-based depth-grayscale images (captured from depth sensors of the famous Microsoft Kinect device) is constructed for gesture intention recognition. In the proposed CNN with wavelet image fusion, a five-level discrete wavelet transformation (DWT) with three different wavelet decomposition merge strategies, namely, max-min, min-max and mean-mean, is employed; the visual geometry group (VGG)-16 CNN is used for deep learning and recognition of the wavelet fused gesture images. Experiments on the classifications of ten hand gesture intention actions (specified in a scenario of laboratory interactions) show that by additionally incorporating depth-grayscale data into CCD RGB-IR gesture recognition one will be able to further increase the averaged recognition accuracy to 83.88% for the VGG-16 CNN with min-max wavelet image fusion of the CCD RGB-IR and depth-grayscale data, which is obviously superior to the 75.33% of VGG-16 CNN with only CCD RGB-IR.

Human Gesture Recognition in Computer Vision Research

Human gesture recognition is a popular issue in the studies of computer vision, since it provides technological expertise required to advance the interaction between people and computers, virtual environments, smart surveillance, motion tracking, as well as other domains. Extraction of the human skeleton is a rather typical gesture recognition approach using existing technologies based on two-dimensional human gesture detection. Likewise, I t cannot be overlooked that objects in the surrounding environment give some information about human gestures. To semantically recognize the posture of the human body, the logic system presented in this research integrates the components recognized in the visual environment alongside the human skeletal position. In principle, it can improve the precision of recognizing postures and semantically represent peoples’ actions. As such, the paper suggests a potential and notion for recognizing human gestures, as well as increasing the quantity of information offered through analysis of images to enhance interaction between humans and computers.

The Image Processing Using Soft Robot Technology in Fitness Motion Detection Under the Internet of Things

In order to study the application of robotic technology in fitness motion detection, this study develops a modular high accuracy and low-latency Human Gesture Recognition (HGR) system. Firstly, an improved HGR algorithm is introduced, which compresses the HGR model through content extraction and reduces the number of parameters. Methods such as Simulated Annealing and Semi-Supervised Learning are introduced to improve content extraction, further improving the compression of the model. Secondly, a human pose recognition system is constructed based on Deep Learning (DL) and robotics, and each module is recommended. Finally, the effectiveness of the system is verified. The results show that when the accuracy is close to HRNet-32. The inference speed of the model is increased by about 2.4 times, and the model parameters are compressed by about 67%. The data demonstrate the effectiveness of the method. The robot-human posture detection technology can effectively solve human posture recognition in fitness exercises.

Towards Domain-Independent and Real-Time Gesture Recognition Using Mmwave Signal

Human gesture recognition using millimeter-wave (mmWave) signals provides attractive applications including smart home and in-car interfaces. While existing works achieve promising performance under controlled settings, practical applications are still limited due to the need of intensive data collection, extra training efforts when adapting to new domains, and poor performance for real-time recognition. In this paper, we propose DI-Gesture, a domain-independent and real-time mmWave gesture recognition system. Specifically, we first derive signal variations corresponding to human gestures with spatial-temporal processing. To enhance the robustness of the system and reduce data collecting efforts, we design a data augmentation framework for mmWave signals based on correlations between signal patterns and gesture variations. Furthermore, a spatial-temporal gesture segmentation algorithm is employed for real-time recognition. Extensive experimental results show DI-Gesture achieves an average accuracy of 97.92\%, 99.18\%, and 98.76\% for new users, environments, and locations, respectively. We also evaluate DI-Gesture in challenging scenarios like real-time recognition and sensing at extreme angles, all of which demonstrate the superior robustness and effectiveness of our system.

AirFi: Empowering WiFi-based Passive Human Gesture Recognition to Unseen Environment via Domain Generalization

AirFi: Empowering WiFi-based Passive Human Gesture Recognition to Unseen Environment via Domain Generalization

Entropy-Based Variational Scheme with Component Splitting for the Efficient Learning of Gamma Mixtures.

Finite Gamma mixture models have proved to be flexible and can take prior information into account to improve generalization capability, which make them interesting for several machine learning and data mining applications. In this study, an efficient Gamma mixture model-based approach for proportional vector clustering is proposed. In particular, a sophisticated entropy-based variational algorithm is developed to learn the model and optimize its complexity simultaneously. Moreover, a component-splitting principle is investigated, here, to handle the problem of model selection and to prevent over-fitting, which is an added advantage, as it is done within the variational framework. The performance and merits of the proposed framework are evaluated on multiple, real-challenging applications including dynamic textures clustering, objects categorization and human gesture recognition.

Ultrastretchable, Adhesive, Fast Self-Healable, and Three-Dimensional Printable Photoluminescent Ionic Skin Based on Hybrid Network Ionogels.

Developing multifunctional stretchable ionic skin (I-Skin) to mimic the sensations of the human skin is of great interest and shows promising potential in wearable sensors and human-machine interfaces (HMIs). However, common ionogels prepared with small-molecule cross-linkers and single networks can hardly satisfy the requirements of adjustable mechanical properties, strong adhesion, fast self-healability, and good stability in extreme environments. Herein, an ultrastretchable (>10,000%), ultrastrong adhesive (>6.8 MPa), ultrafast self-healable (10 s), high thermally stable (-60 to 250 °C), and three-dimensional (3D)-printable photoluminescent ionogel with shape memory properties has been designed. The ionogel consists of hyperbranched polymer covalent-cross-linked poly(zwitterionic ionic liquid)-co-poly(acrylic acid) and multiple dynamic bonding cross-linked networks. The excellent performance of the ionogel-based high-stretchable strain sensor and the triboelectric nanogenerator (TENG)-based self-powered touch sensor is further demonstrated over a wide temperature range (-40 to 150 °C). More importantly, ionogel-based I-Skin can work as an HMI for human gesture recognition and real-time wireless control of robots under extreme vacuum conditions and can also self-heal immediately along with function recovery after mechanical damage.

Gesture recognition using dual-stream CNN based on fusion of sEMG energy kernel phase portrait and IMU amplitude image

Since the surface electromyography signal (sEMG) is a weak and unstable signal, human action recognition based on sEMG time domain or frequency domain feature extraction has poor stability and low recognition rate. Based on modeling analysis of sEMG, the dual-stream convolutional neural network (CNN) is built to perform feature fusion of sEMG energy kernel phase portrait and inertial measurement unit(IMU) data for gesture recognition in this paper. Firstly, methods of matrix counting method and time window amplitude are utilized to represent the sEMG and IMU data as images. Secondly, the moving average method is presented to filter the images. Thirdly, the dual-stream CNN is used to perform feature extraction, fusion and classification of sEMG and IMU images respectively. Finally, the experiments yield an average recognition accuracy of 95.78% for 6 gestures performed by 5 subjects, which demonstrates that the proposed method has better performance for human gestures recognition.

Single Image Depth Map Estimation for Improving Posture Recognition

Image-based posture recognition is a very challenging problem since it is difficult to acquire rich 3D information from the posture in color image. To address this issue, we present a novel and unified framework for human posture recognition, applying single image depth map estimation from color images. The proposed method includes two stages. The first stage estimates the depth map from the single-color image by an improved Pix2Pix generation module. The generation module is equipped with a hybrid loss function that captures the high-level features and recovers the sharp depth discontinuities, thus improving the depth estimation results. The second stage (the recognition stage) improves the color image-based recognition performance by incorporating the estimated depth map. Thereby, a two-stream CNN architecture that separately processes the color image and its estimated depth image is developed for robust posture recognition. To verify its effectiveness, we first test the proposed method on a novel pose dataset, which contains 13800 samples of paired color-and-depth of 6 subjects with 15 poses. The dataset used in this work is been created and released, is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">http://media.ritsumei.ac.jp/iipl/database/pose/</uri> . Extensive experiments are also performed on the public OUHANDS hand gesture dataset. Experiments demonstrate that the proposed method achieves superior performance on both human pose and hand gesture recognition tasks.

Daily Gesture Recognition During Human-Robot Interaction Combining Vision and Wearable Systems

The recognition of human gestures is crucial for improving the quality of human-robot cooperation. This article presents a system composed of a Pepper robot that mounts an RGB-D camera and an inertial device called SensHand. The system acquired data from twenty people who performed five daily living activities (i.e., Having Lunch, Personal Hygiene, Working, House Cleaning, Relax). The activities were composed of at least two “basic” gestures for a total of 10 gestures. The data acquisition was performed by two cameras positioned laterally and frontally to mimic the real conditions. The acquired data were off-line classified considering different combinations of sensors to evaluate how the sensor fusion approach improves the recognition abilities. Specifically, the article presents an experimental study that evaluated four algorithms often used in computer vision, i.e., three classical machine learning and one belonging to the field of deep learning, namely Support Vector Machine, Random Forest, K-Nearest Neighbor and Long Short-Term Memory Recurrent Neural Network. The comparative analysis of the results shows a significant improvement of the accuracy when fusing camera and sensors data, i.e., 0.81 for the whole system configuration when the robot is in a frontal position with respect to the user (0.79 if we consider only the index finger sensors) and equal to 0.75 when the robot is in a lateral position. Interestingly, the system performs well in recognising the transitions between gestures when these are presented one after the other, a common event in the real-life that was often neglected in the previous studies.

Flexible triboelectric nanogenerator for human motion tracking and gesture recognition

Nowadays human motion tracking and gesture recognition have gained increasing attention as they allow humans to control machines through natural and intuitive behaviors by recognizing meaningful expressions. Here, we propose a stretchable, high durability and deformable triboelectric nanogenerator (F-TENG) on the base of silica gel, which can be used to not only collect mechanical energy from our ambient environment, but also monitor human motion and behaviors through the highly sensitive response to the complex states of human motion, such as stretching, folding, extrusion and hitting. The clutter in the signals can be effectively collected by the integrated filtering circuit, thus the function of real-time statistics of human motion and behaviors can be recorded and interpreted. This work promotes the broad application prospects of TENGs in artificial intelligence, sporting monitoring, and big data collection.

A New Data Augmentation Method for Time Series Wearable Sensor Data Using a Learning Mode Switching-Based DCGAN

This letter describes a new image augmentation method based on a DCGAN considering mode switching in terms of transient learning accuracy threshold on the problem of human gesture recognition through imaging of wearable sensor time-series data and a deep CNN structure. Because the discriminator in GANs learns faster than the generator, it is known that mode collapse occurs, in which only image modes biased to a specific image type are augmented among various image forms. In this study, to solve the mode collapse caused by the learning difficulty mismatch between networks, we add a learning mode switching layer between the generator and discriminator and receive feedback from both networks’ transient learning accuracies to switch the learning mode if predefined thresholds are exceeded. We confirm that the proposed approach balanced the learning rate between the generator and discriminator networks, resolved the mode collapse problem, and increased the test accuracy of a deep CNN trained with an augmented image set by approximately 20.35% compared to a conventional DCGAN. In addition, it showed better accuracy on a performance comparison with other improved DCGAN methods.

Deep Neural Networks for Human Action Recognition

Abstract: In deep neural networks, human action detection is one of the most demanding and complex tasks. Human gesture recognition is the same as human action recognition. Gesture is defined as a series of bodily motions that communicate a message. Gestures are a more natural and preferable way for humans to engage with computers, thereby bridging the gap between humans and robots. The finest communication platform for the deaf and dumb is human action recognition. We propose in this work to create a system for hand gesture identification that recognizes hand movements, hand characteristics such as peak calculation and angle calculation, and then converts gesture photos into text. Index Terms: Human action recognition, Deaf and dumb, CNN.

Spatio-temporal continuous gesture recognition under degraded environments: performance comparison between 3D integral imaging (InIm) and RGB-D sensors

In this paper, we introduce a deep learning-based spatio-temporal continuous human gesture recognition algorithm under degraded conditions using three-dimensional (3D) integral imaging. The proposed system is shown as an efficient continuous human gesture recognition system for degraded environments such as partial occlusion. In addition, we compare the performance between the 3D integral imaging-based sensing and RGB-D sensing for continuous gesture recognition under degraded environments. Captured 3D data serves as the input to a You Look Only Once (YOLOv2) neural network for hand detection. Then, a temporal segmentation algorithm is employed to segment the individual gestures from a continuous video sequence. Following segmentation, the output is fed to a convolutional neural network-based bidirectional long short-term memory network (CNN-BiLSTM) for gesture classification. Our experimental results suggest that the proposed deep learning-based spatio-temporal continuous human gesture recognition provides substantial improvement over both RGB-D sensing and conventional 2D imaging system. To the best of our knowledge, this is the first report of 3D integral imaging-based continuous human gesture recognition with deep learning and the first comparison between 3D integral imaging and RGB-D sensors for this task.

XGest: Enabling Cross-Label Gesture Recognition with RF Signals

Extensive efforts have been devoted to human gesture recognition with radio frequency (RF) signals. However, their performance degrades when applied to novel gesture classes that have never been seen in the training set. To handle unseen gestures, extra efforts are inevitable in terms of data collection and model retraining. In this article, we present XGest, a cross-label gesture recognition system that can accurately recognize gestures outside of the predefined gesture set with zero extra training effort. The key insight of XGest is to build a knowledge transfer framework between different gesture datasets. Specifically, we design a novel deep neural network to embed gestures into a high-dimensional Euclidean space. Several techniques are designed to tackle the spatial resolution limits imposed by RF hardware and the specular reflection effect of RF signals in this model. We implement XGest on a commodity mmWave device, and extensive experiments have demonstrated the significant recognition performance.

Convolutional neural network human gesture recognition algorithm based on phase portrait of surface electromyography energy kernel

Surface electromyography (sEMG) is a weak signal which is non-stationary and non-periodic. The sEMG classification methods based on time domain and frequency domain features have low recognition rate and poor stability. Based on the modeling and analysis of sEMG energy kernel, this paper proposes a new method to recognize human gestures utilizing convolutional neural network (CNN) and phase portrait of sEMG energy kernel. Firstly, the matrix counting method is used to process the sEMG energy kernel phase portrait into a grayscale image. Secondly, the grayscale image is preprocessed by moving average method. Finally, CNN is used to recognize sEMG of gestures. Experiments on gesture sEMG signal data set show that the effectiveness of the recognition framework and the recognition method of CNN combined with the energy kernel phase portrait have obvious advantages in recognition accuracy and computational efficiency over the area extraction methods. The algorithm in this paper provides a new feasible method for sEMG signal modeling analysis and real-time identification.