Approach For Gesture Recognition Research Articles

A Random Forest-based Approach for Hand Gesture Recognition with Wireless Wearable Motion Capture Sensors

Abstract Gesture Recognition has a prominent importance in smart environment and home automation. Thanks to the availability of Machine Learning approaches it is possible for users to define gestures that can be associated with commands for the smart environment. In this paper we propose a Random Forest-based approach for Gesture Recognition of hand movements starting from wireless wearable motion capture data. In the presented approach, we evaluate different feature extraction procedures to handle gestures and data with different duration. To enhance reproducibility of our results and to foster research in the Gesture Recognition area, we share the dataset that we have collected and exploited for the present work.

Data Augmentation in Deep Learning-Based Fusion of Depth and Inertial Sensing for Action Recognition

This article covers a deep learning-based decision fusion approach for action or gesture recognition via simultaneous utilization of a depth camera and a wearable inertial sensor. The deep learning approach involves using a convolutional neural network (CNN) for depth images captured by a depth camera and a combination of CNN and long short–term memory network for inertial signals captured by a wearable inertial sensor, followed by a decision-level fusion. Due to the limited size of the training data, a data augmentation procedure is carried out by generating depth images corresponding to different orientations of the depth camera and by generating inertial signals corresponding to different orientations of the inertial sensor placement on the body. The results obtained indicate the positive impact of the decision-level fusion as well as the data augmentation on the recognition accuracies.

TS-I3D Based Hand Gesture Recognition Method With Radar Sensor

Aiming at the problems of the noise impact on the parametric image of hand gestures, the difficulty of gesture feature extraction, and the inefficient utilization of continuous gesture time sequential information, we propose a time sequential inflated 3 dimensions (TS-I3D) convolutional neural network approach for hand gesture recognition based on frequency modulated continuous wave (FMCW) radar sensor. Specifically, the FMCW radar is used to acquire the hand gesture data, and the range and speed of the gesture in each frame signal are calculated by 2 dimensions fast Fourier transform. Then, the range-Doppler map (RDM) is generated based on the relationship between motion parameters and frequency. The interference in RDM caused by people and the external environment is filtered out and the peak of hand gesture in RDM is further enhanced by wavelet transform. Finally, TS-I3D network is designed to extract the range and speed change information of the continuous gestures. The experimental results show that the average recognition accuracy rate of the hand gestures of the proposed method is 96.17%.

GestureVLAD: Combining Unsupervised Features Representation and Spatio-Temporal Aggregation for Doppler-Radar Gesture Recognition

In this paper we propose a novel framework to process Doppler-radar signals for hand gesture recognition. Doppler-radar sensors provide many advantages over other emerging sensing modalities, including low development costs and high sensitivity to capture subtle gestures with precision. Furthermore, they have attractive properties for ubiquitous deployment and can be conveniently embedded into different devices. In this scope, current recognition methods still rely in deep CNN-LSTM and $3D$ CNN-LSTM structures that require sufficient labelled data to optimize millions of parameters and significant amount of computational resources for inference; which limits their deployment. Indeed, subtle gestures recognition is a challenging task due to the high variability of gestures among different subjects. To overcome the challenges in the recognition task and the limitations of the current methods, we propose a shallow learning approach for gesture recognition, that is based on unsupervised range-Doppler features representation, along with a learnable pooling aggregation via NetVLAD. The proposed framework can encode extremely valuable information across time, and results in features that are highly discriminative for hand gesture recognition. Experimentation on publicly available Doppler-radar data shows that the proposed framework outperforms state-of-the-art approaches in terms of recognition accuracy and speed for sequence-level hand gesture classification.

A spatiotemporal approach for vision-based hand gesture recognition using Hough transform and neural network

Recognition of hand gesture is a crucial task as far as accuracy of recognition and real-time performance is concerned. The recognition accuracy of hand gesture systems used in applications from virtual reality to sign language recognition depends not only on the features extracted but also on the soft computing techniques used to recognize the gesture. This requires recognition system, with robust features which are unaffected by rotation, scaling, translation, variation in illumination and view of the gesture within the image. Accordingly, the research work presented in this paper aims to recognize the dynamic hand gestures using Hough transform-based spatiotemporal feature extraction for preprocessing and artificial neural network for identification. The system is implemented and tested using benchmark Cambridge hand gesture database and Sebastien dynamic hand posture database. As many as 32 features comprising of 19 Fourier descriptors, 3 geometrical features and 10 temporal features are obtained on each gesture sequence and are given as input to NN. The experimental results revealed that this approach is able to recognize dynamic hand gestures with an average recognition rate (RR) higher than 94% on Cambridge database which is better than that reported by earlier researchers while 98% RR on Sebastien database. The noteworthy feature of the proposed approach is that it relieves from the burden of foreground detection, generally required for other video processing systems.

Continuous Arm Gesture Recognition Based on Natural Features and Logistic Regression

In this paper, we propose a new gesture modeling method by combining natural features with logistic regression to recognize continuous natural gestures, which is considered as a challenge. Instead of manually modeling gestures with criteria, gesture models can be derived from a few gesture instances. Arm gestures are modeled with three types of intuitive features, and logistic regression is employed to obtain the optimal weights to linearly combine all the features, which can deal with the differences among instances with the same gesture automatically and facilitate distinguishing different gestures, and as a result, improve the recognition accuracy. With these features, numbers of natural arm gestures can be modeled with good comprehensibility. To evaluate our method, we defined twelve natural gestures, including three static gestures and nine dynamic gestures. A continuous gesture database which includes 154024 data frames (102 minutes) and 1628 gesture instances was collected from 10 subjects with a wearable multi-IMU sensing system. Both subject-dependent and subject-mixed recognition tests were conducted with the database. In the subject-dependent test, models were trained with about 30% data, and the average accuracy over all subjects was 91.54%. In the subject-mixed test, only about 10% data was used for training, and the accuracy is 88.75%. The recognition accuracy is highly competitive with previous gesture recognition approaches.

Feasibility of Wrist-Worn, Real-Time Hand, and Surface Gesture Recognition via sEMG and IMU Sensing

While most wearable gesture recognition approaches focus on the forearm or fingers, the wrist may be a more suitable location for practical use. We present the design and validation of a real-time gesture recognition wristband based on surface electromyography and inertial measurement unit sensing fusion, which can recognize 8 air gestures and 4 surface gestures with 2 distinct force levels. Ten healthy subjects performed an initial gesture recognition experiment, followed by a second experiment 1 h later and a third experiment 1 day later. Classification accuracies for the initial experiment were 92.6% and 88.8% for air and surface gestures, respectively, and there were no changes in accuracy results during testing 1 h. and 1 day later ( $p$ $>$ 0.05). These results demonstrate the feasibility of wrist-based gesture recognition paving the way for potential future integration in to a smart watch or other wrist-worn wearable for intuitive human computer interaction.

A Review of Computational Approaches for Human Behavior Detection

Computer vision techniques capable of detecting human behavior are gaining interest. Several researchers have provided their review on behavior detection, however most of the reviews are focused on activity recognition only, and reviews on gesture and facial expression recognition are very few. Therefore, all of them lack to cover complete human behavior analysis. In this study, we provide a comprehensive review of human behavior detection approaches. The framework of this review is based on activity, gesture and facial expression recognition since these are the most important cues for behavior detection. These three areas are further classified in existing computational approaches. One can easily recognize from this review that hidden Markov model is widely exploited for activity recognition while motion history image is still a developing area. Haar-like features can be a valid alternative for gesture recognition. For facial expression recognition, local binary patterns feature is a very popular choice. We have reviewed behavior detection techniques, mostly developed after year 2009. The explicit advantages of this review are: (1) it provides a deep analysis of computational approaches for activity, gesture and facial expression recognition. (2) It includes both types of techniques that include single human as well as multiple human activities. (3) It considers techniques developed in the last decade only pertaining to information about the most recent techniques. (4) It provides a brief description of popular datasets used for activity, gesture and facial expression recognition. (5) It discusses open issues to provide an insight for future also.

Deep Fisher discriminant learning for mobile hand gesture recognition

Gesture recognition becomes a popular analytics tool for extracting the characteristics of user movement and enables numerous practical applications in the biometrics field. Despite recent advances in this technique, complex user interaction and the limited amount of data pose serious challenges to existing methods. In this paper, we present a novel approach for hand gesture recognition based on user interaction on mobile devices. We have developed two deep models by integrating Bidirectional Long-Short Term Memory (BiLSTM) network and Bidirectional Gated Recurrent Unit (BiGRU) with Fisher criterion, termed as F-BiLSTM and F-BiGRU respectively. These two Fisher discriminative models can classify user’s gesture effectively by analyzing the corresponding acceleration and angular velocity data of hand motion. In addition, we build a large Mobile Gesture Database (MGD) containing 5547 sequences of 12 gestures. With extensive experiments, we demonstrate the superior performance of the proposed method compared to the state-of-the-art BiLSTM and BiGRU on MGD database and two other benchmark databases (i.e., BUAA mobile gesture and SmartWatch gesture). The source code and MGD database will be made publicly available at https://github.com/bczhangbczhang/Fisher-Discriminant-LSTM.

A Posteriori Processing Optimization Method for Gesture Interaction in Aseptic Operation Room

Gesture recognition plays an important role in the applications of human-centered smart technologies. This paper designs a gesture recognition approach in a special human-computer interaction environment. Generally, in an aseptic operation room, doctors cannot access the patient’s image data directly, and they have to let the nurse operate the system under their instructions. After the nurses finish their task, they also need to re-dress the surgical gown, disinfect their hands, then they can go back to the operating room. If the doctor could access the image directly by gestures during the operation, they would be able to save a lot of time and work more efficiently. Although some template matching algorithms, such as dynamic time warping (DTW), have been widely used in gesture recognition, those traditional methods are not accurate enough, and they are unable to identify continuous repetitive gestures. This paper proposes an improved DTW method based on posteriori processing, which achieved some key operations, including parameter regulations, invalid area determination, and static gesture processing. This method will address the problem of continuous repetitive gesture recognition, with higher accuracy. The experimental results show that the method can improve the accuracy of recognition by over 40%, and it can also perform real-time dynamic gesture recognition tasks effectively.

Toward an Unsupervised Approach for Daily Gesture Recognition in Assisted Living Applications

Activity Recognition is important in assisted living applications to monitor people at home. Over the past, inertial sensors have been used to recognize different activities, spanning from physical activities to eating ones. Over the last years, supervised methods have been widely used, but they require an extensive labeled data set to train the algorithms and this may represent a limitation of concrete approaches. This paper presents a comparison of unsupervised and supervised methods in recognizing nine gestures by means of two inertial sensors placed on the index finger and on the wrist. Three supervised classification techniques, namely random forest, support vector machine, and multilayer perceptron, as well as three unsupervised classification techniques, namely k-Means, hierarchical clustering, and self-organized maps, were compared with the recognition of gestures made by 20 subjects. The obtained results show that the support vector machine classifier provided the best performances (0.94 accuracy) compared with the other supervised algorithms. However, the outcomes show that even in an unsupervised context, the system is able to recognize the gestures with an average accuracy of ~0.81. The proposed system may be therefore involved in future telecare services that could monitor the activities of daily living, allowing an unsupervised approach that does not require labelled data.

Vision-based Hand Recognition Based on ToF Depth Camera

The current gesture recognition methods mostly adopt the classification-based approaches such as Neural Network (NN), Support Vector Machine (SVM), Hidden Markov Model (HMM) etc. As for the input image features, most research studies combined the color and depth images (ex. RGB-D) to obtain more accurate information of hand area, and such techniques may cost high computational resources and energy consumptions. To provide a low-cost gesture recognition method for wearable devices, this thesis used merely the Time-of-Flight depth camera to achieve a lightweight gesture recognition method. In most traditional gesture recognition methods, users have to wear gloves or bracelets to let depth cameras being able to accurately capture hands areas, and so that the hand contours, palm’s distances, and angle feature can be obtained. Moreover, the Earth Mover’s Distance (EMD) algorithm, which is adopted in most gesture recognition approaches, costs high computational times. In this study, to avoid wearing gloves or bracelets, we propose a new algorithm that can compute the wrist cutting edges and capture the palm areas. In addition, this thesis proposes an efficient finger detection algorithm to judge the number of fingers, and significantly reduce the computing times. In the experimental results, our proposed method achieves a recognition rate of 90% and the performance has 5 frames per second on NVIDIA TX1 embedded platforms.

Multi-objective optimization for hand posture recognition

Hand gestures are an intuitive way for humans to interact with computers. They are becoming increasingly popular in several applications, such as smart houses, games, vehicle infotainment systems, kitchens and operating theaters. An effective human–computer interaction system should aim at both good recognition accuracy and speed. This paper proposes a new approach for static hand gesture recognition. A benchmark database with 36 gestures is used, containing variations in scale, illumination and rotation. Several common image descriptors, such as Fourier, Zernike moments, pseudo-Zernike moments, Hu moments, complex moments and Gabor features are comprehensively compared in terms of their respective accuracy and speed. Gesture recognition is undertaken by a multilayer perceptron which has a flexible structure and fast recognition. In order to achieve improved accuracy and minimize computational cost, both the feature vector and the neural network are tuned by a multi-objective evolutionary algorithm based on the Nondominated Sorting Genetic Algorithm II (NSGA-II). The proposed method is compared with state-of-the-art methods. A real-time gesture recognition system based on the proposed descriptor is constructed and evaluated. Experimental results show a good recognition rate, using a descriptor with low computational cost and reduced size.

Laban movement analysis and hidden Markov models for dynamic 3D gesture recognition

In this paper, we propose a new approach for body gesture recognition. The body motion features considered quantify a set of Laban Movement Analysis (LMA) concepts. These features are used to build a dictionary of reference poses, obtained with the help of a k-medians clustering technique. Then, a soft assignment method is applied to the gesture sequences to obtain a gesture representation. The assignment results are used as input in a Hidden Markov Models (HMM) scheme for dynamic, real-time gesture recognition purposes. The proposed approach achieves high recognition rates (more than 92% for certain categories of gestures), when tested and evaluated on a corpus including 11 different actions. The high recognition rates obtained on two other datasets (Microsoft Gesture dataset and UTKinect-Human Detection dataset) show the relevance of our method.

Combining point context and dynamic time warping for online gesture recognition

Previous gesture recognition methods usually focused on recognizing gestures after the entire gesture sequences were obtained. However, in many practical applications, a system has to identify gestures before they end to give instant feedback. We present an online gesture recognition approach that can realize early recognition of unfinished gestures with low latency. First, a curvature buffer-based point context (CBPC) descriptor is proposed to extract the shape feature of a gesture trajectory. The CBPC descriptor is a complete descriptor with a simple computation, and thus has its superiority in online scenarios. Then, we introduce an online windowed dynamic time warping algorithm to realize online matching between the ongoing gesture and the template gestures. In the algorithm, computational complexity is effectively decreased by adding a sliding window to the accumulative distance matrix. Lastly, the experiments are conducted on the Australian sign language data set and the Kinect hand gesture (KHG) data set. Results show that the proposed method outperforms other state-of-the-art methods especially when gesture information is incomplete.

Hand Motion and Posture Recognition in a Network of Calibrated Cameras

This paper presents a vision-based approach for hand gesture recognition which combines both trajectory and hand posture recognition. The hand area is segmented by fixed-range CbCr from cluttered and moving backgrounds and tracked by Kalman Filter. With the tracking results of two calibrated cameras, the 3D hand motion trajectory can be reconstructed. It is then modeled by dynamic movement primitives and a support vector machine is trained for trajectory recognition. Scale-invariant feature transform is employed to extract features on segmented hand postures, and a novel strategy for hand posture recognition is proposed. A gesture vector is introduced to recognize hand gesture as an entirety which combines the recognition results of motion trajectory and hand postures where a support vector machine is trained for gesture recognition based on gesture vectors.

A Fuzzy Framework for Real-Time Gesture Spotting and Recognition

A vital requirement of any recognition system claiming to be real time is the capability to perform feature extraction in real time. In this paper, we propose an innovative fuzzy approach for real-time dynamic gesture recognition and spotting, where a compact local descriptor is designed to model moving gesture skeletons as a time series of fuzzy statistical features. Then, a set of one-vs-rest SVMs is trained on these features for gesture recognition and spotting. In this approach, the meaningful hand movements are successfully spotted while concurrently removing unintentional hand movements from an input video sequence. When evaluated on a gesture data set incorporating a relatively large and diverse collection of video data, the method proposed yields promising results that compare very favorably with those reported in the literature, while retaining real-time performance.

Hand gesture recognition based on concentric circular scan lines and weighted K-nearest neighbor algorithm

Human-computer interactions based on hand gestures are of the most popular natural interactive modes, which severely depends on real-time hand gesture recognition approaches. In this paper, a simple but effective hand feature extraction method is described, and the corresponding hand gesture recognition method is proposed. First, based on a simple tortoise model, we segment the human hand images by skin color features and tags on the wrist, and normalize them to create the training dataset. Second, feature vectors are computed by drawing concentric circular scan lines (CCSL) according to the center of the palm, and linear discriminant analysis (LDA) algorithm is used to deal with those vectors. Last, a weighted k-nearest neighbor (W-KNN) algorithm is presented to achieve real-time hand gesture classification and recognition. Besides the efficiency and effectiveness, we make sure that the whole gesture recognition system can be easily implemented and extended. Experimental results with a user-defined hand gesture dataset and multi-projector display system show the effectiveness and efficiency of the new approach.

Natural Gesture Modeling and Recognition Approach Based on Joint Movements and Arm Orientations

Gesture recognition, as one of the most promising human–robot interaction approaches, has attracted research interests for a long time. Though many feasible methods have been proposed in this field, the current gesture modeling and recognition methods are deficient in expressiveness, naturalness, and efficiency. In this paper, we present a novel method of modeling and recognizing natural gestures in 3-D space. Inspired by the joint space modeling, which is frequently used in robotics, we decompose the gestures into information of joint movements, joint angles, and arm orientations. We calculate joint angles and detect joint movements with the orientation data measured by inertial measurement units. Numbers of gestures can be modeled by combining three types of criteria, namely, joint angle criteria, arm orientation criteria, and joint movement criteria. Our system can cope with the repetitive movements, which are common in gestures while rarely considered before. Different from the previous statistical learning or template matching approaches, our approach does not need training. User can add new gesture to the gesture set or edit existing gesture definitions conveniently. To evaluate our system, we conducted real-time gesture recognition experiments with ten subjects. Twelve gestures, including three static gestures and nine dynamic gestures, were modeled for the experiments. The average recognition accuracy of totaling 1560 gesture samples was 91.86%.

다중 사용자를 위한 Dynamic Time Warping 기반의 특징 강조형 제스처 인식 모델

최근 제안된 FsGr 모델은 가속도 센서 기반의 제스처 인식을 위한 방법으로 DTW 알고리즘을 두 단계로 적용하여 인식률을 개선하였다. FsGr 모델에서는 유사제스처 집합 개념을 정의하는데 훈련과정에서 유사제스처 집합들을 생성한다. 제스처 인식의 1차 인식 시도에서 유사제스처 집합이 정의된 제스처로 판정되면, 이 유사제스처 집합의 제스처들에 대해 특징이 강조된 부분들을 추출해 DTW를 통한 2차 인식을 시도한다. 그러나 동일 제스처도 사용자의 신체 크기, 나이, 성별, 등의 신체적인 특징에 따라 매우 다른 특성을 보이고 있어 FsGr 모델을 다중 사용자 환경에 적용하기에는 한계가 있다. 본 논문에서는 이를 다중 사용자 환경으로 확장한 FsGrM 모델을 제안하고 이를 사용한 스마트TV의 채널 및 볼륨 제어 프로그램을 보인다. FsGr model, which has been proposed recently, is an approach of accelerometer-based gesture recognition by applying DTW algorithm in two steps, which improved recognition success rate. In FsGr model, sets of similar gestures will be produced through training phase, in order to define the notion of a set of similar gestures. At the 1st attempt of gesture recognition, if the result turns out to belong to a set of similar gestures, it makes the 2nd recognition attempt to feature-strengthened parts extracted from the set of similar gestures. However, since a same gesture show drastically different characteristics according to physical traits such as body size, age, and sex, FsGr model may not be good enough to apply to multi-user environments. In this paper, we propose FsGrM model that extends FsGr model for multi-user environment and present a program which controls channel and volume of smart TV using FsGrM model.