Approach For Gesture Recognition Research Articles

Gesture Feature Extraction and Recognition Based on Image Processing

Gesture recognition has become increasingly popular, in response to the growing demand for intelligent and personalized human-computer interaction (HCI) and human-to-human interaction. However, gesture recognition raises a high requirement on the background color of the gesture image, and faces difficulty in extracting multiple gesture features. To solve these problems, this paper presents a novel approach for gesture feature extraction and recognition based on image processing. Firstly, the workflow of the proposed gesture recognition method was given, and a series of preprocessing was performed on the original gesture image, prior to formal extraction and recognition. Next, the authors detailed the extraction of features from gesture boundaries and fingertips. Finally, a convolutional neural network (CNN) was constructed for gesture recognition, and a gesture recognition model was developed based on residual network. The proposed approach was proved to be valid through experiments. The research results provide a reference for the application of CNN in the recognition of various postures or shapes.

Active Shooter Detection in Multiple-Person Scenario Using RF-Based Machine Vision

Emerging applications of radio frequency (RF) vision sensors for security and gesture recognition primarily target single individual scenarios which restricts potential applications. In this article, we present the design of a cyber-physical framework that analyzes RF micro-Doppler signatures for individual anomaly detection, such as a hidden rifle among multiple individuals. RF avoids certain limitations of video surveillance, such as recognizing concealed objects and privacy concerns. Current RF-based approaches for human activity detection or gesture recognition usually consider single individual scenarios, and the features extracted for such scenarios are not applicable for multi-person cases. From a machine learning perspective, the RF sensor spectrogram images are conducible for training using deep convolutional neural networks. However, generating a large labeled training dataset with an exhaustive variety of multi-person scenarios is extremely time consuming and nearly impossible due to the wide range of combinations possible. We present approaches for multi-person spectrogram generation based on individual person spectrograms that can augment the training dataset and increase the accuracy of prediction. Our results show that the spectrogram generated by RF sensors can be harnessed by artificial intelligence algorithms to detect anomalies such as a concealed weapon for single and multiple people scenarios. The proposed system can aid as a standalone tool, or be complemented by video surveillance for anomaly detection, in scenarios involving single or multiple individuals.

Deep signature-based isolated and large scale continuous gesture recognition approach

In recent years, recognition of hand gestures has received considerable attention from the research community, with an increased interest in advanced decision support systems. In applications of this type of system, recognition of hand gestures based on vision is used in the design and development of interfaces such as that of automobile users. This type of interface allows drivers to interact with the vehicle, without affecting their concentrations. This also increases driver comfort, without compromising safety. However, the video-based gesture recognition problem is not trivial and poses problems due to: (i) intra and inter-person variations in the movement of human hand gestures; (ii) variations between people in the shape and size of the human hand; and (iii) variations in lighting; and background noise.In this paper, a new recognition system is proposed to address the problem of large scale continuous gesture recognition with depth and gray-scale input videos. The proposed recognition system contains 3 main phases. Firstly continuous gesture sequences are segmented into isolated gesture using mean of velocity information calculated based on the estimation of deep optical flow. For each isolated segment a set of relevant descriptors named deep signature features is extracted in order to characterize different intensity and spatial information describing the location, the velocity and the orientation of the movement. The features constructed for both depth and gray-scale sequences are fed to linear SVM for classification. To evaluate the effectiveness of our recognition model, it was compared with other methods using two challenging datasets: KTH and Chalearn. Indeed, a first experimental study has shown that, using the KTH dataSet and the SVM classifier, results given by our method outperforms previously published methods and it reaches 95.6% in term of accuarcy and 96% in term of precision. The second experimental study, carried out on the ChaLearn LAP ConGD data set, also showed a superiority of our method from the point of view of precision by codelivery over other methods with mean Jaccard Index of 0.5011.

Gesture Recognition Based on 3D Human Pose Estimation and Body Part Segmentation for RGB Data Input

This paper presents a novel approach for dynamic gesture recognition using multi-features extracted from RGB data input. Most of the challenges in gesture recognition revolve around the axis of the presence of multiple actors in the scene, occlusions, and viewpoint variations. In this paper, we develop a gesture recognition approach by hybrid deep learning where RGB frames, 3D skeleton joint information, and body part segmentation are used to overcome such problems. Extracted from the RGB images are the multimodal input observations, which are combined by multi-modal stream networks suited to different input modalities: residual 3D convolutional neural networks based on ResNet architecture (3DCNN_ResNet) for RGB images and color body part segmentation modalities; long short-term memory network (LSTM) for 3D skeleton joint modality. We evaluated the proposed model on four public datasets: UTD multimodal human action dataset, gaming 3D dataset, NTU RGB+D dataset, and MSRDailyActivity3D dataset and the experimental results on these datasets proves the effectiveness of our approach.

A Multi-Modular Approach for Gesture Recognition and Text Formulation in American Sign Language

A Multi-Modular Approach for Gesture Recognition and Text Formulation in American Sign Language

A novel hybrid bidirectional unidirectional LSTM network for dynamic hand gesture recognition with Leap Motion

Due to the recent development of machine learning and sensor innovations, hand gesture recognition systems become promisingfor the digital entertainment field. In this paper, we propose a dynamic hand gesture recognition approach using touchless hand motions over a Leap Motion device. First, we analyze the sequential time series data gathered from Leap Motion using Long Short-Term Memory (LSTM) recurrent neural networks for recognition purposes. We exploit basic unidirectional LSTM and bidirectional LSTM separately. Then, we propound novel architecture by combining the aforementioned models with additional components to give a final prediction network, named Hybrid Bidirectional Unidirectional LSTM (HBU-LSTM). The suggested network improves the model performance significantly by considering the spatial and temporal dependencies between the Leap Motion data and the network layers during the forward and backward pass. The recognition models are examined on two available benchmark datasets, named the LeapGestureDB dataset and the RIT dataset. Experiments demonstrate the potential of the proposed HBU-LSTM network for dynamic hand gesture recognition, with an average recognition rate reaching approximately 90%. Our suggested approach reaches superior performance, in terms of accuracy and computational complexity, over some existing methods for hand gesture recognition.

Design and evaluation of a hand gesture recognition approach for real-time interactions

Hand gestures are a natural and intuitive form for human-environment interaction and can be used as an input alternative in human-computer interaction (HCI) to enhance usability and naturalness. Many existing approaches have employed vision -based systems to detect and recognize hand gestures. However, vision-based systems usually require users to move their hands within restricted space, where the optical device can capture the motion of hands. Also, vision-based systems may suffer from self-occlusion issues due to sophisticated finger movements. In this work, we use a sensor-based motion tracking system to capture 3D hand and finger motions. To detect and recognize hand gestures, we propose a novel angular-velocity method, which is directly applied to real-time 3D motion data streamed by the sensor-based system. Our approach is capable of recognizing both static and dynamic gestures in real-time. We assess the recognition accuracy and execution performance with two interactive applications that require gesture input to interact with the virtual environment. Our experimental results show high recognition accuracy, high execution performance, and high-levels of usability.

Multimodal hand gesture recognition combining temporal and pose information based on CNN descriptors and histogram of cumulative magnitudes

In this paper, we present a new approach for dynamic hand gesture recognition. Our goal is to integrate spatiotemporal features extracted from multimodal data captured by the Kinect sensor. In case the skeleton data is not provided, we apply a novel skeleton estimation method to compute temporal features. Furthermore, we introduce an effective method to extract a fixed number of keyframes to reduce the processing time. To extract pose features from RGB-D data, we take advantage of two different approaches: (1) Convolutional Neural Networks and (2) Histogram of Cumulative Magnitudes. We test different integration methods to fuse the extracted spatiotemporal features to boost recognition performance in a linear SVM classifier. Extensive experiments prove the effectiveness and feasibility of the proposed framework for hand gesture recognition.

Hand Calculator System using Convolutional Neural Networks

Computer vision has great attention in recent years as it identifies and similarly processes images that human vision does and they provide suitable output. In computer vision, hand gesture recognition is one of the important and fundamental problems. The hand gesture recognition system has gained significant importance in the recent few years because of its manifoldness applications. This paper aims to give a new approach for vision-based, fast and real time hand gesture recognition, a new light that can be used in many HCI applications. The proposed algorithm first detects and segments the hand region and by using our innovative approach, it finds the fingers and classifies the gesture. The proposed algorithm is invariant to orientation, hand position or distance from the webcam. Based on this proposed algorithm we have progressively developed a gesture-based mathematical tool (calculator) as a practical application.

Hand Gesture Recognition for Sign Language Using 3DCNN

Recently, automatic hand gesture recognition has gained increasing importance for two principal reasons: the growth of the deaf and hearing-impaired population, and the development of vision-based applications and touchless control on ubiquitous devices. As hand gesture recognition is at the core of sign language analysis a robust hand gesture recognition system should consider both spatial and temporal features. Unfortunately, finding discriminative spatiotemporal descriptors for a hand gesture sequence is not a trivial task. In this study, we proposed an efficient deep convolutional neural networks approach for hand gesture recognition. The proposed approach employed transfer learning to beat the scarcity of a large labeled hand gesture dataset. We evaluated it using three gesture datasets from color videos: 40, 23, and 10 classes were used from these datasets. The approach obtained recognition rates of 98.12%, 100%, and 76.67% on the three datasets, respectively for the signer-dependent mode. For the signer-independent mode, it obtained recognition rates of 84.38%, 34.9%, and 70% on the three datasets, respectively.

A brief Review of Deep Learning Based Approaches for Facial Expression and Gesture Recognition Based on Visual Information

Psychological researchers and Many Other Researchers have found that body language of a human can provide substantial information in detecting and interpreting emotions. It could express explicit and implicit information mutually of one’s emotional state and intentions over multi-channel modalities. These imperative channels include eye gaze, head movement, facial expression, body posture and gesture and so on. This learning focuses on detecting emotional states from the body language of the hand and face using computer vision and soft computing techniques. The facial expression and gesture recognition have widely used and challenging task in the present scenario. This paper describes brief review of all approaches which are majorly categorized into two -categories where the first category belongs to Conventional approaches and other based on Deep-learning or Non - Conventional. This paper is basically used as a survey of deep-learning approaches for hand-gesture and Facial Expression Recognitions. We describe all review through the taxonomy of deep-learning approaches proposed architecture details, fusion strategies, datasets, way to treat temporal-dimensions of data, its main features, basic knowledge & general understanding of its challenges

LimbMotion

Wearable-based human-computer interaction is a promising technology to enable various applications. This paper aims to track the 3D posture of the entire limb, both wrist/ankle and elbow/knee, of a user wearing a smart device. This limb tracking technology can trace the geometric motion of the limb, without introducing any training stage usually required in gesture recognition approaches. Nonetheless, the tracked limb motion can also be used as a generic input for gesture-based applications. The 3D posture of a limb is defined by the relative positions among main joints, e.g., shoulder, elbow, and wrist for an arm. When a smartwatch is worn on the wrist of a user, its position is affected by both elbow and shoulder motions. It is challenging to infer the entire 3D posture when only given a single point of sensor data from the smartwatch. In this paper, we propose LimbMotion, an accurate and real-time limb tracking system. The performance gain of LimbMotion comes from multiple key technologies, including an accurate attitude estimator based on a novel two-step filter, fast acoustic ranging, and point clouds-based positioning. We implemented LimbMotion and evaluated its performance using extensive experiments, including different gestures, moving speeds, users, and limbs. Results show that LimbMotion achieves real-time tracking with a median error of 7.5cm to 8.9cm, which outperforms the state-of-the-art approach by about 32%.

Integrative Approach of Voice and Gesture Recognition for Interactive Robot

The instructing technique is never again reasonable for the creation mode with human – robot cooperation. A web based showing technique with the combination of discourse and motion, this application is described by the educator to control the robot camera based video data or using MEMS hardware interface using Zigbee – voice heading can be seen using Android application. The training procedure is never again sensible for the creation mode. An electronic appearing with the blend of discourse, change of the endeavor is to teach the robot to compose by giving three evident sources of information like Voice direction, Camera based information or utilizing MEMS equipment interface utilizing Zigbee. Voice direction can be perceived utilizing android application.

The Inertial Gesture Recognition System

The paper is devoted to the implementation of the gestures recognition system. Existing gesture recognition approaches are considered. Among them the system of inertial type is chosen for research, since the systems of this type are less difficult to implement and require relatively low computational complexity. Thus, the proposed system is based on two components: the hardware module for gesture capturing and the subsystem for the recognition of captured gestures. The hardware component of the system employs MEMS accelerometer and gyroscope for motion acquisition. The prototype of this component is implemented using the STM32F401RE microcontroller and the MEMS sensor MPU9250. The communication between the microcontroller and the MEMS sensor is carried out via the I2C bus. The microcontroller initializes the sensor and acquires the motion data. Then it sends the collected data via the virtual COM port to the PC, where the captured motions are recognized. The system component for the recognition of gestures is implemented as the neural network in Matlab environment. This network consists of four layers: the input layer, the BLSTM layer, the fully-connected layer, and the Softmax layer. The key component of the selected neural network architecture is the BLSTM layer, which (due to its properties) provides some invariance to the length and the shift of the gesture signal in time. The suggested system was tested on three gesture classes: circular motion, tapping and other/unknown gesture. Using the hardware module the training and the testing sets were collected. In course of experiments, it was established that the best training results for the selected network architecture and the collected training set are achieved using the Adam optimizer. The experiment with different number of hidden units in the BLSTM layer showed that the highest recognition rate may be achieved when 100 units are used. Particularly, the suggested system with the specified number of hidden units in BLSTM layer was able to attain the recognition error less than 3.5% (recognition rate of 96.7%) on the test set.

One-shot gesture recognition with attention-based DTW for human-robot collaboration

PurposeThis paper aims to present a one-shot gesture recognition approach which can be a high-efficient communication channel in human–robot collaboration systems.Design/methodology/approachThis paper applies dynamic time warping (DTW) to align two gesture sequences in temporal domain with a novel frame-wise distance measure which matches local features in spatial domain. Furthermore, a novel and robust bidirectional attention region extraction method is proposed to retain information in both movement and hold phase of a gesture.FindingsThe proposed approach is capable of providing efficient one-shot gesture recognition without elaborately designed features. The experiments on a social robot (JiaJia) demonstrate that the proposed approach can be used in a human–robot collaboration system flexibly.Originality/valueAccording to previous literature, there are no similar solutions that can achieve an efficient gesture recognition with simple local feature descriptor and combine the advantages of local features with DTW.

Hand gesture recognition based on motor unit spike trains decoded from high-density electromyography

ObjectiveMethods for surface electromyographic (EMG) signal decomposition have been developed in the past decade, to extract neural information transferred from the spinal cord to muscles. Here, we characterize the accuracy in the identification of motor unit activities during hand postures from high-density EMG signals and we propose a mapping approach between these neural signals and hand gestures. MethodsHigh-density EMG signals were recorded during 11 hand gesture tasks from 11 able-bodied subjects. EMG signals were offline decomposed into motor unit spike trains (MUSTs) with a blind source separation algorithm. A gesture recognition approach based on motor unit classification was proposed. MUSTs were first pooled into groups corresponding to the 11 motions. Then the activation level of the neural drive to each motion was estimated as the summed discharge timings of MUSTs in each group. The output gesture class was determined by comparing the estimated activation level of each motion. ResultsOn average, 29 ± 8 MUSTs were identified for each motion with an estimated decomposition accuracy >90%. The average classification accuracy for 11 hand gestures based on the proposed approach was >95% and outperformed the classic approach of using global EMG features. Conclusion and significanceThese results indicate the possibility of identifying motor unit activities during intended motor tasks and demonstrate high classification accuracy of the hand gestures, with perspectives for human–machine interfacing.

Systolic hand gesture recognition/detection system based on FPGA with multi-port BRAMs

A new systolic approach of real-time vision-based hand gesture recognition is proposed. Hand gesture recognition is classified into two type, static and dynamic. The proposed system can be employed in both types. The system can detect and classify either a static hand shape in one image or a static hand shape at the start, within, and the end of gesture in a dynamic sequence in video stream. An efficient template matching (TM) based similarity measure is used to recognize gesture within the inter frame sequence of video frames. The template can be used to store the static class of different hand gesture shapes, while fast scanning approach of window over an image frame dynamically detects or recognizes the hand gesture. A suitable hardware systolic model of TM is designed. The systolic architecture is expanded to be used with multiple windows (k-templates) simultaneously. The required templates are equal to the number (k) of gestures that is to be recognized. They are stored in FPGA RAM blocks (BRAMs). To increase the bandwidth of templates BRAMs, the BRAM is multi-ported. A considerable reduction of memory access operations is achieved for the parallel systolic model in comparison to sequential/parallel software models. Consequently, the hand gesture is detected, tracked and recognized within the inter frame space of video. An acceptable recognition accuracy of up to 95% is achieved under variant observation conditions of three hand gesture classes dataset. A high frame rate video can be processed in real time.

Real-time gesture recognition based on feature recalibration network with multi-scale information

Gesture recognition is important in human–machine interaction. Current methods for solving gesture recognition have several disadvantages such as low recognition rate, slow speed and poor performance on recognizing multiple targets or long-distance targets in complex environments. In view of the above problems, we propose a gesture recognition approach that can recognize gestures quickly and accurately from complex background. This approach works on a deep convolutional network, which consists of a basic network module for extracting feature information, a squeeze-and-excitation networks for increasing feature channel affinity and a feature pyramid attention module for fusing context information with different scales. To test the proposed approach, we make a data set that contains 3289 images from difference complex scenes. Generally gestures in those images can be generally classified into 16 types. We have uploaded this data set for researchers use. Experimental results demonstrate that the recognition accuracy and speed of the proposed method can achieve 83.45% and 32.2 frames per second respectively, which has better comprehensive performance compared with other state-of-the-art recognition algorithms.

Inferring Static Hand Poses from a Low-Cost Non-Intrusive sEMG Sensor.

Every year, a significant number of people lose a body part in an accident, through sickness or in high-risk manual jobs. Several studies and research works have tried to reduce the constraints and risks in their lives through the use of technology. This work proposes a learning-based approach that performs gesture recognition using a surface electromyography-based device, the Myo Armband released by Thalmic Labs, which is a commercial device and has eight non-intrusive low-cost sensors. With 35 able-bodied subjects, and using the Myo Armband device, which is able to record data at about 200 MHz, we collected a dataset that includes six dissimilar hand gestures. We used a gated recurrent unit network to train a system that, as input, takes raw signals extracted from the surface electromyography sensors. The proposed approach obtained a 99.90% training accuracy and 99.75% validation accuracy. We also evaluated the proposed system on a test set (new subjects) obtaining an accuracy of 77.85%. In addition, we showed the test prediction results for each gesture separately and analyzed which gestures for the Myo armband with our suggested network can be difficult to distinguish accurately. Moreover, we studied for first time the gated recurrent unit network capability in gesture recognition approaches. Finally, we integrated our method in a system that is able to classify live hand gestures.

Development of a hierarchical dynamic keyboard character recognition system using trajectory features and scale-invariant holistic modeling of characters

A robust, intuitive, effortless, and novel dynamic hand gesture based virtual keyboard system is developed in this study. Firstly, a new hierarchical approach is applied, which, based on self-coarticulation and position features, effectively sub-groups a large gesture vocabulary. Additionally, new trajectory features are proposed which shall extract the local structural statistics of the gestures. All state-of-art models are based on temporal trajectory features which are based on the frame-wise 2D sequential path it followed. Due to this, the trajectory features are path dependent and vulnerable to trajectory noises or any other variations in pattern, speed, or scale. In contrast to this, an image-based approach of gesture recognition has been proposed in this study, which is independent of the sequential gesturing path of the gesture. Since the image-models (a holistic view) are not obtained frame-wise, unlike existing image-models, they are pattern, speed, and scale invariant in nature and also immune to trajectory distortions. To this end, image-based features and significant trajectory features are fused to develop a hybrid hierarchical classification model which exhibits an exceptional increase in accuracy by 3.9% as compared to baseline non-hierarchical trajectory based model using an Artificial neural network (ANN). Classification models such as Voronoi diagram based classifier (VDBC) and neuro-fuzzy (NF) classifier have also been explored and displayed motivating performance. Reduction in misclassification has been observed for gestures such as ‘(and)’, ‘{and}’, ‘0 and O’, ‘Z and 2’. The present system can also identify any static/dynamic imposters present in the gesture environment.