Hand Gesture Recognition Method Research Articles

A novel interactive assembly teaching aid using multi-template augmented reality

Augmented Reality (AR) technology is a fast growing field in the academics and industry. This paper presents a novel design of an interactive assembly teaching aid based on a multi-template AR system, which consists of three units: a multi-template AR unit, an online three-dimension (3D) model assembly unit, a hand-gesture interaction unit. The design of the multi-template AR unit employs an efficient multi-template pose tracking method to detect and track multiple template images simultaneously. The online 3D model assembly unit is enabled, when the pose of each target template is tracked and computed by the multi-template pose tracking method. This method measures the distance between the two templates to determine the 3D rendering mode of the virtual object. The third unit aims to realize a vision-based human-computer interactive system, which combines the AR rendering system with the real-time hand-gesture recognition method to decide the status of AR rendering of a 3D dynamic animation according to the user’s hand gesture. Based on the feedback of twenty-one users of the AR-based interactive teaching-aid, it can be concluded that the system creates an interactive experiences that engages the users and facilitates them to increase their learning interest significantly. In future, we plan to port the proposed AR system to Android and iOS mobile devices while improving the functionality, interactivity, and entertaining qualities of the teaching aid system.

Design of Image Extraction Hardware for Hand Gesture Vision Recognition

In this paper, we propose a system that can detect the shape of a hand at high speed using an FPGA. The hand-shape detection system is designed using Verilog HDL, a hardware language that can process in parallel instead of sequentially running C++ because real-time processing is important. There are several methods for hand gesture recognition, but the image processing method is used. Since the human eye is sensitive to brightness, the YCbCr color model was selected among various color expression methods to obtain a result that is less affected by lighting. For the CbCr elements, only the components corresponding to the skin color are filtered out from the input image by utilizing the restriction conditions. In order to increase the speed of object recognition, a median filter that removes noise present in the input image is used, and this filter is designed to allow comparison of values and extraction of intermediate values at the same time to reduce the amount of computation. For parallel processing, it is designed to locate the centerline of the hand during scanning and sorting the stored data. The line with the highest count is selected as the center line of the hand, and the size of the hand is determined based on the count, and the hand and arm parts are separated. The designed hardware circuit satisfied the target operating frequency and the number of gates.

Multidimensional Feature Representation and Learning for Robust Hand-Gesture Recognition on Commercial Millimeter-Wave Radar

This article presents a robust hand-gesture recognition method via multidimensional feature representation and learning specifically designed for commercial frequency-modulated continuous wave (FMCW) multi-input multi-output (MIMO) millimeter-wave radar. First, the optimal configuration of the radar system parameters for the hand-gesture recognition scenario is investigated and a standard procedure to determine the system configuration is given. Then a moving scattering center model is proposed to represent the 3-D point cloud in the range–Doppler (RD)-angular multidimensional feature space. A scattering point detection and tracking algorithm is presented based on a set of motion constraints in terms of position, velocity, and acceleration. It is derived from the space-time continuity of a nonrigid target. Finally, a lightweight multichannel convolutional neural network (CNN) is designed to learn and classify multidimensional gesture features including radial RD and tangential azimuth–elevation. Extensive experiments are carried out with the developed system and a large data set is obtained to train and test the classifier. The results show that the proposed gesture recognition method can effectively distinguish gestures that are easily confused in the RD domain and achieve robust performances under various conditions.

Hand Gesture Recognition Using Mechanomyography Signal Based on LDA Classifier

The growing number of amputees in Iraq with multiple degrees of amputations makes it necessary to provide them with prosthetic hands with an easy to use control system that meets their aspirations. The Mechanomyography (MMG) signal has been proposed as an alternative or assisting method for hand gesture recognition. Electromyography (EMG) which is used as control signal in the commercial prosthetic hands faces many challenges such as electrical interference, non-stationery and electrode displacement. The MMG signal has been presented as a method to deal with the existing challenges of EMG. In this paper, MMG based hand gesture recognition is proposed with Pattern Recognition (PR) system. MMG signal have been collected from six healthy subjects, using accelerometers and microphones, which performed seven classes of hand movements. Classification accuracy of approximately 89% was obtained with PR method, consisting of time domain and Wavelet feature extraction and Linear Discernment Analysis (LDA) for classification. The results showed that the proposed method has a promising way for detecting and classifying hand gestures by low-cost MMG sensors which can be used for the control of prosthetic hand.

A novel hybrid bidirectional unidirectional LSTM network for dynamic hand gesture recognition with Leap Motion

Due to the recent development of machine learning and sensor innovations, hand gesture recognition systems become promisingfor the digital entertainment field. In this paper, we propose a dynamic hand gesture recognition approach using touchless hand motions over a Leap Motion device. First, we analyze the sequential time series data gathered from Leap Motion using Long Short-Term Memory (LSTM) recurrent neural networks for recognition purposes. We exploit basic unidirectional LSTM and bidirectional LSTM separately. Then, we propound novel architecture by combining the aforementioned models with additional components to give a final prediction network, named Hybrid Bidirectional Unidirectional LSTM (HBU-LSTM). The suggested network improves the model performance significantly by considering the spatial and temporal dependencies between the Leap Motion data and the network layers during the forward and backward pass. The recognition models are examined on two available benchmark datasets, named the LeapGestureDB dataset and the RIT dataset. Experiments demonstrate the potential of the proposed HBU-LSTM network for dynamic hand gesture recognition, with an average recognition rate reaching approximately 90%. Our suggested approach reaches superior performance, in terms of accuracy and computational complexity, over some existing methods for hand gesture recognition.

Chronological pattern indexing: An efficient feature extraction method for hand gesture recognition with Leap Motion

Recently, Hand-Gesture-Recognition (HGR) systems has appreciably change the way of interaction between humans and computers thanks to advanced sensor technologies like the Leap-Motion-Controller (LMC). Despite the success achieved by many state-of-the-art methods, they have not worked on the rich temporal information existing in the sequential hand gesture data and characterizing the discriminative representation of different hand gesture classes. In this paper, we suggest a novel Chronological-Pattern-Indexing (CPI) approach which encodes the temporal orders of patterns for hand gesture time series data acquired by the LMC sensor. We extract a set of temporal patterns from different optimized projections. Then, we compare their temporal order and we encode the whole sequence with the index of the first coming pattern. We repeat these steps until we generate an efficient feature vector modeling the chronological dynamics of the hand gesture. The experiments demonstrate the potential of the proposed CPI approach for HGR systems.

A robust method for VR-based hand gesture recognition using density-based CNN

Many VR-based medical purposes applications have been developed to help patients with mobility decrease caused by accidents, diseases, or other injuries to do physical treatment efficiently. VR-based applications were considered more effective helper for individual physical treatment because of their low-cost equipment and flexibility in time and space, less assistance of a physical therapist. A challenge in developing a VR-based physical treatment was understanding the body part movement accurately and quickly. We proposed a robust pipeline to understanding hand motion accurately. We retrieved our data from movement sensors such as HTC vive and leap motion. Given a sequence position of palm, we represent our data as binary 2D images of gesture shape. Our dataset consisted of 14 kinds of hand gestures recommended by a physiotherapist. Given 33 3D points that were mapped into binary images as input, we trained our proposed density-based CNN. Our CNN model concerned with our input characteristics, having many 'blank block pixels', 'single-pixel thickness' shape and generated as a binary image. Pyramid kernel size applied on the feature extraction part and classification layer using softmax as loss function, have given 97.7% accuracy.

RETRACTED ARTICLE: An efficient method for human hand gesture detection and recognition using deep learning convolutional neural networks

The physical movement of the human hand produces gestures, and hand gesture recognition leads to the advancement in automated vehicle movement system. In this paper, the human hand gestures are detected and recognized using convolutional neural networks (CNN) classification approach. This process flow consists of hand region of interest segmentation using mask image, fingers segmentation, normalization of segmented finger image and finger recognition using CNN classifier. The hand region of the image is segmented from the whole image using mask images. The adaptive histogram equalization method is used as enhancement method for improving the contrast of each pixel in an image. In this paper, connected component analysis algorithm is used in order to segment the finger tips from hand image. The segmented finger regions from hand image are given to the CNN classification algorithm which classifies the image into various classes. The proposed hand gesture detection and recognition methodology using CNN classification approach with enhancement technique stated in this paper achieves high performance with state-of-the-art methods.

Hand Gesture Recognition in Complex Background Based on Convolutional Pose Machine and Fuzzy Gaussian Mixture Models

Hand gesture is one of the most intuitive and natural ways for human to communicate with computers, and it has been widely adopted in many human–computer interaction applications. However, it is still a challenging problem when confronted with complex background, illumination variation and occlusion in real-world scenarios. In this paper, a two-stage hand gesture recognition method is proposed to tackle these problems. At the first stage, hand pose estimation is developed to locate the hand keypoints using the convolutional pose machine, which can effectively localize hand keypoints even in a complex background. At the second stage, the Fuzzy Gaussian mixture models (FGMMs) are tailored to reject the nongesture patterns and classify the gestures based on the estimated hand keypoints. Extensive experiments are conducted to evaluate the performance of the proposed method, and the result demonstrates that the proposed algorithm is effective, robust, and satisfactory in real-time scenarios.

Real-Time Hand Gesture Spotting and Recognition Using RGB-D Camera and 3D Convolutional Neural Network

Using hand gestures is a natural method of interaction between humans and computers. We use gestures to express meaning and thoughts in our everyday conversations. Gesture-based interfaces are used in many applications in a variety of fields, such as smartphones, televisions (TVs), video gaming, and so on. With advancements in technology, hand gesture recognition is becoming an increasingly promising and attractive technique in human–computer interaction. In this paper, we propose a novel method for fingertip detection and hand gesture recognition in real-time using an RGB-D camera and a 3D convolution neural network (3DCNN). This system can accurately and robustly extract fingertip locations and recognize gestures in real-time. We demonstrate the accurateness and robustness of the interface by evaluating hand gesture recognition across a variety of gestures. In addition, we develop a tool to manipulate computer programs to show the possibility of using hand gesture recognition. The experimental results showed that our system has a high level of accuracy of hand gesture recognition. This is thus considered to be a good approach to a gesture-based interface for human–computer interaction by hand in the future.

Finger Gesture Spotting from Long Sequences Based on Multi-Stream Recurrent Neural Networks.

Gesture spotting is an essential task for recognizing finger gestures used to control in-car touchless interfaces. Automated methods to achieve this task require to detect video segments where gestures are observed, to discard natural behaviors of users’ hands that may look as target gestures, and be able to work online. In this paper, we address these challenges with a recurrent neural architecture for online finger gesture spotting. We propose a multi-stream network merging hand and hand-location features, which help to discriminate target gestures from natural movements of the hand, since these may not happen in the same 3D spatial location. Our multi-stream recurrent neural network (RNN) recurrently learns semantic information, allowing to spot gestures online in long untrimmed video sequences. In order to validate our method, we collect a finger gesture dataset in an in-vehicle scenario of an autonomous car. 226 videos with more than 2100 continuous instances were captured with a depth sensor. On this dataset, our gesture spotting approach outperforms state-of-the-art methods with an improvement of about 10% and 15% of recall and precision, respectively. Furthermore, we demonstrated that by combining with an existing gesture classifier (a 3D Convolutional Neural Network), our proposal achieves better performance than previous hand gesture recognition methods.

Hand gesture recognition using multimodal data fusion and multiscale parallel convolutional neural network for human–robot interaction

AbstractHand gesture recognition plays an important role in human–robot interaction. The accuracy and reliability of hand gesture recognition are the keys to gesture‐based human–robot interaction tasks. To solve this problem, a method based on multimodal data fusion and multiscale parallel convolutional neural network (CNN) is proposed in this paper to improve the accuracy and reliability of hand gesture recognition. First of all, data fusion is conducted on the sEMG signal, the RGB image, and the depth image of hand gestures. Then, the fused images are generated to two different scale images by downsampling, which are respectively input into two subnetworks of the parallel CNN to obtain two hand gesture recognition results. After that, hand gesture recognition results of the parallel CNN are combined to obtain the final hand gesture recognition result. Finally, experiments are carried out on a self‐made database containing 10 common hand gestures, which verify the effectiveness and superiority of the proposed method for hand gesture recognition. In addition, the proposed method is applied to a seven‐degree‐of‐freedom bionic manipulator to achieve robotic manipulation with hand gestures.

Dynamic Gesture Controlled User Interface Expert HCI System using Adaptative Background Masking: An Aid to Prevent Cross Infections

Human-Computer Interaction (HCI) interfaces need unambiguous instructions in the form of mouse clicks or keyboard taps from the user and thus gets complex. To simplify this monotonous task, a real-time hand gesture recognition method using computer vision, image, and video processing techniques has been proposed. Controlling infections has turned out to be the major concern of the healthcare environment. Several input devices such as keyboards, mouse, touch screens can be considered as a breeding ground for various micro pathogens and bacteria. Direct use of hands as an input device is an innovative method for providing natural HCI ensuring minimal physical contact with the devices i.e., less transmission of bacteria and thus can prevent cross infections. Convolutional Neural Network (CNN) has been used for object detection and classification. CNN architecture for 3d object recognition has been proposed which consists of two models: 1) A detector, a CNN architecture for detection of gestures; and 2) A classifier, a CNN for classification of the detected gestures. By using dynamic hand gesture recognition to interact with the system, the interactions can be increased with the help of multidimensional use of hand gestures as compared to other input methods. The dynamic hand gesture recognition method focuses to replace the mouse for interaction with the virtual objects. This work centralises the efforts of implementing a method that employs computer vision algorithms and gesture recognition techniques for developing a low-cost interface device for interacting with objects in the virtual environment such as screens using hand gestures.

A Novel Detection and Recognition Method for Continuous Hand Gesture Using FMCW Radar

In this article, a novel method for continuous hand gesture detection and recognition is proposed based on a frequency modulated continuous wave (FMCW) radar. Firstly, we adopt the 2-Dimensional Fast Fourier Transform (2D-FFT) to estimate the range and Doppler parameters of the hand gesture raw data, and construct the range-time map (RTM) and Doppler-time map (DTM). Meanwhile, we apply the Multiple Signal Classification (MUSIC) algorithm to calculate the angle and construct the angle-time map (ATM). Secondly, a hand gesture detection method is proposed to segment the continuous hand gestures using a decision threshold. Thirdly, the central time-frequency trajectory of each hand gesture spectrogram is clustered using the k-means algorithm, and then the Fusion Dynamic Time Warping (FDTW) algorithm is presented to recognize the hand gestures. Finally, experiments show that the accuracy of the proposed hand gesture detection method can reach 96.17%. The hand gesture average recognition accuracy of the proposed FDTW algorithm is 95.83%, while its time complexity is reduced by more than 50%.

Hand sign recognition from depth images with multi-scale density features for deaf mute persons

Among many of the fastest growing research fields, sign language recognition is one of the top. Deaf and dumb community uses sign language to express their ideas or views. Sign Language is a methodical coded language where meanings are assigned to every gestures. Many techniques have been developed with the advancement of science and technology to minimize the problem for speech and hearing disabled. The mode of such communication is part of human computer interaction. Hand gesture plays an important role here. The interaction with computer through gesture removes the use of conventional input devices like mouse and keyboards. To create a strong interface between user and computer, recognition of gesture is important. In this paper, a hand gesture recognition method based on multiscale density features is proposed. Depth images of numerals of American Sign Language are considered in this work and recognition rate of 98.20% is obtained, which is comparable with related state-of-the art methods.

Skeleton-Based Dynamic Hand Gesture Recognition Using a Part-Based GRU-RNN for Gesture-Based Interface

Recent improvements in imaging sensors and computing units have led to the development of a range of image-based human-machine interfaces (HMIs). An important approach in this direction is the use of dynamic hand gestures for a gesture-based interface, and some methods have been developed to provide real-time hand skeleton generation from depth images for dynamic hand gesture recognition. Towards this end, we propose a skeleton-based dynamic hand gesture recognition method that divides geometric features into multiple parts and uses a gated recurrent unit-recurrent neural network (GRU-RNN) for each feature part. Because each divided feature part has fewer dimensions than an entire feature, the number of hidden units required for optimization is reduced. As a result, we achieved similar recognition performance as the latest methods with fewer parameters.

Study on User Interface for a Capacitive-Sensor Based Smart Device

In this paper, we designed HW / SW interfaces for processing the signals of capacitive sensors like Electric Potential Sensor (EPS) to detect the surrounding electric field disturbance as feature signals in motion recognition systems. We implemented a smart light control system with those interfaces. In the system, the on/off switch and brightness adjustment are controlled by hand gestures using the designed and fabricated interface circuits. PWM (Pulse Width Modulation) signals of the controller with a driver IC are used to drive the LED and to control the brightness and on/off operation. Using the hand-gesture signals obtained through EPS sensors and the interface HW/SW, we can not only construct a gesture instructing system but also accomplish the faster recognition speed by developing dedicated interface hardware including control circuitry. Finally, using the proposed hand-gesture recognition and signal processing methods, the light control module was also designed and implemented. The experimental result shows that the smart light control system can control the LED module properly by accurate motion detection and gesture classification.

Spatial temporal graph convolutional networks for skeleton-based dynamic hand gesture recognition

Hand gesture recognition methods play an important role in human-computer interaction. Among these methods are skeleton-based recognition techniques that seem to be promising. In literature, several methods have been proposed to recognize hand gestures with skeletons. One problem with these methods is that they consider little the connectivity between the joints of a skeleton, constructing simple graphs for skeleton connectivity. Observing this, we built a new model of hand skeletons by adding three types of edges in the graph to finely describe the linkage action of joints. Then, an end-to-end deep neural network, hand gesture graph convolutional network, is presented in which the convolution is conducted only on linked skeleton joints. Since the training dataset is relatively small, this work proposes expanding the coordinate dimensionality so as to let models learn more semantic features. Furthermore, relative coordinates are employed to help hand gesture graph convolutional network learn the feature representation independent of the random starting positions of actions. The proposed method is validated on two challenging datasets, and the experimental results show that it outperforms the state-of-the-art methods. Furthermore, it is relatively lightweight in practice for hand skeleton-based gesture recognition.

PutEMG-A Surface Electromyography Hand Gesture Recognition Dataset.

In this paper, we present a putEMG dataset intended for the evaluation of hand gesture recognition methods based on sEMG signal. The dataset was acquired for 44 able-bodied subjects and include 8 gestures (3 full hand gestures, 4 pinches and idle). It consists of uninterrupted recordings of 24 sEMG channels from the subject’s forearm, RGB video stream and depth camera images used for hand motion tracking. Moreover, exemplary processing scripts are also published. The putEMG dataset is available under a Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0). The dataset was validated regarding sEMG amplitudes and gesture recognition performance. The classification was performed using state-of-the-art classifiers and feature sets. An accuracy of 90% was achieved for SVM classifier utilising RMS feature and for LDA classifier using Hudgin’s and Du’s feature sets. Analysis of performance for particular gestures showed that LDA/Du combination has significantly higher accuracy for full hand gestures, while SVM/RMS performs better for pinch gestures. The presented dataset can be used as a benchmark for various classification methods, the evaluation of electrode localisation concepts, or the development of classification methods invariant to user-specific features or electrode displacement.

Monocular Multi-feature Fusion Hand Gesture Recognition Method Based on Multi-core Learning

Based on multi-core learning, this paper uses the method of skin color segmentation, fingertip distance detection and extraction and gesture contour extraction, which realizes the fusion of local features and overall features and makes the accuracy of hand gesture classification, reach 92%. Compared with traditional gesture recognition based on binocular camera, our algorithm using multi-core learning has the same accuracy as the recognition based on binocular camera, and the complexity and cost of our algorithm are lower. Thus, our algorithm has broader application prospects. Compared with gesture recognition based on monocular camera, our algorithm is more suitable for the background environment of gesture classification. In other words, it has better generalization ability.