Continuous Hand Gesture Research Articles

Continuous hand gesture segmentation and acknowledgement of hand gesture path for innovative effort interfaces

Human-computer interaction (HCI) has revolutionized the way we interact with computers, making it more intuitive and user-friendly. It is a dynamic field that has found it is applications in various industries, including multimedia and gaming, where hand gestures are at the forefront. The advent of ubiquitous computing has further heightened the interest in using hand gestures as input. However, recognizing continuous hand gestures presents a set of challenges, primarily stemming from the variable duration of gestures and the lack of clear starting and ending points. Our main objective is to propose a solution: the framework for “continuous palm motion analysis and retrieval” based on “Spatial-temporal and path knowledge”. Framework harnesses the power of cognitive deep learning networks (DLN), offering a significant advancement in the continuous hand gesture recognition domain. we conducted rigorous experiments using a diverse video dataset capturing hand gestures for boasting an impressive F-score of up to 0.99. The potential of our framework to significantly enhance the accuracy and reliability of hand gesture recognition in real-world applications.

Hand Gestures Recognition Model Using Adaptive Feature Extraction with Attention-Based Hybrid Deep Learning via Optimization Strategy

Research on continuous hand gesture recognition is motivated by applications of deaf people to provide effective communication between one another. Numerous techniques are adapted for effectively identifying the hand gesture. Moreover, the collection of data becomes a challenging task in the existing models. In some instances, hand gesture in low-resolution images becomes a complicated task. In this research work, a deep learning-based hand gesture detection model is designed using hand data points to uncover patterns that are used to convey information. Initially, hand gesture images are obtained from public sources. The collected inputs are preprocessed through Contrast-Limited Adaptive Histogram Equalization and filtering to eliminate the unnecessary blurs in the images. The preprocessed images undergo hand segmentation using four methods color space transformation, skin color detection, active contour, and morphological operation. The segmented image is subjected to data point feature extraction, where the Adaptive Weighted Scale-Invariant Feature Transform is utilized for further enhancement. The hand data point-extracted features are given to the recognition of hand gestures with the support of an attention-based hybrid network, where the Attention-based Hybrid 1D Convolutional Neural Network with Recurrent Neural Network recognizes the hand gestures from the hand data points. In the hybridization network, the values are optimized with the help of the investigated Adaptive Dandelion White Shark Optimizer to enhance recognition effectiveness. The test results are validated with the existing hand gesture recognition models using diverse evaluation metrics. The findings of the recommended method show 97% and 99% in terms of accuracy and NPV. Thus, the developed model is validated to outperform significant performance rather than the existing models.

Continuous Hand Gestures Detection and Recognition in Emergency Human-Robot Interaction Based on the Inertial Measurement Unit

Continuous Hand Gestures Detection and Recognition in Emergency Human-Robot Interaction Based on the Inertial Measurement Unit

Soft Wrist-Worn Multi-Functional Sensor Array for Real-Time Hand Gesture Recognition

Soft electronics have been widely applied to wearable electronics, hand gesture detection and human robot interaction (HRI). Soft wrist-worn sensor system (SWSS) with electromyography (EMG), strain/pressure sensing abilities has been created for continuous hand gesture recognition, which includes multi-source data sensing, data collection and process, and wireless communication. SWSS is conformal contact with skin surface as the softness of SWSS, and it would improve the wearability with the wrist during long-term hand gesture monitoring. Linear discriminant analysis (LDA) and support vector machines (SVM) algorithms are applied to hand gesture recognition with average accuracy 83.67%, 86.8% for Group #1 (wrist flexion (WF), wrist extension (WE), finger flexion (FF), finger extension (FE), radial deviation (RD), and ulnar deviation (UD)) and 84.71%, 88.53% for Group #2 (sign language 0–9 digits), respectively. It is found that the recognition accuracy has just a little decreased during the long-term hand gesture detection at different sessions (at initial time, one day after, three days after and seven days after). This paper demonstrated the feasibility of gesture recognition via SWSS on wrist, and it could be integrated into wrist-worn electronic system for human robot interaction.

A Novel Detection and Recognition Method for Continuous Hand Gesture Using FMCW Radar

In this article, a novel method for continuous hand gesture detection and recognition is proposed based on a frequency modulated continuous wave (FMCW) radar. Firstly, we adopt the 2-Dimensional Fast Fourier Transform (2D-FFT) to estimate the range and Doppler parameters of the hand gesture raw data, and construct the range-time map (RTM) and Doppler-time map (DTM). Meanwhile, we apply the Multiple Signal Classification (MUSIC) algorithm to calculate the angle and construct the angle-time map (ATM). Secondly, a hand gesture detection method is proposed to segment the continuous hand gestures using a decision threshold. Thirdly, the central time-frequency trajectory of each hand gesture spectrogram is clustered using the k-means algorithm, and then the Fusion Dynamic Time Warping (FDTW) algorithm is presented to recognize the hand gestures. Finally, experiments show that the accuracy of the proposed hand gesture detection method can reach 96.17%. The hand gesture average recognition accuracy of the proposed FDTW algorithm is 95.83%, while its time complexity is reduced by more than 50%.

LSTM‐based dynamic probability continuous hand gesture trajectory recognition

In the field of continuous hand-gesture trajectory recognition, aiming at the problems of existing a lot of noise for handwriting trajectories, and difficult to segment multiple continuous hand gestures accurately, a long short-term memory-based dynamic probability (DP-LSTM) method is proposed. Firstly, obtain the classification result for each sub-period in the whole time period by using LSTM; secondly, cluster the classification results by non-maximum suppression for trajectory algorithm to eliminate interference of invalid subsets; Finally, the end point of the valid trajectory is obtained according to the characteristics of the probability change, thus realising dynamic trajectory segmentation and recognition. In order to evaluate the performance of the DP-LSTM, this method is evaluated by using an Arabic numerals gesture database. The experiments show that the DP-LSTM has a high recognition rate for continuous hand gestures and can recognise its in real time.

FLGR: Fixed Length Gists Representation Learning for RNN-HMM Hybrid-Based Neuromorphic Continuous Gesture Recognition.

A neuromorphic vision sensors is a novel passive sensing modality and frameless sensors with several advantages over conventional cameras. Frame-based cameras have an average frame-rate of 30 fps, causing motion blur when capturing fast motion, e.g., hand gesture. Rather than wastefully sending entire images at a fixed frame rate, neuromorphic vision sensors only transmit the local pixel-level changes induced by the movement in a scene when they occur. This leads to advantageous characteristics, including low energy consumption, high dynamic range, a sparse event stream and low response latency. In this study, a novel representation learning method was proposed: Fixed Length Gists Representation (FLGR) learning for event-based gesture recognition. Previous methods accumulate events into video frames in a time duration (e.g., 30 ms) to make the accumulated image-level representation. However, the accumulated-frame-based representation waives the friendly event-driven paradigm of neuromorphic vision sensor. New representation are urgently needed to fill the gap in non-accumulated-frame-based representation and exploit the further capabilities of neuromorphic vision. The proposed FLGR is a sequence learned from mixture density autoencoder and preserves the nature of event-based data better. FLGR has a data format of fixed length, and it is easy to feed to sequence classifier. Moreover, an RNN-HMM hybrid was proposed to address the continuous gesture recognition problem. Recurrent neural network (RNN) was applied for FLGR sequence classification while hidden Markov model (HMM) is employed for localizing the candidate gesture and improving the result in a continuous sequence. A neuromorphic continuous hand gestures dataset (Neuro ConGD Dataset) was developed with 17 hand gestures classes for the community of the neuromorphic research. Hopefully, FLGR can inspire the study on the event-based highly efficient, high-speed, and high-dynamic-range sequence classification tasks.

Sensor-Based Continuous Hand Gesture Recognition by Long Short-Term Memory

This article aims to present a novel sensor-based continuous hand gesture recognition algorithm by long short-term memory (LSTM). Only the basic accelerators and/or gyroscopes are required by the algorithm. Given a sequence of input sensory data, a many-to-many LSTM scheme is adopted to produce an output path. A maximum a posteriori estimation is then carried out based on the observed path to obtain the final classification results. A prototype system based on smartphones has been implemented for the performance evaluation. Experimental results show that the proposed algorithm is an effective alternative for robust and accurate hand-gesture recognition.

Latern: Dynamic Continuous Hand Gesture Recognition Using FMCW Radar Sensor

Recently, hand gesture recognition systems have become increasingly interesting to researchers in the field of human–computer interfaces. Real-world systems for human dynamic hand gesture recognition is challenging as: 1) the system must be robust to various conditions; 2) there is a rich diversity in how people perform hand gestures, making hand gesture recognition difficult; and 3) the system must detect and recognize hand gestures continuously using unsegmented input streams in order to avoid noticeable lag between performing a gesture and its classification. In this paper, to address these challenges, we present Latern, a novel system for dynamic continuous hand gesture recognition based on a frequency-modulated continuous wave radar sensor. The radar system does not depend on lighting, noise, or atmospheric conditions. We employ a recurrent 3-D convolutional neural network to perform the classification of dynamic hand gestures. To enhance the processing performance, a connectionist temporal classification algorithm is used to train the network to predict class labels from inprogress gestures in unsegmented input streams. The experimental results show that Latern is able to achieve high recognition rates of 96%, which is higher than state-of-the-art hand gesture recognition systems. In addition, the conclusion in this paper can be used for a real-time hand gesture recognition system design.

Structured dynamic time warping for continuous hand trajectory gesture recognition

Continuous hand gesture recognition is an important area of HCI and challenged by various writing habits and unconstrained hand movement. In this paper, we propose a Structured Dynamic Time Warping (SDTW) approach for continuous hand trajectory recognition. We first propose an automatic continuous trajectory segmentation approach which combines templates and velocity information to spot the beginning and ending points in hand gesture trajectories. Then we assign different weights to feature sequences based on the structured information, from the positions of corner points in the arbitrary trajectories. Finally, we evaluate the SDTW on the Continuous Letter Trajectory (CLT) database. Experimental results show that the proposed approach is robust to the diversity of same handwritten letter, and significantly outperforms state-of-the-art approaches.

Continuous hand gesture recognition based on trajectory shape information

In this paper, we propose a continuous hand gesture recognition method based on trajectory shape information. A key issue in recognizing continuous gestures is that performance of conventional recognition algorithms may be lowered by such factors as, unknown start and end points of a gesture or variations in gesture duration. These issues become particularly difficult for those methods that rely on temporal information. To alleviate the issues of continuous gesture recognition, we propose a framework that simultaneously performs both segmentation and recognition. Each component of the framework applies shape-based information to ensure robust performance for gestures with large temporal variation. A gesture trajectory is divided by a set of key frames by thresholding its tangential angular change. Variable-sized trajectory segments are then generated using the selected key frames. For recognition, these trajectory segments are examined to determine whether the segment belongs to a class among intended gestures or a non-gesture class based on fusion of shape information and temporal features. In order to assess performance, the proposed algorithm was evaluated with a database of digit hand gestures. The experimental results indicate that the proposed algorithm has a high recognition rate while maintaining its performance in the presence of continuous gestures.

Survey on 3D Hand Gesture Recognition

Three-dimensional hand gesture recognition has attracted increasing research interests in computer vision, pattern recognition, and human-computer interaction. The emerging depth sensors greatly inspired various hand gesture recognition approaches and applications, which were severely limited in the 2D domain with conventional cameras. This paper presents a survey of some recent works on hand gesture recognition using 3D depth sensors. We first review the commercial depth sensors and public data sets that are widely used in this field. Then, we review the state-of-the-art research for 3D hand gesture recognition in four aspects: 1) 3D hand modeling; 2) static hand gesture recognition; 3) hand trajectory gesture recognition; and 4) continuous hand gesture recognition. While the emphasis is on 3D hand gesture recognition approaches, the related applications and typical systems are also briefly summarized for practitioners.

A Continuous Hand Gestures Recognition Technique for Human-Machine Interaction Using Accelerometer and Gyroscope Sensors

Recent advances in smart devices have sustained them as a better alternative for the design of human–machine interaction (HMI), because they are equipped with accelerometer sensor, gyroscope sensor, and an advanced operating system. This paper presents a continuous hand gestures recognition technique that is capable of continuous recognition of hand gestures using three-axis accelerometer and gyroscope sensors in a smart device. To reduce the influence of unstableness of a hand making the gesture and compress the data, a gesture coding algorithm is developed. An automatic gesture spotting algorithm is developed to detect the start and end points of meaningful gesture segments. Finally, a gesture is recognized by comparing the gesture code with gesture database using dynamic time warping algorithm. In addition, a prototype system is developed to recognize the continuous hand gestures-based HMI. With the smartphone, the user is able to perform the predefined gestures and control smart appliances using the Samsung AllShare protocol.

Dynamic Gesture Recognition based on Image Sequence

This paper proposed an algorithm for 3D hands tracking on the learned hierarchical latent variable space, which employs a Hierarchical Gaussian Process Latent Variable Model(HGPLVM) to learn the hierarchical latent space of hands motion and the nonlinear mapping from the hierarchical latent space to the pose space simultaneously. Nonlinear mappings from the hierarchical latent space to the space of hand images are constructed using radial basis function interpolation method. With these mappings, particles can be projected into hand images and measured in the image space directly. Particle filters with fewer particles are used to track the hand on the learned hierarchical low-dimensional space. Then the Hierarchical Conditional Random Field, which can capture extrinsic class dynamics and learn the relationship between motions of hand parts and different hand gestures simultaneously, is presented to model the continuous hand gestures. Experimental results show that our proposed method can track articulated hand robustly and approving recognition performance has also been achieved on the user-defined hand gesture dataset.

Adaptation to visual feedback delay in a redundant motor task.

The goal of this study was to examine the reorganization of hand movements during adaptation to delayed visual feedback in a novel and redundant environment. In most natural behaviors, the brain must learn to invert a many-to-one map from high-dimensional joint movements and muscle forces to a low-dimensional goal. This spatial "inverse map" is learned by associating motor commands to their low-dimensional consequences. How is this map affected by the presence of temporal delays? A delay presents the brain with a new set of kinematic data, and, because of redundancy, the brain may use these data to form a new inverse map. We consider two possible responses to a novel visuomotor delay. In one case, the brain updates the previously learned spatial map, building a new association between motor commands and visual feedback of their effects. In the alternative case, the brain preserves the original map and learns to compensate the delay by a temporal shift of the motor commands. To test these alternative possibilities, we developed a virtual reality game in which subjects controlled the two-dimensional coordinates of a cursor by continuous hand gestures. Two groups of subjects tracked a target along predictable paths by wearing an instrumented data glove that recorded finger motions. The 19-dimensional glove signals controlled a cursor on a 2-dimensional computer display. The experiment was performed on 2 consecutive days. On the 1st day, subjects practiced tracking movements without delay. On the 2nd day, the test group performed the same task with a delay of 300 ms between the glove signals and the cursor display, whereas the control group continued practicing the nondelayed trials. We found evidence that to compensate for the delay, the test group relied on the coordination patterns established during the baseline, e.g., their hand-to-cursor inverse map was robust to the delay perturbation, which was counteracted by an anticipation of the motor command.

A novel set of features for continuous hand gesture recognition

Applications requiring the natural use of the human hand as a human–computer interface motivate research on continuous hand gesture recognition. Gesture recognition depends on gesture segmentation to locate the starting and end points of meaningful gestures while ignoring unintentional movements. Unfortunately, gesture segmentation remains a formidable challenge because of unconstrained spatiotemporal variations in gestures and the coarticulation and movement epenthesis of successive gestures. Furthermore, errors in hand image segmentation cause the estimated hand motion trajectory to deviate from the actual one. This research moves toward addressing these problems. Our approach entails using gesture spotting to distinguish meaningful gestures from unintentional movements. To avoid the effects of variations in a gesture’s motion chain code (MCC), we propose instead to use a novel set of features: the (a) orientation and (b) length of an ellipse least-squares fitted to motion-trajectory points and (c) the position of the hand. The features are designed to support classification using conditional random fields. To evaluate the performance of the system, 10 participants signed 10 gestures several times each, providing a total of 75 instances per gesture. To train the system, 50 instances of each gesture served as training data and 25 as testing data. For isolated gestures, the recognition rate using the MCC as a feature vector was only 69.6 % but rose to 96.0 % using the proposed features, a 26.1 % improvement. For continuous gestures, the recognition rate for the proposed features was 88.9 %. These results show the efficacy of the proposed method.

A novel approach for dynamic hand gesture recognition using contour-based similarity images

A novel approach is proposed for the recognition of moving hand gestures based on the representation of hand motions as contour-based similarity images (CBSIs). The CBSI was constructed by calculating the similarity between hand contours in different frames. The input CBSI was then matched with CBSIs in the database to recognize the hand gesture. The proposed continuous hand gesture recognition algorithm can simultaneously divide the continuous gestures into disjointed gestures and recognize them. No restrictive assumptions were considered for the motion of the hand between the disjointed gestures. The proposed algorithm was tested using hand gestures from American Sign Language and the results showed a recognition rate of 91.3% for disjointed gestures and 90.4% for continuous gestures. The experimental results illustrate the efficiency of the algorithm for noisy videos.

A COLORED FINGER TIP-BASED TRACKING METHOD FOR CONTINUOUS HAND GESTURE RECOGNITION

Hand gesture recognition system can be used for human-computer interaction (HCI). Proper hand segmentation from the background and other body parts of the video is the primary requirement for the design of a hand-gesture based application. These video frames can be captured from a low cost webcam (camera) for use in a vision based gesture recognition technique. This paper discusses about the continuous hand gesture recognition. The aim of this paper is to report a robust and efficient hand segmentation algorithm where a new method, wearing glove on the hand is utilized. After that a new idea called “Finger-Pen”, is developed by segmenting only one finger from the hand for proper tracking. In this technique only a finger tip is segmented in spite of the full hand part. Hence this technique allows the hand (excepting the segmented finger tip) to move freely during the tracking time also. Problems such as skin colour detection, complexity from large numbers of people in front of the camera, complex background removal and variable lighting condition are found to be efficiently handled by the system. Noise present in the segmented image due to dynamic background can be removed with the help of this adaptive technique which is found to be effective for the application conceived.

Spatio-temporal 3D surface matching for hand gesture recognition using ICP algorithm

With emerging new applications like virtual reality, different algorithms for human action and gesture recognition have been proposed. In this paper, a new method for the recognition of moving hand gestures is presented. The proposed algorithm is based on the representation of hand motion as spatio-temporal 3D surfaces. Then, 3D surface matching is used to recognize the hand gesture. To form the spatio-temporal 3D surface of hand motion, we first apply the necessary preprocessing to video frames and extract hand contours. Then, by normalizing and overlapping hand contours in different frames, we construct spatio-temporal 3D surface of the hand gesture. To recognize hand gesture, we match the input 3D surface with surfaces in the database. For this purpose, we utilize ICP algorithm to find and compensate for 3D transformation between surfaces as well as the similarity measure between them. In real-world applications, hand motion is continuous and results in a sequence of disjointed hand gestures, which is called continuous hand gesture. To recognize continuous hand gestures, we propose an algorithm which first estimates probable disjointed gestures in the continuous gesture and then divides iteratively continuous gestures to true disjointed gestures. Finally, by applying a robust algorithm, the continuous gesture is recognized. We tested the proposed algorithm with hand gestures of American sign language and results showed the recognition rate of 94 % for disjointed gestures and 93.9 % for continuous gestures. The experimental results showed the efficiency of the proposed algorithm for hand gestures with noise as well.