Hand Gesture Recognition Model Research Articles

TQU-HG dataset and comparative study for hand gesture recognition of RGB-based images using deep learning

Hand gesture recognition has great applications in human-computer interaction (HCI), human-robot interaction (HRI), and supporting the deaf and mute. To build a hand gesture recognition model using deep learning (DL) with high results then needs to be trained on many data and in many different conditions and contexts. In this paper, we publish the TQU-HG dataset of large RGB images with low resolution (640×480) pixels, low light conditions, and fast speed (16 fps). TQU-HG dataset includes 60,000 images collected from 20 people (10 male, 10 female) with 15 gestures of both left and right hands. A comparative study with two branches: i) based on Mediapipe TML and ii) Based on convolutional neural networks (CNNs) (you only look once (YOLO); YOLOv5, YOLOv6, YOLOv7, YOLOv8, YOLO-Nas, single shot multiBox detector (SSD) VGG16, residual network (ResNet)18, ResNext50, ResNet152, ResNext50, MobileNet V3 small, and MobileNet V3 large), the architecture and operation of CNNs models are also introduced in detail. We especially fine-tune the model and evaluate it on TQU-HG and HaGRID datasets. The quantitative results of the training and testing are presented (F1-score of YOLOv8, YOLO-Nas, MobileNet V3 small, ResNet50 is 98.99%, 98.98%, 99.27%, 99.36%, respectively on the TQU-HG dataset and is 99.21%, 99.37%, 99.36%, 86.4%, 98.3%, respectively on the HaGRID dataset). The computation time of YOLOv8 is 6.19 fps on the CPU and 18.28 fps on the GPU.

A Hybrid Image Augmentation Technique for User- and Environment-Independent Hand Gesture Recognition Based on Deep Learning

This research stems from the increasing use of hand gestures in various applications, such as sign language recognition to electronic device control. The focus is the importance of accuracy and robustness in recognizing hand gestures to avoid misinterpretation and instruction errors. However, many experiments on hand gesture recognition are conducted in limited laboratory environments, which do not fully reflect the everyday use of hand gestures. Therefore, the importance of an ideal background in hand gesture recognition, involving only the signer without any distracting background, is highlighted. In the real world, the use of hand gestures involves various unique environmental conditions, including differences in background colors, varying lighting conditions, and different hand gesture positions. However, the datasets available to train hand gesture recognition models often lack sufficient variability, thereby hindering the development of accurate and adaptable systems. This research aims to develop a robust hand gesture recognition model capable of operating effectively in diverse real-world environments. By leveraging deep learning-based image augmentation techniques, the study seeks to enhance the accuracy of hand gesture recognition by simulating various environmental conditions. Through data duplication and augmentation methods, including background, geometric, and lighting adjustments, the diversity of the primary dataset is expanded to improve the effectiveness of model training. It is important to note that the utilization of the green screen technique, combined with geometric and lighting augmentation, significantly contributes to the model’s ability to recognize hand gestures accurately. The research results show a significant improvement in accuracy, especially with implementing the proposed green screen technique, underscoring its effectiveness in adapting to various environmental contexts. Additionally, the study emphasizes the importance of adjusting augmentation techniques to the dataset’s characteristics for optimal performance. These findings provide valuable insights into the practical application of hand gesture recognition technology and pave the way for further research in tailoring techniques to datasets with varying complexities and environmental variations.

FFCSLT: A Deep Learning Model for Traffic Police Hand Gesture Recognition Using Surface Electromyographic Signals

FFCSLT: A Deep Learning Model for Traffic Police Hand Gesture Recognition Using Surface Electromyographic Signals

Transferable non-invasive modal fusion-transformer (NIMFT) for end-to-end hand gesture recognition.

Objective.Recent studies have shown that integrating inertial measurement unit (IMU) signals with surface electromyographic (sEMG) can greatly improve hand gesture recognition (HGR) performance in applications such as prosthetic control and rehabilitation training. However, current deep learning models for multimodal HGR encounter difficulties in invasive modal fusion, complex feature extraction from heterogeneous signals, and limited inter-subject model generalization. To address these challenges, this study aims to develop an end-to-end and inter-subject transferable model that utilizes non-invasively fused sEMG and acceleration (ACC) data.Approach.The proposed non-invasive modal fusion-transformer (NIMFT) model utilizes 1D-convolutional neural networks-based patch embedding for local information extraction and employs a multi-head cross-attention (MCA) mechanism to non-invasively integrate sEMG and ACC signals, stabilizing the variability induced by sEMG. The proposed architecture undergoes detailed ablation studies after hyperparameter tuning. Transfer learning is employed by fine-tuning a pre-trained model on new subject and a comparative analysis is performed between the fine-tuning and subject-specific model. Additionally, the performance of NIMFT is compared to state-of-the-art fusion models.Main results.The NIMFT model achieved recognition accuracies of 93.91%, 91.02%, and 95.56% on the three action sets in the Ninapro DB2 dataset. The proposed embedding method and MCA outperformed the traditional invasive modal fusion transformer by 2.01% (embedding) and 1.23% (fusion), respectively. In comparison to subject-specific models, the fine-tuning model exhibited the highest average accuracy improvement of 2.26%, achieving a final accuracy of 96.13%. Moreover, the NIMFT model demonstrated superiority in terms of accuracy, recall, precision, and F1-score compared to the latest modal fusion models with similar model scale.Significance.The NIMFT is a novel end-to-end HGR model, utilizes a non-invasive MCA mechanism to integrate long-range intermodal information effectively. Compared to recent modal fusion models, it demonstrates superior performance in inter-subject experiments and offers higher training efficiency and accuracy levels through transfer learning than subject-specific approaches.

CNN-LSTM and post-processing for EMG-based hand gesture recognition

Hand Gesture Recognition (HGR) using electromyography (EMG) signals is a challenging problem due to the variability and noise in the signals across individuals. This study addresses this challenge by examining the effect of incorporating a post-processing algorithm, which filters the sequence of predictions and removes spurious labels, on the performance of a HGR model based on spectrograms and Convolutional Neural Networks (CNN). The study also compares CNN vs CNN-LSTM to assess the influence of the memory cells on the model. The EMG-EPN-612 dataset, which contains measurements of EMG signals for 5 hand gestures from 612 subjects, was used for training and testing. The results showed that the post-processing algorithm increased the recognition accuracy by 41.86% for the CNN model and 24.77% for the CNN-LSTM model. The inclusion of the memory cells increased accuracy by 3.29%, but at the cost of 53 times more learnables. The CNN-LSTM model with post-processing achieved a mean recognition accuracy of 90.55% (SD=9.45%). These findings suggest new paths for research in HGR architectures beyond the traditional focus on the classification and feature extraction stages. For reproducibility purposes, we made publicly available the source code in Github.

WITHDRAWN: Sewing-driving training optimization-enabled feature selection and hand gesture recognition with deep residual network using sEMG signal

An innovation of an accurate model along with low latency for hand gesture identification is still a huge challenge especially for rehabilitation. This research provides a unified solution for such limitations by constructing an efficient system for hand gesture recognition by employing designed Sewing Driving Training based Optimization (SDTO)_Deep Residual Network (DRN). In this work, pre-processing is done using Gaussian filtering, whereas feature selection is carried out based on SDTO. In order to attain efficient results, appropriate features are extracted at the feature extraction phase. The last step is the hand gesture recognition phase, which is accomplished effectively using DRN and the network is efficiently trained using designed SDTO and it is achieved by the integration of STBO and DTBO. The introduced SDTO_DRN model for hand gesture recognition has attained high accuracy of 0.943, TPR of 0.929, TNR of 0.919, PPV of 0.924, and NPV of 0.924.

Empowering human-robot interaction using sEMG sensor: Hybrid deep learning model for accurate hand gesture recognition

In this paper, a novel approach using a Henry Gas Solubility-based Stacked Convolutional Neural Network (HGS-SCNN) for hand gesture recognition using surface electromyography (sEMG) sensors is proposed. The stacked architecture of the CNN model helps to capture both low-level and high-level features, enabling effective representation learning. To begin, we generated a dataset comprising 600 samples of hand gestures. Next, we applied the Discrete Wavelet Transform (DWT) technique to extract features from the filtered sEMG signal. This step allowed us to capture both spatial and frequency information, enhancing the discriminative power of the extracted features. Extensive experiments are conducted to evaluate the performance of the proposed HGS-SCNN model. In addition, the obtained results are compared with state-of-the-art techniques, namely AOA-SCNN, GWO-SCNN, and WOA-SCNN. The comparative analysis demonstrates that the HGS-SCNN outperforms these existing methods, achieving an impressive accuracy of 99.3%. The experimental results validate the effectiveness of our proposed approach in accurately detecting hand gestures. The combination of DWT-based feature extraction and the HGS-SCNN model offers robust and reliable hand gesture recognition, thereby opening new possibilities for intuitive human-machine interaction and applications requiring gesture-based control.

GESTURE RECOGNITION OF MACHINE LEARNING AND CONVOLUTIONAL NEURAL NETWORK METHODS FOR KAZAKH SIGN LANGUAGE

Recently, there has been a growing interest in machine learning and neural networks among the public, largely due to advancements in technology which have led to improved methods of computer recognition of objects, sounds, texts, and other data types. As a result, human-computer interactions are becoming more natural and comprehensible to the average person. The progress in computer vision has enabled the use of increasingly sophisticated models for object recognition in images and videos, which can also be applied to recognize hand gestures. In this research, popular hand gesture recognition models, such as the Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and Support Vector Machine (SVM) were examined. These models vary in their approaches, processing time, and training data size. The important feature of this research work is the use of various machine learning algorithms and methods such as CNN, LSTM, and SVM. Experiments showed different results when training neural networks for sign language recognition in the Kazakh sign language based on the dactyl alphabet. This article provides a detailed description of each method, their respective purposes, and effectiveness in terms of performance and training. Numerous experimental results were recorded in a table, demonstrating the accuracy of recognizing each gesture. Additionally, specific hand gestures were isolated for testing in front of the camera to recognize the gesture and display the result on the screen. An important feature was the use of mathematical formulas and functions to explain the working principle of the machine learning algorithm, as well as the logical scheme and structure of the LSTM algorithm.

GESTURE RECOGNITION FOR INTERACTIVE DASHBOARD: A 3D CONVOLUTIONAL NEURAL NETWORKS WITH LONG-SHORT-TERM-MEMORY APPROACH

Gesture recognition technology has become increasingly popular in various fields due to its potential for controlling consumer devices, robotics, and even translating sign language. However, there is still a lack of practical applications in daily life, especially for consumer products. Previous studies primarily focused on gesture recognition and did not delve into exploring the potential applications of gesture recognition in human interaction. This work aims to address this gap by deploying a gesture recognition model into practical applications and exploring its potential in human interaction. The proposed model combines the use of 3D convolutional neural networks (3DCNN) and long-short-term-memory networks (LSTM) for spatiotemporal feature learning. Six common interactive dashboard gestures were extracted from the 20BN-Jester dataset to train the model. The validation accuracy reached 80.47% after ablation studies. Then, two test cases were conducted to evaluate the effectiveness of the model in controlling Spotify and the Artificial Intelligence of Things (AIoT) smart classroom. The response time for each recognition was approximately 100 to 150 milliseconds. The proposed research aims to develop a simple yet efficient model for hand gesture recognition, which has the potential to be applied in various fields beyond just music and smart classrooms. With the growing popularity of gesture recognition technology, this study contributes to the advancement of practical applications and product innovations that can be integrated into daily life, thereby making it more accessible to the general public.

Saliency-Driven Hand Gesture Recognition Incorporating Histogram of Oriented Gradients (HOG) and Deep Learning.

Hand gesture recognition is a vital means of communication to convey information between humans and machines. We propose a novel model for hand gesture recognition based on computer vision methods and compare results based on images with complex scenes. While extracting skin color information is an efficient method to determine hand regions, complicated image backgrounds adversely affect recognizing the exact area of the hand shape. Some valuable features like saliency maps, histogram of oriented gradients (HOG), Canny edge detection, and skin color help us maximize the accuracy of hand shape recognition. Considering these features, we proposed an efficient hand posture detection model that improves the test accuracy results to over 99% on the NUS Hand Posture Dataset II and more than 97% on the hand gesture dataset with different challenging backgrounds. In addition, we added noise to around 60% of our datasets. Replicating our experiment, we achieved more than 98% and nearly 97% accuracy on NUS and hand gesture datasets, respectively. Experiments illustrate that the saliency method with HOG has stable performance for a wide range of images with complex backgrounds having varied hand colors and sizes.

An enhanced artificial neural network for hand gesture recognition using multi-modal features

ABSTRACT The development of automatic application fields enhanced the advancement in Hand Gesture Recognition (HGR), which enhances human-computer interaction. In particular, hand gestures find a valuable application in hand games, virtual reality, and gesture-based signalling systems. Thus, detecting and recognising hand gestures gain technical significance. Although existing models highlight remarkable recognition outcomes, handling dynamic gestures is a challenging task. Hence, in this research, an enhanced flashfly optimisation-based artificial neural network is developed for the effective identification of hand gestures, and to overcome the existing challenges, the enhanced flash fly optimisation-based ANN technique is proposed. The main objective of the optimisation strategy is to effectively tune the parameters involved in the artificial neural network (ANN) by the attained optimal solution. The optimised ANN classifier is well trained by the pre-processed statistical and the Residual neural Networks (ResNet) features for consistently identifying hand movements. The attained performance of the enhanced flash fly optimisation-based ANN classifier during the representation of hand gestures based on the precision, recall, and F1 score is 0.940 %, 0.791 %, and 0.927 % for the training percentages 50 and 90. Hence, it is illustrated that the proposed HGR model exceeds the competent technique, demonstrating its effectiveness.

A comparison of EMG-based hand gesture recognition systems based on supervised and reinforcement learning

Hand gesture recognition (HGR) based on electromyography signals (EMGs) has been one of the most relevant research topics in the human–machine interfaces field in recent years. The HGR systems are aimed at identifying the moment in which a hand gesture was performed as well as the gesture category. To date, several HGR state-of-the-art methods are based mainly on supervised machine learning (ML) techniques. However, the use of reinforcement learning (RL) approaches to classify EMGs has not yet been thoroughly evaluated. Moreover, the behavior of HGR systems based on ML and RL methods on large datasets for user-general HGR systems is still an open research problem. In the present work, we compare a supervised learning with a reinforcement learning HGR system, which are composed of the following stages: pre-processing, feature extraction, classification, and post-processing. We compared the performance of using both a supervised and a reinforcement learning method to classify and recognize EMGs for six different hand gestures. We performed experiments by using training, validation, and test sets on the EMG-EPN-612 public dataset, and the results were evaluated for user-general HGR models. The final accuracy results on the test set demonstrate that the best model was obtained for the supervised learning method, reaching up to 90.49%±9.7% and 86.83%±11.30% for classification and recognition accuracy respectively. The results obtained in this work demonstrated that supervised learning methods outperform reinforcement learning methods for user-general HGR systems based on EMGs for the EMG-EPN-612 dataset distribution.

Intelligent hybrid hand gesture recognition system using deep recurrent neural network with chaos game optimisation

ABSTRACT Hand gesture recognition is considered an essential task in various human-computer interaction (HCI) applications. Therefore, developing a robust system for hand gesture recognition is a challenge. This work proposed a new hybrid approach named hybrid deep recurrent neural network (RNN) incorporated with a chaos game optimisation (CGO) algorithm for efficiently recognising hand gestures. The main objective of a hybrid recurrent neural network with chaos game optimisation (RNN-CGO) is to achieve the recognition of alphabet signs from 2D gesture images. The different stages included in this technique are pre-processing, feature extraction, feature selection, and classification. The proposed approach is executed using American Sign Language (ASL) dataset with the help of the PYTHON platform. The precision, accuracy, f1-score, and recall obtained for the proposed approach are 99.28%, 99.96%, 99.25%, and 99.28%, respectively. The inference time required for the proposed approach is 0.121 s. The proposed technique has less computational complexity. The results showed that the proposed approach is better than the existing hand gesture recognition model approaches.

Hand gesture recognition and voice conversion for deaf and Dumb

In this paper, we purpose a Hand gesture recognition model which can be used in real time application. This model is based on the mediapipe frame work of the google, Tensor flow in openCv and python and classification using feed forward neural network with keras model. The structure of the proposed work consists of 3 modules: Grabbing the frames, detecting hand landmarks and classification. The proposed model has the accuracy 95.7% at recognizing 10 kinds of hand gestures(Thumbs up, Thumbs down, Peace, Smile, Rock, Ok, Fist, livelong, call me, stop). A hand gesture recognition model that reacts rapidly and with generally acceptable accuracy is one of this work’s primary achievements and pre trained model for feature extraction. The unique approach of the suggested approach is that it detects hand landmarks using Google’s MediaPipe, which is faster and more accurate than traditional methods that rely on geometry, form, and edge data. For modelling sequence data and for recognising gestures, the LSTM model has proven to be quite successful.

Alignment-Enhanced Interactive Fusion Model for Complete and Incomplete Multimodal Hand Gesture Recognition.

Hand gesture recognition (HGR) based on surface electromyogram (sEMG) and Accelerometer (ACC) signals is increasingly attractive where fusion strategies are crucial for performance and remain challenging. Currently, neural network-based fusion methods have gained superior performance. Nevertheless, these methods typically fuse sEMG and ACC either in the early or late stages, overlooking the integration of entire cross-modal hierarchical information within each individual hidden layer, thus inducing inefficient inter-modal fusion. To this end, we propose a novel Alignment-Enhanced Interactive Fusion (AiFusion) model, which achieves effective fusion via a progressive hierarchical fusion strategy. Notably, AiFusion can flexibly perform both complete and incomplete multimodal HGR. Specifically, AiFusion contains two unimodal branches and a cascaded transformer-based multimodal fusion branch. The fusion branch is first designed to adequately characterize modality-interactive knowledge by adaptively capturing inter-modal similarity and fusing hierarchical features from all branches layer by layer. Then, the modality-interactive knowledge is aligned with that of unimodality using cross-modal supervised contrastive learning and online distillation from embedding and probability spaces respectively. These alignments further promote fusion quality and refine modality-specific representations. Finally, the recognition outcomes are set to be determined by available modalities, thus contributing to handling the incomplete multimodal HGR problem, which is frequently encountered in real-world scenarios. Experimental results on five public datasets demonstrate that AiFusion outperforms most state-of-the-art benchmarks in complete multimodal HGR. Impressively, it also surpasses the unimodal baselines in the challenging incomplete multimodal HGR. The proposed AiFusion provides a promising solution to realize effective and robust multimodal HGR-based interfaces.

Hardware Architecture Design for Hand Gesture Recognition System on FPGA

Hand gesture recognition (HGR) is one of the widely-used human-computer interaction technology. With HGR, the user can operate the interaction system without touching any devices. For a better experience, recognition accuracy and computational speed are especially important. In this work, a small-footprint HGR model and its hardware architecture design are proposed. The model first processes the hand segmentation and uses the feature to recognize the hand gesture. The model mainly consists of depthwise separable convolution to reduce the overall parameters and computations. We transfer the hand segmentation task with some features to the hand gesture recognition task as a single-stage model. Based on this hardware-efficient model, we propose the hardware architecture of the whole neural model including depthwise convolution, pointwise convolution, batch normalization, and max-pooling. We also demonstrate it on the evaluation board. The whole system is implemented on the Xilinx ZCU106 evaluation board. The implemented system can achieve the performance of 52.6 fps and 65.6 GOPS based on the evaluation board.

An Underwater Human-Robot Interaction Using a Visual-Textual Model for Autonomous Underwater Vehicles.

The marine environment presents a unique set of challenges for human-robot interaction. Communicating with gestures is a common way for interacting between the diver and autonomous underwater vehicles (AUVs). However, underwater gesture recognition is a challenging visual task for AUVs due to light refraction and wavelength color attenuation issues. Current gesture recognition methods classify the whole image directly or locate the hand position first and then classify the hand features. Among these purely visual approaches, textual information is largely ignored. This paper proposes a visual-textual model for underwater hand gesture recognition (VT-UHGR). The VT-UHGR model encodes the underwater diver's image as visual features, the category text as textual features, and generates visual-textual features through multimodal interactions. We guide AUVs to use image-text matching for learning and inference. The proposed method achieves better performance than most existing purely visual methods on the dataset CADDY, demonstrating the effectiveness of using textual patterns for underwater gesture recognition.

Two-stream fusion model using 3D-CNN and 2D-CNN via video-frames and optical flow motion templates for hand gesture recognition.

Hand gestures are useful tools for many applications in the human-computer interaction community. Here, the objective is to track the movement of the hand irrespective of the shape, size and color of the hand. And, for this, a motion template guided by optical flow (OFMT) is proposed. OFMT is a compact representation of the motion information of a gesture encoded into a single image. Recently, deep networks have shown impressive improvements as compared to conventional hand-crafted feature-based techniques. Moreover, it is seen that the use of different streams with informative input data helps to increase the recognition performance. This work basically proposes a two-stream fusion model for hand gesture recognition. The two-stream network consists of two layers—a 3D convolutional neural network (C3D) that takes gesture videos as input and a 2D-CNN that takes OFMT images as input. C3D has shown its efficiency in capturing spatiotemporal information of a video, whereas OFMT helps to eliminate irrelevant gestures providing additional motion information. Though each stream can work independently, they are combined with a fusion scheme to boost the recognition results. We have shown the efficiency of the proposed two-stream network on two databases.

A 3DCNN-LSTM Multi-Class Temporal Segmentation for Hand Gesture Recognition

This paper introduces a multi-class hand gesture recognition model developed to identify a set of hand gesture sequences from two-dimensional RGB video recordings, using both the appearance and spatiotemporal parameters of consecutive frames. The classifier utilizes a convolutional-based network combined with a long-short-term memory unit. To leverage the need for a large-scale dataset, the model deploys training on a public dataset, adopting a technique known as transfer learning to fine-tune the architecture on the hand gestures of relevance. Validation curves performed over a batch size of 64 indicate an accuracy of 93.95% (±0.37) with a mean Jaccard index of 0.812 (±0.105) for 22 participants. The fine-tuned architecture illustrates the possibility of refining a model with a small set of data (113,410 fully labelled image frames) to cover previously unknown hand gestures. The main contribution of this work includes a custom hand gesture recognition network driven by monocular RGB video sequences that outperform previous temporal segmentation models, embracing a small-sized architecture that facilitates wide adoption.

Helping Hearing-Impaired in Emergency Situations: A Deep Learning-Based Approach

Hearing-impaired people use sign language to express their thoughts and emotions and reinforce information delivered in daily conversations. Though they make a significant percentage of any population, the majority of people can’t interact with them due to limited or no knowledge of sign languages. Sign language recognition aims to detect the significant motions of the human body, especially hands, analyze them and understand them. Such systems may become life-saving when hearing-challenged people are in desperate situations like heart attacks, accidents, etc. In the present study, deep learning-based hand gesture recognition models are developed to accurately predict the emergency signs of Indian Sign Language (ISL). The dataset used contains the videos for eight different emergency situations. Several frames were extracted from the videos and are fed to three different models. Two models are designed for classification, while one is an object detection model, applied after annotating the frames. The first model consists of a three-dimensional convolutional neural network (3D CNN), while the second comprises of a pre-trained VGG-16 and a recurrent neural network with a long short-term memory (RNN-LSTM) scheme. The last model is based on YOLO (You Only Look Once) v5, an advanced object detection algorithm. The prediction accuracies of the classification models were 82% and 98%, respectively. YOLO based model outperformed the rest and achieved an impressive mean average precision of 99.6%.