Hand Gesture Classification Research Articles

A novel neuroevolution model for emg-based hand gesture classification

A novel neuroevolution model for emg-based hand gesture classification

Fighting for a future free from violence: A framework for real-time detection of “Signal for Help”

In April 2020, by the start of isolation all around the world to counter the spread of COVID-19, an increase in violence against women and kids has been observed such that it has been named The Shadow Pandemic. To fight against this phenomenon, a Canadian foundation proposed the “Signal for Help” gesture to help people in danger to alert others of being in danger, discreetly. Soon, this gesture became famous among people all around the world, and even after COVID-19 isolation, it has been used in public places to alert them of being in danger and abused. However, the problem is that the signal works if people recognize it and know what it means. To address this challenge, we present a workflow for real-time detection of “Signal for Help” based on two lightweight CNN architectures, dedicated to hand palm detection and hand gesture classification, respectively. Moreover, due to the lack of a “Signal for Help” dataset, we create the first video dataset representing the “Signal for Help” hand gesture for detection and classification applications which includes 200 videos. While the hand-detection task is based on a pre-trained network, the classifying network is trained using the publicly available Jesture dataset, including 27 classes, and fine-tuned with the “Signal for Help” dataset through transfer learning. The proposed platform shows an accuracy of 91.25% with a video processing capability of 16 fps executed on a machine with an Intel i9-9900K@3.6 GHz CPU, 31.2 GB memory, and NVIDIA GeForce RTX 2080 Ti GPU, while it reaches 6 fps when running on Jetson Nano NVIDIA developer kit as an embedded platform. The high performance and small model size of the proposed approach ensure great suitability for resource-limited devices and embedded applications which has been confirmed by implementing the developed framework on the Jetson Nano Developer Kit. A comparison between the developed framework and the state-of-the-art hand detection and classification models shows a negligible reduction in the validation accuracy, around 3%, while the proposed model required 4 times fewer resources for implementation, and inference has a speedup of about 50% on Jetson Nano platform, which make it highly suitable for embedded systems. The developed platform as well as the created dataset are publicly available.

LSTM-AE for Domain Shift Quantification in Cross-day Upper-limb Motion Estimation Using Surface Electromyography.

Although deep learning (DL) techniques have been extensively researched in upper-limb myoelectric control, system robustness in cross-day applications is still very limited. This is largely caused by non-stable and time-varying properties of surface electromyography (sEMG) signals, resulting in domain shift impacts on DL models. To this end, a reconstruction-based method is proposed for domain shift quantification. Herein, a prevalent hybrid framework that combines a convolutional neural network (CNN) and a long short-term memory network (LSTM), i.e. CNN-LSTM, is selected as the backbone. The paring of auto-encoder (AE) and LSTM, abbreviated as LSTM-AE, is proposed to reconstruct CNN features. Based on reconstruction errors (RErrors) of LSTM-AE, domain shift impacts on CNN-LSTM can be quantified. For a thorough investigation, experiments were conducted in both hand gesture classification and wrist kinematics regression, where sEMG data were both collected in multi-days. Experiment results illustrate that, when the estimation accuracy degrades substantially in between-day testing sets, RErrors increase accordingly and can be distinct from those obtained in within-day datasets. According to data analysis, CNN-LSTM classification/regression outcomes are strongly associated with LSTM-AE errors. The average Pearson correlation coefficients could reach -0.986±0.014 and -0.992±0.011, respectively.

In-Situ Measuring sEMG and Muscle Shape Change with a Flexible and Stretchable Hybrid Sensor for Hand Gesture Recognition.

The accurate recognition of hand motion intentions is an essential prerequisite for efficient human-machine interaction (HMI) systems such as multifunctional prostheses and rehabilitation robots. Surface electromyography (sEMG) signals and muscle shape change (MSC) signals which are usually detected with different types of sensors have been used for human hand motion intention recognition. However, using different sensors to measure sEMG and MSC respectively, it would be inconvenient and add deploying difficulty for human-machine interaction systems. In this study, a novel flexible and stretchable sensor was fabricated with a nano gold conductive material, which could simultaneously sense both sEMG and MSC signals. Accordingly, a wireless signal acquisition device was developed to record both sEMG and MSC signals with the fabricated hybrid sensors. The performance of the proposed in-situ dual-mode signal measurement (IDSM) system was evaluated by the recording signal quality and the accuracy of hand gesture recognition. The results demonstrated that by using two pairs of the hybrid sensors, the proposed IDSM system could obtain two-channel sEMG at a noise level of about 0.89 μVrms and four-channel MSC with a resolution of about 0.1 Ω. For a recognition task of 11 classes of hand gestures, the results showed that only with two pairs of the hybrid sensors, the average accuracy over all the subjects was 95.6 ± 2.9%, which was about 7% higher than that with two-channel sEMG and six-channel accelerometer signals. These results suggest that the proposed IDSM method would be an efficient way to simplify the human-machine interaction system with fewer sensors for high recognition accuracy of hand motions.

SSEC: Semantic Segmentation and Ensemble Classification Framework for Static Hand Gesture Recognition using RGB-D Data

Hand Gesture Recognition (HGR) refers to identifying various hand postures used in Sign Language Recognition (SLR) and Human Computer Interaction (HCI) applications. Complex background in uncontrolled environmental condition is the major challenging issue which impacts the recognition accuracy of HGR system. This can be effectively addressed by discarding the background using suitable semantic segmentation method, where it predicts the hand region pixels into foreground and rest of the pixels into background. In this paper, we have analyzed and evaluated well known semantic segmentation architectures for hand region segmentation using both RGB and depth data. Further, ensemble of segmented RGB and depth stream is used for hand gesture classification through probability score fusion. Experimental results shows that the proposed novel framework of Semantic Segmentation and Ensemble Classification (SSEC) is suitable for static hand gesture recognition and achieved F1-score of 88.91% on OUHANDS test dataset.

Analysis of the Efficacy of Real-Time Hand Gesture Detection with Hog and Haar-Like Features Using SVM Classification

The field of hand gesture recognition has recently reached new heights thanks to its widespread use in domains like remote sensing, robotic control, and smart home appliances, among others. Despite this, identifying gestures is difficult because of the intransigent features of the human hand, which make the codes used to decode them illegible and impossible to compare. Differentiating regional patterns is the job of pattern recognition. Pattern recognition is at the heart of sign language. People who are deaf or mute may understand the spoken language of the rest of the world by learning sign language. Any part of the body may be used to create signs in sign language. The suggested system employs a gesture recognition system trained on Indian sign language. The methods of preprocessing, hand segmentation, feature extraction, gesture identification, and classification of hand gestures are discussed in this work as they pertain to hand gesture sign language. A hybrid approach is used to extract the features, which combines the usage of Haar-like features with the application of Histogram of Oriented Gradients (HOG).The SVM classifier is then fed the characteristics it has extracted from the pictures in order to make an accurate classification. A false rejection error rate of 8% is achieved while the accuracy of hand gesture detection is improved by 93.5%.

How do sEMG segmentation parameters influence pattern recognition process? An approach based on wearable sEMG sensor

Processing surface electromyography (sEMG) data in real-time to control robotic devices in applications involving upper-limb prostheses is challenging, especially when the problem involves multi-class recognition. A typical processing pipeline in this kind of system presents the following steps: segmentation, feature extraction, and classification. An under-exploited issue is the segmentation step, usually a windowing technique whose parameters are mostly heuristically defined. In this context, this work examines some segmentation parameters and how it affects the accuracy in the classification of hand gestures using a commercial sEMG wearable device. The main contribution of the work is to point out recommendations and insights about the segmentation step. Our findings show that for most feature sets and classifiers, there is no significant difference in using window lengths over 500 ms; however, for some cases, this value can be smaller without depreciating the accuracy. Moreover, there is no significant difference in using the truncated signal, which means using the first half of the signal in relation to the whole signal; no gesture is privileged in relation to the others due to the change in the segmentation parameters; data augmentation caused by changes on segmentation could affect the classification error; and the best performance was observed using TD9 feature set, SVMR classifier with an overlap fraction of 25%.

Dynamic Hand Gesture Classification Based on Multichannel Radar Using Multistream Fusion 1-D Convolutional Neural Network

Radar-based dynamic hand gesture classification has been an active research field in recent years. Deep learning methods using radar sensors are widely used to classify dynamic hand gestures. The existing deep learning methods need the overhead of performing the slow- and fast-time Fourier transforms to obtain the various spectrum images as the input of the deep convolutional neural network (DCNN). In this article, a dynamic hand gesture classification method based on the multichannel radar using a multistream fusion 1-D convolutional neural network (MSF-1-D-CNN) is proposed. The proposed MSF-1-D-CNN has four branches in parallel, and each branch has the inception modules to extract features from the raw echo data of each receiving antenna. Then, the extracted features from each branch are concatenated, and the long short-term memory (LSTM) layer is utilized to extract the temporal characteristic of the concatenated features. Finally, the dense layer with the softmax function is utilized to obtain the classification result of hand gestures. The experimental results show that, compared with existing methods, the proposed method using multichannel radar data can provide improved classification accuracy when the hand has a large incident angle and distance from the radar. Moreover, the proposed method can also reduce the network parameters and computational complexity, which has the potential to be implemented on commercial embedded systems.

Hand Gesture Identification for Improving Accuracy Using Convolutional Neural Network(CNN)

Hand gestures are a type of non-verbal communication that uses visible body movementsto convey important messages. This paper presents a much better approach of hand gestureprediction. Image Identification is an important step in most of the modern hand gestureprediction system. A convolutional neural network are used for improving the accuracy of thesystem. Proposed system tested for large number of hand gesture images using Tensor flowtool. . The convolutional neural network (ConvNet) is a deep learning algorithm for learningand classifying hand gestures and achieved accuracy 93.61%.

Exploiting domain transformation and deep learning for hand gesture recognition using a low-cost dataglove

Hand gesture recognition is one of the most widely explored areas under the human–computer interaction domain. Although various modalities of hand gesture recognition have been explored in the last three decades, in recent years, due to the availability of hardware and deep learning algorithms, hand gesture recognition research has attained renewed momentum. In this paper, we evaluate the effectiveness of a low-cost dataglove for classifying hand gestures in the light of deep learning. We have developed a cost-effective dataglove using five flex sensors, an inertial measurement unit, and a powerful microcontroller for onboard processing and wireless connectivity. We have collected data from 25 subjects for 24 static and 16 dynamic American sign language gestures for validating our system. Moreover, we proposed a novel Spatial Projection Image-based technique for dynamic hand gesture recognition. We also explored a parallel-path neural network architecture for handling multimodal data more effectively. Our method produced an F1-score of 82.19% for static gestures and 97.35% for dynamic gestures from a leave-one-out-cross-validation approach. Overall, this study demonstrates the promising performance of a generalized hand gesture recognition technique in hand gesture recognition. The dataset used in this work has been made publicly available.

ASL Hand Gesture Classification and Localization Using Deep Ensemble Neural Network

ASL Hand Gesture Classification and Localization Using Deep Ensemble Neural Network

Optimization of HD-sEMG-Based Cross-Day Hand Gesture Classification by Optimal Feature Extraction and Data Augmentation

Human–machine interaction requires accurate recognition of human intentions (e.g., via hand gestures). Here, we assessed the cross-day robustness of widely used hand gesture classification techniques applied to high-density surface electromyogram (HD-sEMG) signals (256 channels). Our evaluation covered techniques in each stage of the classification framework: first, 50 temporal-spectral-spatial domain features, second, 15 feature optimization techniques, and third, seven classifiers. Moreover, although HD-sEMG provides sufficient neuromuscular information, some of the channels may present low signal-to-noise ratio and should therefore be treated as outliers. Accordingly, we performed our evaluation with, first, all outlier channels retained, and second, removal of the features corresponding to poor-quality channels and substitution with interpolated values from neighbor channels. The impact of sliding window and data augmentation was also investigated. We examined the results on a 35-gesture classification task using HD-sEMG acquired from 20 subjects on two sessions in separate days. The results showed that interpolation of features from outlier channels significantly improved the performance in most cases. Use of a sliding window and of data augmentation contributed to a higher classification accuracy. For the classification of 11 selected gestures of common daily use, the support vector machine classifier achieved the highest classification accuracy of 91.9% in a cross-day validation protocol using an optimal combination of 13 features (each extracted from sliding windows), feature optimization by linear discriminant analysis, and data augmentation. Our work can serve as a technique-screening tool on cross-day applications of human–machine interactions.

Development of a gesture-based system of interaction with the interface of adaptive information systems

The implementation of gesture control is an actual direction in the field of organizing interaction with adaptive information systems. This study considers the development of a neural network-based gesture control system for hand recognition and gesture classification. When implementing such systems, it is necessary to ensure the maximum accuracy of gesture recognition, as well as high performance (video stream processing speed). At the first stage of the study, the task of optimizing gesture control system was posed, including analysis of the information processing algorithms used for optimization according to the criteria of classification accuracy and performance. In the course of solving the optimization problem, the structure and parameters of the neural network were determined that provide the best accuracy indicators, and the necessary changes were made in the software of the gesture control system to achieve the required level of performance. The developed software for the gestural interaction system was tested when integrating with the interface of adaptive information systems implemented on the basis of the Unity game engine. The results obtained can be used in the implementation of interfaces for interaction with virtual reality with a high level of immersion.

Static hand gesture recognition for American sign language using neuromorphic hardware

In this paper, we develop four spiking neural network (SNN) models for two static American sign language (ASL) hand gesture classification tasks, i.e., the ASL alphabet and ASL digits. The SNN models are deployed on Intel’s neuromorphic platform, Loihi, and then compared against equivalent deep neural network (DNN) models deployed on an edge computing device, the Intel neural compute stick 2 (NCS2). We perform a comprehensive comparison between the two systems in terms of accuracy, latency, power consumption, and energy. The best DNN model achieves an accuracy of 99.93% on the ASL alphabet dataset, whereas the best performing SNN model has an accuracy of 99.30%. For the ASL-digits dataset, the best DNN model achieves an accuracy of 99.76% accuracy while the SNN achieves 99.03%. Moreover, our obtained experimental results show that the Loihi neuromorphic hardware implementations achieve up to 20.64× and 4.10× reduction in power consumption and energy, respectively, when compared to NCS2.

Hand Gestures Classification Using Electrical Impedance Tomography Images

Human–computer communication using hand gestures has always been difficult. More than half a century ago, people used differentways of interactionwith computers from the early mediums such as perforated game cards. Nowadays, if a richer lexicon of gestures is given, people can communicate more effectively with computers. Machine learning is now used to recognize and classify hand gestures in amore preciseway. In order to increase the communication between computers and humans, we proposed a technique, which uses a wearable low-cost device to generate the electrical impedance tomography (EIT) images to recover the inner impedance structure of a user’s wrist. This is done by measuring the transverse impedance between all the 16 pairs of electrodesofwrist band that lie on the skin of the user hand. The proposedtechnique is enough to integrate the technology into the prototypewrist band tomonitor and classify gestures in real time. We have conducted a study of 16 gestures with a focus on gross hand and pinch finger gestures. The results evaluation shows that the gross hand gestures achieved 90% accuracy inwrist position, while pinch gestures achieved 93% accuracy.

Hand Gesture Recognition with Symmetric Pattern under Diverse Illuminated Conditions Using Artificial Neural Network

This paper investigated the effects of variant lighting conditions on the recognition process. A framework is proposed to improve the performance of gesture recognition under variant illumination using the luminosity method. To prove the concept, a workable testbed has been developed in the laboratory by using a Microsoft Kinect sensor to capture the depth images for the purpose of acquiring diverse resolution data. For this, a case study was formulated to achieve an improved accuracy rate in gesture recognition under diverse illuminated conditions. For data preparation, American Sign Language (ASL) was used to create a dataset of all twenty-six signs, evaluated in real-time under diverse lighting conditions. The proposed method uses a set of symmetric patterns as a feature set in order to identify human hands and recognize gestures extracted through hand perimeter feature-extraction methods. A Scale-Invariant Feature Transform (SIFT) is used in the identification of significant key points of ASL-based images with their relevant features. Finally, an Artificial Neural Network (ANN) trained on symmetric patterns under different lighting environments was used to classify hand gestures utilizing selected features for validation. The experimental results showed that the proposed system performed well in diverse lighting effects with multiple pixel sizes. A total aggregate 97.3% recognition accuracy rate is achieved across 26 alphabet datasets with only a 2.7% error rate, which shows the overall efficiency of the ANN architecture in terms of processing time.

Comparison and Evaluation of Machine Learning-Based Classification of Hand Gestures Captured by Inertial Sensors

Gesture recognition is a topic in computer science and language technology that aims to interpret human gestures with computer programs and many different algorithms. It can be seen as the way computers can understand human body language. Today, the main interaction tools between computers and humans are still the keyboard and mouse. Gesture recognition can be used as a tool for communication with the machine and interaction without any mechanical device such as a keyboard or mouse. In this paper, we present the results of a comparison of eight different machine learning (ML) classifiers in the task of human hand gesture recognition and classification to explore how to efficiently implement one or more tested ML algorithms on an 8-bit AVR microcontroller for on-line human gesture recognition with the intention to gesturally control the mobile robot. The 8-bit AVR microcontrollers are still widely used in the industry, but due to their lack of computational power and limited memory, it is a challenging task to efficiently implement ML algorithms on them for on-line classification. Gestures were recorded by using inertial sensors, gyroscopes, and accelerometers placed at the wrist and index finger. One thousand and eight hundred (1800) hand gestures were recorded and labelled. Six important features were defined for the identification of nine different hand gestures using eight different machine learning classifiers: Decision Tree (DT), Random Forests (RF), Logistic Regression (LR), Linear Discriminant Analysis (LDA), Support Vector Machine (SVM) with linear kernel, Naïve Bayes classifier (NB), K-Nearest Neighbours (KNN), and Stochastic Gradient Descent (SGD). All tested algorithms were ranged according to Precision, Recall, and F1-score (abb.: P-R-F1). The best algorithms were SVM (P-R-F1: 0.9865, 0.9861, and 0.0863), and RF (P-R-F1: 0.9863, 0.9861, and 0.0862), but their main disadvantage is their unusability for on-line implementations in 8-bit AVR microcontrollers, as proven in the paper. The next best algorithms have had only slightly poorer performance than SVM and RF: KNN (P-R-F1: 0.9835, 0.9833, and 0.9834) and LR (P-R-F1: 0.9810, 0.9810, and 0.9810). Regarding the implementation on 8-bit microcontrollers, KNN has proven to be inadequate, like SVM and RF. However, the analysis for LR has proved that this classifier could be efficiently implemented on targeted microcontrollers. Having in mind its high F1-score (comparable to SVM, RF, and KNN), this leads to the conclusion that the LR is the most suitable classifier among tested for on-line applications in resource-constrained environments, such as embedded devices based on 8-bit AVR microcontrollers, due to its lower computational complexity in comparison with other tested algorithms.

Static hand gesture recognition in sign language based on convolutional neural network with feature extraction method using ORB descriptor and Gabor filter

In this paper, a new structure of deep learning neural network is introduced to identify the static hand gesture in the sign language. The proposed structure includes the convolutional neural network (CNN) and the classical non-intelligent feature extraction method. In the proposed structure, the hand gesture image, after preprocessing and removing its background, passes through three different streams of feature extraction, to well extract of effective features and determine the hand gesture class. These three streams, that independently extract their own specific features, consist of three widely used methods in the hand gesture classification named CNN, Gabor filter and ORB feature descriptor. Then these features are merged and formed the final feature vector. By combining these efficient methods, in addition to achieving a very high accuracy in hand gestures classifying, the proposed structure becomes more resistant to uncertainties such as rotation and ambiguity in the hand gestures. Another prominent feature of the proposed structure is its comprehensiveness on different image databases, compared to the similar methods. The transfer learning technique demonstrates that the proposed structure has the ability to be used as a pre-trained structure for any type of image database. Finally, the proposed structure is applied to the three different databases of Massey, ASL Alphabet and ASL, which have 2520, 87,000 and 23,400 of hand gesture images, respectively. The results show the mean accuracy of the proposed structure for the Massey test set of 758 images, ASL with 7020 test images, and ASL Alphabet with 26,100 test images, at 99.92%, 99.8%, and 99.80% respectively.

Increased Accuracy on Image Classification of Game Rock Paper Scissors using CNN

Rock Paper Scissors is one of the most popular games in the world, because of their easy and simple way to play among young and elderly people. The point of this game is to do the draw or just to find out who loses or wins. The pandemic conditions made people unable to meet face-to-face and could only play this game virtually. To carry out this activity in a virtual way, this research facilitates a model in the form of image classification to distinguish the hand gestures s in the form of rock, paper, and scissors. This classification process utilizes the Convolutional Neural Network (CNN) method. This method is one type of artificial neural network in terms of image classification. CNN uses three stages, namely convolutional layer, pooling layer, and fully connected layer. The implementation of this method for hand gesture classification in the form of rock, scissors, and paper images in this study shows an increased average accuracy towards the previous study from 97.66% to 99%.

Deteksi Gestur Tangan Berbasis Pengolahan Citra

Hand sign language is a medium of communication for people with disabilities (deaf and speech impaired). However, in social practice, persons with disabilities may have to communicate with non-disable persons who do not understand sign language. These problems can be overcome with the help of translators or normal people learning sign language through existing media such as videos. Unfortunately, this method will probably cost a lot of money and time. In respons to this issue, the present study designed a sistem to detect hand gestures based on image processing. The method used is the You Only Look Once (YOLO) algorithm. The YOLO algorithm can detect and classify objects at once without being influenced by the light intensity and background of the object. This algorithm is a deep learning method that is more accurate than other deep learning methods. From this research, the system can detect and classify hand gestures with different backgrounds, light intensity, and distances with an accuracy rate above 90%.