Static Hand Gesture Recognition Research Articles

Robust static hand gesture recognition: harnessing sparsity of deeply learned features

Robust static hand gesture recognition: harnessing sparsity of deeply learned features

A Survey of Sign Language Recognition

Sign Language is mainly used by deaf (hard hearing) and dumb people to exchange information between their own community and with other people. It is a language where people use their hand gestures to communicate as they can't speak or hear. The goal of sign language recognition (SLR) is to identify acquired hand motions and to continue until related hand gestures are translated into text and speech. Here, static and dynamic hand gestures for sign language can be distinguished. The human community values both types of recognition, even if static hand gesture recognition is easier than dynamic hand gesture recognition. By creating Deep Neural Network designs (Convolution Neural Network designs), where the model will learn to detect the hand motions images throughout an epoch, we are using Deep Learning Computer Vision to recognize the hand gestures. After the model successfully recognizes the motion, an English text file is created that can subsequently be translated to speech. The user can choose from a variety of translations for this paragraph. This application can be used without an internet connection and is entirely offline. With this model's improved efficiency, communication will be easier for the deaf (hard of hearing) and disabled people. We shall discuss the use of deep learning for sign language recognition in this paper. Key Words: sign language, convolutional neural network, computer vision.

Static Hand Gesture Recognition Based on Millimeter-Wave Near-Field FMCW-SAR Imaging

To address the limitations of wireless sensing in static gesture recognition and the issues of Computer Vision’s dependence on lighting conditions, we propose a method that utilizes millimeter-wave near-field SAR (Synthetic Aperture Radar) imaging for static gesture recognition. First, a millimeter-wave near-field SAR imaging system is used to scan the defined static gestures to obtain data. Then, based on the distance plane, the three-dimensional gesture is divided into multiple two-dimensional planes, constructing an imaging dataset. Finally, an HOG (Histogram of Oriented Gradients) is used to extract features from the imaging results, PCA (Principal Component Analysis) is applied for feature dimensionality reduction, and RF (Random Forest) performs classification. Experimental verification shows that the proposed method achieves an average recognition precision of 97% in unobstructed situations and 93% in obstructed situations, providing an effective means for wireless-sensing-based static gesture recognition.

Static Hand Gesture Recognition Using Capacitive Sensing and Machine Learning

Automated hand gesture recognition is a key enabler of Human-to-Machine Interfaces (HMIs) and smart living. This paper reports the development and testing of a static hand gesture recognition system using capacitive sensing. Our system consists of a 6×18 array of capacitive sensors that captured five gestures-Palm, Fist, Middle, OK, and Index-of five participants to create a dataset of gesture images. The dataset was used to train Decision Tree, Naïve Bayes, Multi-Layer Perceptron (MLP) neural network, and Convolutional Neural Network (CNN) classifiers. Each classifier was trained five times; each time, the classifier was trained using four different participants' gestures and tested with one different participant's gestures. The MLP classifier performed the best, achieving an average accuracy of 96.87% and an average F1 score of 92.16%. This demonstrates that the proposed system can accurately recognize hand gestures and that capacitive sensing is a viable method for implementing a non-contact, static hand gesture recognition system.

Static hand gesture recognition method based on the Vision Transformer

Static hand gesture recognition method based on the Vision Transformer

Static hand gesture recognition using multi-dilated DenseNet-based deep learning architecture

ABSTRACT Hand Gesture Recognition (HGR), a Human-Robot Interaction (HRI) create user interfaces to enhance safety and control. Several approaches are introduced to recognize static hand gestures effectively, however the classification efficiency suffered due to various lightings like natural light, artificial light and dark rooms increases the difficulty of accurate HGR. To tackle these problems, a Multi-Dilated Convolution-based DenseNet (MDCDN) architecture, a combination of multi-dilated convolution and DenseNet is proposed which extracts the features automatically. Finally, the benefits of high-level deep learning techniques are leveraged for the gesture recognition from hand. Python is used for architecture evaluation. The outcome of proposed is estimated in terms of accuracy, recall, F-measure, precision, etc, using ASL, ISL, Massey and HSR real datasets. Each dataset contains huge amount of gesture classes and their pictures have an equal amount of uniform and complex backgrounds. The proposed results are promising and provide better performance than existing methods.

Using VGG Models with Intermediate Layer Feature Maps for Static Hand Gesture Recognition

A hand gesture recognition system provides a robust and innovative solution to nonverbal communication through human–computer interaction. Deep learning models have excellent potential for usage in recognition applications. To overcome related issues, most previous studies have proposed new model architectures or have fine-tuned pre-trained models. Furthermore, these studies relied on one standard dataset for both training and testing. Thus, the accuracy of these studies is reasonable. Unlike these works, the current study investigates two deep learning models with intermediate layers to recognize static hand gesture images. Both models were tested on different datasets, adjusted to suit the dataset, and then trained under different methods. First, the models were initialized with random weights and trained from scratch. Afterward, the pre-trained models were examined as feature extractors. Finally, the pre-trained models were fine-tuned with intermediate layers. Fine-tuning was conducted on three levels: the fifth, fourth, and third blocks, respectively. The models were evaluated through recognition experiments using hand gesture images in the Arabic sign language acquired under different conditions. This study also provides a new hand gesture image dataset used in these experiments, plus two other datasets. The experimental results indicated that the proposed models can be used with intermediate layers to recognize hand gesture images. Furthermore, the analysis of the results showed that fine-tuning the fifth and fourth blocks of these two models achieved the best accuracy results. In particular, the testing accuracies on the three datasets were 96.51%, 72.65%, and 55.62% when fine-tuning the fourth block and 96.50%, 67.03%, and 61.09% when fine-tuning the fifth block for the first model. The testing accuracy for the second model showed approximately similar results.

DeReFNet: Dual-stream Dense Residual Fusion Network for static hand gesture recognition

Vision-based hand gesture recognition (HGR) system provides the most effective and natural way of interaction between humans and machines. However, the recognition performance of such an HGR system is challenging due to the variations in illumination, complex backgrounds, the shape of the user’s hand, and inter-class similarity. This work proposes a compact dual-stream dense residual fusion network (DeReFNet) to address the above challenges. The proposed convolutional neural network architecture mainly utilizes the strength of global features from each residual block of the residual stream and spatial information from the other stream using dense connectivity. Both the streams are fused to gather enriched information using the feature concatenation module. The efficacy of the DeReFNet is validated using a subject-independent cross-validation technique on four publicly available benchmark datasets. Furthermore, the qualitative and quantitative analysis of the benchmarked datasets illustrates that the DeReFNet outperforms state-of-the-art methods in terms of accuracy and computational time.

Noise-Against Skeleton Extraction Framework and Application on Hand Gesture Recognition

Extracting stable skeletons from noisy images is a challenging problem since the skeletonization method is prone to be affected by inner and border noise. Although many methods have been proposed in the past for increasing the antinoise ability of skeletonization methods, most of them either only overcome border noise or, at the cost of lost topology, degrade the effects of two noises. In this paper, we propose a skeleton extraction framework to enhance the robustness of the existing skeletonization method against both inner and border noise. In our approach, we first use the different scales of Gaussian filters to smooth the input image and obtain multiple representations. Then, binarization and skeletonization were performed to produce a series of binary images and a series of skeletal images. Next, we use our measure on these binary and skeletal images to find the most suitable skeleton. Since our measure considers both the skeleton image changes and binary image changes caused by using a filter, the selected skeleton is sufficiently robust and has all the necessary skeletal branches. The inner noise experiment and border noise experiment are conducted for comparison. From the perspective of the measure of the rate of variation in the skeleton, the proposed framework can reduce the inner noise by approximately 92% and the border noise by approximately 40%. In addition, the experiment on static hand gesture recognition has demonstrated that the introduction of our framework can increase up to 11% mean recognition accuracy.

SSEC: Semantic Segmentation and Ensemble Classification Framework for Static Hand Gesture Recognition using RGB-D Data

Hand Gesture Recognition (HGR) refers to identifying various hand postures used in Sign Language Recognition (SLR) and Human Computer Interaction (HCI) applications. Complex background in uncontrolled environmental condition is the major challenging issue which impacts the recognition accuracy of HGR system. This can be effectively addressed by discarding the background using suitable semantic segmentation method, where it predicts the hand region pixels into foreground and rest of the pixels into background. In this paper, we have analyzed and evaluated well known semantic segmentation architectures for hand region segmentation using both RGB and depth data. Further, ensemble of segmented RGB and depth stream is used for hand gesture classification through probability score fusion. Experimental results shows that the proposed novel framework of Semantic Segmentation and Ensemble Classification (SSEC) is suitable for static hand gesture recognition and achieved F1-score of 88.91% on OUHANDS test dataset.

An improved hand gesture recognition system using keypoints and hand bounding boxes

Hand gesture recognition is a significant problem for human–computer interaction. One form of hand gesture recognition is static hand gestures. This study developed a static hand gesture recognition system, consisting of three modules: Feature extraction Module, Processing Module, and Classification Module. The feature extraction module uses human pose estimation with a top-down method to extract not only the keypoints but also body and hand bounding boxes. After being normalized and processed in the processing module, its output will be used as the input for the classification module in which we proposed an architecture called Two-pipeline architecture. In this module, we also employ different methods to find the most suitable one for this task. Experiments were conducted on three datasets called HANDS, OUHANDS, and SHAPE. Results showed that the proposed Two-pipeline architecture with 2.5 million parameters obtained accuracy of 94%, 98%, and 94% on three datasets. In addition, the lightweight version with 0.22 million parameters also achieved accuracy of 91%, 94%, and 96%.

Static hand gesture recognition for American sign language using neuromorphic hardware

In this paper, we develop four spiking neural network (SNN) models for two static American sign language (ASL) hand gesture classification tasks, i.e., the ASL alphabet and ASL digits. The SNN models are deployed on Intel’s neuromorphic platform, Loihi, and then compared against equivalent deep neural network (DNN) models deployed on an edge computing device, the Intel neural compute stick 2 (NCS2). We perform a comprehensive comparison between the two systems in terms of accuracy, latency, power consumption, and energy. The best DNN model achieves an accuracy of 99.93% on the ASL alphabet dataset, whereas the best performing SNN model has an accuracy of 99.30%. For the ASL-digits dataset, the best DNN model achieves an accuracy of 99.76% accuracy while the SNN achieves 99.03%. Moreover, our obtained experimental results show that the Loihi neuromorphic hardware implementations achieve up to 20.64× and 4.10× reduction in power consumption and energy, respectively, when compared to NCS2.

Static hand gesture recognition in sign language based on convolutional neural network with feature extraction method using ORB descriptor and Gabor filter

In this paper, a new structure of deep learning neural network is introduced to identify the static hand gesture in the sign language. The proposed structure includes the convolutional neural network (CNN) and the classical non-intelligent feature extraction method. In the proposed structure, the hand gesture image, after preprocessing and removing its background, passes through three different streams of feature extraction, to well extract of effective features and determine the hand gesture class. These three streams, that independently extract their own specific features, consist of three widely used methods in the hand gesture classification named CNN, Gabor filter and ORB feature descriptor. Then these features are merged and formed the final feature vector. By combining these efficient methods, in addition to achieving a very high accuracy in hand gestures classifying, the proposed structure becomes more resistant to uncertainties such as rotation and ambiguity in the hand gestures. Another prominent feature of the proposed structure is its comprehensiveness on different image databases, compared to the similar methods. The transfer learning technique demonstrates that the proposed structure has the ability to be used as a pre-trained structure for any type of image database. Finally, the proposed structure is applied to the three different databases of Massey, ASL Alphabet and ASL, which have 2520, 87,000 and 23,400 of hand gesture images, respectively. The results show the mean accuracy of the proposed structure for the Massey test set of 758 images, ASL with 7020 test images, and ASL Alphabet with 26,100 test images, at 99.92%, 99.8%, and 99.80% respectively.

Reducing the Number of Sensors in the Data Glove for Recognition of Static Hand Gestures

Data glove devices, apart from being widely used in industry and entertainment, can also serve as a means for communication with the environment. This is possible thanks to the advancement in electronic technology and machine learning algorithms. In this paper, the results of the study using a designed data glove equipped with 10 piezoelectric sensors are reported, and the designed glove is validated on a recognition task of hand gestures based on 16 static signs of the Polish Sign Language (PSL) alphabet. The main result of the study is that recognition of 16 PSL static gestures is possible with a reduced number of piezoelectric sensors. This result has been achieved by applying the decision tree classifier that can rank the importance of the sensors for the recognition performance. Other machine learning algorithms were also tested, and it was showed that for the Support Vector Machines, k-NN and Bagged Trees classifiers, a recognition rate of the signs exceeding 90% can be achieved just for three preselected sensors. Such a result is important for a reduction in design complexity and costs of such a data glove with sustained reliability of the device.

Static Hand Gesture Angle Recognition via Aggregated Channel Features (ACF) Detector

Static hand gesture recognition is critical in the development of a system for human-computer interaction. Many human-computer interactions, such as human-robot interaction, game control, control of smart home devices, and others, use hand gestures as a fundamental and natural language of the body. The direction of rotation of static hand gestures is the subject of this research, and the focus is on six degrees of rotation (0°, 45°, 90°, 180°, 270°, and 315°). This work presents an ideal approach that can recognize the angle of hand gestures based on the Aggregate Channel Features (ACF) detector. This approach consists of three main stages: preprocessing (image labelling), computer training, and hand angle detection based on the ACF detector. The training process consists of 25 stages. The static hand gesture dataset contained 569 images (361 for training and 208 for testing). The average time cost to detect all hand gesture angles was 0.9445 seconds, and all hand angles were recognized with 100% accuracy. This is a strong indication that supports our approach.

RBI-2RCNN: Residual Block Intensity Feature using a Two-stage Residual Convolutional Neural Network for Static Hand Gesture Recognition

RBI-2RCNN: Residual Block Intensity Feature using a Two-stage Residual Convolutional Neural Network for Static Hand Gesture Recognition

Static hand gesture recognition based on hierarchical decision and classification of finger features.

AbstractConsidering that the distinctions among static hand gestures are the difference between fingers sticking out, a method of grouping and classifying hand gestures step by step by using the information of the quantity, direction, position and shape of the outstretched fingers was proposed this paper. Firstly, the gesture region was segmented by using the skin color information of the hand, and the gesture direction was normalized by using the direction information of the gesture contour lines. Secondly, the finger was segmented one by one by using convex decomposition in the hand gesture image based on the convex characteristic of the gesture shape. Thirdly, the features of quantity, direction, position and shape of the segmented fingers were extracted. Lastly, a hierarchical decision classifier embedded with deep sparse autoencoders was constructed. The quantity of fingers was used to divide the gesture images into groups first, then the direction, position and shape features of the fingers were used to subdivide and recognize gestures within each group. The experimental results show that the proposed method is robust as lighting, direction and scale changes and significantly superior to the traditional method both in the recognition rate and the recognition stability.

Robust Hand Gesture Recognition Based on RGB-D Data for Natural Human–Computer Interaction

To naturally interact with virtual environment by hand gesture, this paper presents a robust RGB-D data based recognition method of static and dynamic hand gesture. Firstly, for static hand gesture recognition, starting from the hand gesture contour extraction, the palm center is identified by Distance Transform (DT) algorithm. The fingertips are localized by employing the K-Curvature-Convex Defects Detection algorithm (K-CCD). On the basis, the distances of the pixels on hand gesture contour to palm center and the angle between fingertips are considered as the auxiliary features to construct a multimodal feature vector, and then recognition algorithm is presented to robustly recognize the static hand gestures. Secondly, combining Euclidean distance between hand joints and shoulder center joint with the modulus ratios of skeleton features, this paper generates a unifying feature descriptor for each dynamic hand gesture and proposes an improved dynamic time warping (IDTW) algorithm to obtain recognition results of dynamic hand gestures. Finally, we conduct extensive experiments to test and verify the static and dynamic hand gesture recognition algorithm and realize a low-cost real-time application of natural interaction with virtual environment by hand gestures.

Performance analysis of static hand gesture recognition approaches using artificial neural network, support vector machine and two stream based transfer learning approach

Performance analysis of static hand gesture recognition approaches using artificial neural network, support vector machine and two stream based transfer learning approach

Static hand gesture recognition of Arabic sign language by using deep CNNs

An Arabic sign language recognition using two concatenated deep convolution neural network models DenseNet121 & VGG16 is presented. The pre-trained models are fed with images, and then the system can automatically recognize the Arabic sign language. To evaluate the performance of concatenated two models in the Arabic sign language recognition, the red-green-blue (RGB) images for various static signs are collected in a dataset. The dataset comprises 220,000 images for 44 categories: 32 letters, 11 numbers (0:10), and 1 for none. For each of the static signs, there are 5000 images collected from different volunteers. The pre-trained models were used and trained on prepared Arabic sign language data. These models were used after some modification. Also, an attempt has been made to adopt two models from the previously trained models, where they are trained in parallel deep feature extractions. Then they are combined and prepared for the classification stage. The results demonstrate the comparison between the performance of the single model and multi-model. It appears that most of the multi-model is better in feature extraction and classification than the single models. And also show that when depending on the total number of incorrect recognize sign image in training, validation and testing dataset, the best convolutional neural networks (CNN) model in feature extraction and classification Arabic sign language is the DenseNet121 for a single model using and DenseNet121 & VGG16 for multi-model using.