Gesture Images Research Articles

Static Hand Gesture Recognition Using Capacitive Sensing and Machine Learning

Automated hand gesture recognition is a key enabler of Human-to-Machine Interfaces (HMIs) and smart living. This paper reports the development and testing of a static hand gesture recognition system using capacitive sensing. Our system consists of a 6×18 array of capacitive sensors that captured five gestures-Palm, Fist, Middle, OK, and Index-of five participants to create a dataset of gesture images. The dataset was used to train Decision Tree, Naïve Bayes, Multi-Layer Perceptron (MLP) neural network, and Convolutional Neural Network (CNN) classifiers. Each classifier was trained five times; each time, the classifier was trained using four different participants' gestures and tested with one different participant's gestures. The MLP classifier performed the best, achieving an average accuracy of 96.87% and an average F1 score of 92.16%. This demonstrates that the proposed system can accurately recognize hand gestures and that capacitive sensing is a viable method for implementing a non-contact, static hand gesture recognition system.

Intelligent hybrid hand gesture recognition system using deep recurrent neural network with chaos game optimisation

ABSTRACT Hand gesture recognition is considered an essential task in various human-computer interaction (HCI) applications. Therefore, developing a robust system for hand gesture recognition is a challenge. This work proposed a new hybrid approach named hybrid deep recurrent neural network (RNN) incorporated with a chaos game optimisation (CGO) algorithm for efficiently recognising hand gestures. The main objective of a hybrid recurrent neural network with chaos game optimisation (RNN-CGO) is to achieve the recognition of alphabet signs from 2D gesture images. The different stages included in this technique are pre-processing, feature extraction, feature selection, and classification. The proposed approach is executed using American Sign Language (ASL) dataset with the help of the PYTHON platform. The precision, accuracy, f1-score, and recall obtained for the proposed approach are 99.28%, 99.96%, 99.25%, and 99.28%, respectively. The inference time required for the proposed approach is 0.121 s. The proposed technique has less computational complexity. The results showed that the proposed approach is better than the existing hand gesture recognition model approaches.

A Machine Learning Based Full Duplex System Supporting Multiple Sign Languages for the Deaf and Mute

This manuscript presents a full duplex communication system for the Deaf and Mute (D-M) based on Machine Learning (ML). These individuals, who generally communicate through sign language, are an integral part of our society, and their contribution is vital. They face communication difficulties mainly because others, who generally do not know sign language, are unable to communicate with them. The work presents a solution to this problem through a system enabling the non-deaf and mute (ND-M) to communicate with the D-M individuals without the need to learn sign language. The system is low-cost, reliable, easy to use, and based on a commercial-off-the-shelf (COTS) Leap Motion Device (LMD). The hand gesture data of D-M individuals is acquired using an LMD device and processed using a Convolutional Neural Network (CNN) algorithm. A supervised ML algorithm completes the processing and converts the hand gesture data into speech. A new dataset for the ML-based algorithm is created and presented in this manuscript. This dataset includes three sign language datasets, i.e., American Sign Language (ASL), Pakistani Sign Language (PSL), and Spanish Sign Language (SSL). The proposed system automatically detects the sign language and converts it into an audio message for the ND-M. Similarities between the three sign languages are also explored, and further research can be carried out in order to help create more datasets, which can be a combination of multiple sign languages. The ND-M can communicate by recording their speech, which is then converted into text and hand gesture images. The system can be upgraded in the future to support more sign language datasets. The system also provides a training mode that can help D-M individuals improve their hand gestures and also understand how accurately the system is detecting these gestures. The proposed system has been validated through a series of experiments resulting in hand gesture detection accuracy exceeding 95%.

Using VGG Models with Intermediate Layer Feature Maps for Static Hand Gesture Recognition

A hand gesture recognition system provides a robust and innovative solution to nonverbal communication through human–computer interaction. Deep learning models have excellent potential for usage in recognition applications. To overcome related issues, most previous studies have proposed new model architectures or have fine-tuned pre-trained models. Furthermore, these studies relied on one standard dataset for both training and testing. Thus, the accuracy of these studies is reasonable. Unlike these works, the current study investigates two deep learning models with intermediate layers to recognize static hand gesture images. Both models were tested on different datasets, adjusted to suit the dataset, and then trained under different methods. First, the models were initialized with random weights and trained from scratch. Afterward, the pre-trained models were examined as feature extractors. Finally, the pre-trained models were fine-tuned with intermediate layers. Fine-tuning was conducted on three levels: the fifth, fourth, and third blocks, respectively. The models were evaluated through recognition experiments using hand gesture images in the Arabic sign language acquired under different conditions. This study also provides a new hand gesture image dataset used in these experiments, plus two other datasets. The experimental results indicated that the proposed models can be used with intermediate layers to recognize hand gesture images. Furthermore, the analysis of the results showed that fine-tuning the fifth and fourth blocks of these two models achieved the best accuracy results. In particular, the testing accuracies on the three datasets were 96.51%, 72.65%, and 55.62% when fine-tuning the fourth block and 96.50%, 67.03%, and 61.09% when fine-tuning the fifth block for the first model. The testing accuracy for the second model showed approximately similar results.

A simple and effective static gesture recognition method based on attention mechanism

To solve the problem of low sign language recognition rate under the condition of small samples, a simple and effective static gesture recognition method based on an attention mechanism is proposed. The method proposed in this paper can enhance the features of both the details and the subject of the gesture image. The input of the proposed method depends on the intermediate feature map generated by the original network. Also, the proposed convolutional model is a lightweight general module, which can be seamlessly integrated into any CNN(Convolutional Neural Network) architecture and achieve significant performance gains with minimal overhead. Experiments on two different datasets show that the proposed method is effective and can improve the accuracy of sign language recognition of the benchmark model, making its performance better than the existing methods.

HAND GESTURE AND DETEKSI WAJAHDETECTION USING RASPBERRY PI

Face detection is currently used for various purposes, one of which is to record employees attendance. This strategy is ineffective since the employees still can hack the attendance by making their own photos and put them in their desks. If they are unable to come to the office,they can always ask their colleagues to submit their already available photos.Therefore, an alternative that can complement the current face detection method is highly needed.One of the methods that can be used is hand gesture detection.This study aims to detect hand gestures made by the employees to ensure whether they really come to work or not,so the chance for manipulation is quite small.For the purpose of hand gesture recognition, this study utilized Local Binary Pattern Histogram algorithm. LBPH is an algorithm used for the image matching process between images that have been given training and images taken in real time.The hand gesture image was first taken using a raspberry pi camera and then processed to examine whether it matches the registered ID or not.The results showed that ID recognition by using hand gestures is detectable and is in accordance with the registered ID.The number recognition in hand gestures includes numbers 1 to 10. The test results showed that, the average time required for reading hand gestures using a laptop was 9.2 seconds, while that of using raspberry was 14,2 seconds.Motion reading using a raspberry takes longer than that of using a laptop because the laptop's performance is higher than that of a raspberry.

Real-time Assamese Sign Language Recognition using MediaPipe and Deep Learning

People lacking the sense of hearing and the ability to speak have undeniable communication problems in their life. People with hearing and speech problems communicate using sign language with themselves and others. Sign language is not essentially known to a more significant portion of the human population who uses spoken and written language for communication. Therefore, it is a necessity to develop technological tools for interpretation of sign language. Much research have been carried out to acknowledge sign language using technology for most global languages. But there are still scopes of development of tools and techniques for sign language development for local dialects. There are 22 modern Indian languages and more than 19000 languages that are spoken regionally as mother tongue. This work attempts to develop a technical approach for recognizing Assamese Sign Language, which is one of the 22 modern languages of India. Using machine learning techniques, this work tried to establish a system for identifying the hand gestures from Assamese Sign Language. A combination of two-dimensional and three-dimensional images of Assamese gestures has been used to prepare a dataset. The MediaPipe framework has been implemented to detect landmarks in the images. An Assamese Sign Language dataset of 2094 data points has been generated, including nine static gestures with vowels and consonants (অ, ই, ঈ, উ, এ, ও, ক, জ, ল) from the Assamese Sign Language. The dataset was used for training of a feed-forward neural network. The model yielded an accuracy of 99%. The results reveal that the method implemented in this work is effective for the recognition of the other alphabets and gestures in the Assamese Sign Language. This method could also be tried and tested for the recognition of signs and gestures for various other local languages of India.

An Efficient and Robust Hand Gesture Recognition System of Sign Language Employing Finetuned Inception-V3 and Efficientnet-B0 Network

Hand Gesture Recognition (HGR) is a promising research area with an extensive range of applications, such as surgery, video game techniques, and sign language translation, where sign language is a complicated structured form of hand gestures. The fundamental building blocks of structured expressions in sign language are the arrangement of the fingers, the orientation of the hand, and the hand’s position concerning the body. The importance of HGR has increased due to the increasing number of touchless applications and the rapid growth of the hearing-impaired population. Therefore, real-time HGR is one of the most effective interaction methods between computers and humans. Developing a user-free interface with good recognition performance should be the goal of real-time HGR systems. Nowadays, Convolutional Neural Network (CNN) shows great recognition rates for different image-level classification tasks. It is challenging to train deep CNN networks like VGG-16, VGG-19, Inception-v3, and Efficientnet-B0 from scratch because only some significant labeled image datasets are available for static hand gesture images. However, an efficient and robust hand gesture recognition system of sign language employing finetuned Inception-v3 and Efficientnet-Bo network is proposed to identify hand gestures using a comparative small HGR dataset. Experiments show that Inception-v3 achieved 90% accuracy and 0.93% precision, 0.91% recall, and 0.90% f1-score, respectively, while EfficientNet-B0 achieved 99% accuracy and 0.98%, 0.97%, 0.98%, precision, recall, and f1-score respectively.

Arabic Sign Language Gesture Classification Using Deer Hunting Optimization with Machine Learning Model

Sign language includes the motion of the arms and hands to communicate with people with hearing disabilities. Several models have been available in the literature for sign language detection and classification for enhanced outcomes. But the latest advancements in computer vision enable us to perform signs/gesture recognition using deep neural networks. This paper introduces an Arabic Sign Language Gesture Classification using Deer Hunting Optimization with Machine Learning (ASLGC-DHOML) model. The presented ASLGC-DHOML technique mainly concentrates on recognising and classifying sign language gestures. The presented ASLGC-DHOML model primarily pre-processes the input gesture images and generates feature vectors using the densely connected network (DenseNet169) model. For gesture recognition and classification, a multilayer perceptron (MLP) classifier is exploited to recognize and classify the existence of sign language gestures. Lastly, the DHO algorithm is utilized for parameter optimization of the MLP model. The experimental results of the ASLGC-DHOML model are tested and the outcomes are inspected under distinct aspects. The comparison analysis highlighted that the ASLGC-DHOML method has resulted in enhanced gesture classification results than other techniques with maximum accuracy of 92.88%.

Raspberry Based Hand Gesture Recognition Using Haar Cascade and Local Binary Pattern Histogram

Many companies and even public institutions for civil servants currently use photo-taking for the attendance. However, this strategy is still considered ineffective since the employees still can hack the attendance by making their own photos and put them in their desks. Therefore, an alternative that can complement the current face detection method is highly needed so that the employee’s attendance can be directly monitored. One of the methods that can be used to detect the attendance is hand gesture detection. This research aims to detect hand gestures made by the employees to ensure whether they really come to work or not. This research make the chance for manipulation using photo or fake GPS is quite small. For the purpose of hand gesture recognition, this study utilized Local Binary Pattern Histogram algorithm. The hand gesture image was first taken using a raspberry pi camera and then processed by the device to examine whether it matches the registered ID or not. The results showed that ID recognition by using hand gestures is detectable. The number recognition in hand gestures includes numbers 1 to 10. The test results showed that for 5 trials, the average time required for reading hand gestures using a laptop was 9.2 seconds, while that of using raspberry was 14.2 seconds. The results of this research show that the system has not been able to distinguish which hand is read first, so numbers that have the same number are considered the same, such as 81 and 18. So, the motion reading using a raspberry takes longer than that of using a laptop because the laptop's performance is higher than that of a raspberry and system cannot distinguish between numbers consisting of the same number.

Reconstruction of Gesture Images by Using Banner as Illumination of Single-Pixel Imaging

Reconstruction of Gesture Images by Using Banner as Illumination of Single-Pixel Imaging

Hand Gesture Identification for Improving Accuracy Using Convolutional Neural Network(CNN)

Hand gestures are a type of non-verbal communication that uses visible body movementsto convey important messages. This paper presents a much better approach of hand gestureprediction. Image Identification is an important step in most of the modern hand gestureprediction system. A convolutional neural network are used for improving the accuracy of thesystem. Proposed system tested for large number of hand gesture images using Tensor flowtool. . The convolutional neural network (ConvNet) is a deep learning algorithm for learningand classifying hand gestures and achieved accuracy 93.61%.

Real-time automated detection of older adults' hand gestures in home and clinical settings.

The online version contains supplementary material available at 10.1007/s00521-022-08090-8.

Recognizing Teachers’ Hand Gestures for Effective Non-Verbal Interaction

Hand gesturing is one of the most useful non-verbal behaviors in the classroom, and can help students activate multi-sensory channels to complement teachers’ verbal behaviors and ultimately enhance teaching effectiveness. The existing mainstream detection algorithms that can be used to recognize hand gestures suffered from low recognition accuracy under complex backgrounds and different backlight conditions. This study proposes an improved hand gesture recognition framework based on key point statistical transformation features. The proposed framework can effectively reduce the sensitivity of images to background and light conditions. We extracted key points of the image and establish a weak classifier to enhance the anti-interference ability of the algorithm in the case of noise and partial occlusion. Then, we used a deep convolutional neural network model with multi-scale feature fusion to recognize teachers’ hand gestures. A series of experiments were conducted on different human gesture datasets to verify the performance of the proposed framework. The results show that the framework proposed in this study has better detection and recognition rates compared to the you only look once (YOLO) algorithm, YOLOv3, and other counterpart algorithms. The proposed framework not only achieved 98.43%, measured by F1 score, for human gesture images in low-light conditions, but also has good robustness in complex lighting environments. We used the proposed framework to recognize teacher gestures in a case classroom setting, and found that the proposed framework outperformed YOLO and YOLOv3 algorithms on small gesture images with respect to recognition performance and robustness.

(Retracted) Deep graph convolutional network-based high-performance detection method for spectral domain gesture image stream

The use of vision-based high-performance detection technology has become an innovative technical methodology. Gestures can express more actions and even emotions, which is more in line with the design idea of large-scale integrated human–computer interaction software, and then assists the development of emotion recognition and other fields. The emerging deep graph convolutional network can capture the interdependence between instances, and then infer the complete information of the image based on specific features. The two-dimensional discrete wavelet and multi-resolution analysis are used to replace the traditional Fourier transform to realize the convolution operation, which improves the accuracy of the generated graph data. This work studies the spectral clustering method of Graph Wavelet Neural Network and adds the local correlation preserving support vector machine as a classifier. This classifier has a simplified structure compared with the cascade classifier and can achieve faster and stable classification results. On the test set, the average accuracy of the algorithm is 93.40%, the recall rate is 96.27%, and the average detection time per frame is 359 ms.

Deep transfer learning base on sequenced edge grid image technique for sign language recognition

Sign language is a visual-gestural language used by hearing impaired person, they modality the gesture to convey meaning. The main problem with sign language communication is ordinary people do not understand the sign language. Therefore, sign language is one of the challenging problems in machine learning. In this paper, researchers focus on visual-based methods and optimize the data preprocessing apply with existing sign language resources. Researchers propose an innovative technique for video processing called Sequenced Edge Grid Images (SEGI) for sign language recognition to interpret hand gesture, body movement, and facial expression. Researchers collected several of sign language data from the internet, the data including Thai sign language utilize in everyday life. The proposed technique was implemented with a convolutional neural network (CNN). The experiments showed SEGI with CNN has increases test accuracy rate with approximately 11% compared to static hand gesture images. Finally, researchers discovered a CNN structure suitable for dataset and examination data by transferring a pre-trained CNN. The fine-tuning with SEGI technique improved 99.8%, thus highest among all the methods. From the results data-preprocesses technique of dataset generation and deep transfer learning was an effective way to improve the accuracy of sign language recognition.

Static hand gesture recognition in sign language based on convolutional neural network with feature extraction method using ORB descriptor and Gabor filter

In this paper, a new structure of deep learning neural network is introduced to identify the static hand gesture in the sign language. The proposed structure includes the convolutional neural network (CNN) and the classical non-intelligent feature extraction method. In the proposed structure, the hand gesture image, after preprocessing and removing its background, passes through three different streams of feature extraction, to well extract of effective features and determine the hand gesture class. These three streams, that independently extract their own specific features, consist of three widely used methods in the hand gesture classification named CNN, Gabor filter and ORB feature descriptor. Then these features are merged and formed the final feature vector. By combining these efficient methods, in addition to achieving a very high accuracy in hand gestures classifying, the proposed structure becomes more resistant to uncertainties such as rotation and ambiguity in the hand gestures. Another prominent feature of the proposed structure is its comprehensiveness on different image databases, compared to the similar methods. The transfer learning technique demonstrates that the proposed structure has the ability to be used as a pre-trained structure for any type of image database. Finally, the proposed structure is applied to the three different databases of Massey, ASL Alphabet and ASL, which have 2520, 87,000 and 23,400 of hand gesture images, respectively. The results show the mean accuracy of the proposed structure for the Massey test set of 758 images, ASL with 7020 test images, and ASL Alphabet with 26,100 test images, at 99.92%, 99.8%, and 99.80% respectively.

An efficient hand gesture recognition based on optimal deep embedded hybrid convolutional neural network‐long short term memory network model

AbstractHand gestures are the nonverbal communication done by individuals who cannot represent their thoughts in form of words. It is mainly used during human‐computer interaction (HCI), deaf and mute people interaction, and other robotic interface applications. Gesture recognition is a field of computer science mainly focused on improving the HCI via touch screens, cameras, and kinetic devices. The state‐of‐art systems mainly used computer vision‐based techniques that utilize both the motion sensor and camera to capture the hand gestures in real‐time and interprets them via the usage of the machine learning algorithms. Conventional machine learning algorithms often suffer from the different complexities present in the visible hand gesture images such as skin color, distance, light, hand direction, position, and background. In this article, an adaptive weighted multi‐scale resolution (AWMSR) network with a deep embedded hybrid convolutional neural network and long short term memory network (hybrid CNN‐LSTM) is proposed for identifying the different hand gesture signs with higher recognition accuracy. The proposed methodology is formulated using three steps: input preprocessing, feature extraction, and classification. To improve the complex visual effects present in the input images, a histogram equalization technique is used which improves the size of the gray level pixel in the image and also their occurrence probability. The multi‐block local binary pattern (MB‐LBP) algorithm is employed for feature extraction which extracts the crucial features present in the image such as hand shape structure feature, curvature feature, and invariant movements. The AWMSR with the deep embedded hybrid CNN–LSTM network is applied in the two‐benchmark datasets namely Jochen Triesch static hand posture and NUS hand posture dataset‐II to detect its stability in identifying different hand gestures. The weight function of the deep embedded CNN‐LSTM architecture is optimized using the puzzle optimization algorithm. The efficiency of the proposed methodology is verified in terms of different performance evaluation metrics such as accuracy, loss, confusion matrix, Intersection over the union, and execution time. The proposed methodology offers recognition accuracy of 97.86% and 98.32% for both datasets.

Convolution neural networks for hand gesture recognation

Hand gestures (not static or fixed positions) are movements of fingers and the arm to communicate messages. Hand gesture recognition is the process of identifying meaningful expressions involving the human hand. Pictorial representation of gestures will enable to understand human computer interaction (HCI). This paper describes a system using convolution neural network (CNN) for recognizing the 26 letters of the English alphabet signaled with hand gestures. A Python program was developed to recognize the gestures made in front of a web camera. The hand gestures obtained are categorized using CNN with a trained model. The model was constructed using 1,100 gestures images. The recognition rate was obtained with 91% of accuracy. The proposed method was found to be highly efficient in distinguishing and classifying gestures.

Gesture Recognition using Conditional Generative Adversarial Networks

Sign language is very useful for giving signs to communication partners, in this case, signs are not only for people with disabilities but can also be used by normal people. Even in children's or adult games, sign language is used as a language or a means of communication with one another. Recognition of sign language using a computer by doing several methods, because the computer does not recognize the image of a sign with a certain meaning. Therefore, it is necessary to train computers to recognize these signs. One of the fields that discusses gesture image recognition is the field of Computer Vision. Where the science of computer vision is able to process the image. In addition, in image processing, it is necessary to carry out deep learning processes such as Convolutional Neural Networks. In the Convolutional Neural Network algorithm, there are also many methods or architectures such as VGG16, VGG19, ResNet-50, DenseNet, Inception_V3, and many more. The use of architecture is used in accordance with existing needs. Therefore, the choice of architecture will determine the model is built or not to build from scratch, only transfer learning or pre-train. Pre-training is done by using the initial model and then using it only. Or do some training. The purpose of this study was to detect sign language using the Generative Adversarial Network (GAN). Actually, the Generative Adversarial Network method is widely used in making synthetic images, but this time the Generative Adversarial Network can also detect images from sign language.