Dynamic Gesture Research Articles

Sign Language Recognition utilizing LSTM and Mediapipe for Dynamic Gestures of ISL

Humans, in general, are social creatures who communicate themselves through an assortment of spoken languages. Deaf and Mute individuals converse in a manner that's comparable, however many others are ignorant of their sign language. As a result, there is a need to develop a system that facilitates communication among the hearing and hard-of-hearing communities. This research offers a real-time Indian Sign Language (ISL) recognition system for 24 dynamic signals using the Mediapipe framework and an LSTM network. The method proposed in the study involves training a LSTM to differentiate between different signs using a dataset created of 24 dynamic gesture signs. To accomplish dataset creation, a pre-trained Holistic model of the Mediapipe framework is used as a feature extractor. The results of the study demonstrate that the above approach achieves 97% test accuracy.

Dynamic gesture recognition based on compressed video for UAV control

As a multifunctional aircraft with small size, low cost and easy control, UAVs can be used in many fields such as gardening, plant protection, mapping, logistics, military, etc. For a more convenient human-machine interaction mode, the user interacts with the UAVs in the form of dynamic gestures. The current traditional approach is to train convolutional neural networks with images as direct inputs to the system for the purpose of controlling UAVs. However, this approach leaves the temporal representation information between images missing, resulting in poor training results or over-reliance on computing resources. Therefore, this paper proposes an efficient processing strategy, that is, for simple gesture tasks, the optimized frame extraction algorithm is adopted to process picture-based gesture recognition; for complex gesture tasks, compressed video is used as system input to complete video-based gesture recognition. Based on the training results in action recognition datasets, UCF-101 and HMDB-51, the efficiency of compressed video in gesture recognition tasks has been verified, which can be applied to UAV dynamic gesture control.

Smart Home Automation-Based Hand Gesture Recognition Using Feature Fusion and Recurrent Neural Network.

Gestures have been used for nonverbal communication for a long time, but human-computer interaction (HCI) via gestures is becoming more common in the modern era. To obtain a greater recognition rate, the traditional interface comprises various devices, such as gloves, physical controllers, and markers. This study provides a new markerless technique for obtaining gestures without the need for any barriers or pricey hardware. In this paper, dynamic gestures are first converted into frames. The noise is removed, and intensity is adjusted for feature extraction. The hand gesture is first detected through the images, and the skeleton is computed through mathematical computations. From the skeleton, the features are extracted; these features include joint color cloud, neural gas, and directional active model. After that, the features are optimized, and a selective feature set is passed through the classifier recurrent neural network (RNN) to obtain the classification results with higher accuracy. The proposed model is experimentally assessed and trained over three datasets: HaGRI, Egogesture, and Jester. The experimental results for the three datasets provided improved results based on classification, and the proposed system achieved an accuracy of 92.57% over HaGRI, 91.86% over Egogesture, and 91.57% over the Jester dataset, respectively. Also, to check the model liability, the proposed method was tested on the WLASL dataset, attaining 90.43% accuracy. This paper also includes a comparison with other-state-of-the art methods to compare our model with the standard methods of recognition. Our model presented a higher accuracy rate with a markerless approach to save money and time for classifying the gestures for better interaction.

A novel static and dynamic hand gesture recognition using self organizing map with deep convolutional neural network

Gesture recognition has gained a lot of popularity as it allows humans to communicate with real or virtual systems through gestures, offering new and natural interaction modalities. Recent technologies, such as augmented reality (AR) and the Internet of Things (IoT), have witnessed enormous growth in computer applications that focus on human–computer interaction (HCI). However, a few of these tactics make use of a combination of methods, such as image segmentation, pre-processing, and classification. The hessian-based multiscale filtering and YCbCr colour space are used to separate the gesture region to be recognized. A modified marker-controlled watershed method is employed to segment the gesture contour along with the eight-connector graph to increase recognition precision. The proposed hand gesture recognition methodology uses Self Organizing Map (SOM) with Deep Convolutional Neural Network (DCNN) provides better results with fast convergence speed. Experiments were carried out on a dataset of 30 static and 6 dynamic gestures and also evaluated on a publicly available IIITA-ROBITA ISL Gesture Database to show the effectiveness. The results show that the suggested method can recognize gesture classes with 95.63% accuracy rate without significantly affecting the recognition time. The proposed algorithm was then implemented to control household appliances.

Dynamic Gesture Recognition Based on FMCW Millimeter Wave Radar: Review of Methodologies and Results.

As a convenient and natural way of human-computer interaction, gesture recognition technology has broad research and application prospects in many fields, such as intelligent perception and virtual reality. This paper summarized the relevant literature on gesture recognition using Frequency Modulated Continuous Wave (FMCW) millimeter-wave radar from January 2015 to June 2023. In the manuscript, the widely used methods involved in data acquisition, data processing, and classification in gesture recognition were systematically investigated. This paper counts the information related to FMCW millimeter wave radar, gestures, data sets, and the methods and results in feature extraction and classification. Based on the statistical data, we provided analysis and recommendations for other researchers. Key issues in the studies of current gesture recognition, including feature fusion, classification algorithms, and generalization, were summarized and discussed. Finally, this paper discussed the incapability of the current gesture recognition technologies in complex practical scenes and their real-time performance for future development.

Exploring a Novel Mexican Sign Language Lexicon Video Dataset

This research explores a novel Mexican Sign Language (MSL) lexicon video dataset containing the dynamic gestures most frequently used in MSL. Each gesture consists of a set of different versions of videos under uncontrolled conditions. The MX-ITESO-100 dataset is composed of a lexicon of 100 gestures and 5000 videos from three participants with different grammatical elements. Additionally, the dataset is evaluated in a two-step neural network model as having an accuracy greater than 99% and thus serves as a benchmark for future training of machine learning models in computer vision systems. Finally, this research provides an inclusive environment within society and organizations, in particular for people with hearing impairments.

An enhanced artificial neural network for hand gesture recognition using multi-modal features

ABSTRACT The development of automatic application fields enhanced the advancement in Hand Gesture Recognition (HGR), which enhances human-computer interaction. In particular, hand gestures find a valuable application in hand games, virtual reality, and gesture-based signalling systems. Thus, detecting and recognising hand gestures gain technical significance. Although existing models highlight remarkable recognition outcomes, handling dynamic gestures is a challenging task. Hence, in this research, an enhanced flashfly optimisation-based artificial neural network is developed for the effective identification of hand gestures, and to overcome the existing challenges, the enhanced flash fly optimisation-based ANN technique is proposed. The main objective of the optimisation strategy is to effectively tune the parameters involved in the artificial neural network (ANN) by the attained optimal solution. The optimised ANN classifier is well trained by the pre-processed statistical and the Residual neural Networks (ResNet) features for consistently identifying hand movements. The attained performance of the enhanced flash fly optimisation-based ANN classifier during the representation of hand gestures based on the precision, recall, and F1 score is 0.940 %, 0.791 %, and 0.927 % for the training percentages 50 and 90. Hence, it is illustrated that the proposed HGR model exceeds the competent technique, demonstrating its effectiveness.

Tracking of dynamic gesture fingertips position in video sequence

Tracking of dynamic gesture fingertips position in video sequence

Continuous dynamic gesture recognition using surface EMG signals based on blockchain-enabled internet of medical things

The integration of blockchain technology in the Internet of Medical Things (IoMT) enhances user security, convenience and interoperability. It introduces a novel approach to gesture recognition. Dynamic surface electromyography (EMG) is essential to address the shortcomings of visual analysis and gesture similarity in discrete pattern recognition. This paper uses an improved comb filter and a Gaussian mixture model to reduce the noise of surface EMG signals. Continuous dynamic gesture recognition models have been improved based on temporal network and generalized regression neural network. These models are applied to blockchain-enabled IoMT and ensure the traceability of data sharing. Experimental results demonstrate that the proposed method effectively reduces the false recognition rate attributed to signal complexity, thus achieving accurate continuous dynamic gesture recognition. This paper's approach lays a crucial foundation for implementing it in the blockchain-enabled IoMT.

Snapture—a Novel Neural Architecture for Combined Static and Dynamic Hand Gesture Recognition

As robots are expected to get more involved in people’s everyday lives, frameworks that enable intuitive user interfaces are in demand. Hand gesture recognition systems provide a natural way of communication and, thus, are an integral part of seamless human-robot interaction (HRI). Recent years have witnessed an immense evolution of computational models powered by deep learning. However, state-of-the-art models fall short of expanding across different gesture domains, such as emblems and co-speech. In this paper, we propose a novel hybrid hand gesture recognition system. Our Snapture architecture enables learning both static and dynamic gestures: by capturing a so-called snapshot of the gesture performance at its peak, we integrate the hand pose and the dynamic movement. Moreover, we present a method for analyzing the motion profile of a gesture to uncover its dynamic characteristics, which allows regulating a static channel based on the amount of motion. Our evaluation demonstrates the superiority of our approach on two gesture benchmarks compared to a state-of-the-art CNNLSTM baseline. Our analysis on a gesture class basis unveils the potential of our Snapture architecture for performance improvements using RGB data. Thanks to its modular implementation, our framework allows the integration of other multimodal data, like facial expressions and head tracking, which are essential cues in HRI scenarios, into one architecture. Thus, our work contributes both to integrative gesture recognition research and machine learning applications for non-verbal communication with robots.

EfficientNetV2-based dynamic gesture recognition using transformed scalogram from triaxial acceleration signal

Abstract In this paper, a dynamic gesture recognition system is proposed using triaxial acceleration signal and image-based deep neural network. With our dexterous glove device, 1D acceleration signal can be measured from each finger and decomposed to time-divided frequency components via wavelet transformation, which is known as scalogram as image-like format. To feed-forward the scalogram with single 2D, convolutional neural networks allows the gesture having temporality to be easily recognized without any complex system such as RNN, LSTM, or spatio-temporal feature as 3D CNN, etc. To classify the image with general input dimension of image RGB channels, we numerically reconstruct fifteen scalograms into one RGB image with various representation methods. In experiments, we employ the off-the-shelf model, EfficientNetV2 small-to-large model as an image classification model with fine-tuning. To evaluate our system, we bulid our custom bicycle hand signals as dynamic gesture dataset under our transformation system, and then qualitatively compare the reconstruction method with matrix representation methods. In addition, we use other signal transformation tools such as the fast Fourier transform and short-time Fourier transform and then explain the advantages of scalogram classification in the terms of time-frequency resolution trade-off issue.

Robust and Context-Aware Real-Time Collaborative Robot Handling via Dynamic Gesture Commands

This paper studies real-time collaborative robot (cobot) handling, where the cobot maneuvers an object under human dynamic gesture commands. Enabling dynamic gesture commands is useful when the human needs to avoid direct contact with the robot or the object handled by the robot. However, the key challenge lies in the heterogeneity in human behaviors and the stochasticity in the perception of dynamic gestures, which requires the robot handling policy to be adaptable and robust. To address these challenges, we introduce <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Conditional Collaborative Handling Process</i> (CCHP) to encode a context-aware cobot handling policy and a procedure to learn such policy from human-human collaboration. We thoroughly evaluate the adaptability and robustness of CCHP and apply our approach to a real-time cobot assembly task with Kinova Gen3 robot arm. Results show that our method leads to significantly less human effort and smoother human-robot collaboration than state-of-the-art rule-based approach even with first-time users.

Human Gesture Recognition Based on Millimeter-Wave Radar Using Improved C3D Convolutional Neural Network

&lt;p&gt;In this paper, we propose a time sequential IC3D convolutional neural network approach for hand gesture recognition based on frequency modulated continuous wave (FMCW) radar. Firstly, the FMCW radar is used to collect the echoes of human hand gestures. A two-dimensional fast Fourier transform calculates the range and velocity information of hand gestures in each frame signal to construct the Range-Doppler heat map dataset of hand gestures. Then, we design an IC3D network for feature extraction and classification of the dynamic gesture heat map. Finally, the experiment results show that the gesture recognition system designed in this paper effectively solves the problems of the difficulty of human gesture feature extraction and low utilization of time series information, and the average recognition accuracy rate can reach more than 99.8%.&lt;/p&gt; &lt;p&gt;&amp;nbsp;&lt;/p&gt;

TripCEAiR: A multi-loss minimization approach for surface EMG based airwriting recognition

Airwriting Recognition refers to the problem of identification of letters written in space with movement of the finger. It can be seen as a special case of dynamic gesture recognition wherein the set of gestures are letters in a particular language. Surface Electromyography (sEMG) is a non-invasive approach used to capture electrical signals generated as a result of contraction and relaxation of the muscles. sEMG has been widely adopted for gesture recognition applications. Unlike static gestures, dynamic gestures are user-friendly and can be used as a method for input with applications in Human Computer Interaction. There has been limited work in recognition of dynamic gestures such as airwriting, using sEMG signals and forms the core of the current work. In this work, a multi-loss minimization framework for sEMG based airwriting recognition is proposed. The proposed framework aims at learning a feature embedding vector that minimizes the triplet loss, while simultaneously learning the parameters of a classifier head to recognize corresponding alphabets. The proposed method is validated on a dataset recorded in the lab comprising of sEMG signals from 50 participants writing English uppercase alphabets. The effect of different variations of triplet loss, triplet mining strategies and feature embedding dimension is also presented. The best-achieved accuracy was 81.26% and 65.62% in user-dependent and independent scenarios respectively by using semihard positive and hard negative triplet mining. The code for our implementation will be made available at https://github.com/ayushayt/TripCEAiR

An Approach for Gesture Recognition Based on a Lightweight Convolutional Neural Network

Gesture recognition, which plays an important role to understand meaningful movements of human bodies, is one of the most effective approaches for humans to interact. Sign language is a fundamental and innate means of communication for hearing-impaired individuals. Though significant progress has been made, the state-of-the-art gesture recognition methods yield week performance for conditions with dynamic gestures in videos. Thus, robust gesture recognition remains a challenging issue because of many barriers of gesture-irrelevant factors. The key to robust gesture recognition is to learn effective and concise spatiotemporal information. Inspired by the great promise of the convolutional neural network (CNN) and its breakthroughs, we introduce an approach for identifying static alphabet gestures in the American Sign Language (ASL). The proposed CNN-based approach has been developed to classify letters of the alphabet from A to Z. It composes of three phases: a preprocessing phase for extracted of the region of interest, a feature extraction, and a classification phase. The performance of proposed gesture recognition approach is evaluated on the common ASL dataset and it achieves 94.83% accuracy, which is good enough to develop a strong translator from gesture-based ASL to spoken language as it is capable to handle a variety of 24 hand gestures.

SIGN LANGUAGE DETECTION USING ACTION RECOGNITION WITH PYTHON

Sign language detection plays a crucial role in bridging the communication gap between individuals with hearing impairments and the rest of the population. This study presents a novel approach for sign language detection using action recognition and LSTM deep learning models implemented in Python. The proposed system aims to accurately recognize and interpret sign language gestures in real-time. The research focuses on leveraging the power of deep learning techniques, specifically the Long Short-Term Memory (LSTM) architecture, to effectively capture temporal dependencies and classify sign language gestures. A comprehensive dataset comprising a wide range of sign language gestures is collected and preprocessed to train and evaluate the LSTM model. The pipeline for sign language detection consists of several stages, including video acquisition, preprocessing, feature extraction, and model training. In the pre-processing stage, the acquired video data is segmented into individual frames, and various image processing techniques are applied to enhance the quality and remove noise. Next, robust features are extracted from the preprocessed frames using techniques like optical flow or deep learning-based feature extraction methods. The LSTM model is then trained on the extracted features to learn the temporal dynamics of sign language gestures. Transfer learning is also explored to leverage pre-trained models on large-scale action recognition datasets. The trained model is evaluated using a comprehensive set of metrics, including accuracy, precision, recall, and F1-score, to assess its performance. Experimental results demonstrate the effectiveness of the proposed approach in accurately recognizing sign language gestures. The LSTM-based deep learning model achieves a high accuracy rate, showcasing its capability to handle the temporal nature of sign language. The system exhibits real-time performance, enabling it to be deployed in various applications, such as real-time interpretation tools or assistive devices for individuals with hearing impairments

MEMS Devices-Based Hand Gesture Recognition via Wearable Computing.

Gesture recognition has found widespread applications in various fields, such as virtual reality, medical diagnosis, and robot interaction. The existing mainstream gesture-recognition methods are primarily divided into two categories: inertial-sensor-based and camera-vision-based methods. However, optical detection still has limitations such as reflection and occlusion. In this paper, we investigate static and dynamic gesture-recognition methods based on miniature inertial sensors. Hand-gesture data are obtained through a data glove and preprocessed using Butterworth low-pass filtering and normalization algorithms. Magnetometer correction is performed using ellipsoidal fitting methods. An auxiliary segmentation algorithm is employed to segment the gesture data, and a gesture dataset is constructed. For static gesture recognition, we focus on four machine learning algorithms, namely support vector machine (SVM), backpropagation neural network (BP), decision tree (DT), and random forest (RF). We evaluate the model prediction performance through cross-validation comparison. For dynamic gesture recognition, we investigate the recognition of 10 dynamic gestures using Hidden Markov Models (HMM) and Attention-Biased Mechanisms for Bidirectional Long- and Short-Term Memory Neural Network Models (Attention-BiLSTM). We analyze the differences in accuracy for complex dynamic gesture recognition with different feature datasets and compare them with the prediction results of the traditional long- and short-term memory neural network model (LSTM). Experimental results demonstrate that the random forest algorithm achieves the highest recognition accuracy and shortest recognition time for static gestures. Moreover, the addition of the attention mechanism significantly improves the recognition accuracy of the LSTM model for dynamic gestures, with a prediction accuracy of 98.3%, based on the original six-axis dataset.

An Approach to Convert Sign to Text for Impaired People

Human Computer Interface is the study of how humans and computers interact. Hand gestures are a great way to communicate with people when they don’t understand exactly what we are saying. Understanding hand gestures is essential to make sure the listener understands what we are saying. The main idea of our project is to try different approaches to hand gesture recognition. This proposed work first with radar data and then with camera sensor to achieve hand gesture recognition. First, we tried to build hand gesture recognition using radar data, and since most people don’t know sign language and few interpreters, we developed an approach to real-time approach for American Sign Language based on neural networks finger spelling followed by another model with Media Pipe. We propose a complex neural network method to detect hand gestures of human behaviour from camera recorded images. The hand gesture first goes through the filter and after applying the filter the gesture goes through a classifier that predicts which type of hand gesture it is. In an existing system radar unable to detect static gestures in our approach, a deep learning-based image captioning algorithm captures both static and dynamic gestures through Media Pipe.

Stretchable array electromyography sensor with graph neural network for static and dynamic gestures recognition system

With advances in artificial intelligence (AI)-based algorithms, gesture recognition accuracy from sEMG signals has continued to increase. Spatiotemporal multichannel-sEMG signals substantially increase the quantity and reliability of the data for any type of study. Here, we report an array of bipolar stretchable sEMG electrodes with a self-attention-based graph neural network to recognize gestures with high accuracy. The array is designed to spatially cover the skeletal muscles to acquire the regional sampling data of EMG activity from 18 different gestures. The system can differentiate individual static and dynamic gestures with ~97% accuracy when training a single trial per gesture. Moreover, a sticky patchwork of holes adhered to an array sensor enables skin-like attributes such as stretchability and water vapor permeability and aids in delivering stable EMG signals. In addition, the recognition accuracy (~95%) remained unchanged even after long-term testing for over 72 h and being reused more than 10 times.

A multimodal human-robot sign language interaction framework applied in social robots.

Deaf-mutes face many difficulties in daily interactions with hearing people through spoken language. Sign language is an important way of expression and communication for deaf-mutes. Therefore, breaking the communication barrier between the deaf-mute and hearing communities is significant for facilitating their integration into society. To help them integrate into social life better, we propose a multimodal Chinese sign language (CSL) gesture interaction framework based on social robots. The CSL gesture information including both static and dynamic gestures is captured from two different modal sensors. A wearable Myo armband and a Leap Motion sensor are used to collect human arm surface electromyography (sEMG) signals and hand 3D vectors, respectively. Two modalities of gesture datasets are preprocessed and fused to improve the recognition accuracy and to reduce the processing time cost of the network before sending it to the classifier. Since the input datasets of the proposed framework are temporal sequence gestures, the long-short term memory recurrent neural network is used to classify these input sequences. Comparative experiments are performed on an NAO robot to test our method. Moreover, our method can effectively improve CSL gesture recognition accuracy, which has potential applications in a variety of gesture interaction scenarios not only in social robots.