Gesture Recognition Model Research Articles

Research on wearable sensor gesture recognition based on CNN_GRU and attention mechanism

Gesture recognition based on wearable sensors has received extensive attention in recent years. This paper proposes a gesture recognition model (CGR_ATT) based on Convolutional Neural Network (CNN) and Gated Recurrent Unit (GRU) fused attention mechanism to improve accuracy rate of wearable sensors. First, CNN serves as a feature extractor, learning features automatically from sensor data by performing multiple layers of convolution and pooling operations, capturing spatial features of gestures. Furthermore, a temporal modeling unit GRU is introduced to capture the temporal dynamics in gesture sequences. By controlling the information flow through gate mechanisms, it effectively handles the temporal relationships in sensor data. Finally, an attention mechanism is introduced to assign different weights to the hidden state of the GRU. By calculating the attention weights for each time period, the model automatically selects key time periods related to gesture movements. The GR-dataset proposed in this paper involves 910 sets of training parameters. The model achieves an ultimate accuracy of 97.57% . In compare with CLA-net, CLT-net, CGR, GRU, LSTM and CNN, the experimental results demonstrate that the proposed method has superior accuracy.

Study on Gesture Recognition Method with Two-Stream Residual Network Fusing sEMG Signals and Acceleration Signals.

Currently, surface EMG signals have a wide range of applications in human-computer interaction systems. However, selecting features for gesture recognition models based on traditional machine learning can be challenging and may not yield satisfactory results. Considering the strong nonlinear generalization ability of neural networks, this paper proposes a two-stream residual network model with an attention mechanism for gesture recognition. One branch processes surface EMG signals, while the other processes hand acceleration signals. Segmented networks are utilized to fully extract the physiological and kinematic features of the hand. To enhance the model's capacity to learn crucial information, we introduce an attention mechanism after global average pooling. This mechanism strengthens relevant features and weakens irrelevant ones. Finally, the deep features obtained from the two branches of learning are fused to further improve the accuracy of multi-gesture recognition. The experiments conducted on the NinaPro DB2 public dataset resulted in a recognition accuracy of 88.25% for 49 gestures. This demonstrates that our network model can effectively capture gesture features, enhancing accuracy and robustness across various gestures. This approach to multi-source information fusion is expected to provide more accurate and real-time commands for exoskeleton robots and myoelectric prosthetic control systems, thereby enhancing the user experience and the naturalness of robot operation.

FFCSLT: A Deep Learning Model for Traffic Police Hand Gesture Recognition Using Surface Electromyographic Signals

FFCSLT: A Deep Learning Model for Traffic Police Hand Gesture Recognition Using Surface Electromyographic Signals

InRes-ACNet: Gesture Recognition Model of Multi-Scale Attention Mechanisms Based on Surface Electromyography Signals

Surface electromyography (sEMG) signals are the sum of action potentials emitted by many motor units; they contain the information of muscle contraction patterns and intensity, so they can be used as a simple and reliable source for grasping mode recognition. This paper introduces the InRes-ACNet (inception–attention–ACmix-ResNet50) model, a novel deep-learning approach based on ResNet50, incorporating multi-scale modules and self-attention mechanisms. The proposed model aims to improve gesture recognition performance by enhancing its ability to extract channel feature information within sparse sEMG signals. The InRes-ACNet model is evaluated on the NinaPro DB1 and NinaPro DB5 datasets; the recognition accuracy for these datasets can reach 87.94% and 87.04%, respectively, and recognition accuracy can reach 88.37% in the grasping mode prediction of an electromyography manipulator. The results show that the fusion of multi-scale modules and self-attention mechanisms endows a strong ability for the task of gesture recognition based on sparse sEMG signals.

A lightweight GRU-based gesture recognition model for skeleton dynamic graphs

A lightweight GRU-based gesture recognition model for skeleton dynamic graphs

Gesture Recognition Based on a Convolutional Neural Network-Bidirectional Long Short-Term Memory Network for a Wearable Wrist Sensor with Multi-Walled Carbon Nanotube/Cotton Fabric Material.

Flexible pressure sensors play a crucial role in detecting human motion and facilitating human-computer interaction. In this paper, a type of flexible pressure sensor unit with high sensitivity (2.242 kPa-1), fast response time (80 ms), and remarkable stability (1000 cycles) is proposed and fabricated by the multi-walled carbon nanotube (MWCNT)/cotton fabric (CF) material based on a dip-coating method. Six flexible pressure sensor units are integrated into a flexible wristband and made into a wearable and portable wrist sensor with favorable stability. Then, seven wrist gestures (Gesture Group #1), five letter gestures (Gesture Group #2), and eight sign language gestures (Gesture Group #3) are performed by wearing the wrist sensor, and the corresponding time sequence signals of the three gesture groups (#1, #2, and #3) from the wrist sensor are collected, respectively. To efficiently recognize different gestures from the three groups detected by the wrist sensor, a fusion network model combined with a convolutional neural network (CNN) and the bidirectional long short-term memory (BiLSTM) neural network, named CNN-BiLSTM, which has strong robustness and generalization ability, is constructed. The three types of Gesture Groups were recognized based on the CNN-BiLSTM model with accuracies of 99.40%, 95.00%, and 98.44%. Twenty gestures (merged by Group #1, #2, and #3) were recognized with an accuracy of 96.88% to validate the applicability of the wrist sensor based on this model for gesture recognition. The experimental results denote that the CNN-BiLSTM model has very efficient performance in recognizing different gestures collected from the flexible wrist sensor.

Integrated block-wise neural network with auto-learning search framework for finger gesture recognition using sEMG signals

Accurate finger gesture recognition with surface electromyography (sEMG) is essential and long-challenge in the muscle-computer interface, and many high-performance deep learning models have been developed to predict gestures. For these models, problem-specific tuning of network architecture is essential for improving the performance, yet it requires substantial knowledge of network architecture design and commitment of time and effort. This process thus imposes a major obstacle to the widespread and flexible application of modern deep learning. To address this issue, we present an auto-learning search framework (ALSF) to generate the integrated block-wised neural network (IBWNN) for sEMG-based gesture recognition. IBWNN contains several feature extraction blocks and dimensional reduction layers, and each feature extraction block integrates two sub-blocks (i.e., multi-branch convolutional block and triplet attention block). Meanwhile, ALSF generates optimal models for gesture recognition through the reinforcement learning method. The results show that the generated models yield state-of-the-art results compared to the modern popular networks on the open dataset Ninapro DB5. Moreover, compared to other networks, the generated models have fewer parameters and can be deployed in practical applications with less resource consumption.

An Exosuit System With Bidirectional Hand Support for Bilateral Assistance Based on Dynamic Gesture Recognition.

Hand motor impairment has seriously affected the daily life of the elderly. We developed an electromyography (EMG) exosuit system with bidirectional hand support for bilateral coordination assistance based on a dynamic gesture recognition model using graph convolutional network (GCN) and long short-term memory network (LSTM). The system included a hardware subsystem and a software subsystem. The hardware subsystem included an exosuit jacket, a backpack module, an EMG recognition module, and a bidirectional support glove. The software subsystem based on the dynamic gesture recognition model was designed to identify dynamic and static gestures by extracting the spatio-temporal features of the patient's EMG signals and to control glove movement. The offline training experiment built the gesture recognition models for each subject and evaluated the feasibility of the recognition model; the online control experiments verified the effectiveness of the exosuit system. The experimental results showed that the proposed model achieve a gesture recognition rate of 96.42% ± 3.26 %, which is higher than the other three traditional recognition models. All subjects successfully completed two daily tasks within a short time and the success rate of bilateral coordination assistance are 88.75% and 86.88%. The exosuit system can effectively help patients by bidirectional hand support strategy for bilateral coordination assistance in daily tasks, and the proposed method can be applied to various limb assistance scenarios.

Gesture recognition of the Kazakh alphabet based on machine and deep learning models

Currently, a growing body of research focuses on addressing problems using computer vision libraries and artificial intelligence tools. The predominant approaches involve employing machine and deep learning models of artificial neural networks to recognize gestures in the Kazakh Sign Alphabet (KSA) via supervised and deep learning techniques for sequential data processing. Pattern recognition in this context involves identifying an object within an image, where the object can be abstract and vary in shape. We have chosen to investigate the field of gesture recognition, specifically. For recognizing Kazakh Sign Language (KSL), the initial step involves mastering the KSA. Training a neural network to recognize KSL necessitates the collection of datasets in the form of images depicting hand gestures. In this research, prominent hand gesture recognition models such as the Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and Support Vector Machine (SVM) were analyzed. These models differ in their methodologies, processing times, and training data requirements. A significant aspect of this study is the application of unsupervised and supervised learning techniques including CNN, LSTM, and SVM. The experiments yielded diverse results when training neural networks for recognizing gestures in Kazakh sign language based on the dactyl alphabet. This article provides a comprehensive overview of each method, their specific purposes, and their effectiveness in terms of performance and training. Numerous experimental outcomes were documented in a table, showcasing the accuracy of recognizing each gesture. Additionally, specific hand gestures were tested in front of a camera to identify the gesture and display the result on the screen. A notable feature was the use of mathematical formulas and functions to elucidate the operating principles of the machine learning methods, as well as the logical structure and design of the LSTM model.

A real-time basketball referee gesture recognition system based on multi-scale spatio-temporal features

Abstract In this paper, the multi-scale spatio-temporal convolution module and the channel attention mechanism of global information synchronization have excellent performance in the field of sports gesture action recognition; therefore, combining the multi-scale spatio-temporal convolution and the channel attention mechanism to construct the dynamic gesture recognition model. The trained, dynamic gesture recognition model is deployed to cell phones and web pages through Java computational programming language, and the application scenarios of the basketball referee gesture recognition system are explained in detail. The real-time basketball referee gesture recognition system processes are analyzed using simulation analysis after determining the dataset. The basketball referee gesture recognition system’s video cropping frame analysis has an accuracy of 87.64% and 71.25%, with a loss value of 6.24 and 5.59, as shown in the results. When compared to the single-resolution features on the basketball referee gesture recognition system, the fused features exhibit some improvement, and the average recognition rate is 92.20%. This study is applicable to the referee gesture tracking method in complex basketball sports scenarios and lays the foundation for higher semantic level research in the field of sports video analytics, such as action recognition, event detection, and content understanding.

WITHDRAWN: Sewing-driving training optimization-enabled feature selection and hand gesture recognition with deep residual network using sEMG signal

An innovation of an accurate model along with low latency for hand gesture identification is still a huge challenge especially for rehabilitation. This research provides a unified solution for such limitations by constructing an efficient system for hand gesture recognition by employing designed Sewing Driving Training based Optimization (SDTO)_Deep Residual Network (DRN). In this work, pre-processing is done using Gaussian filtering, whereas feature selection is carried out based on SDTO. In order to attain efficient results, appropriate features are extracted at the feature extraction phase. The last step is the hand gesture recognition phase, which is accomplished effectively using DRN and the network is efficiently trained using designed SDTO and it is achieved by the integration of STBO and DTBO. The introduced SDTO_DRN model for hand gesture recognition has attained high accuracy of 0.943, TPR of 0.929, TNR of 0.919, PPV of 0.924, and NPV of 0.924.

Empowering human-robot interaction using sEMG sensor: Hybrid deep learning model for accurate hand gesture recognition

In this paper, a novel approach using a Henry Gas Solubility-based Stacked Convolutional Neural Network (HGS-SCNN) for hand gesture recognition using surface electromyography (sEMG) sensors is proposed. The stacked architecture of the CNN model helps to capture both low-level and high-level features, enabling effective representation learning. To begin, we generated a dataset comprising 600 samples of hand gestures. Next, we applied the Discrete Wavelet Transform (DWT) technique to extract features from the filtered sEMG signal. This step allowed us to capture both spatial and frequency information, enhancing the discriminative power of the extracted features. Extensive experiments are conducted to evaluate the performance of the proposed HGS-SCNN model. In addition, the obtained results are compared with state-of-the-art techniques, namely AOA-SCNN, GWO-SCNN, and WOA-SCNN. The comparative analysis demonstrates that the HGS-SCNN outperforms these existing methods, achieving an impressive accuracy of 99.3%. The experimental results validate the effectiveness of our proposed approach in accurately detecting hand gestures. The combination of DWT-based feature extraction and the HGS-SCNN model offers robust and reliable hand gesture recognition, thereby opening new possibilities for intuitive human-machine interaction and applications requiring gesture-based control.

Blended English Teaching Model in Higher Education School Environment AR Constructive Technology

China's education modernization requirements clearly suggest exploring new teaching methods to improve teaching effectiveness. Based on this, this article investigates AR construction technology as a blended English teaching model in the university school environment. Mobile terminals are used to build the blended English teaching model, modeling tools are used to achieve split modeling, and the mobile terminals themselves are used to achieve the management and network synchronization work for the cloud classroom. For the gesture recognition model construction, 3D convolutional neural networks are used to separate and optimize the parameters. Finally, experiments are designed to simulate and analyze the AR construct of the blended English teaching model to determine accuracy. The simulation results show that the model can realize the simultaneous display of teacher explanation, and the improved algorithm of gesture recognition is improved in recognition rate, which can improve the effectiveness of blended English teaching.

Piano Playing Gesture Recognition Based on Multiple Intelligences Theory

Abstract In this paper, firstly, in order to solve the piano playing recognition problem in the field of artificial intelligence, based on the theory of multiple intelligences, the VGG-16 deep network migration learning algorithm is applied to estimate and acquire the piano playing gesture posture. Secondly, combined with the Iterative Update Extended Kalman Filter (IUEKF) algorithm, the micro-inertial sensor fixation of the piano-playing gesture is realized, which in turn is conducive to improving the piano-playing gesture recognition accuracy. Then, we obtain real-time piano-playing gesture information through a Kinect somatosensory device, construct a piano-playing gesture recognition model based on migration learning on the basis of obtaining piano-playing gesture features, and confirm the effectiveness of the model through the experimental study of piano-playing recognition. The results show that in piano-playing gesture recognition, the recognition accuracy of this paper’s method remains above 0.9, and the application of this paper’s method can effectively improve the recognition accuracy of piano-playing gestures. On piano playing pedal action recognition, this paper’s method shows that the average F-measure scores of these two strategies are 0.924 and 0.944, respectively, which are better compared to other methods. This study provides an effective case for applying AI techniques to piano performance recognition and broadens the scope of AI applications.

Data glove-based gesture recognition using CNN-BiLSTM model with attention mechanism

As a novel form of human machine interaction (HMI), hand gesture recognition (HGR) has garnered extensive attention and research. The majority of HGR studies are based on visual systems, inevitably encountering challenges such as depth and occlusion. On the contrary, data gloves can facilitate data collection with minimal interference in complex environments, thus becoming a research focus in fields such as medical simulation and virtual reality. To explore the application of data gloves in dynamic gesture recognition, this paper proposes a data glove-based dynamic gesture recognition model called the Attention-based CNN-BiLSTM Network (A-CBLN). In A-CBLN, the convolutional neural network (CNN) is employed to capture local features, while the bidirectional long short-term memory (BiLSTM) is used to extract contextual temporal features of gesture data. By utilizing attention mechanisms to allocate weights to gesture features, the model enhances its understanding of different gesture meanings, thereby improving recognition accuracy. We selected seven dynamic gestures as research targets and recruited 32 subjects for participation. Experimental results demonstrate that A-CBLN effectively addresses the challenge of dynamic gesture recognition, outperforming existing models and achieving optimal gesture recognition performance, with the accuracy of 95.05% and precision of 95.43% on the test dataset.

Smart-Data-Glove-Based Gesture Recognition for Amphibious Communication.

This study has designed and developed a smart data glove based on five-channel flexible capacitive stretch sensors and a six-axis inertial measurement unit (IMU) to recognize 25 static hand gestures and ten dynamic hand gestures for amphibious communication. The five-channel flexible capacitive sensors are fabricated on a glove to capture finger motion data in order to recognize static hand gestures and integrated with six-axis IMU data to recognize dynamic gestures. This study also proposes a novel amphibious hierarchical gesture recognition (AHGR) model. This model can adaptively switch between large complex and lightweight gesture recognition models based on environmental changes to ensure gesture recognition accuracy and effectiveness. The large complex model is based on the proposed SqueezeNet-BiLSTM algorithm, specially designed for the land environment, which will use all the sensory data captured from the smart data glove to recognize dynamic gestures, achieving a recognition accuracy of 98.21%. The lightweight stochastic singular value decomposition (SVD)-optimized spectral clustering gesture recognition algorithm for underwater environments that will perform direct inference on the glove-end side can reach an accuracy of 98.35%. This study also proposes a domain separation network (DSN)-based gesture recognition transfer model that ensures a 94% recognition accuracy for new users and new glove devices.

Overcoming the effect of muscle fatigue on gesture recognition based on sEMG via generative adversarial networks

Surface electromyography (sEMG)-controlled bionic prostheses have been extensively investigated recently. However, the majority of investigations pertaining to gesture recognition grounded in sEMG predominantly center their focus on the human body in a static and non-fatigued condition, thus neglecting the ramifications of muscle fatigue on the precision of gesture recognition. This study explores the effect of muscle fatigue on recognition accuracy in a gesture recognition task based on sEMG. The muscle fatigue induction experiment was designed, and eight subjects were recruited to participate in the experiment to collect the sEMG signal data sets of seven gesture actions under non-fatigue and fatigue conditions. Seven gesture actions under non-fatigue and fatigue conditions were identified by four classifiers, namely K-nearest neighbor (K-NN), support vector machine (SVM), decision tree (DT), and deep convolution neural network (CNN). The experimental results show that muscle fatigue has a great impact on the accuracy of gesture recognition. Specifically, using the four classifiers K-NN, SVM, DT and CNN trained in the non-fatigue state, the test accuracy of sEMG signals in the non-fatigue state is 96.7 %, 89.0 %, 87.3 % and 97.5 % respectively, while the test accuracy in the fatigue state is reduced to 53.3 %, 55.4 %, 45.8 % and 64.8 % respectively. In this regard, we propose a new method to overcome the effects of muscle fatigue, namely, the data enhancement method based on Wasserstein General Adversary Networks-Gradient Penalty (WGAN-GP), which is used to enhance the sEMG signal under fatigue. Through the data enhancement of the fatigue sEMG signal, the experimental results show that the final test accuracy in the fatigue state is improved by more than 20 %, which can reach 72.3 %, 80.9 %, 69.9 % and 92.1 % respectively. This shows that the method proposed by us can effectively overcome the influence of muscle fatigue on the accuracy of gesture recognition, and has made a great contribution to the improvement of the robustness of gesture recognition model based on sEMG signal.

Lightweight hybrid model based on MobileNet-v2 and Vision Transformer for human–robot interaction

Within convolutional neural networks, convolutional operations are good at extracting local features, but have difficulty in capturing global representations. For Vision Transformer, multi-head self-attention can capture feature dependencies over long distance, but can destruct local feature details. Based on this, we propose a novel lightweight model, named HybridNet, based on MobileNet-v2 and Vision Transformer, capable of combining the advantages of both CNNs and Vision Transformer. In addition, to enhance the capability of HybridNet for temporal information interaction, we incorporate temporal-channel attention in HybridNet. We conducted experiments on Kinetics-400, Jester, and EgoGesture datasets to validate the effectiveness of HybridNet. The experimental results demonstrate that the lightweight model HybridNet achieves 96.3% and 93.9% accuracy on Jester and EgoGesture, respectively, obtaining the performance close to or even comparable with the state-of-the-art methods. Last but not least, we take HybridNet as the real-time gesture recognition model and use the recognition results as commands to control robots in the simulation environment to achieve human–robot interaction. The use of gesture interaction between humans and robots improves communication, facilitates physical collaboration, enables non-verbal expression, enhances accessibility, and creates a more engaging user experience. It adds a dimension of intuitiveness and efficiency to human–robot interaction, making it more dynamic and interactive.

Visual-semantic network: a visual and semantic enhanced model for gesture recognition

Gesture recognition has attracted considerable attention and made encouraging progress in recent years due to its great potential in applications. However, the spatial and temporal modeling in gesture recognition is still a problem to be solved. Specifically, existing works lack efficient temporal modeling and effective spatial attention capacity. To efficiently model temporal information, we first propose a long- and short-term temporal shift module (LS-TSM) that models the long-term and short-term temporal information simultaneously. Then, we propose a spatial attention module (SAM) that focuses on where the change primarily occurs to obtain effective spatial attention capacity. In addition, the semantic relationship among gestures is helpful in gesture recognition. However, this is usually neglected by previous works. Therefore, we propose a label relation module (LRM) that takes full advantage of the relationship among classes based on their labels’ semantic information. To explore the best form of LRM, we design four different semantic reconstruction methods to incorporate the semantic relationship information into the class label’s semantic space. We perform extensive ablation studies to analyze the best settings of each module. The best form of LRM is utilized to build our visual-semantic network (VS Network), which achieves the state-of-the-art performance on two gesture datasets, i.e., EgoGesture and NVGesture.

GESTURE RECOGNITION OF MACHINE LEARNING AND CONVOLUTIONAL NEURAL NETWORK METHODS FOR KAZAKH SIGN LANGUAGE

Recently, there has been a growing interest in machine learning and neural networks among the public, largely due to advancements in technology which have led to improved methods of computer recognition of objects, sounds, texts, and other data types. As a result, human-computer interactions are becoming more natural and comprehensible to the average person. The progress in computer vision has enabled the use of increasingly sophisticated models for object recognition in images and videos, which can also be applied to recognize hand gestures. In this research, popular hand gesture recognition models, such as the Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and Support Vector Machine (SVM) were examined. These models vary in their approaches, processing time, and training data size. The important feature of this research work is the use of various machine learning algorithms and methods such as CNN, LSTM, and SVM. Experiments showed different results when training neural networks for sign language recognition in the Kazakh sign language based on the dactyl alphabet. This article provides a detailed description of each method, their respective purposes, and effectiveness in terms of performance and training. Numerous experimental results were recorded in a table, demonstrating the accuracy of recognizing each gesture. Additionally, specific hand gestures were isolated for testing in front of the camera to recognize the gesture and display the result on the screen. An important feature was the use of mathematical formulas and functions to explain the working principle of the machine learning algorithm, as well as the logical scheme and structure of the LSTM algorithm.