Gesture Recognition Task Research Articles

Enhancement algorithm for surface electromyographic-based gesture recognition based on real-time fusion of muscle fatigue features

This study aims to optimize surface electromyography-based gesture recognition technique, focusing on the impact of muscle fatigue on the recognition performance. An innovative real-time analysis algorithm is proposed in the paper, which can extract muscle fatigue features in real time and fuse them into the hand gesture recognition process. Based on self-collected data, this paper applies algorithms such as convolutional neural networks and long short-term memory networks to provide an in-depth analysis of the feature extraction method of muscle fatigue, and compares the impact of muscle fatigue features on the performance of surface electromyography-based gesture recognition tasks. The results show that by fusing the muscle fatigue features in real time, the algorithm proposed in this paper improves the accuracy of hand gesture recognition at different fatigue levels, and the average recognition accuracy for different subjects is also improved. In summary, the algorithm in this paper not only improves the adaptability and robustness of the hand gesture recognition system, but its research process can also provide new insights into the development of gesture recognition technology in the field of biomedical engineering.

Zero-shot prompt-based video encoder for surgical gesture recognition.

In order to produce a surgical gesture recognition system that can support a wide variety of procedures, either a very large annotated dataset must be acquired, or fitted models must generalize to new labels (so-called zero-shot capability). In this paper we investigate the feasibility of latter option. Leveraging the bridge-prompt framework, we prompt-tune a pre-trained vision-text model (CLIP) for gesture recognition in surgical videos. This can utilize extensive outside video data such as text, but also make use of label meta-data and weakly supervised contrastive losses. Our experiments show that prompt-based video encoder outperforms standard encoders in surgical gesture recognition tasks. Notably, it displays strong performance in zero-shot scenarios, where gestures/tasks that were not provided during the encoder training phase are included in the prediction phase. Additionally, we measure the benefit of inclusion text descriptions in the feature extractor training schema. Bridge-prompt and similar pre-trained+prompt-tuned video encoder models present significant visual representation for surgical robotics, especially in gesture recognition tasks. Given the diverse range of surgical tasks (gestures), the ability of these models to zero-shot transfer without the need for any task (gesture) specific retraining makes them invaluable.

FM‐YOLOv8:Lightweight gesture recognition algorithm

AbstractIn practical production applications, the efficiency and success rate of gesture recognition directly affect the user experience and work efficiency. However, the existing gesture recognition models have the problem of a large number of model parameters and high computational complexity, which makes them unable to meet the needs of end‐to‐end industrial deployment. To solve these problems, this article proposes a gesture recognition model based on YOLOv8. First, FasterNet is adopted as the backbone of YOLOv8, which significantlydecrease the number of parameters and made the model more lightweight. By reducing the number of parameters, the computational complexity of the model can be reduced while maintaining the performance of the model, and the operation efficiency of the model can be improved. Second, recombination convolution ScConv is introduced to replace common convolution operations to further improve the model's efficiency. Recombination convolution can reduce the computation and make up for the loss of precision to some extent. Finally, the MDPIoU loss function is used to optimize target location and prediction, to improve the accuracy of the model. The MDPIoU loss function can better deal with the problem of target boundary frame positioning and prediction so that the model can locate and predict gestures more accurately in gesture recognition tasks. Experiments on a data set containing 10 types of gestures show that the number of parameters and floating point calculations of the improved network model are reduced by 45% and 42.7%, respectively, while the accuracy is unchanged. The improved model can be deployed on edge terminals, providing efficient and accurate gesture recognition.

EvHandPose: Event-Based 3D Hand Pose Estimation With Sparse Supervision.

Event camera shows great potential in 3D hand pose estimation, especially addressing the challenges of fast motion and high dynamic range in a low-power way. However, due to the asynchronous differential imaging mechanism, it is challenging to design event representation to encode hand motion information especially when the hands are not moving (causing motion ambiguity), and it is infeasible to fully annotate the temporally dense event stream. In this paper, we propose EvHandPose with novel hand flow representations in Event-to-Pose module for accurate hand pose estimation and alleviating the motion ambiguity issue. To solve the problem under sparse annotation, we design contrast maximization and hand-edge constraints in Pose-to-IWE (Image with Warped Events) module and formulate EvHandPose in a weakly-supervision framework. We further build EvRealHands, the first large-scale real-world event-based hand pose dataset on several challenging scenes to bridge the real-synthetic domain gap. Experiments on EvRealHands demonstrate that EvHandPose outperforms previous event-based methods under all evaluation scenes, achieves accurate and stable hand pose estimation with high temporal resolution in fast motion and strong light scenes compared with RGB-based methods, generalizes well to outdoor scenes and another type of event camera, and shows the potential for the hand gesture recognition task.

InRes-ACNet: Gesture Recognition Model of Multi-Scale Attention Mechanisms Based on Surface Electromyography Signals

Surface electromyography (sEMG) signals are the sum of action potentials emitted by many motor units; they contain the information of muscle contraction patterns and intensity, so they can be used as a simple and reliable source for grasping mode recognition. This paper introduces the InRes-ACNet (inception–attention–ACmix-ResNet50) model, a novel deep-learning approach based on ResNet50, incorporating multi-scale modules and self-attention mechanisms. The proposed model aims to improve gesture recognition performance by enhancing its ability to extract channel feature information within sparse sEMG signals. The InRes-ACNet model is evaluated on the NinaPro DB1 and NinaPro DB5 datasets; the recognition accuracy for these datasets can reach 87.94% and 87.04%, respectively, and recognition accuracy can reach 88.37% in the grasping mode prediction of an electromyography manipulator. The results show that the fusion of multi-scale modules and self-attention mechanisms endows a strong ability for the task of gesture recognition based on sparse sEMG signals.

Human-machine interactions based on hand gesture recognition using deep learning methods

Human interaction with computers and other machines is becoming an increasingly important and relevant topic in the modern world. Hand gesture recognition technology is an innovative approach to managing computers and electronic devices that allows users to interact with technology through gestures and hand movements. This article presents deep learning methods that allow you to efficiently process and classify hand gestures and hand gesture recognition technologies for interacting with computers. This paper discusses modern deep learning methods such as convolutional neural networks (CNN) and recurrent neural networks (RNN), which show excellent results in gesture recognition tasks. Next, the development and implementation of a human-machine interaction system based on hand gesture recognition is discussed. System architectures are described, as well as technical and practical aspects of their application. In conclusion, the article summarizes the research results and outlines the prospects for the development of hand gesture recognition technology to improve human-machine interaction. The advantages and limitations of the technology are analyzed, as well as possible areas of its application in the future.

Overcoming the effect of muscle fatigue on gesture recognition based on sEMG via generative adversarial networks

Surface electromyography (sEMG)-controlled bionic prostheses have been extensively investigated recently. However, the majority of investigations pertaining to gesture recognition grounded in sEMG predominantly center their focus on the human body in a static and non-fatigued condition, thus neglecting the ramifications of muscle fatigue on the precision of gesture recognition. This study explores the effect of muscle fatigue on recognition accuracy in a gesture recognition task based on sEMG. The muscle fatigue induction experiment was designed, and eight subjects were recruited to participate in the experiment to collect the sEMG signal data sets of seven gesture actions under non-fatigue and fatigue conditions. Seven gesture actions under non-fatigue and fatigue conditions were identified by four classifiers, namely K-nearest neighbor (K-NN), support vector machine (SVM), decision tree (DT), and deep convolution neural network (CNN). The experimental results show that muscle fatigue has a great impact on the accuracy of gesture recognition. Specifically, using the four classifiers K-NN, SVM, DT and CNN trained in the non-fatigue state, the test accuracy of sEMG signals in the non-fatigue state is 96.7 %, 89.0 %, 87.3 % and 97.5 % respectively, while the test accuracy in the fatigue state is reduced to 53.3 %, 55.4 %, 45.8 % and 64.8 % respectively. In this regard, we propose a new method to overcome the effects of muscle fatigue, namely, the data enhancement method based on Wasserstein General Adversary Networks-Gradient Penalty (WGAN-GP), which is used to enhance the sEMG signal under fatigue. Through the data enhancement of the fatigue sEMG signal, the experimental results show that the final test accuracy in the fatigue state is improved by more than 20 %, which can reach 72.3 %, 80.9 %, 69.9 % and 92.1 % respectively. This shows that the method proposed by us can effectively overcome the influence of muscle fatigue on the accuracy of gesture recognition, and has made a great contribution to the improvement of the robustness of gesture recognition model based on sEMG signal.

Unsupervised Domain Adaptation by Causal Learning for Biometric Signal-based HCI

Biometric signal based human-computer interface (HCI) has attracted increasing attention due to its wide application in healthcare, entertainment, neurocomputing, and so on. In recent years, deep learning-based approaches have made great progress on biometric signal processing. However, the state-of-the-art (SOTA) approaches still suffer from model degradation across subjects or sessions. In this work, we propose a novel unsupervised domain adaptation approach for biometric signal-based HCI via causal representation learning. Specifically, three kinds of interventions on biometric signals (i.e., subjects, sessions, and trials) can be selected to generalize deep models across the selected intervention. In the proposed approach, a generative model is trained for producing intervened features that are subsequently used for learning transferable and causal relations with three modes. Experiments on the EEG-based emotion recognition task and sEMG-based gesture recognition task are conducted to confirm the superiority of our approach. An improvement of +0.21% on the task of inter-subject EEG-based emotion recognition is achieved using our approach. Besides, on the task of inter-session sEMG-based gesture recognition, our approach achieves improvements of +1.47%, +3.36%, +1.71%, and +1.01% on sEMG datasets including CSL-HDEMG, CapgMyo DB-b, 3DC, and Ninapro DB6, respectively. The proposed approach also works on the task of inter-trial sEMG-based gesture recognition and an average improvement of +0.66% on Ninapro databases is achieved. These experimental results show the superiority of the proposed approach compared with the SOTA unsupervised domain adaptation methods on HCIs based on biometric signal.

Dynamic gesture recognition based on compressed video for UAV control

As a multifunctional aircraft with small size, low cost and easy control, UAVs can be used in many fields such as gardening, plant protection, mapping, logistics, military, etc. For a more convenient human-machine interaction mode, the user interacts with the UAVs in the form of dynamic gestures. The current traditional approach is to train convolutional neural networks with images as direct inputs to the system for the purpose of controlling UAVs. However, this approach leaves the temporal representation information between images missing, resulting in poor training results or over-reliance on computing resources. Therefore, this paper proposes an efficient processing strategy, that is, for simple gesture tasks, the optimized frame extraction algorithm is adopted to process picture-based gesture recognition; for complex gesture tasks, compressed video is used as system input to complete video-based gesture recognition. Based on the training results in action recognition datasets, UCF-101 and HMDB-51, the efficiency of compressed video in gesture recognition tasks has been verified, which can be applied to UAV dynamic gesture control.

Visual Static Hand Gesture Recognition Using Convolutional Neural Network

Hand gestures are an essential part of human-to-human communication and interaction and, therefore, of technical applications. The aim is increasingly to achieve interaction between humans and computers that is as natural as possible, for example, by means of natural language or hand gestures. In the context of human-machine interaction research, these methods are consequently being explored more and more. However, the realization of natural communication between humans and computers is a major challenge. In the field of hand gesture recognition, research approaches are being pursued that use additional hardware, such as special gloves, to classify gestures with high accuracy. Recently, deep learning techniques using artificial neural networks have been increasingly proposed for the problem of gesture recognition without using such tools. In this context, we explore the approach of convolutional neural network (CNN) in detail for the task of hand gesture recognition. CNN is a deep neural network that can be used in the fields of visual object processing and classification. The goal of this work is to recognize ten types of static hand gestures in front of complex backgrounds and different hand sizes based on raw images without the use of extra hardware. We achieved good results with a CNN network architecture consisting of seven layers. Through data augmentation and skin segmentation, a significant increase in the model’s accuracy was achieved. On public benchmarks, two challenging datasets have been classified almost perfectly, with testing accuracies of 96.5% and 96.57%.

Hand Gesture Recognition and Cursor Control

Abstract: Convolutional neural network in this process to overcome the issue, use the network AutoGesNet for the gesture recognition task that creating effectiveneural network architecture is challenging. To be more precise, we first combine and preprocess three sets of gesture recognition data. The AutoGesNet search space and general architecture are then designed. Additionally, we employ transfer learning and reinforcement learning techniques to automatically create the intricate AutoGesNet architecture. Finally, the searched neural network is adjusted and retrained for two alternative input sizes. The retrained model performs accurately on both our data set and the NUS Hand Posture Dataset II, according to experiments. network that performs wellin terms of recognition accuracy. We will contrast andmerge AutoGesNet in further work

A Strain-Sensitive Flexible MoTe2-Based Memristor for Gesture Recognition

Conventional flexible electronics suffer from complex fabrication processes and great energy consumption due to the separated tactile sensors, storage devices, and processing units. In-sensor computing based on multifunctional devices allows efficient computation by directly sensing and processing signals and is considered an effective way to achieve text large-scale integration and low-power devices in the future. Herein, a strain-sensitive multifunctional memristor is prepared based on MoTe2. The device shows good retention (>105 s) and switching stability (>500 cycles). The neuromorphic computing capability of the device was verified using a gesture recognition task with accuracy reaches above 70% even at 50% background noise. The present results demonstrate the attractive multifunction integrated behaviors of MoTe2 devices, opening a new avenue for designing artificially intelligent electronic skins.

LST-EMG-Net: Long short-term transformer feature fusion network for sEMG gesture recognition.

With the development of signal analysis technology and artificial intelligence, surface electromyography (sEMG) signal gesture recognition is widely used in rehabilitation therapy, human-computer interaction, and other fields. Deep learning has gradually become the mainstream technology for gesture recognition. It is necessary to consider the characteristics of the surface EMG signal when constructing the deep learning model. The surface electromyography signal is an information carrier that can reflect neuromuscular activity. Under the same circumstances, a longer signal segment contains more information about muscle activity, and a shorter segment contains less information about muscle activity. Thus, signals with longer segments are suitable for recognizing gestures that mobilize complex muscle activity, and signals with shorter segments are suitable for recognizing gestures that mobilize simple muscle activity. However, current deep learning models usually extract features from single-length signal segments. This can easily cause a mismatch between the amount of information in the features and the information needed to recognize gestures, which is not conducive to improving the accuracy and stability of recognition. Therefore, in this article, we develop a long short-term transformer feature fusion network (referred to as LST-EMG-Net) that considers the differences in the timing lengths of EMG segments required for the recognition of different gestures. LST-EMG-Net imports multichannel sEMG datasets into a long short-term encoder. The encoder extracts the sEMG signals' long short-term features. Finally, we successfully fuse the features using a feature cross-attention module and output the gesture category. We evaluated LST-EMG-Net on multiple datasets based on sparse channels and high density. It reached 81.47, 88.24, and 98.95% accuracy on Ninapro DB2E2, DB5E3 partial gesture, and CapgMyo DB-c, respectively. Following the experiment, we demonstrated that LST-EMG-Net could increase the accuracy and stability of various gesture identification and recognition tasks better than existing networks.

Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices.

Audio-visual speech recognition (AVSR) is one of the most promising solutions for reliable speech recognition, particularly when audio is corrupted by noise. Additional visual information can be used for both automatic lip-reading and gesture recognition. Hand gestures are a form of non-verbal communication and can be used as a very important part of modern human-computer interaction systems. Currently, audio and video modalities are easily accessible by sensors of mobile devices. However, there is no out-of-the-box solution for automatic audio-visual speech and gesture recognition. This study introduces two deep neural network-based model architectures: one for AVSR and one for gesture recognition. The main novelty regarding audio-visual speech recognition lies in fine-tuning strategies for both visual and acoustic features and in the proposed end-to-end model, which considers three modality fusion approaches: prediction-level, feature-level, and model-level. The main novelty in gesture recognition lies in a unique set of spatio-temporal features, including those that consider lip articulation information. As there are no available datasets for the combined task, we evaluated our methods on two different large-scale corpora-LRW and AUTSL-and outperformed existing methods on both audio-visual speech recognition and gesture recognition tasks. We achieved AVSR accuracy for the LRW dataset equal to 98.76% and gesture recognition rate for the AUTSL dataset equal to 98.56%. The results obtained demonstrate not only the high performance of the proposed methodology, but also the fundamental possibility of recognizing audio-visual speech and gestures by sensors of mobile devices.

Adversarial Unsupervised Domain Adaptation for Hand Gesture Recognition Using Thermal Images

Hand gesture recognition has a wide range of applications, including in the automotive and industrial sectors, health assistive systems, authentication, and so on. Thermal images are more resistant to environmental changes than red–green–blue (RGB) images for hand gesture recognition. However, one disadvantage of using thermal images for the aforementioned task is the scarcity of labeled thermal datasets. To tackle this problem, we propose a method that combines unsupervised domain adaptation (UDA) techniques with deep-learning (DL) technology to remove the need for labeled data in the learning process. There are several types and methods for implementing UDA, with adversarial UDA being one of the most common. In this article, the first time in this field, we propose a novel adversarial UDA model that uses channel attention and bottleneck layers to learn domain-invariant features across RGB and thermal domains. Thus, the proposed model leverages the information from the labeled RGB data to solve the hand gesture recognition task using thermal images. We evaluate the proposed model on two hand gesture datasets, namely, Sign Digit Classification and Alphabet Gesture Classification, and compare it to other benchmark models in terms of accuracy, model size, and model parameters. Our model outperforms the other state-of-the-art methods on the Sign Digit Classification and Alphabet Gesture Classification datasets and achieves 91.32% and 80.91% target test accuracy, respectively.

Static Gesture Recognition Algorithm Based on Improved YOLOv5s

With the government’s increasing support for the virtual reality (VR)/augmented reality (AR) industry, it has developed rapidly in recent years. Gesture recognition, as an important human-computer interaction method in VR/AR technology, is widely used in the field of virtual reality. The current static gesture recognition technology has several shortcomings, such as low recognition accuracy and low recognition speed. A static gesture recognition algorithm based on improved YOLOv5s is proposed to address these issues. The content-aware re-assembly of features (CARAFE) is used to replace the nearest neighbor up-sampling method in YOLOv5s to make full use of the semantic information in the feature map and improve the recognition accuracy of the model for gesture regions. The adaptive spatial feature fusion (ASFF) method is introduced to filter out useless information and retain useful information for efficient feature fusion. The bottleneck transformer method is initially introduced into the gesture recognition task, reducing the number of model parameters and increasing the accuracy while accelerating the inference speed. The improved algorithm achieved an mAP(mean average precision) of 96.8%, a 3.1% improvement in average accuracy compared with the original YOLOv5s algorithm; the confidence level of the actual detection results was higher than the original algorithm.

Hand Gesture-based Teleoperation Control of a Mecanum-wheeled Mobile Robot

The aim of this study is to design an interface for the teleoperation control of a Mecanum-wheeled Mobile Robot (MMR). To make a cost-effective and non-invasive teleoperation interface approach, a hand gesture recognition strategy via a webcam is adopted. In this study, seven hand gestures are recognized on a desktop PC and then the corresponding velocity commands are sent to the MMR via WiFi. The SqueezeNet architecture is employed for the hand gesture recognition and MMR teleoperation task after various Convolutional Neural Network (CNN) models are compared. The comparison considers some aspects, i.e., accuracy, memory usage, and recognition speed, to find the most feasible method for real-time hand gesture recognition and teleoperation control. Experimental results are presented to show the excellence of the proposed SqueezeNet architecture.

A Novel SE-CNN Attention Architecture for sEMG-Based Hand Gesture Recognition

In this article, to reduce the complexity and improve the generalization ability of current gesture recognition systems, we propose a novel SE-CNN attention architecture for sEMG-based hand gesture recognition. The proposed algorithm introduces a temporal squeeze-and-excite block into a simple CNN architecture and then utilizes it to recalibrate the weights of the feature outputs from the convolutional layer. By enhancing important features while suppressing useless ones, the model realizes gesture recognition efficiently. The last procedure of the proposed algorithm is utilizing a simple attention mechanism to enhance the learned representations of sEMG signals to perform multi-channel sEMG-based gesture recognition tasks. To evaluate the effectiveness and accuracy of the proposed algorithm, we conduct experiments involving multi-gesture datasets Ninapro DB4 and Ninapro DB5 for both inter-session validation and subject-wise cross-validation. After a series of comparisons with the previous models, the proposed algorithm effectively increases the robustness with improved gesture recognition performance and generalization ability.

Hardware Architecture Design for Hand Gesture Recognition System on FPGA

Hand gesture recognition (HGR) is one of the widely-used human-computer interaction technology. With HGR, the user can operate the interaction system without touching any devices. For a better experience, recognition accuracy and computational speed are especially important. In this work, a small-footprint HGR model and its hardware architecture design are proposed. The model first processes the hand segmentation and uses the feature to recognize the hand gesture. The model mainly consists of depthwise separable convolution to reduce the overall parameters and computations. We transfer the hand segmentation task with some features to the hand gesture recognition task as a single-stage model. Based on this hardware-efficient model, we propose the hardware architecture of the whole neural model including depthwise convolution, pointwise convolution, batch normalization, and max-pooling. We also demonstrate it on the evaluation board. The whole system is implemented on the Xilinx ZCU106 evaluation board. The implemented system can achieve the performance of 52.6 fps and 65.6 GOPS based on the evaluation board.

Convolutional Transformer Fusion Blocks for Multi-Modal Gesture Recognition

Gesture recognition defines an important information channel in human-computer interaction. Intuitively, combining inputs from multiple modalities improves the recognition rate. In this work, we explore multi-modal video-based gesture recognition tasks by fusing spatio-temporal representation of relevant distinguishing features from different modalities. We present a self-attention based transformer fusion architecture to distill the knowledge from different modalities in two-stream convolutional neural networks (CNNs). For this, we introduce convolutions into the self-attention function and design the Convolutional Transformer Fusion Blocks (CTFB) for multi-modal data fusion. These fusion blocks can be easily added at different abstraction levels of the feature hierarchy in existing two-stream CNNs. In addition, the information exchange between two-stream CNNs along the feature hierarchy has so far been barely explored. We propose and evaluate different architectures for multi-level fusion pathways using CTFB to gain insights into the information flow between both streams. Our method achieves state-of-the-art or competitive performance on three benchmark gesture recognition datasets: a) IsoGD, b) NVGesture, and c) IPN hand. Extensive evaluation demonstrates the effectiveness of the proposed CTFB both in terms of recognition rate as well as resource efficiency.