Hand Gesture Recognition Research Articles

STCNet: Spatio-Temporal Cross Network with subject-aware contrastive learning for hand gesture recognition in surface EMG.

This paper introduces the Spatio-Temporal Cross Network (STCNet), a novel deep learning architecture tailored for robust hand gesture recognition in surface electromyography (sEMG) across multiple subjects. We address the challenges associated with the inter-subject variability and environmental factors such as electrode shift and muscle fatigue, which traditionally undermine the robustness of gesture recognition systems. STCNet integrates a convolutional-recurrent architecture with a spatio-temporal block that extracts features over segmented time intervals, enhancing both spatial and temporal analysis. Additionally, a rolling convolution technique designed to reflect the circular band structure of the sEMG measurement device is incorporated, thus capturing the inherent spatial relationships more effectively. We further propose a subject-aware contrastive learning framework that utilizes both subject and gesture label information to align the representation of vector space. Our comprehensive experimental evaluations demonstrate the superiority of STCNet under aggregated conditions, achieving state-of-the-art performance on benchmark datasets and effectively managing the variability among different subjects. The implemented code can be found at https://github.com/KNU-BrainAI/STCNet.

AirCanvas using OpenCV and MediaPipe

Human-Computer Interaction (HCI) has undergone significant transformations with the advent of Artificial Intelligence (AI) and Machine Learning (ML), enhancing the ways in which users engage with computing systems. This paper introduces AirCanvas, a novel hands-free digital interaction tool that leverages air gestures for intuitive and seamless computer control. The system uses advanced image processing techniques, specifically OpenCV for visual data analysis and MediaPipe for accurate hand gesture recognition, enabling users to manipulate virtual environments without physical touch. By integrating a standard webcam as the primary sensor, AirCanvas offers an accessible solution for gesture-based interaction, eliminating the need for specialized external hardware like motion sensors or gloves. Users can perform a variety of tasks such as virtual drawing, cursor control, and presentation navigation with simple hand gestures in 3D space. The system's gesture recognition capabilities are powered by deep learning models trained on large datasets of hand movements, ensuring robust performance in diverse lighting and environmental conditions. The potential applications of AirCanvas are far-reaching, ranging from interactive art creation to more practical uses in assistive technology, where people with mobility impairments can benefit from hands-free control. In the field of robotics, the tool can enable more natural human-robot interaction, while in gaming, it can introduce new forms of immersive gameplay through gesture-based interfaces. Furthermore, the tool's open-source nature allows for further customization and enhancement, fostering innovation and collaboration in various industries. As human-computer interaction continues to evolve, AirCanvas represents a significant step forward in making technology more intuitive, engaging, and accessible for all users.

A novel approach to frame reduction in robust hand gesture recognition

Abstract Hand gesture recognition is becoming an increasingly integral part of our daily lives, enabling seamless communication, enhancing interaction, and revolutionizing multiple industries. To ensure a more precise and efficient system, the key aspect of hand gestures lies in detecting hand patterns and retrieving the hand gestures. However, as the volume of video data increases, extracting the essential hand patterns while excluding unnecessary frames becomes a challenge. Addressing this issue, a novel Harris Hawk Optimization K-Means frame reduction is proposed, inspired by the hunting behavior of Harris Hawks in nature. This proposed approach combines the Harris Hawk Optimization algorithm with the K-Means clustering method. The algorithm simulates the hunting behavior of Harris Hawks and utilizes Euclidean distance as a fitness function to determine the optimal frames. Subsequently, the K-Means clustering method is employed to group similar frames together based on these optimal selections. An average frame is generated and aggregated for each cluster to form a reduced set of frames. These reduced frames are then classified using the modified Mobilenet V2 model, outperforming other state-of-the-art techniques by achieving an exceptional accuracy rate of 99.93%. The experiment results lay the groundwork for incorporating the novel framework of hand gesture recognition into a range of applications, including sign language interpretation, human-computer interaction, and virtual reality systems.

Hand Gesture Recognition

Hand gesture recognition system received great attention in the recent few years because of its manifoldness applications and the ability to interact with machine efficiently through human computer interaction. In this paper a survey of recent hand gesture recognition systems is presented. Key issues of hand gesture recognition system are presented with challenges of gesture system. Review methods of recent postures and gestures recognition system presented as well. Summary of research results of hand gesture methods, databases, and comparison between main gesture recognition phases are also given. Advantages and drawbacks of the discussed systems are explained finally

Vision-Based Gesture-Driven Drone Control in a Metaverse-Inspired 3D Simulation Environment

Unlike traditional remote control systems for controlling unmanned aerial vehicles (UAVs) and drones, active research is being carried out in the domain of vision-based hand gesture recognition systems for drone control. However, contrary to static and sensor based hand gesture recognition, recognizing dynamic hand gestures is challenging due to the complex nature of multi-dimensional hand gesture data, present in 2D images. In a real-time application scenario, performance and safety is crucial. Therefore we propose a hybrid lightweight dynamic hand gesture recognition system and a 3D simulator based drone control environment for live simulation. We used transfer learning-based computer vision techniques to detect dynamic hand gestures in real-time. The gestures are recognized, based on which predetermine commands are selected and sent to a drone simulation environment that operates on a different computer via socket connectivity. Without conventional input devices, hand gesture detection integrated with the virtual environment offers a user-friendly and immersive way to control drone motions, improving user interaction. Through a variety of test situations, the efficacy of this technique is illustrated, highlighting its potential uses in remote-control systems, gaming, and training. The system is tested and evaluated in real-time, outperforming state-of-the-art methods. The code utilized in this study are publicly accessible. Further details can be found in the “Data Availability Statement”.

Spatio-Temporal Transformer with Kolmogorov–Arnold Network for Skeleton-Based Hand Gesture Recognition

Manually crafted features often suffer from being subjective, having an inadequate accuracy, or lacking in robustness in recognition. Meanwhile, existing deep learning methods often overlook the structural and dynamic characteristics of the human hand, failing to fully explore the contextual information of joints in both the spatial and temporal domains. To effectively capture dependencies between the hand joints that are not adjacent but may have potential connections, it is essential to learn long-term relationships. This study proposes a skeleton-based hand gesture recognition framework, the ST-KT, a spatio-temporal graph convolution network, and a transformer with the Kolmogorov–Arnold Network (KAN) model. It incorporates spatio-temporal graph convolution network (ST-GCN) modules and a spatio-temporal transformer module with KAN (KAN–Transformer). ST-GCN modules, which include a spatial graph convolution network (SGCN) and a temporal convolution network (TCN), extract primary features from skeleton sequences by leveraging the strength of graph convolutional networks in the spatio-temporal domain. A spatio-temporal position embedding method integrates node features, enriching representations by including node identities and temporal information. The transformer layer includes a spatial KAN–Transformer (S-KT) and a temporal KAN–Transformer (T-KT), which further extract joint features by learning edge weights and node embeddings, providing richer feature representations and the capability for nonlinear modeling. We evaluated the performance of our method on two challenging skeleton-based dynamic gesture datasets: our method achieved an accuracy of 97.5% on the SHREC’17 track dataset and 94.3% on the DHG-14/28 dataset. These results demonstrate that our proposed method, ST-KT, effectively captures dynamic skeleton changes and complex joint relationships.

Enhanced dynamic hand gesture recognition for finger disabilities using deep learning and an optimized Otsu threshold method

Abstract Hand gestures serve as a powerful means of communication, capable of conveying extensive information across various public health domains, including medicine and education. The process of hand gesture recognition involves the use of mathematical algorithms to identify human gestures and finds applications in areas such as communication for the deaf, human-computer interaction, intelligent driving, and virtual reality. This study introduces a robust method aimed at recognizing dynamic hand gestures, particularly for individuals with finger disabilities. The approach begins with segmenting hand gestures from intricate backgrounds using an advanced Otsu segmentation algorithm, while also integrating motion data from RGB video sequences. Hand gestures are transformed into texture and contour features, which are utilized as input for a hybrid model that merges a convolutional neural network (CNN) with a recurrent neural network (RNN). The model employs Inception-v3 for feature extraction complemented by an LSTM layer for classification. The focus of the study is on recognizing six dynamic gestures, with particular emphasis on "scroll right" and "scroll down" due to their high accuracy in recognition. The model demonstrated an average precision of 84.34% across all gestures, achieving 87.57% for gestures involving finger impairments. These results highlight the model’s effectiveness in practical applications for dynamic hand gesture recognition.

An Ensemble Approach with Evolutionary Algorithm for Hand Posture Classification

Grasping is a fundamental action in daily life and particularly evident during mealtime situations where various grasping actions occur with tableware such as chopsticks, spoons, forks, bowls, and cups, each serving specific purposes. While tableware usage varies across regions and cultures, recognizing grasping actions is crucial for assessing performance in daily activities. In this study, we focus on assessing grasping functionality in terms of tableware usage during meals and propose a method for identifying hand movements. In recent years, there has been a surge in developing approaches for hand pose estimation and gesture recognition using deep learning. However, these approaches encounter common challenges, including the need for large-scale datasets, hyperparameter tuning, significant time and computational costs, and limited applicability to incremental learning. To address these challenges, we propose an ensemble approach employing extreme learning machines to recognize grasp postures. In addition, we apply spatiotemporal modeling to extract the relationship between grasp postures and the surrounding tools during mealtimes.

Hand Gesture Recognition from Surface Electromyography Signals with Graph Convolutional Network and Attention Mechanisms

Hand Gesture Recognition from Surface Electromyography Signals with Graph Convolutional Network and Attention Mechanisms

Gesture Driven Gaming: A Deep Dive into Computer Vision-Based Hand Gesture Recognition

Gesture Driven Gaming: A Deep Dive into Computer Vision-Based Hand Gesture Recognition

Hand gestures classification of sEMG signals based on BiLSTM-metaheuristic optimization and hybrid U-Net-MobileNetV2 encoder architecture

Surface electromyography (sEMG) data has been extensively utilized in deep learning algorithms for hand movement classification. This paper aims to introduce a novel method for hand gesture classification using sEMG data, addressing accuracy challenges seen in previous studies. We propose a U-Net architecture incorporating a MobileNetV2 encoder, enhanced by a novel Bidirectional Long Short-Term Memory (BiLSTM) and metaheuristic optimization for spatial feature extraction in hand gesture and motion recognition. Bayesian optimization is employed as the metaheuristic approach to optimize the BiLSTM model’s architecture. To address the non-stationarity of sEMG signals, we employ a windowing strategy for signal augmentation within deep learning architectures. The MobileNetV2 encoder and U-Net architecture extract relevant features from sEMG spectrogram images. Edge computing integration is leveraged to further enhance innovation by enabling real-time processing and decision-making closer to the data source. Six standard databases were utilized, achieving an average accuracy of 90.23% with our proposed model, showcasing a 3–4% average accuracy improvement and a 10% variance reduction. Notably, Mendeley Data, BioPatRec DB3, and BioPatRec DB1 surpassed advanced models in their respective domains with classification accuracies of 88.71%, 90.2%, and 88.6%, respectively. Experimental results underscore the significant enhancement in generalizability and gesture recognition robustness. This approach offers a fresh perspective on prosthetic management and human-machine interaction, emphasizing its efficacy in improving accuracy and reducing variance for enhanced prosthetic control and interaction with machines through edge computing integration.

Multimodal Hand Gesture Recognition Using Surface Electromyography and Inertial Measurement Units with 3D-CNN and Transfer Learning

Multimodal Hand Gesture Recognition Using Surface Electromyography and Inertial Measurement Units with 3D-CNN and Transfer Learning

Recognition of Hand Gestures Using Image with Histogram Feature Extraction and Euclidean Distance Classification Method

Recognition of Hand Gestures Using Image with Histogram Feature Extraction and Euclidean Distance Classification Method

AirStrum: A virtual guitar using real-time hand gesture recognition and strumming technique

AirStrum: A virtual guitar using real-time hand gesture recognition and strumming technique

Hand Gesture Recognition Using Frequency‐Modulated Continuous Wave Radar on Tactile Displays for the Visually Impaired

Touchscreens are essential parts of many electronics in daily lives of sighted people in the digital information era. On the other hand, visually impaired users rely on tactile displays as one of the key communication devices to interact with the digital world. However, due to their working mechanism and the uneven surface of tactile displays, one of the key features of screens for sighted users is surprisingly challenging to implement: precision touch input. To overcome this, a hand gesture recognition system is developed using a frequency‐modulated continuous wave millimeter‐wave radar. A multifeature encoder method is used to obtain the range and velocity information from the radar to translate the data into spectrogram images. Gesture recognition is implemented for common input gestures: single/double‐click, swipe‐right/left, scroll‐up/down, zoom‐in/out, and rotate‐anticlockwise/clockwise. The gesture recognition and classification are based on machine learning, support vector machines, deep learning, and convolutional neural network approaches. The chosen model You‐Only‐Look‐Once (YOLOv8) shows a high accuracy of 97.1% by iterating only 30 epochs with only 500 collected data samples per gesture. This research paves the way toward using radar sensors not only for tactile displays but also for other digital devices in human–computer interaction.

Gesture Recognition for Interactive Presentation Control: A Deep Learning and Edge Computing Approach on Raspberry Pi

Gesture recognition is one of the important tech- nologies for intuitive HCI applications, for ex- ample, in interactive presentations. This re- view focuses on recent work on gesture recog- nition systems, deep learning-based methods for presentation control by hand gestures, on edge devices like the Raspberry Pi. This work de- scribes both static and dynamic frameworks for gesture recognition and human-machine interac- tion techniques based on deep learning.Recent studies show the usability of real-time hand gesture recognition using Python-based systems via Raspberry Pi, stressing portability and ef- ficiency in solutions. The review discusses var- ious deep learning frameworks such as Tensor- Flow, which can be used in image preprocess- ing and model training to enhance the accuracy of gesture detection. Techniques cover wear- able devices to computer vision and demon- strate the flexibility of gesture recognition in a variety of hardware platforms. Advances in edge computing enable complex gesture recog- nition on low- power devices, with reduced la- tency, and enables improved accessibility.The re- view addresses challenges such as gesture type distinction, responsiveness, and environmental variability in developing reliable gesture-based systems. In this regard, it emphasizes the Rasp- berry Pi as an edge computing solution for in- teractive presentation control in HCI. Keywords Gesture Recognition Human-Computer Inter- action (HCI) Edge Computing Deep Learning Real-Time Gesture Detection

Comparative analysis of force sensitive resistor circuitry for use in force myography systems for hand gesture recognition

Wearable technologies for hand gesture classification are becoming increasingly prominent due to the growing need for more natural, human-centered control of complex devices. This need is particularly evident in emerging fields such as virtual reality and bionic prostheses, which require precise control with minimal delay. One method used for hand gesture recognition is force myography (FMG), which utilizes non-invasive pressure sensors to measure radial muscle forces on the skin’s surface of the forearm during hand movements. These sensors, typically force-sensitive resistors (FSRs), require additional circuitry to generate analog output signals, which are then classified using machine learning to derive corresponding control signals for the device. The performance of hand gesture classification can be influenced by the characteristics of this output signal, which may vary depending on the circuitry used. Our study examined three commonly used circuits in FMG systems: the voltage divider (VD), unity gain amplifier (UGA), and transimpedance amplifier (TIA). We first conducted benchtop testing of FSRs to characterize the impact of this circuitry on linearity, deadband, hysteresis, and drift, all metrics with the potential to influence an FMG system’s performance. To evaluate the circuit’s performance in hand gesture classification, we constructed an FMG band with 8 FSRs, using an adjustable Velcro strap and interchangeable circuitry. Wearing the FMG band, participants (N = 15) were instructed to perform 10 hand gestures commonly used in daily living. Our findings indicated that the UGA circuit outperformed others in minimizing hysteresis, drift and deadband with comparable results to the VD, while the TIA circuit excelled in ensuring linearity. Further, contemporary machine learning algorithms used to detect hand gestures were unaffected by the circuitry employed. These results suggest that applications of FMG requiring precise sensing of force values would likely benefit from use of the UGA. Alternatively, if hand gesture state classification is the only use case, developers can take advantage of benefits offered from using less complex circuitry such as the VD.

Posture-invariant myoelectric control with self-calibrating random forests.

Myoelectric control systems translate different patterns of electromyographic (EMG) signals into the control commands of diverse human-machine interfaces via hand gesture recognition, enabling intuitive control of prosthesis and immersive interactions in the metaverse. The effect of arm position is a confounding factor leading to the variability of EMG characteristics. Developing a model with its characteristics and performance invariant across postures, could largely promote the translation of myoelectric control into real world practice. Here we propose a self-calibrating random forest (RF) model which can (1) be pre-trained on data from many users, then one-shot calibrated on a new user and (2) self-calibrate in an unsupervised and autonomous way to adapt to varying arm positions. Analyses on data from 86 participants (66 for pre-training and 20 in real-time evaluation experiments) demonstrate the high generalisability of the proposed RF architecture to varying arm positions. Our work promotes the use of simple, explainable, efficient and parallelisable model for posture-invariant myoelectric control.

Wi-TCG: a WiFi gesture recognition method based on transfer learning and conditional generative adversarial networks

Abstract With the rapid progress of WiFi technology, WiFi-based wireless sensing technology has opened up new ways for contactless human-computer interaction. However, hand gesture recognition technology faces low-quality data sets, insufficient model robustness, poor universality when the application scenario changes, high training costs, and weak generalization ability. To this end, this study innovatively proposes the Wi-TCG method, which combines transfer learning and conditional generative adversarial network (CGAN) to optimize WiFi gesture recognition. This method uses commercial Wi-Fi devices to collect channel state information (CSI) of gesture actions. It innovatively extracts Doppler shift image data as the input of CGAN to generate virtual data with similar characteristics to expand the training sample set. The network is fine-tuned using transfer learning techniques to recognize multiple gesture action categories in different scenarios accurately. In tests of two new natural scenes and six new gesture categories, the Wi-TCG method achieved a high recognition accuracy of 93.1%, providing strong support for applying WiFi-based wireless sensing technology in contactless human-computer interaction.

A Node Point Approach for Real-Time Hand Gesture Recognition Using Support Vector Machines (SVM)

Gesture recognition systems have become increasingly popular in recent years due to their ability to improve the user experience and accessibility in real time. However, accurate and efficient hand gesture recognition remains a challenge, especially in dynamic environments. This study presents real-time hand gestures and poses using computer vision. Fundamentally, the system can identify and categorize a wide range of hand movements through real-time analysis of the node connect points of the user's fingers. Every successive point is rendered in such a way that associations between several points have been oriented, and differences between each gesture may be classified. Apart from this, the machine learning model is to be trained on such joint points of the user’s fingers using angle approximation in between them and gradient as per pixels per inch according to a hand gesture-capturing adapter like a camera. For this task to be done, we aim to use the Support Vector Machine (SVM) algorithm for recognizing the classifying parameters of live gestures. The study utilizes support vector machines (SVM) to deliberately cluster signals according to their node focuses. To enhance the accuracy and resilience of the whole model, a support vector machine is incorporated into the classifier to maximize the classification function of the courier-based domain.