Hand Gesture Recognition Method Research Articles

Contagious infection-free medical interaction system with machine vision controlled by remote hand gesture during an operation

Background and objectiveMedical image visualization is a requirement in many types of surgery such as orthopaedic, spinal, thoracic procedures or tumour resection to eliminate risk such as “wrong level surgery”. However, direct contact with physical devices such as mice or touch screens to control images is a challenge because of the potential risk of infection. To prevent the spread of infection in sterile environments, a contagious infection-free medical interaction system has been developed for manipulating medical images. MethodsWe proposed an integrated system with three key modules: hand landmark detection, hand pointing, and hand gesture recognition. A proposed depth enhancement algorithm is combined with a deep learning hand landmark detector to generate hand landmarks. Based on the designed system, a proposed hand-pointing system combined with projection and ray-pointing techniques allows for reducing fatigue during manipulation. A proposed landmark geometry constraint algorithm and deep learning method were applied to detect six gestures including click, open, close, zoom, drag, and rotation. Additionally, a control menu was developed to effectively activate common functions. ResultsThe proposed hand-pointing system allowed for a large control range of up to 1200 mm in both vertical and horizontal direction. The proposed hand gesture recognition method showed high accuracy of over 97% and real-time response. ConclusionThis paper described the contagious infection-free medical interaction system that enables precise and effective manipulation of medical images within the large control range, while minimizing hand fatigue.

Improving sEMG-Based Hand Gesture Recognition through Optimizing Parameters and Sliding Voting Classifiers

In this paper, we present a preliminary study that proposes to improve surface electromyography (sEMG)-based hand gesture recognition through optimizing parameters and sliding voting classifiers. Targeting the high-performing myoelectric control system, the traditional methods for hand gesture recognition still need to further improve the classification accuracy and utilization rate for sEMG signals. Therefore, the proposed method first optimizes parameters to reduce redundant information by selecting the proper values for the window length, the overlapping rate, the number of channels, and the features of sEMG signals. In addition, the random forest (RF) classifier is an advanced classifier for sEMG-based hand gesture recognition. To further improve classification performance, this paper proposes a sliding voting random forest (SVRF) classifier which can reduce potential pseudo decisions made by the RF classifier. Finally, experiments were conducted using two sEMG datasets, named DB2 and DB4, from the NinaPro database, as well as self-collected data. The results illustrate a certain improvement in classification accuracy based on the optimized values for window length, overlapping rate, number of channels, and features of sEMG signals. And the SVRF classifier can significantly improve performance with higher accuracy compared with the traditional linear discriminate analysis (LDA), k-nearest neighbors (KNN), support vector machine (SVM), and RF classifiers.

Point cloud based hand gesture recognition using template matching

Hand gesture recognition is important in human-computer interaction with wide applications in many fields. Different from common hand gesture recognition based on 2D images acquired from RGB camera, the utilization of 3D images provides additional spatial information about the target and attracts more and more attention in hand gesture recognition. However, most 3D images for hand gesture recognition are based on depth maps, which only take the distance information as a channel of 2D images, without taking full use of the 3D information. Besides, greater data volume of 3D images brings challenges to the arithmetic facility of hand gesture recognition. Here, we proposed a point cloud based method for hand gesture recognition. To fully use the 3D information, plane points for template matching were extracted based on their normal distributions, which leads to the average recognition rate over 97%. Pre-classification was implemented to ensure a high-efficient recognition without additional requirements for the computer. The proposed method may provide approach for accurate and efficient hand gesture recognition based on 3D images.

SEMG and IMU Data-Based Hand Gesture Recognition Method Using Multistream CNN With a Fine-Tuning Transfer Framework

sEMG and IMU Data-Based Hand Gesture Recognition Method Using Multistream CNN With a Fine-Tuning Transfer Framework

“I Can See Your Password”: A Case Study About Cybersecurity Risks in Mid-Air Interactions of Mixed Reality-Based Smart Manufacturing Applications

Abstract This paper aims to present a potential cybersecurity risk existing in mixed reality (MR)-based smart manufacturing applications that decipher digital passwords through a single RGB camera to capture the user’s mid-air gestures. We first created a test bed, which is an MR-based smart factory management system consisting of mid-air gesture-based user interfaces (UIs) on a video see-through MR head-mounted display. To interact with UIs and input information, the user’s hand movements and gestures are tracked by the MR system. We setup the experiment to be the estimation of the password input by users through mid-air hand gestures on a virtual numeric keypad. To achieve this goal, we developed a lightweight machine learning-based hand position tracking and gesture recognition method. This method takes either video streaming or recorded video clips (taken by a single RGB camera in front of the user) as input, where the videos record the users’ hand movements and gestures but not the virtual UIs. With the assumption of the known size, position, and layout of the keypad, the machine learning method estimates the password through hand gesture recognition and finger position detection. The evaluation result indicates the effectiveness of the proposed method, with a high accuracy of 97.03%, 94.06%, and 83.83% for 2-digit, 4-digit, and 6-digit passwords, respectively, using real-time video streaming as input with known length condition. Under the unknown length condition, the proposed method reaches 85.50%, 76.15%, and 77.89% accuracy for 2-digit, 4-digit, and 6-digit passwords, respectively.

Online cross session electromyographic hand gesture recognition using deep learning and transfer learning

In recent years, hand gesture recognition in human-computer interfaces is usually based on surface electromyography because the signals are non-intrusive and are not affected by the variations of light, position, and orientation of the hand. Deep learning algorithms have become increasingly more prominent in gesture recognition for the ability to automatically learn features from large amounts of data. However, delicate and complicated network structures brought by deep learning, which are elaborately designed for cross session tasks, need more computing time to be trained and tested, which can hardly be applied to the online system. In this study, an online electromyographic hand gesture recognition method using deep learning and transfer learning is proposed. The deep learning model includes a feature extractor, a label classifier, and a gesture predictor. The feature extractor is based on the temporal convolutional network, which is designed to learn high-level discriminant features from the input signals. The label classifier includes three fully connected layers, designed to classify hand gesture labels using the feature vector which is produced by the feature extractor. The gesture predictor uses a threshold voting algorithm to predict the gesture, used at the stage of testing to perform the online recognition. Transfer learning technique is used to transfer model parameters from one pre-trained model, which costs less time and can be applied for online applications. The proposed model is verified on both the Myo dataset and the public NinaPro database. The proposed transfer learning scheme is shown to systematically and significantly enhance the performance of the proposed model on the two datasets, only using no more than three sessions to retrain the label predictor can achieve the accuracy of more than 90% of that obtained though the normal training of the whole parts of the model using full training sessions.

Mask-guided multiscale feature aggregation network for hand gesture recognition

Hand gesture recognition from images is a longstanding computer vision task that can be used to build a potential bridge for human-computer interaction and sign language translation. For number of methods proposed for hand gesture recognition (HGR); however, difficult scenarios such as different scales of hand gestures and complex backgrounds exist, making them less effective. In this paper, we propose an end-to-end multiscale feature learning network for HGR, which consists of a CNN-based backbone network, a feature aggregation pyramid network (FAPN) embedded with a two-stage expansion-squeeze-aggregation (ESA) module, and three task-specific prediction branches. First, the backbone network extracts multiscale features from the original hand gesture images. Furthermore, the FAPN embedded with two-stage ESA extensively exploits multiscale feature information and learns hand gesture-specific features at different scales. Then, the mask loss guides the network to locate hand-specific regions during the training stage, and finally, the classification and regression branches output the category and location of a hand gesture during the model training and prediction. The experimental results on two publicly available datasets show that the proposed method outperforms most state-of-the-art HGR methods.

Hand gesture recognition method using FMCW radar based on multidomain fusion

Hand gesture recognition method using FMCW radar based on multidomain fusion

Real-Time Hand Gesture Recognition by Decoding Motor Unit Discharges Across Multiple Motor Tasks From Surface Electromyography.

Surface electromyography (EMG) decomposition techniques have been developed to decode motor neuron activities non-invasively in the past decades, showing superior performance in human-machine interfaces such as gesture recognition and proportional control. However, neural decoding across multiple motor tasks and in real-time remains challenging, which limits its wide application. In this work, we proposed a real-time hand gesture recognition method by decoding motor unit (MU) discharges across multiple motor tasks ( 10) in a motion-wise way. The EMG signals were first divided into numerous segments related to motions. The convolution kernel compensation algorithm was applied for each segment individually. The local MU filters, which indicate the MU-EMG correlation for each motion, were calculated iteratively in each segment and reused for global EMG decomposition to trace the MU discharges across motor tasks in real-time. The motion-wise decomposition method was applied on the high-density EMG signals recorded during twelve hand gesture tasks from eleven non-disabled participants. The neural feature of discharge count was extracted for gesture recognition based on five common classifiers. On average, 164 ±34 MUs were identified for twelve motions from each subject, with a pulse-to-noise ratio of 32.1 ±5.6 dB. The average time cost of EMG decomposition in a sliding window of 50 ms was less than 5 ms. The average classification accuracy using a linear discriminant analysis classifier was 94.6 ±8.1%, which was significantly higher than that of a time-domain feature called root mean square. The superiority of the proposed method was also validated with a previously published EMG database comprising 65 gestures. These results indicate the feasibility and superiority of the proposed method for MU identification and hand gesture recognition across multiple motor tasks, extending the potential applications of neural decoding in human-machine interfaces.

A comparative analysis of neural network models for hand gesture recognition methods

In recent years, gesture recognition methods have undergone major changes. Because the very demand for it has reached a different level. Humans have increasingly begun using various areas of human activity. The purpose of gesture recognition is to record gestures formed in a certain way and then tracked by a device such as a camera. Hand gestures can be used as a form of communication within many different applications. It can be used by people with various disabilities, including those with hearing and speech impairments, to communicate and interact socially with others. Our research demonstrates various methods for implementing hand gesture recognition based on Hidden Markov Model (HMM), Convolutional Neural Network (CNN), Diffractive Deep Neural Network (D2NN) and other neural networks. This research reviews previous approaches and results of hand gesture recognition methods, hypotheses, diagrams, as well as a comparative review between various gesture recognition methods are given in this paper.

HandFormer: A Dynamic Hand Gesture Recognition Method Based on Attention Mechanism

The application of dynamic gestures is extensive in the field of automated intelligent manufacturing. Due to the temporal and spatial complexity of dynamic gesture data, traditional machine learning algorithms struggle to extract accurate gesture features. Existing dynamic gesture recognition algorithms have complex network designs, high parameter counts, and inadequate gesture feature extraction. In order to solve the problems of low accuracy and high computational complexity in current dynamic gesture recognition, a network model based on the MetaFormer architecture and an attention mechanism was designed. The proposed network fuses a CNN (convolutional neural network) and Transformer model by embedding spatial attention convolution and temporal attention convolution into the Transformer model. Specifically, the token mixer in the MetaFormer block is replaced by the Spatial Attention Convolution Block and Temporal Attention Convolution Block to obtain the Spatial Attention Former Block and Temporal Attention Former Block. Firstly, each frame of the input image is quickly down-sampled by the PoolFormer block and then input to the Spatial Attention Former Block to learn spatial feature information. Then, the spatial feature maps learned from each frame are concatenated along the channel dimension and input to the Temporal Attention Former Block to learn the temporal feature information of the gesture action. Finally, the learned overall feature information is classified to obtain the category of dynamic gestures. The model achieves an average recognition accuracy of 96.72% and 92.16% on two publicly available datasets, Jester and NVGesture, respectively.

Gesture Detection and Recognition Based on Object Detection in Complex Background

In practical human–computer interaction, a hand gesture recognition method based on improved YOLOv5 is proposed to address the problem of low recognition accuracy and slow speed with complex backgrounds. By replacing the CSP1_x module in the YOLOv5 backbone network with an efficient layer aggregation network, a richer combination of gradient paths can be obtained to improve the network’s learning and expressive capabilities and enhance recognition speed. The CBAM attention mechanism is introduced to filtering gesture features in channel and spatial dimensions, reducing various types of interference in complex background gesture images and enhancing the network’s robustness against complex backgrounds. Experimental verification was conducted on two complex background gesture datasets, EgoHands and TinyHGR, with recognition accuracies of mAP0.5:0.95 at 75.6% and 66.8%, respectively, and a recognition speed of 64 FPS for 640 × 640 input images. The results show that the proposed method can recognize gestures quickly and accurately with complex backgrounds, and has higher recognition accuracy and stronger robustness compared to YOLOv5l, YOLOv7, and other comparative algorithms.

Static hand gesture recognition method based on the Vision Transformer

Static hand gesture recognition method based on the Vision Transformer

MediaPipe Hands 모델 기반의 손 제스처 인식을 이용한 사용자 인터페이스

This paper is related to a user interface based on hand gestures. The purpose of this paper is to propose a real-time hand gesture recognition method that recognizes hand point, hand mouse, hand dial, and handwriting gestures using MediaPipe Hands model. In this paper, various static poses are detected using landmarks, which are the output values of MediaPipe Hands model, and then dynamic gestures are detected through the movement and change of these static poses. Then, using these dynamic gestures, hand point, hand mouse, hand dial, and hand writing functions are implemented. In this paper, when a usability test was performed on the proposed hand gesture recognition method, an average recognition rate of 96.4% was obtained. A computer simulation was performed focusing on the scenario of controlling a virtual smart home device using the proposed hand gesture recognition method.

Дослідження моделей для розпізнавання жестів з використанням 3D конволюційних нейронних мереж та візуальних трансформерів

The work primarily focuses on addressing the contemporary challenge of hand gesture recognition, driven by the overarching objectives of revolutionizing military training methodologies, enhancing human-machine interactions, and facilitating improved communication between individuals with disabilities and machines. In-depth scrutiny of the methods for hand gesture recognition involves a comprehensive analysis, encompassing both established historical computer vision approaches and the latest deep learning trends available in the present day. This investigation delves into the fundamental principles that underpin the design of models utilizing 3D convolutional neural networks and visual transformers. Within the 3D-CNN architecture that was analyzed, a convolutional neural network with two convolutional layers and two pooling layers is considered. Each 3D convolution is obtained by convolving a 3D filter kernel and summing multiple adjacent frames to create a 3D cube. The visual transformer architecture that is consisting of a visual transformer with Linear Projection, a Transformer Encoder, and two sub-layers: the Multi-head Self-Attention (MSA) layer and the feedforward layer, also known as the Multi-Layer Perceptron (MLP), is considered. This research endeavors to push the boundaries of hand gesture recognition by deploying models trained on the ASL and NUS-II datasets, which encompass a diverse array of sign language images. The performance of these models is assessed after 20 training epochs, drawing insights from various performance metrics, including recall, precision, and the F1 score. Additionally, the study investigates the impact on model performance when adopting the ViT architecture after both 20 and 40 training epochs were performed. This analysis unveils the scenarios in which 3D convolutional neural networks and visual transformers achieve superior accuracy results. Simultaneously, it sheds light on the inherent constraints that accompany each approach within the ever-evolving landscape of environmental variables and computational resources. The research identifies cutting-edge architectural paradigms for hand gesture recognition, rooted in deep learning, which hold immense promise for further exploration and eventual implementation and integration into software products.

Hand Gesture-based Teleoperation Control of a Mecanum-wheeled Mobile Robot

The aim of this study is to design an interface for the teleoperation control of a Mecanum-wheeled Mobile Robot (MMR). To make a cost-effective and non-invasive teleoperation interface approach, a hand gesture recognition strategy via a webcam is adopted. In this study, seven hand gestures are recognized on a desktop PC and then the corresponding velocity commands are sent to the MMR via WiFi. The SqueezeNet architecture is employed for the hand gesture recognition and MMR teleoperation task after various Convolutional Neural Network (CNN) models are compared. The comparison considers some aspects, i.e., accuracy, memory usage, and recognition speed, to find the most feasible method for real-time hand gesture recognition and teleoperation control. Experimental results are presented to show the excellence of the proposed SqueezeNet architecture.

Unsupervised Domain Adaptation for Gesture Identification Against Electrode Shift

Surface electromyogram (sEMG)-based hand gesture recognition, which interprets commands given by humans through sEMG signals, performs well in many studies. However, its recognition accuracy drops dramatically due to electrode shift since the distributions of motion classes are changed. Although calibrating the system with newly collected samples after electrode shift maintains the accuracy, collecting labeled samples is inconvenient and time-consuming since the procedure is rigid. However, the calibration may not work properly without label, especially when the change is significant. This study proposes a user friendly and convenient calibration method for hand gesture recognition by an unsupervised domain adaptation method, which only obtains the unlabeled samples of preselected benchmark classes from users in calibration. The change of benchmark classes is captured by unlabeled samples by a clustering method. The other classes are estimated based on the benchmark classes by regression models. As a result, the information of all classes is used to calibrate the system. Linear discriminant analysis is used to demonstrate our model. A dataset with ten subjects is collected to verify the performance empirically. Experimental results confirm that our method utilizes the unlabeled benchmark class samples in calibration and achieves 75.55% average accuracy. Our method is more robust to electrode shift and improves around 8.5% accuracy consistently on all subjects compared with the methods without calibration or label information in calibration. Although the accuracy of our method is slightly less than the ones using label calibration samples, our calibration data collection is more convenient and less complicated.

Clustering-Driven DGS-Based Micro-Doppler Feature Extraction for Automatic Dynamic Hand Gesture Recognition.

We propose in this work a dynamic group sparsity (DGS) based time-frequency feature extraction method for dynamic hand gesture recognition (HGR) using millimeter-wave radar sensors. Micro-Doppler signatures of hand gestures show both sparse and structured characteristics in time-frequency domain, but previous study only focus on sparsity. We firstly introduce the structured prior when modeling the micro-Doppler signatures in this work to further enhance the features of hand gestures. The time-frequency distributions of dynamic hand gestures are first modeled using a dynamic group sparse model. A DGS-Subspace Pursuit (DGS-SP) algorithm is then utilized to extract the corresponding features. Finally, the support vector machine (SVM) classifier is employed to realize the dynamic HGR based on the extracted group sparse micro-Doppler features. The experiment shows that the proposed method achieved 3.3% recognition accuracy improvement over the sparsity-based method and has a better recognition accuracy than CNN based method in small dataset.

Dynamic Hand Gesture Recognition Based on 3D Hand Pose Estimation for Human–Robot Interaction

Dynamic hand gesture recognition is a challenging problem in the area of hand-based human–robot interaction (HRI), such as issues of a complex environment and dynamic perception. In the context of this problem, we learn from the principle of the data-glove-based hand gesture recognition method and propose a dynamic hand gesture recognition method based on 3D hand pose estimation. This method uses 3D hand pose estimation, data fusion and deep neural network to improve the recognition accuracy of dynamic hand gestures. First, a 2D hand pose estimation method based on OpenPose is improved to obtain a fast 3D hand pose estimation method. Second, the weighted sum fusion method is utilized to combine the RGB, depth and 3D skeleton data of hand gestures. Finally, a 3DCNN + ConvLSTM framework is used to identify and classify the combined dynamic hand gesture data. In the experiment, the proposed method is verified on a developed dynamic hand gesture database for HRI and gets 92.4% accuracy. Comparative experiment results verify the reliability and efficiency of the proposed method.

Robust Dynamic Hand Gesture Recognition Based on Millimeter Wave Radar Using Atten-TsNN

The millimeter-wave radar can sense the subtle movement of the hand. However, the traditional hand gesture recognition methods do not work well in dynamic interference scenarios. To address this issue, a robust hand gesture recognition method is proposed based on the self-attention time-series neural networks. Firstly, the original radar echo is constructed in terms of frame, sequence, and channel at the input terminal of the network. A one-dimensional time-series neural network is built, and the time-distributed layer is used as the wrapper to extract the feature from each frame sequence independently. Then the self-attention mechanism is employed to assign the adequate weights to the sequence of frames entered in parallel to obtain the inter-frame correlation and suppress the random interference. Finally, the Global AvgPooling layer is used to reduce the number of channels, and the fully connected layer outputs the label of the gesture. The experimental results show that the proposed method can achieve a high recognition rate in the presence of random dynamic interference.