Gesture Recognition Accuracy Research Articles

Adoption of Artificial Intelligence Along with Gesture Interactive Robot in Musical Perception Education Based on Deep Learning Method

The present work aims to improve students’ interest in music teaching and promote modern teaching. A distributed application system of artificial intelligence gesture interactive robot is designed through deep learning technology and applied to music perception education. First, the user’s gesture instruction data is collected through the double channel convolution neural network (DCCNN). It uses the double-size convolution kernel to extract feature information in the image and collect the video frame’s gesture instruction. Secondly, a two-stream convolutional neural network (two-stream CNN) recognizes the collected gesture instruction data. The spatial and temporal information is extracted from RGB color mode (RGB) images and optical flow images and input into the two-stream CNN to fuse the prediction results of each network as the final detection result. Then, the distributed system used by the interactive robot is introduced. This structure can improve the stability of the interactive systems and reduce the requirements for local hardware performance. Finally, experiments are conducted to test the gesture command acquisition and recognition network, and the performance of the gesture interactive robot in practice. The results indicate that combining convolution kernels of [Formula: see text] and [Formula: see text] can increase the recognition accuracy of DCCNN to 98% and effectively collect gesture instruction data. The gesture recognition accuracy of two-stream CNN after training reaches 90%, higher than the mainstream dynamic gesture recognition algorithm trained with the same data set. Finally, the recognition test of gesture instructions is carried out on the gesture interactive robot reported here. The results show that the recognition accuracy of the gesture interactive robots is more than 90%, meeting the routine interaction needs. Therefore, the interactive gesture robot has good reliability and stability and is applicable to music perception teaching. The research reported here has guiding significance for establishing music teaching with multiple perception modes.

SEMG-Based Gesture Classifier for a Rehabilitation Glove

Human hand gesture recognition from surface electromyography (sEMG) signals is one of the main paradigms for prosthetic and rehabilitation device control. The accuracy of gesture recognition is correlated with the control mechanism. In this work, a new classifier based on the Bayesian neural network, pattern recognition networks, and layer recurrent network is presented. The online results obtained with this architecture represent a promising solution for hand gesture recognition (98.7% accuracy) in sEMG signal classification. For real time classification performance with rehabilitation devices, a new simple and efficient interface is developed in which users can re-train the classification algorithm with their own sEMG gesture data in a few minutes while enables shape memory alloy-based rehabilitation device connection and control. The position of reference for the rehabilitation device is generated by the algorithm based on the classifier, which is capable of detecting user movement intention in real time. The main aim of this study is to prove that the device control algorithm is adapted to the characteristics and necessities of the user through the proposed classifier with high accuracy in hand gesture recognition.

Real-Time Control of Intelligent Prosthetic Hand Based on the Improved TCN.

Intelligent prosthetic hand is an important branch of intelligent robotics. It can remotely replace humans to complete various complex tasks and also help humans to complete rehabilitation training. In human-computer interaction technology, the prosthetic hand can be accurately controlled by surface electromyography (sEMG). This paper proposes a new multichannel fusion scheme (MSFS) to extend the virtual channels of sEMG and improve the accuracy of gesture recognition. In addition, the Temporal Convolutional Network (TCN) in deep learning has been improved to enhance the performance of the network. Finally, the sEMG is collected by the Myo armband and the prosthetic hand is controlled in real time to validate the new method. The experimental results show that the method proposed in this paper can improve the accuracy of the control intelligent prosthetic hand, and the accuracy rate is 93.69%.

A lightweight hand gesture recognition in complex backgrounds

Hand Gesture Recognition (HGR) is widely used in human–computer interaction due to its convenience. However, there are still some challenges in real-world scenarios, such as recognizing hand gestures in the complex backgrounds. To this end, the paper proposes a two-stage HGR system to solve the above issue. Specifically, the first stage performs accurate segmentation to segment the hand from the background. The segmentation network combines dilated residual network, atrous spatial pyramid pooling module and a simplified decoder. The segmentation network can effectively determine hand region even in challenging backgrounds. In the second stage, the double-channel Convolutional Neural Networks (CNNs) are presented to improve the recognition performance. The double-channel CNNs can learn features from the RGB input images and the segmented hand images separately. Experiment results show that the proposed method has an accuracy of 91.17% with the model size of 1.8 MB, both of which are better than other state-of-arts in hand gesture recognition. The method successfully constructed a lightweight model while keeping a high gesture recognition accuracy at final.

SoDar: Multitarget Gesture Recognition Based on SIMO Doppler Radar

In recent years, various intelligent activity recognition systems have been developed based on radio frequency signals such as radar, Wi-Fi, and radio frequency identification (RFID). When only one target is present, these systems can often provide high accuracy in recognizing different activities. However, such activity identification systems often fail to work due to signal interference when multiple targets coexist. To address this problem, we propose a multitarget gesture recognition system, named SoDar, based on a commercial single-input multi-output (SIMO) dual-channel Doppler radar. First, we employ endpoint detection, low-pass filtering, and discrete wavelet transform for data preprocessing. Then, we design a multitarget signal separation algorithm by maximizing the signal-to-noise ratio, and further refine the obtained signal based on principle component analysis. Afterward, we put forward a two-stage feature extraction method to extract both static and dynamic features from each separated signal. Finally, a classification model is trained to recognize the gestures of multiple targets. To verify the performance of SoDar, we selected nine different combinations of six gestures for two targets and collected more than 8000 data samples. Experimental results showed that the accuracy of two-target gesture recognition is above 90%.

Fatigue-Sensitivity Comparison of sEMG and A-Mode Ultrasound based Hand Gesture Recognition.

Though physiological signal based human-machine interfaces (HMIs) have recently developed rapidly, their practical use is restricted by many real-world environmental factors, one of which is muscle fatigue. This paper explores the sensitivities between surface electromyography (sEMG) and A-mode ultrasound (AUS) sensing modalities subject to muscle fatigue in the context of hand gesture recognition tasks. Two metrics, mean classification accuracy ( mCA) and decline rate ( DR), are proposed to evaluate the accuracy and muscle fatigue sensitivity between sEMG and AUS based HMIs. Muscle fatigue inducing experiment was designed and eight subjects were recruited to participate in the experiment. The gesture recognition accuracies of sEMG and AUS under non-fatigue state and fatigue state are compared through Mahalanobis distance based classifier linear discriminant analysis (LDA). In addition, Mahalanobis distance based metrics, repeatability index ( RI) and separability index ( SI), are introduced to evaluate the changes in the feature distribution during muscle fatigue and reveal the cause of the fatigue sensitivity difference between sEMG and AUS signals. The experimental results demonstrate that the fatigue robustness of AUS signal is better than that of sEMG signal. Specifically, with the employment of the LDA classifier trained under non-fatigue state, the testing accuracy of the sEMG signal on the non-fatigue state is 94.96%, while reduce to 68.26% on the fatigue state. The testing accuracy of the AUS signal on the corresponding states is 99.68% and 91.24% respectively. AUS signal attains higher mCA and lower DR, indicating that it has advantages over sEMG signal in terms of both accuracy and muscle fatigue sensitivity. In addition, the RI and RI/SI analysis reveal that before and after muscle fatigue, the consistency of AUS feature distribution is better than that of sEMG. These research outcomes validate that AUS is more tolerant to feature migration caused by muscle fatigue than sEMG.

Gesture Recognition Algorithm Introducing Ghost Feature Mapping and Channel Attention Mechanism

To solve the problem that the lightweight object detection network has insufficient ability to extract static gesture features, high false detection rate and missed detection rate, a lightweight gesture recognition algorithm is proposed based on YOLOv4-tiny network structure. First, a more powerful and low-cost ghost feature mapping is introduced to enhance the ability of network to obtain multi-scale gesture features. Then, the embedded channel attention mechanism realizes feature recalibration and achieves the purpose of reducing background interference. Finally, Swish is used as the main activation function to further improve the accuracy of gesture recognition. The experimental results on the gesture dataset show that the proposed algorithm has better recognition performance than YOLOv4-tiny. For multi-scale gestures under different environmental conditions, the algorithm achieves accurate classification as well as real-time detection, and has better recognition performance for small-scale gestures.

Wrist-Worn Hand Gesture Recognition While Walking via Transfer Learning.

Walking, one of the most common daily activities, causes unwanted movement artifacts which can significantly deteriorate hand gesture recognition accuracy. However, traditional hand gesture recognition algorithms are typically developed and validated with wrist-worn devices only during static human poses, neglecting the critical importance of dynamic effects on gesture accuracy. Thus, we developed and validated a signal decomposition approach via empirical mode decomposition to accurately segment target gestures from coupled raw signals during dynamic walking and a transfer learning method based on distribution adaptation to enable gesture recognition through domain transfer between dynamic walking and static standing scenarios. Ten healthy subjects performed seven hand gestures during both walking and standing experiments while wearing an IMU wrist-worn device. Experimental results showed that the signal decomposition approach reduced the gesture detection error by 83.8%, and the transfer learning approach (20% transfer rate) improved hand gesture recognition accuracy by 15.1%. This ground-breaking work demonstrates the feasibility of hand gesture recognition while walking via wrist-worn sensing. These findings serve to inform real-life and ubiquitous adoption of wrist-worn hand gesture recognition for intuitive human-machine interaction in dynamic walking situations.

A Novel Myoelectric Control Scheme Supporting Synchronous Gesture Recognition and Muscle Force Estimation.

Aiming to provide feasible solutions for the realization of the robust and natural myoelectric control systems, a novel myoelectric control scheme supporting gesture recognition and muscle force estimation is proposed in this study. Eleven grasping gestures abstracted from daily life are selected as the target gesture set. The high-density surface electromyography (HD-sEMG) of the forearm flexor and the grasping force signal are collected simultaneously. The synchronous prediction of gesture category and instantaneous force is realized by the multi-task learning (MTL) technique. Especially, a post-processing algorithm based on threshold method is conducted to overcome the influence of force variation on the accuracy of gesture recognition. The experimental results show that the proposed post-processing method can decrease the classification error significantly. Specifically, the overall gesture classification error is reduced by 27 ~ 30 percent compared with not using the post-processing method; and 16 ~ 24 percent compared with using classical post-processing methods. The whole scheme can realize the synchronous gesture recognition and force estimation with 9.35 ± 11.48% gesture classification error and 0.1479 ± 0.0436 root-mean-square deviation force estimation accuracy. Meanwhile, it is feasible in different number of electrodes and well meets the real-time requirement of the EMG control system in response time delay (about 28.22 ~ 113.16ms on average). The proposed framework provides the possibility for myoelectric control supporting synchronous gesture recognition and force estimation, which can be extended and applied in the fields of myoelectric prosthesis and exoskeleton devices.

Cross Modality Knowledge Distillation Between A-Mode Ultrasound and Surface Electromyography

Surface electromyography (sEMG) and A-mode ultrasound (AUS) are two widely employed sensing modalities to detect muscle activities. By comparison, AUS modality shows the characteristics of higher decoding accuracy than sEMG modality. However, AUS is far less reliable than sEMG in actual long-term use. To resolve this contradiction, we considered leveraging AUS as a teacher to supervise sEMG training better and learning an augmented sEMG representation. Firstly, a novel network architecture MINDS (MultI-branch Network with Diverse focuS) was proposed for gesture recognition, which was suitable for both sEMG and AUS modalities. Secondly, a cross modality knowledge distillation (CMKD) framework was proposed, to transfer the latent knowledge of AUS to sEMG through Kullback-Leibler divergence loss. The gesture recognition accuracies were compared between MINDS and the existing networks. The experimental results demonstrated that MINDS outperforms other networks under both sEMG and AUS modalities. Furthermore, the feasibility of the CMKD framework was evaluated on the proposed MINDS and other existing networks. The results revealed that with knowledge distillation from AUS, the accuracy of the sEMG modality obtained a significant improvement, regardless of the employed network architecture. This work confirms the superiority of the proposed MINDS network and the feasibility of the proposed CMKD framework.

Real-time and Accurate Gesture Recognition with Commercial RFID Devices

Gesture recognition based on radio frequency identification (RFID) has attracted much research attention in recent years. Most existing RFID-based gesture recognition approaches use signal profile matching to distinguish different gestures, which incur large recognition latency and fail to support real-time applications. In this paper, we design and implement ReActor, a real-time and accurate gesture recognition system that recognizes a user's gestures with low latency and high accuracy even when the gestures'speed varies. ReActor combines the time-domain statistical features and the frequency-domain features to precisely represent the signal profile corresponding to different gestures. To maintain high accuracy across different environments, we preprocess the signals to remove reflection signals from surrounding objects and use only the signals related to gestures to train the classifier. Moreover, we train a classifier to predict the speed of the gesture and feed the extracted features to different classifiers according to the speed. We implement ReActor and evaluate its performance in different scenarios. Experimental results show that ReActor achieves an average accuracy of 97.2% in recognizing 18 different gestures with an average latency of 72 ms, more than two orders of magnitude faster than approaches based on profile template matching.

Разработка Python приложения для распознавания жестов рук из видеопотока RGB и RGBD камер

Gesture recognition systems have changed a lot recently, due to the development of modern data capture devices (sensors) and the development of new recognition algorithms. The article presents the results of a study for recognizing static and dynamic hand gestures from a video stream from RGB and RGBD cameras, namely from the Logitech HD Pro Webcam C920 webcam and from the Intel RealSense D435 depth camera. Software implementation is done using Python 3.6 tools. Open source Python libraries provide robust implementations of image processing and segmentation algorithms. The feature extraction and gesture classification subsystem is based on the VGG-16 neural network architecture implemented using the TensorFlow and Keras deep learning frameworks. The technical characteristics of the cameras are given. The algorithm of the application is described. The research results aimed at comparing data capture devices under various experimental conditions (distance and illumination) are presented. Experimental results show that using the Intel RealSense D435 depth camera provides more accurate gesture recognition under various experimental conditions.

Reliability Analysis for Finger Movement Recognition With Raw Electromyographic Signal by Evidential Convolutional Networks.

Hand gesture recognition with surface electromyography (sEMG) is indispensable for Muscle-Gesture-Computer Interface. The usual focus of it is upon performance evaluation involving the accuracy and robustness of hand gesture recognition. However, addressing the reliability of such classifiers has been absent, to our best knowledge. This may be due to the lack of consensus on the definition of model reliability in this field. An uncertainty-aware model has the potential to self-evaluate the quality of its inference, thereby making it more reliable. Moreover, uncertainty-based rejection has been shown to improve the performance of sEMG-based hand gesture recognition. Therefore, we first define model reliability here as the quality of its uncertainty estimation and propose an offline framework to quantify it. To promote reliability analysis, we propose a novel end-to-end uncertainty-aware finger movement classifier, i.e., evidential convolutional neural network (ECNN), and illustrate the advantages of its multidimensional uncertainties such as vacuity and dissonance. Extensive comparisons of accuracy and reliability are conducted on NinaPro Database 5, exercise A, across CNN and three variants of ECNN based on different training strategies. The results of classifying 12 finger movements over 10 subjects show that the best mean accuracy achieved by ECNN is 76.34%, which is slightly higher than the state-of-the-art performance. Furthermore, ECNN variants are more reliable than CNN in general, where the highest improvement of reliability of 19.33% is observed. This work demonstrates the potential of ECNN and recommends using the proposed reliability analysis as a supplementary measure for studying sEMG-based hand gesture recognition.

Design of Human-Computer Interaction Product Interface of Intelligent Service System based on User Experience

The current intelligent service platform for human-computer interaction products and services on the user experience is not comprehensive enough, resulting in user satisfaction cannot reach the ideal level. Therefore, a new human-computer interface of intelligent service system based on user experience is designed. Build a user experience based intelligent service system hardware platform, the introduction of the full name of hypertext markup language, so that more personalized design to be met. Design user experience PC terminal and Bluetooth/RS-485 gateway module to achieve two-way signal conversion between Bluetooth and RS-485. Based on ARM processor, the speech recognition of human-computer interaction is completed, the features of collected data are extracted, and the hand gesture recognition is completed. In order to optimize the human-computer interaction effect, Kinect is used to track and identify moving objects, and the 3D interactive image is simulated by fused texture. Experimental results show that the proposed method has a higher probability of receiving data, and the recognition rate of gesture features and recognition accuracy can reach more than 90%.

Chinese Traffic Police Gesture Recognition Based on Graph Convolutional Network in Natural Scene

For an automated driving system to be robust, it needs to recognize not only fixed signals such as traffic signs and traffic lights, but also gestures used by traffic police. With the aim to achieve this requirement, this paper proposes a new gesture recognition technology based on a graph convolutional network (GCN) according to an analysis of the characteristics of gestures used by Chinese traffic police. To begin, we used a spatial–temporal graph convolutional network (ST-GCN) as a base network while introducing the attention mechanism, which enhanced the effective features of gestures used by traffic police and balanced the information distribution of skeleton joints in the spatial dimension. Next, to solve the problem of the former graph structure only representing the physical structure of the human body, which cannot capture the potential effective features, this paper proposes an adaptive graph structure (AGS) model to explore the hidden feature between traffic police gesture nodes and a temporal attention mechanism (TAS) to extract features in the temporal dimension. In this paper, we established a traffic police gesture dataset, which contained 20,480 videos in total, and an ablation study was carried out to verify the effectiveness of the method we proposed. The experiment results show that the proposed method improves the accuracy of traffic police gesture recognition to a certain degree; the top-1 is 87.72%, and the top-3 is 95.26%. In addition, to validate the method’s generalization ability, we also carried out an experiment on the Kinetics–Skeleton dataset in this paper; the results show that the proposed method is better than some of the existing action-recognition algorithms.

Finger-Gesture Recognition for Visible Light Communication Systems Using Machine Learning

Gesture recognition (GR) has many applications for human-computer interaction (HCI) in the healthcare, home, and business arenas. However, the common techniques to realize gesture recognition using video processing are computationally intensive and expensive. In this work, we propose to task existing visible light communications (VLC) systems with gesture recognition. Different finger movements are identified by training on the light transitions between fingers using the long short-term memory (LSTM) neural network. This paper describes the design and implementation of the gesture recognition technique for a practical VLC system operating over a distance of 48 cm. The platform uses a single low-cost light-emitting diode (LED) and photo-diode sensor at the receiver side. The system recognizes gestures from interruptions in the direct light transmission, and is therefore suitable for high-speed communication. Gesture recognition accuracies were conducted for five gestures, and results demonstrate that the proposed system is able to accurately identify the gestures in up to 88% of cases.

Agricultural Machinery Virtual Assembly System Using Dynamic Gesture Recognitive Interaction Based on a CNN and LSTM Network

The agricultural machinery experiment is restricted by the crop production season. Missing the crop growth cycle will extend the machine development period. The use of virtual reality technology to complete preassembly and preliminary experiments can reduce the loss caused by this problem. To improve the intelligence and stability of virtual assembly, this paper proposed a more stable dynamic gesture cognition framework: the TCP/IP protocol constituted the network communication terminal, the leap motion-based vision system constituted the gesture data collection terminal, and the CNN-LSTM network constituted the dynamic gesture recognition classification terminal. The dynamic gesture recognition framework and the harvester virtual assembly platform formed a virtual assembly system to achieve gesture interaction. Through experimental analysis, the improved CNN-LSTM network had a small volume and could quickly establish a stable and accurate gesture recognition model with an average accuracy of 98.0% (±0.894). The assembly efficiency of the virtual assembly system with the framework was improved by approximately 15%. The results showed that the accuracy and stability of this model met the requirements, the corresponding assembly parts were robust in the virtual simulation environment of the whole machine, and the harvesting behaviour in the virtual reality scene was close to the real scene. The virtual assembly system under this framework provided technical support for unmanned farms and virtual experiments on agricultural machinery.

Dynamic Gesture Recognition Using Surface EMG Signals Based on Multi-Stream Residual Network.

Gesture recognition technology is widely used in the flexible and precise control of manipulators in the assisted medical field. Our MResLSTM algorithm can effectively perform dynamic gesture recognition. The result of surface EMG signal decoding is applied to the controller, which can improve the fluency of artificial hand control. Much current gesture recognition research using sEMG has focused on static gestures. In addition, the accuracy of recognition depends on the extraction and selection of features. However, Static gesture research cannot meet the requirements of natural human-computer interaction and dexterous control of manipulators. Therefore, a multi-stream residual network (MResLSTM) is proposed for dynamic hand movement recognition. This study aims to improve the accuracy and stability of dynamic gesture recognition. Simultaneously, it can also advance the research on the smooth control of the Manipulator. We combine the residual model and the convolutional short-term memory model into a unified framework. The architecture extracts spatiotemporal features from two aspects: global and deep, and combines feature fusion to retain essential information. The strategy of pointwise group convolution and channel shuffle is used to reduce the number of network calculations. A dataset is constructed containing six dynamic gestures for model training. The experimental results show that on the same recognition model, the gesture recognition effect of fusion of sEMG signal and acceleration signal is better than that of only using sEMG signal. The proposed approach obtains competitive performance on our dataset with the recognition accuracies of 93.52%, achieving state-of-the-art performance with 89.65% precision on the Ninapro DB1 dataset. Our bionic calculation method is applied to the controller, which can realize the continuity of human-computer interaction and the flexibility of manipulator control.

A study on the algorithm of ultrasonic detection and recognition based on DAG‐SVMs mixed HMM of teleoperation gestures for intelligent manufacturing devices

Remote control for the position and status of a machine or an equipment can often be teleoperated by gestures in an intelligent manufacturing environment. In order to solve the problems that gestures with two directions such as left and right cannot be detected by single ultrasonic frequency, double different ultrasonic frequencies are used to detect gestures by the Doppler shift, and an algorithm of the recognition gesture based on the DAG-SVMs mixed Hidden Markov Model (HMM) is proposed to identify and classify the extracted feature sequences. Thus, four more types of gestures are expanded other than that of reading display screen information, and the comparative experiments to classify and recognise gestures of teleoperation are made with DAG-SVMs, the HMM, the DAG-SVMs mixed HMM, and other improved HMM algorithms. The test results have shown that the mean rate of gesture recognition for the algorithm based on the DAG-SVMs mixed HMM is 94.917%, which is 9.497% higher than that of the unimproved HMM, and its recognition accuracy of complex teleoperation gestures is improved by 2.3% compared with other improved HMM algorithms. The experimental results show that the DAG-SVMs mixed HMM algorithm has a good effect on recognition for the gestures of teleoperation and it can perform gesture recognition accurately.

Hand character gesture recognition based on a single millimetre‐wave radar chip

In this work, a new method for hand character gesture recognition based on a single millimetre-wave radar chip is proposed. Most current radar-based gesture recognition approaches use Doppler spectrograms or the feature extraction of Doppler spectrograms. In contrast, the authors propose using the trajectory points of gesture movements. Compared with the previous methods, the variations in the gesture movement speed, distance and direction during data acquisition have relatively little impact on the generated trajectories. The character trajectories drawn by gestures in the air output a fixed shape, and this fixed output facilitates the classification of gestures at a later stage. In this work, a single millimetre-wave radar is used to collect data and a three-dimensional fast Fourier Transform (3D-FFT) algorithm is used to obtain the gesture trajectories. For the trajectory points, a trajectory conversion image method is proposed and finally the improved Xception network proposed in this work is used for image recognition. Compared with traditional methods, the proposed method does not require large-scale data acquisition to facilitate different users. This work is concluded by recognising hand character gestures and comparing them with other commonly used classification methods to verify the gesture recognition accuracy of the proposed method.