Continuous Gesture Recognition Research Articles

WiCGesture: Meta-Motion-Based Continuous Gesture Recognition With Wi-Fi

WiCGesture: Meta-Motion-Based Continuous Gesture Recognition With Wi-Fi

Optimizing echo state networks for continuous gesture recognition in mobile devices: A comparative study

Continuous gesture recognition can be used to enhance human-computer interaction. This can be accomplished by capturing human movement with the use of the Inertial Measurement Units in smartphones and using machine learning algorithms to predict the intended gestures. Echo State Networks (ESNs) consist of a fixed internal reservoir that is able to generate rich and diverse nonlinear dynamics in response to input signals that capture temporal dependencies within the signal. This makes ESNs well-suited for time series prediction tasks, such as continuous gesture recognition. However, their application has not been rigorously explored, with regard to gesture recognition. In this study, we sought to enhance the efficacy of ESN models in continuous gesture recognition by exploring diverse model structures, fine-tuning hyperparameters, and experimenting with various training approaches. We used three different training schemes that used the Leave-one-out Cross-validation (LOOCV) protocol to investigate the performance in real-world scenarios with different levels of data availability: Leaving out data from one user to use for testing (F1-score: 0.89), leaving out a fraction of data from all users to use in testing (F1-score: 0.96), and training and testing using LOOCV on a single user (F1-score: 0.99). The obtained results outperformed the Long Short-Term Memory (LSTM) performance from past research (F1-score: 0.87) while maintaining a low training time of approximately 13 seconds compared to 63 seconds for the LSTM model. Additionally, we further explored the performance of the ESN models through behaviour space analysis using memory capacity, Kernel Rank, and Generalization Rank. Our results demonstrate that ESNs can be optimized to achieve high performance on gesture recognition in mobile devices on multiple levels of data availability. These findings highlight the practical ability of ESNs to enhance human-computer interaction.

Real-time continuous gesture recognition system based on PSO-PNN

In the field of natural human–machine interaction, wearable gesture interaction technologies have received considerable attention, particularly continuous gesture (CG) recognition. However, CG faces several challenges, including the impact of motion characteristics on gesture recognition and performance that is not sufficiently robust. Traditional algorithms are highly dependent on samples, thus meeting the requirements of low sample volume and high accuracy simultaneously is challenging. To address these challenges, we propose a real-time CG recognition system based on particle swarm optimization (PSO) and probabilistic neural network (PNN). This system employs principal component analysis for signal dimensionality reduction to alleviate computational burden and uses K-means clustering and Pearson correlation coefficient to extract optimal features for gesture classification. In offline gesture recognition experiments involving six CGs, the algorithm achieved an accuracy rate of 97% with a training set of 300 samples and a runtime of just 31.25 ms. Compared to other five algorithms, the proposed algorithm improved accuracy by at least 9% and reduced the runtime by 40.475 ms. Moreover, gesture recognition experiments were conducted using different datasets, with the PSO-PNN algorithm achieving an average recognition rate of 90.17%, at least 9.84% higher than other algorithms. Finally, in experiments on online CG control for robots in complex environments, the PSO-PNN demonstrated real-time performance of 28.56 ms and a task completion rate of 90.67%, validating the feasibility of PSO-PNN. This research provides a substantial theoretical and technical foundation for the ongoing enhancement and application of CG recognition technology.

Multi-stream fusion network for continuous gesture recognition based on sEMG

Multi-stream fusion network for continuous gesture recognition based on sEMG

Multi-stream fusion network for continuous gesture recognition based on sEMG

Multi-stream fusion network for continuous gesture recognition based on sEMG

Continuous Gesture Recognition and Force Estimation Using sEMG Signal

Continuous Gesture Recognition and Force Estimation Using sEMG Signal

Continuous Gesture Sequences Recognition Based on Few-Shot Learning

A large number of demands for space on-orbit services to ensure the on-orbit system completes its specified tasks are foreseeable, and the efficiency and the security are the most significant factors when we carry out an on-orbit mission. And it can improve human-computer interaction efficiency in operations with proper gesture recognition solutions. In actual situations, the operations are complex and changeable, so the gestures used in interaction are also difficult to predict in advance due to the compounding of multiple consecutive gestures. To recognize such gestures based on computer vision (CV) requires complex models trained by a large amount of datasets, it is often unable to obtain enough gesture samples for training a complex model in real tasks, and the cost of labeling the collected gesture samples is quite expensive. Aiming at the problems mentioned above, we propose a few-shot continuous gesture recognition scheme based on RGB video. The scheme uses Mediapipe to detect the key points of each frame in the video stream, decomposes the basic components of gesture features based on certain human palm structure, and then extracts and combines the above basic gesture features by a lightweight autoencoder network. Our scheme can achieve 89.73% recognition accuracy on the 5-way 1-shot gesture recognition task which randomly selected 142 gesture instances of 5 categories from the RWTH German fingerspelling dataset.

A Continuous Gesture Segmentation and Recognition Method for Human-Robot Interaction

The process of human-computer cooperation using gesture recognition can make people get rid of the limitations of traditional input devices such as mouse and keyboard, and control artificial intelligence devices more efficiently and naturally. As a new way of human-robot interaction (HRI), gesture recognition has made some progress. There are many ways to realize gesture recognition combined with visual recognition, motion information acquisition and EMG signal. The research on isolated language gesture recognition has been quite mature, but the expression semantics of isolated gestures is single. In order to improve the interaction efficiency, the application of continuous gesture recognition is essential. This paper studies its application in continuous sign language sentence recognition based on inertial sensor and rule matching recognition algorithm. The recognition rate of nine single HRI gestures is 92.7%, and the HRI of combined gestures is realized.

Development of a low-resource wearable continuous gesture-to-speech conversion system

Purpose Mild to profound hearing impairment places limits on effective communication and day-to-day interaction. Sign language, being the primary mode of communication for people with hearing loss, lacks communicative efficacy. A wearable assistive device that aims to convert sign language into speech is proposed to facilitate communication between the unimpaired population (untrained in sign language) and the hearing impaired population. However, the wide use of geo-centric sign languages in India has resulted in the lack of standardised sign-language datasets. In the proposed work, a compact, low-resource, motion sensor-based, wireless, single and double hand-gesture recognition module is designed to address this issue. Material and methods The proposed module is designed to perform a two-step process with a Hidden Markov Model (HMM) based gesture-to-text conversion and a bilingual text-to-speech synthesis. Multi-threading based parallel processing is implemented to enable simultaneous working of the two systems to reduce the delay. In the proposed continuous-gesture recognition system, non-gesture hand motions are modelled using ergodic HMMs that are trained by concatenating all the states of gesture models, allowing equiprobable transitions. The proposed system is modelled and tested for American Sign Language (ASL) and user-defined gestures. Results The maximum performance of the proposed system in recognising single-handed and double-handed gestures in terms of F1-score is 98.17% and 84.85%, respectively. Further, the proposed system achieves a maximum F1-score of 98% and 83% in recognising isolated and continuous gestures, respectively. Conclusion The gesture-to-speech conversion system is ported on Raspberry Pi making the proposed system wireless, and highly mobile. Implications for rehabilitation The research work proposes to develop a gesture-to-speech conversion system to enable the deaf-mute population in communication. The major implications of the proposed work are: •A light-weight raspberry pi 3B + module hosts the entire hardware andsoftware, and is sufficient to train and test the gesture-to-speech conversion system, thereby ensuring greater mobility. •The proposed system can be customised to recognise user-defined gestures with just 5 examples of the new gesture. •The proposed system can be expanded to control home appliances (IoT applications) by combining the output of the proposed gesture recognition system with appropriate control interfaces.

G3-2d: A Continuous Gesture Segmentation and Recognition Model to Shoot Gestures from 3d Space to 2d Coordinate System

G3-2d: A Continuous Gesture Segmentation and Recognition Model to Shoot Gestures from 3d Space to 2d Coordinate System

Spatio-temporal continuous gesture recognition under degraded environments: performance comparison between 3D integral imaging (InIm) and RGB-D sensors

In this paper, we introduce a deep learning-based spatio-temporal continuous human gesture recognition algorithm under degraded conditions using three-dimensional (3D) integral imaging. The proposed system is shown as an efficient continuous human gesture recognition system for degraded environments such as partial occlusion. In addition, we compare the performance between the 3D integral imaging-based sensing and RGB-D sensing for continuous gesture recognition under degraded environments. Captured 3D data serves as the input to a You Look Only Once (YOLOv2) neural network for hand detection. Then, a temporal segmentation algorithm is employed to segment the individual gestures from a continuous video sequence. Following segmentation, the output is fed to a convolutional neural network-based bidirectional long short-term memory network (CNN-BiLSTM) for gesture classification. Our experimental results suggest that the proposed deep learning-based spatio-temporal continuous human gesture recognition provides substantial improvement over both RGB-D sensing and conventional 2D imaging system. To the best of our knowledge, this is the first report of 3D integral imaging-based continuous human gesture recognition with deep learning and the first comparison between 3D integral imaging and RGB-D sensors for this task.

Two Stage Continuous Gesture Recognition Based on Deep Learning

The paper proposes an effective continuous gesture recognition method, which includes two modules: segmentation and recognition. In the segmentation module, the video frames are divided into gesture frames and transitional frames by using the information of hand motion and appearance, and continuous gesture sequences are segmented into isolated sequences. In the recognition module, our method exploits the spatiotemporal information embedded in RGB and depth sequences. For the RGB modality, our method adopts Convolutional Long Short-Term Memory Networks to learn long-term spatiotemporal features from short-term spatiotemporal features obtained from a 3D convolutional neural network. For the depth modality, our method converts a sequence into Dynamic Images and Motion Dynamic Images through weighted rank pooling and feed them into Convolutional Neural Networks, respectively. Our method has been evaluated on both ChaLearn LAP Large-scale Continuous Gesture Dataset and Montalbano Gesture Dataset and achieved state-of-the-art performance.

Towards wide-scale continuous gesture recognition model for in-depth and grayscale input videos

In recent years, gesture recognition in video sequences has aroused growing interest in the fields of computer vision and behavioral understanding, for example in the control of robots and video games, in the field of video surveillance, automatic video indexing or content-based video retrieval. Processing large-scale continuous gesture data with in-depth, grayscale input videos remains a primary challenge for academic researchers. A wide range of recognition models have been proposed to solve this problem but have not proven their great performance. The main contribution of this article to address this problem is to segment the sequences of continuous gestures into isolated gestures, using the average of the velocity information calculated on the basis of the estimate of the deep optical flow, and to extract a set of relevant descriptors, called characteristics. signature, in order to characterize different intensities and spatial information describing the location, speed and orientation of movement. Finally, to transmit to a linear SVM the characteristics built for the depth and gray scale sequences, for each isolated segment for its classification. The experimental study carried out on the various standard data collections namely KTH, Chalearn and Weizmann, on our model and on the main models that we have studied in the literature, as well as the analysis of the results, which we obtained, clearly show the limits of these studied models and confirms the performance of our model as well as efficiency in terms of precision, recall and robustness.

Approach for Improving User Interface Based on Gesture Recognition

Gesture recognition technology based on visual detection to acquire gestures information is obtained in a non-contact manner. There are two types of gesture recognition: independent and continuous gesture recognition. The former aims to classify videos or other types of gesture sequences that only contain one isolated gesture instance in each sequence (e.g., RGB-D or skeleton data). In this study, we review existing research methods of visual gesture recognition and will be grouped according to the following family: static, dynamic, based on the supports (Kinect, Leap…etc), works that focus on the application of gesture recognition on robots and works on dealing with gesture recognition at the browser level. Following that, we take a look at the most common JavaScript-based deep learning frameworks. Then we present the idea of defining a process for improving user interface control based on gesture recognition to streamline the implementation of this mechanism.

TMMF: Temporal Multi-Modal Fusion for Single-Stage Continuous Gesture Recognition.

Gesture recognition is a much studied research area which has myriad real-world applications including robotics and human-machine interaction. Current gesture recognition methods have focused on recognising isolated gestures, and existing continuous gesture recognition methods are limited to two-stage approaches where independent models are required for detection and classification, with the performance of the latter being constrained by detection performance. In contrast, we introduce a single-stage continuous gesture recognition framework, called Temporal Multi-Modal Fusion (TMMF), that can detect and classify multiple gestures in a video via a single model. This approach learns the natural transitions between gestures and non-gestures without the need for a pre-processing segmentation step to detect individual gestures. To achieve this, we introduce a multi-modal fusion mechanism to support the integration of important information that flows from multi-modal inputs, and is scalable to any number of modes. Additionally, we propose Unimodal Feature Mapping (UFM) and Multi-modal Feature Mapping (MFM) models to map uni-modal features and the fused multi-modal features respectively. To further enhance performance, we propose a mid-point based loss function that encourages smooth alignment between the ground truth and the prediction, helping the model to learn natural gesture transitions. We demonstrate the utility of our proposed framework, which can handle variable-length input videos, and outperforms the state-of-the-art on three challenging datasets: EgoGesture, IPN hand and ChaLearn LAP Continuous Gesture Dataset (ConGD). Furthermore, ablation experiments show the importance of different components of the proposed framework.

A Review of the Hand Gesture Recognition System: Current Progress and Future Directions

This paper reviewed the sign language research in the vision-based hand gesture recognition system from 2014 to 2020. Its objective is to identify the progress and what needs more attention. We have extracted a total of 98 articles from well-known online databases using selected keywords. The review shows that the vision-based hand gesture recognition research is an active field of research, with many studies conducted, resulting in dozens of articles published annually in journals and conference proceedings. Most of the articles focus on three critical aspects of the vision-based hand gesture recognition system, namely: data acquisition, data environment, and hand gesture representation. We have also reviewed the performance of the vision-based hand gesture recognition system in terms of recognition accuracy. For the signer dependent, the recognition accuracy ranges from 69% to 98%, with an average of 88.8% among the selected studies. On the other hand, the signer independent’s recognition accuracy reported in the selected studies ranges from 48% to 97%, with an average recognition accuracy of 78.2%. The lack in the progress of continuous gesture recognition could indicate that more work is needed towards a practical vision-based gesture recognition system.

Real-time Arm Gesture Recognition in Smart Home Scenarios via Millimeter Wave Sensing

"In air" gesture recognition using millimeter wave (mmWave) radar and its applications in natural human-computer-interaction for smart home has shown its potential. However, the state-of-the-art works still fall short in terms of limited gesture space, vulnerable to surrounding interference, and off-line recognition. In this paper, we propose mHomeGes, a real-time mmWave arm gesture recognition system for practical smart home-usage. To this end, we first distill arm gesture's position and dynamic variation, and then custom-design a lightweight convolution neural network to recognize fine-grained gestures. Next, we propose a user discovery method to focus on the target human gesture, thus eliminating the adverse impact of surrounding interference. Finally, we design a hidden Markov model-based voting mechanism to handle continuous gesture signals at run-time, which leads to continuous gesture recognition in real-time. We implement mHomeGes on a commodity mmWave radar and also perform a user study, which demonstrates that mHomeGes achieves high recognition accuracy above 95.30% in real-time across various smart home scenarios, regardless of the impact of surrounding movements, concurrent gestures, human physiological conditions, and outer packing materials. Moreover, we have also publicly archived a mmWave gesture data-set collected during developing mHomeGes, which consists of about 22,000 instances from 25 persons and may have an independent value of facilitating future research.

Echo State Networks and Long Short-Term Memory for Continuous Gesture Recognition: a Comparative Study

Recent developments of sensors that allow tracking of human movements and gestures enable rapid progress of applications in domains like medical rehabilitation or robotic control. Especially the inertial measurement unit (IMU) is an excellent device for real-time scenarios as it rapidly delivers data input. Therefore, a computational model must be able to learn gesture sequences in a fast yet robust way. We recently introduced an echo state network (ESN) framework for continuous gesture recognition (Tietz et al., 2019) including novel approaches for gesture spotting, i.e., the automatic detection of the start and end phase of a gesture. Although our results showed good classification performance, we identified significant factors which also negatively impact the performance like subgestures and gesture variability. To address these issues, we include experiments with Long Short-Term Memory (LSTM) networks, which is a state-of-the-art model for sequence processing, to compare the obtained results with our framework and to evaluate their robustness regarding pitfalls in the recognition process. In this study, we analyze the two conceptually different approaches processing continuous, variable-length gesture sequences, which shows interesting results comparing the distinct gesture accomplishments. In addition, our results demonstrate that our ESN framework achieves comparably good performance as the LSTM network but has significantly lower training times. We conclude from the present work that ESNs are viable models for continuous gesture recognition delivering reasonable performance for applications requiring real-time performance as in robotic or rehabilitation tasks. From our discussion of this comparative study, we suggest prospective improvements on both the experimental and network architecture level.

Deep signature-based isolated and large scale continuous gesture recognition approach

In recent years, recognition of hand gestures has received considerable attention from the research community, with an increased interest in advanced decision support systems. In applications of this type of system, recognition of hand gestures based on vision is used in the design and development of interfaces such as that of automobile users. This type of interface allows drivers to interact with the vehicle, without affecting their concentrations. This also increases driver comfort, without compromising safety. However, the video-based gesture recognition problem is not trivial and poses problems due to: (i) intra and inter-person variations in the movement of human hand gestures; (ii) variations between people in the shape and size of the human hand; and (iii) variations in lighting; and background noise.In this paper, a new recognition system is proposed to address the problem of large scale continuous gesture recognition with depth and gray-scale input videos. The proposed recognition system contains 3 main phases. Firstly continuous gesture sequences are segmented into isolated gesture using mean of velocity information calculated based on the estimation of deep optical flow. For each isolated segment a set of relevant descriptors named deep signature features is extracted in order to characterize different intensity and spatial information describing the location, the velocity and the orientation of the movement. The features constructed for both depth and gray-scale sequences are fed to linear SVM for classification. To evaluate the effectiveness of our recognition model, it was compared with other methods using two challenging datasets: KTH and Chalearn. Indeed, a first experimental study has shown that, using the KTH dataSet and the SVM classifier, results given by our method outperforms previously published methods and it reaches 95.6% in term of accuarcy and 96% in term of precision. The second experimental study, carried out on the ChaLearn LAP ConGD data set, also showed a superiority of our method from the point of view of precision by codelivery over other methods with mean Jaccard Index of 0.5011.

ChaLearn Looking at People: IsoGD and ConGD Large-Scale RGB-D Gesture Recognition.

The ChaLearn large-scale gesture recognition challenge has run twice in two workshops in conjunction with the International Conference on Pattern Recognition (ICPR) 2016 and International Conference on Computer Vision (ICCV) 2017, attracting more than 200 teams around the world. This challenge has two tracks, focusing on isolated and continuous gesture recognition, respectively. It describes the creation of both benchmark datasets and analyzes the advances in large-scale gesture recognition based on these two datasets. In this article, we discuss the challenges of collecting large-scale ground-truth annotations of gesture recognition and provide a detailed analysis of the current methods for large-scale isolated and continuous gesture recognition. In addition to the recognition rate and mean Jaccard index (MJI) as evaluation metrics used in previous challenges, we introduce the corrected segmentation rate (CSR) metric to evaluate the performance of temporal segmentation for continuous gesture recognition. Furthermore, we propose a bidirectional long short-term memory (Bi-LSTM) method, determining video division points based on skeleton points. Experiments show that the proposed Bi-LSTM outperforms state-of-the-art methods with an absolute improvement of 8.1% (from 0.8917 to 0.9639) of CSR.