Problem Of Gesture Recognition Research Articles

Ultra-Range Gesture Recognition using a web-camera in Human–Robot Interaction

Hand gestures play a significant role in human interactions where non-verbal intentions, thoughts and commands are conveyed. In Human–Robot Interaction (HRI), hand gestures offer a similar and efficient medium for conveying clear and rapid directives to a robotic agent. However, state-of-the-art vision-based methods for gesture recognition have been shown to be effective only up to a user-camera distance of seven meters. Such a short distance range limits practical HRI with, for example, service robots, search and rescue robots and drones. In this work, we address the Ultra-Range Gesture Recognition (URGR) problem by aiming for a recognition distance of up to 25 m and in the context of HRI. We propose the URGR framework, a novel deep-learning, using solely a simple RGB camera. Gesture inference is based on a single image. First, a novel super-resolution model termed High-Quality Network (HQ-Net) uses a set of self-attention and convolutional layers to enhance the low-resolution image of the user. Then, we propose a novel URGR classifier termed Graph Vision Transformer (GViT ) which takes the enhanced image as input. GViT combines the benefits of a Graph Convolutional Network (GCN) and a modified Vision Transformer (ViT). Evaluation of the proposed framework over diverse test data yields a high recognition rate of 98.1%. The framework has also exhibited superior performance compared to human recognition in ultra-range distances. With the framework, we analyze and demonstrate the performance of an autonomous quadruped robot directed by human gestures in complex ultra-range indoor and outdoor environments, acquiring 96% recognition rate on average.

Research of Stages, Types of Modeling and Methods of Gesture Recognition

It is impossible to leave aside the problem of communication with the outside world and the social integration of people with disabilities in modern society, where there is a rapid development of communication methods between people. Gesture is the only method of communication for people with disabilities (auditory and language). The research provides information on static and dynamic gestures that allow people with communication problems to exchange information. It is noted that the problem of automatic gesture recognition is solved by various mathematical methods, algorithms and computer systems. The advantages and disadvantages of 2D and 3D models for sequential recognition of hand gestures are shown. It is noted that the creation of a national electronic database that recognizes the national dactyl alphabet and gestures in the Republic of Azerbaijan is an urgent problem. The well-known methods of this category are shown. Artificial neural network method, Hidden Markov method, Random Forest method and method of designating a hand gesture using gloves painted with markers of different colors. The creation of a national electronic database that recognizes the national dactyl alphabet and gestures is proposed, which will partially eliminate the existing barrier in education and problems of communication between people.

Visual Static Hand Gesture Recognition Using Convolutional Neural Network

Hand gestures are an essential part of human-to-human communication and interaction and, therefore, of technical applications. The aim is increasingly to achieve interaction between humans and computers that is as natural as possible, for example, by means of natural language or hand gestures. In the context of human-machine interaction research, these methods are consequently being explored more and more. However, the realization of natural communication between humans and computers is a major challenge. In the field of hand gesture recognition, research approaches are being pursued that use additional hardware, such as special gloves, to classify gestures with high accuracy. Recently, deep learning techniques using artificial neural networks have been increasingly proposed for the problem of gesture recognition without using such tools. In this context, we explore the approach of convolutional neural network (CNN) in detail for the task of hand gesture recognition. CNN is a deep neural network that can be used in the fields of visual object processing and classification. The goal of this work is to recognize ten types of static hand gestures in front of complex backgrounds and different hand sizes based on raw images without the use of extra hardware. We achieved good results with a CNN network architecture consisting of seven layers. Through data augmentation and skin segmentation, a significant increase in the model’s accuracy was achieved. On public benchmarks, two challenging datasets have been classified almost perfectly, with testing accuracies of 96.5% and 96.57%.

On the Gesture Recognition of a Faint Phantom Motion for the Control of a Transradial Prosthesis amidst varying Contraction Forces

The variation of the contraction force associated with the phantom motion used for the actuation of a bionic upper-limb prosthesis represents a scenario encountered regularly by amputees, while prior research appears to not have been able to succinctly address this problem. In this study, an extended prosthesis control system is proposed which is able to recognise gesture intent motions alongside the prediction of an associated contraction force as part of an advanced pattern recognition system. As part of this research topic, this paper introduces the proposed control architecture and is based on the solving of the gesture recognition problem amidst varying contraction forces for a transradial amputee with a seemingly faint phantom motion.The work involves the application of a novel decomposition algorithm and the use of a set of computationally effective features, alongside the contrast of the recognition capabilities of the proposed approach using various classification models. The results show an enhanced recognition of gesture motion intent with the use of the decomposition method, despite the faint phantom motion signal from the amputee.

On the Gesture Recognition of a Faint Phantom Motion for the Control of a Transradial Prosthesis amidst varying Contraction Forces

The variation of the contraction force associated with the phantom motion used for the actuation of a bionic upper-limb prosthesis represents a scenario encountered regularly by amputees, while prior research appears to not have been able to succinctly address this problem. In this study, an extended prosthesis control system is proposed which is able to recognise gesture intent motions alongside the prediction of an associated contraction force as part of an advanced pattern recognition system. As part of this research topic, this paper introduces the proposed control architecture and is based on the solving of the gesture recognition problem amidst varying contraction forces for a transradial amputee with a seemingly faint phantom motion.The work involves the application of a novel decomposition algorithm and the use of a set of computationally effective features, alongside the contrast of the recognition capabilities of the proposed approach using various classification models. The results show an enhanced recognition of gesture motion intent with the use of the decomposition method, despite the faint phantom motion signal from the amputee.

A General Pipeline for Online Gesture Recognition in Human–Robot Interaction

Recent advances in robotics have allowed the introduction of robots assisting and working together with human subjects. To promote their use and diffusion, intuitive and user-friendly interaction means should be adopted. In particular, gestures have become an established way to interact with robots since they allow to command them in an intuitive manner. In this article, we focus on the problem of gesture recognition in human–robot interaction (HRI). While this problem has been largely studied in the literature, it poses specific constraints when applied to HRI. We propose a framework consisting in a pipeline devised to take into account these specific constraints. We implement the proposed pipeline considering, as an example, an evaluation use case. To this end, we consider standard machine learning algorithms for the classification stage and evaluate their performance considering different performance metrics for a thorough assessment.

A fuzzy granular logistic regression algorithm for sEMG-based cross-individual prosthetic hand gesture classification

Objective. Prosthetic systems are used to improve the quality of life of post-amputation patients, and research on surface electromyography (sEMG)-based gesture classification has yielded rich results. Nonetheless, current gesture classification algorithms focus on the same subject, and cross-individual classification studies that overcome physiological factors are relatively scarce, resulting in a high abandonment rate for clinical prosthetic systems. The purpose of this research is to propose an algorithm that can significantly improve the accuracy of gesture classification across individuals. Approach. Eight healthy adults were recruited, and sEMG data of seven daily gestures were recorded. A modified fuzzy granularized logistic regression (FG_LogR) algorithm is proposed for cross-individual gesture classification. Main results. The results show that the average classification accuracy of the four features based on the FG_LogR algorithm is 79.7%, 83.6%, 79.0%, and 86.1%, while the classification accuracy based on the logistic regression algorithm is 76.2%, 79.5%, 71.1%, and 81.3%, the overall accuracy improved ranging from 3.5% to 7.9%. The performance of the FG_LogR algorithm is also superior to the other five classic algorithms, and the average prediction accuracy has increased by more than 5%. Conclusion. The proposed FG_LogR algorithm improves the accuracy of cross-individual gesture recognition by fuzzy and granulating the features, and has the potential for clinical application. Significance. The proposed algorithm in this study is expected to be combined with other feature optimization methods to achieve more precise and intelligent prosthetic control and solve the problems of poor gesture recognition and high abandonment rate of prosthetic systems.

Sign Language Recognition and Response via Virtual Reality

The main aim of this project is to provide a voice to voiceless people with the help of Python and Unity prototype systems. Aforesaid project work focuses on the problem of gesture recognition in real time that sign language is used by the community of deaf people. The problem addressed is based on Digital image processing using color segmentation, Skin detection, Image Segmentation, Image Filtering, and Template Matching techniques. This system recognizes gestures of ISL(Indian Sign Language) including the alphabet and a subset of its words.

Early Detection of Gestures using Context-and-Gap-aware Predictive Pose Framework

With a growing interest in autonomous vehicles’ operation, there is an equally increasing need for efficient anticipatory gesture recognition systems for human-vehicle interaction. Existing gesture recognition algorithms have been primarily restricted to historical data. In this paper, we propose a novel Context-and-Gap-aware Pose Prediction Framework (CGAP2), which predicts the future pose data for anticipatory recognition of gestures in an online fashion. CGAP2 implements an encoder–decoder architecture paired with a pose prediction module to anticipate future frames followed by a shallow classifier. CGAP2’s pose prediction module uses 3D convolutional layers and is dependent on the number of pose frames supplied, the time difference between subsequent pose frames, and the number of predicted pose frames. The performance of CGAP2 is evaluated on the Human3.6M dataset with the MPJPE metric. For the pose prediction of 15 frames in advance, an error of 79 mm is achieved. The pose prediction module consists of only 26M parameters and can run at 50 FPS on the NVIDIA TITAN RTX. CGAP2 has a 1-s time advantage compared to other gesture recognition systems, which can be crucial for autonomous vehicles. Furthermore, this work formalizes the problem of anticipatory gesture recognition by introducing two hyper-parameters namely context and gap and a relationship is established between the two using carefully designed experiments in ablation testing. As per our knowledge, this work is the first to tackle all the challenges faced by anticipatory gesture recognition and also the first to post a performance benchmark on Human3.6M dataset.

HandGest: Hierarchical Sensing for Robust-in-the-Air Handwriting Recognition With Commodity WiFi Devices

Recent advances in wireless sensing techniques have made it possible to recognize hand gestures using channel state information (CSI) in commodity WiFi devices. Existing WiFi-based gesture recognition systems mainly use learning-based pattern recognition methods to recognize different gestures, however, these methods fail to work well when the locations of transceivers, the relative location and orientation of the hand with respect to transceivers, and/or the hand gesturing size change, leading to inconsistent signal patterns caused by those factors. Although some recent efforts have been made to address the so-called “domain-dependent” gesture recognition problem, they either require prior knowledge on initial locations of the hand and WiFi devices or need to train several classifiers for the specific domains. Different from the state-of-the-art methods, we construct two distinct features from a hand-oriented view (rather than from a transceiver’s view), namely, the dynamic phase vector (DPV) and motion rotation variable (MRV), which are quite consistent in characterizing a big set of handwriting gestures, despite significant change in locations of transceivers, the relative location and orientation of the hand with respect to transceivers, and the drawing sizes. We further incorporate a hierarchical sensing framework and develop HandGest—a real-time handwriting gesture recognition system using commodity WiFi devices, to precisely recognize a great number of “in-the-air” handwritings based on the aforementioned two domain-independent features and a pipeline of specific features. Extensive experiments have been done in practical settings with 20 volunteers, evaluation results demonstrate that HandGest outperforms state-of-the-art methods on a large number of handwritings with different transceivers’ location, different initial hand locations and orientations, as well as different drawing sizes. Given its superior performance, we believe that HandGest paves a new way to enhance the real-world practicality of WiFi-based gesture recognition.

Head and Eye Egocentric Gesture Recognition for Human-Robot Interaction Using Eyewear Cameras

Non-verbal communication plays a particularly important role in a wide range of scenarios in Human-Robot Interaction (HRI). Accordingly, this work addresses the problem of human gesture recognition. In particular, we focus on head and eye gestures, and adopt an egocentric (first-person) perspective using eyewear cameras. We argue that this egocentric view may offer a number of conceptual and technical benefits over scene- or robot-centric perspectives. A motion-based recognition approach is proposed, which operates at two temporal granularities. Locally, frame-to-frame homographies are estimated with a convolutional neural network (CNN). The output of this CNN is input to a long short-term memory (LSTM) to capture longer-term temporal visual relationships, which are relevant to characterize gestures. Regarding the configuration of the network architecture, one particularly interesting finding is that using the output of an internal layer of the homography CNN increases the recognition rate with respect to using the homography matrix itself. While this work focuses on action recognition, and no robot or user study has been conducted yet, the system has been designed to meet real-time constraints. The encouraging results suggest that the proposed egocentric perspective is viable, and this proof-of-concept work provides novel and useful contributions to the exciting area of HRI.

Conversion of Sign Language Video to Text and Speech

Abstract: Sign Language recognition (SLR) is a significant and promising technique to facilitate communication for hearingimpaired people. Here, we are dedicated to finding an efficient solution to the gesture recognition problem. This work develops a sign language (SL) recognition framework with deep neural networks, which directly transcribes videos of SL sign to word. We propose a novel approach, by using Video sequences that contain both the temporal as well as the spatial features. So, we have used two different models to train both the temporal as well as spatial features. To train the model on the spatial features of the video sequences we use the (Convolutional Neural Networks) CNN model. CNN was trained on the frames obtained from the video sequences of train data. We have used RNN(recurrent neural network) to train the model on the temporal features. A trained CNN model was used to make predictions for individual frames to obtain a sequence of predictions or pool layer outputs for each video. Now this sequence of prediction or pool layer outputs was given to RNN to train on the temporal features. Thus, we perform sign language translation where input video will be given, and by using CNN and RNN, the sign shown in the video is recognized and converted to text and speech. Keywords: CNN (Convolutional Neural Network), RNN(Recurrent Neural Network), SLR(Sign Language Recognition), SL(Sign Language).

Multi-Stream Convolutional Neural Network-Based Wearable, Flexible Bionic Gesture Surface Muscle Feature Extraction and Recognition.

Surface electromyographic (sEMG) signals are weak physiological electrical signals, which are highly susceptible to coupling external noise and cause major difficulties in signal acquisition and processing. The study of using sEMG signals to analyze human motion intention mainly involves data preprocessing, feature extraction, and model classification. Feature extraction is an extremely critical part; however, this often involves many manually designed features with specialized domain knowledge, so the experimenter will spend time and effort on feature extraction. To address this problem, deep learning methods that can automatically extract features are applied to the sEMG-based gesture recognition problem, drawing on the success of deep learning for image classification. In this paper, sEMG is captured using a wearable, flexible bionic device, which is simple to operate and highly secure. A multi-stream convolutional neural network algorithm is proposed to enhance the ability of sEMG to characterize hand actions in gesture recognition. The algorithm virtually augments the signal channels by reconstructing the sample structure of the sEMG to provide richer input information for gesture recognition. The methods for noise processing, active segment detection, and feature extraction are investigated, and a basic method for gesture recognition based on the combination of multichannel sEMG signals and inertial signals is proposed. Suitable filters are designed for the common noise in the signal. An improved moving average method based on the valve domain is used to reduce the segmentation error rate caused by the short resting signal time in continuous gesture signals. In this paper, three machine learning algorithms, K-nearest neighbor, linear discriminant method, and multi-stream convolutional neural network, are used for hand action classification experiments, and the effectiveness of the multi-stream convolutional neural network algorithm is demonstrated by comparison of the results. To improve the accuracy of hand action recognition, a final 10 gesture classification accuracy of up to 93.69% was obtained. The separability analysis showed significant differences in the signals of the two cognitive-behavioral tasks when the optimal electrode combination was used. A cross-subject analysis of the test set subjects illustrated that the average correct classification rate using the pervasive electrode combination could reach 93.18%.

Real-Time Analysis of Hand Gesture Recognition with Temporal Convolutional Networks

In recent years, the successful application of Deep Learning methods to classification problems has had a huge impact in many domains. (1) Background: In biomedical engineering, the problem of gesture recognition based on electromyography is often addressed as an image classification problem using Convolutional Neural Networks. Recently, a specific class of these models called Temporal Convolutional Networks (TCNs) has been successfully applied to this task. (2) Methods: In this paper, we approach electromyography-based hand gesture recognition as a sequence classification problem using TCNs. Specifically, we investigate the real-time behavior of our previous TCN model by performing a simulation experiment on a recorded sEMG dataset. (3) Results: The proposed network trained with data augmentation yields a small improvement in accuracy compared to our existing model. However, the classification accuracy is decreased in the real-time evaluation, showing that the proposed TCN architecture is not suitable for such applications. (4) Conclusions: The real-time analysis helps in understanding the limitations of the model and exploring new ways to improve its performance.

Analytical review of models and methods for automatic recognition of gestures and sign languages

Introduction: Currently, the recognition of gestures and sign languages is one of the most intensively developing areas in computer vision and applied linguistics. The results of current investigations are applied in a wide range of areas, from sign language translation to gesture-based interfaces. In that regard, various systems and methods for the analysis of gestural data are being developed. Purpose: A detailed review of methods and a comparative analysis of current approaches in automatic recognition of gestures and sign languages. Results: The main gesture recognition problems are the following: detection of articulators (mainly hands), pose estimation and segmentation of gestures in the flow of speech. The authors conclude that the use of two-stream convolutional and recurrent neural network architectures is generally promising for efficient extraction and processing of spatial and temporal features, thus solving the problem of dynamic gestures and coarticulations. This solution, however, heavily depends on the quality and availability of data sets. Practical relevance: This review can be considered a contribution to the study of rapidly developing sign language recognition, irrespective to particular natural sign languages. The results of the work can be used in the development of software systems for automatic gesture and sign language recognition.

A Gait Energy Image-Based System for Brazilian Sign Language Recognition

Sign language is the main type of communication of the deaf community. However, most people do not know this language, which causes communication problems for many people. Many technological solutions have been proposed to overcome this issue. Some use the concept of wearable devices, but gesture recognition in a video sequence is a cheaper and less intrusive solution. In this work, we address the problem of gesture recognition in video. To do so, we employ a two-step method with feature space mapping and classification. First, the body parts of each subject in a video are segmented through a deep neural network architecture. Then, we use Gait Energy Image to encode the motion of the body parts in a compact feature space. Small datasets are usually a problem in this type of application, leading to sparse representation in the feature space. To contour this problem, we evaluate SMOTE as a data augmentation technique in the feature space and classical dimensionality reduction techniques. We evaluate our method on three challenging Brazilian sign language (Libras) datasets, CEFET/RJ-Libras, MINDS-Libras, and LIBRAS-UFOP, achieving global accuracies of 85.40±3.13%, 84.66±1.78%, and 64.91±3.79%, respectively, with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">singular value decomposition</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">support vector machine</i> .

A New Data Augmentation Method for Time Series Wearable Sensor Data Using a Learning Mode Switching-Based DCGAN

This letter describes a new image augmentation method based on a DCGAN considering mode switching in terms of transient learning accuracy threshold on the problem of human gesture recognition through imaging of wearable sensor time-series data and a deep CNN structure. Because the discriminator in GANs learns faster than the generator, it is known that mode collapse occurs, in which only image modes biased to a specific image type are augmented among various image forms. In this study, to solve the mode collapse caused by the learning difficulty mismatch between networks, we add a learning mode switching layer between the generator and discriminator and receive feedback from both networks’ transient learning accuracies to switch the learning mode if predefined thresholds are exceeded. We confirm that the proposed approach balanced the learning rate between the generator and discriminator networks, resolved the mode collapse problem, and increased the test accuracy of a deep CNN trained with an augmented image set by approximately 20.35% compared to a conventional DCGAN. In addition, it showed better accuracy on a performance comparison with other improved DCGAN methods.

Sign Language Recognition using CNN

Abstract: This research work presents a prototype system that helps to recognize hand gesture to normal people in order to communicate more effectively with the special people. Aforesaid research work focuses on the problem of gesture recognition in real time that sign language used by the community of deaf people. The problem addressed is based on Digital Image Processing using CNN (Convolutional Neural Networks), Skin Detection and Image Segmentation techniques. This system recognizes gestures of ASL (American Sign Language) including the alphabet and a subset of its words. Keywords: gesture recognition, digital image processing, CNN (Convolutional Neural Networks), image segmentation, ASL (American Sign Language), alphabet

A Cross-Domain Augmentation-Based AI Learning Framework for In-Network Gesture Recognition

This article studies the problem of RFID-based gesture recognition, which is practically important in various human-computer interaction scenarios, for example, smart homes, intelligent logistics, and smart cities. However, the existing solutions normally suffer from two major limitations: the model-driven methods are sensitive to specific environmental factors, and usually do not adapt well to a complex scenario that is full of multipath; the data-driven methods normally need the collection of massive RFID training data, and deploying the model in the remote cloud leads to long response delay. To overcome the above limitations, this article proposes a cross-domain augmentation-based AI learning (CAL) framework in the context of cloud-edge computing. In the CAL framework, we can simulate massive RFID phase profiles by converting the computer vision data that contains the gesture movement information, instead of costing lots of manpower to actually collect RFID training data. The simulated RFID phase profiles are used to train an AI model in the high-performance cloud. Note that since many sources of this kind of computer vision data are available online, we actually do not even need any manpower to collect training data. To achieve time-efficient recognition, knowledge distillation is applied to get a light and accurate model, which is deployed at the edge side. Thus, recognition response delay can be significantly reduced because the edge server where the AI model is actually deployed is much closer to users than the cloud server. We use commercial off-the-shelf RFID, Kinect, a high-performance server, and a laptop to implement the CAL framework. Extensive experiments are conducted to evaluate the performance of CAL. The results reveal that gesture recognition accuracy of CAL can reach nearly 90 percent without collection of any RFID training data.

Анализ эффективности методов машинного обучения в задаче распознавания жестов на основе данных электромиографических сигналов

При разработке систем человеко-машинных интерфейсов актуальной является задача распознавания жестов. Для выявления наиболее эффективного метода распознавания жестов был проведен анализ различных методов машинного обучения, используемых для классификации движений на основе электромиографических сигналов мышц. Были рассмотрены такие методы, как наивный байесовский классификатор (НБК), дерево решений, случайный лес, градиентный бустинг, метод опорных векторов, метод $k$-ближайших соседей, а также ансамбли методов (НБК и дерево решений, НБК и градиентный бустинг, градиентный бустинг и дерево решений). В качестве метода получения информации о жестах была выбрана электромиография. Такое решение не требует расположения руки в поле зрения камеры и может быть использовано для распознавания движений пальцев рук. Для проверки эффективности выбранных методов распознавания жестов было разработано устройство регистрации электромиографического сигнала мышц предплечья, которое включает в себя три электрода и ЭМГ-датчик, соединенный с микрокон- троллером и блоком питания. В качестве жестов были выбраны: сжатие кулака, знак «большой палец», знак «Виктория», сжатие указательного пальца и взмах рукой справа налево. Оценка эффективности методов классификации проводилась на основе значений доли правильных ответов, точности, полноты, а также среднего значения времени работы классификатора. Данные параметры были рассчитаны для трех вариантов расположения электромиографических электродов на предплечье. По результатам тести- рования, наиболее эффективными методами являются метод $k$-ближайших соседей, случайный лес и ансамбль НБК и градиентного бустинга, средняя точность которого для трех положений электродов составила 81,55 %. Также было определено положение электродов, при котором методы машинного обучения достигают максимального значения точности распознавания. При таком положении один из дифференциальных электродов располагается на месте пересечения глубокого сгибателя пальцев и длинного сгибателя большого пальца, второй — над поверхностным сгибателем пальцев