Dynamic Gesture Recognition Research Articles

3D-CNN based Dynamic Gesture Recognition for Indian Sign Language Modeling

Hand gestures used in Indian Sign Language (ISL) are static and dynamic in the time domain. The Indian Sign Language is available as a standard but is still not very common among peoples. In this paper, we have used a 3-dimensional convolutional based Convolution Neural Network to model the most utilized gestures of the Indian community. The trained model can provide a natural language output corresponding to the signs of the ISL. This in turn will help in reducing the problems faced while communicating with deaf and dumb peoples. Moreover, these dynamic gestures can be used in medical, industrial and various other fields. We took 20 gestures from standard Indian Sign Language (ISL) and trained our model on the dataset made by replicating the actions of those gestures. Ten subjects volunteered to make the dataset in distinct backgrounds, light conditions and orientations. Network model used produced good results in terms of accuracy, precision, recall and f1-scores.

Intelligent Dynamic Gesture Recognition Using CNN Empowered by Edit Distance

Intelligent Dynamic Gesture Recognition Using CNN Empowered by Edit Distance

Dynamic Gesture Recognition Based on iCPM and RNN

Aim to solve the problem that continuous complex actions are difficult to be recognized by computer vision technology in actual production, this paper collects gesture video from single-and double-viewpoint, constructs gesture joint point coordinate recognition network model by improved Convolutional Pose Machine(iCPM) model, obtains gesture Gauss heat map and joint point coordinates in each frame of video, and then inputs joint point coordinates with time series into gesture sequence recognition network. Finally, the gesture sequence is obtained. The experimental results show that the recognition accuracy of gesture recognition model based on double-viewpoint is 4.18% ∼ 6.92% higher than that of the single-viewpoint gesture recognition model, which verifies that the gesture recognition model based on dual-viewpoint has better recognition effect. In addition, the video samples of Tin-Lead Soldering including continuous and complex actions verify that the model has ideal recognition ability for dynamic gesture in actual production process.

HSFE Network and Fusion Model based Dynamic Hand Gesture Recognition

HSFE Network and Fusion Model based Dynamic Hand Gesture Recognition

Gesture Recognition Based on 3D Human Pose Estimation and Body Part Segmentation for RGB Data Input

This paper presents a novel approach for dynamic gesture recognition using multi-features extracted from RGB data input. Most of the challenges in gesture recognition revolve around the axis of the presence of multiple actors in the scene, occlusions, and viewpoint variations. In this paper, we develop a gesture recognition approach by hybrid deep learning where RGB frames, 3D skeleton joint information, and body part segmentation are used to overcome such problems. Extracted from the RGB images are the multimodal input observations, which are combined by multi-modal stream networks suited to different input modalities: residual 3D convolutional neural networks based on ResNet architecture (3DCNN_ResNet) for RGB images and color body part segmentation modalities; long short-term memory network (LSTM) for 3D skeleton joint modality. We evaluated the proposed model on four public datasets: UTD multimodal human action dataset, gaming 3D dataset, NTU RGB+D dataset, and MSRDailyActivity3D dataset and the experimental results on these datasets proves the effectiveness of our approach.

Dynamic gesture recognition based on feature fusion network and variant ConvLSTM

Gesture is a natural form of human communication, and it is of great significance in human–computer interaction. In the dynamic gesture recognition method based on deep learning, the key is to obtain comprehensive gesture feature information. Aiming at the problem of inadequate extraction of spatiotemporal features or loss of feature information in current dynamic gesture recognition, a new gesture recognition architecture is proposed, which combines feature fusion network with variant convolutional long short-term memory (ConvLSTM). The architecture extracts spatiotemporal feature information from local, global and deep aspects, and combines feature fusion to alleviate the loss of feature information. Firstly, local spatiotemporal feature information is extracted from video sequence by 3D residual network based on channel feature fusion. Then the authors use the variant ConvLSTM to learn the global spatiotemporal information of dynamic gesture, and introduce the attention mechanism to change the gate structure of ConvLSTM. Finally, a multi-feature fusion depthwise separable network is used to learn higher-level features including depth feature information. The proposed approach obtains very competitive performance on the Jester dataset with the classification accuracies of 95.59%, achieving state-of-the-art performance with 99.65% accuracy on the SKIG (Sheffifield Kinect Gesture) dataset.

Connectionist Temporal Classification Model for Dynamic Hand Gesture Recognition using RGB and Optical flow Data

Automatic classification of dynamic hand gesture is challenging due to the large diversity in a different class of gesture, Low resolution, and it is performed by finger. Due to a number of challenges many researchers focus on this area. Recently deep neural network can be used for implicit feature extraction and Soft Max layer is used for classification. In this paper, we propose a method based on a two-dimensional convolutional neural network that performs detection and classification of hand gesture simultaneously from multimodal Red, Green, Blue, Depth (RGBD) and Optical flow Data and passes this feature to Long-Short Term Memory (LSTM) recurrent network for frame-to-frame probability generation with Connectionist Temporal Classification (CTC) network for loss calculation. We have calculated an optical flow from Red, Green, Blue (RGB) data for getting proper motion information present in the video. CTC model is used to efficiently evaluate all possible alignment of hand gesture via dynamic programming and check consistency via frame-to-frame for the visual similarity of hand gesture in the unsegmented input stream. CTC network finds the most probable sequence of a frame for a class of gesture. The frame with the highest probability value is selected from the CTC network by max decoding. This entire CTC network is trained end-to-end with calculating CTC loss for recognition of the gesture. We have used challenging Vision for Intelligent Vehicles and Applications (VIVA) dataset for dynamic hand gesture recognition captured with RGB and Depth data. On this VIVA dataset, our proposed hand gesture recognition technique outperforms competing state-of-the-art algorithms and gets an accuracy of 86%

A Research Review on Indian Sign Language Recognition using Machine Learning

Sign language is the best communication media to interact with the deaf and dumb people. As sign language is completely different from our regular languages, someone has to learn that language and they are called as sign language interpreters. As per the survey of Indian Sign Language Research and Training Center (ISLRTC), there are around 466 million people worldwide with hearing loss and 34 million of these are youth. If we will think about India, only 250 interpreters are available from India, roughly 1 for 20284 deaf and dumb people. So, education of deaf and dumb people is very big question as interpreters, schools/colleges/any learning institutes available for them are very few and not available at every place. So, this problem can be solved by having online educational system for deaf and dumb people. Because it is one of the important purposes of this system that system should be accessed by deaf and dumb people from anywhere or at any place. Already online systems are available for deaf and dumb people. But problem in existing systems is that, more focus is given on just character recognition, gesture recognition etc. But I would like to contribute more additional work regarding educational courses (ex. like NPTEL, Coursera, Udemy) for deaf and dumb people instead of just gesture recognition in existing systems. So, more focus is given on Dynamic gesture recognition and Web-based technology for the implantation point of view. As well as vision-based method is chosen for implementation as it is cheaper and more comfortable for user than sensor-based method. As per the Persons with Disability Act 1995, one percent jobs in government and private sectors are reserved for them. But those jobs have conditions like people with disability should have degree certificates from known universities across India. With the help of this application, they can learn as well as can give exams using mobile or laptop camera. If they will clear the exams successfully then certificates will be provided to them. On the basis of those certificates, they can get a job as they will get certified education. This system itself can test the deaf and dumb users in exams as it is using Machine Learning instead of any external help of sign language interpreters. Various machine learning algorithms like Support Vector Machine (SVM) and Artificial Neural Network (ANN), Hidden Markov Model (HMM), Convolutional Neural Network (CNN), Finite State Machine (FSM) are used in existing systems.

Virtual reality gesture recognition based on depth information

With the rapid development of computer technology, the information revolution is constantly affecting and changing our lives. Today’s virtual reality technology is no longer just a virtual model for viewing. Experiencers are more willing to personally interact with the virtual environment in a virtual scene to increase their sense of gaming experience.Today’s virtual reality interaction methods mainly interact through the handles that come with VR helmets. Previously, some scholars have researched gesture interaction in virtual environments. They use data gloves to obtain information about hand movements. This method makes users have to wear heavy data gloves, which limits the natural movement of their hands. To overcome this shortcoming, the need for low‐cost vision‐based virtual reality gesture interaction is becoming more and more urgent.Traditional gesture recognition technologies based on artificial modeling include template‐based matching, data‐based gloves, and machine learning. As machine learning gesture classifiers, Hidden Markov Model (HMM) and Support Vector Machine (SVM) [1] have been widely used. However, in the hand motion detection of ordinary cameras, the image is prone to speckles, lighting changes, and other factors, which seriously affects the performance of the classifier, making it difficult to accurately and efficiently detect the first image when the background is complex and the lighting is affected. In recent years, deep learning‐based object detection methods have achieved great breakthroughs [2]. In the past decade, many deep learning‐based dynamic gesture recognition algorithms have been introduced. In the case of changing lighting conditions, in order to improve the classification accuracy of gesture recognition [3]. Therefore, this article chooses a depth camera combined with HTC VIVE device for gesture recognition in virtual reality. First obtain the depth image of the hand through the depth camera, then label the depth image, and then learn through the deep learning network hand movement training [4], preprocess the detected human hand image, and recognize it by hand detection The position of the hand in the image is obtained, and the movement of the hand is recognized as a result of recognition.The algorithm in this paper uses the advanced deep learning network YOLOv3, and changes the network structure and multi‐scale structure based on the original network. This algorithm is compared with other deep learning algorithms for gesture recognition, and the recognition effect is better than other algorithms in the same category.

Gesture recognition based on deep deformable 3D convolutional neural networks

Dynamic gesture recognition, which plays an essential role in human-computer interaction, has been widely investigated but not yet fully addressed. The challenge mainly lies in three folders: 1) to model both of the spatial appearance and the temporal evolution simultaneously; 2) to address the interference from the varied and complex background; 3) the requirement of real-time processing. In this paper, we address the above challenges by proposing a novel deep deformable 3D convolutional neural network for end-to-end learning, which not only gains impressive accuracy in challenging datasets but also can meet the requirement of the real-time processing. We propose three types of very deep 3D CNNs for gesture recognition, which can directly model the spatiotemporal information with their inherent hierarchical structure. To eliminate the background interference, a light-weight spatiotemporal deformable convolutional module is specially designed to augment the spatiotemporal sampling locations of the 3D convolution by learning additional offsets according to the preceding feature map. It can not only diversify the shape of the convolution kernel to better fit the appearance of the hands and arms, but also help the models pay more attention to the discriminative frames in the video sequence. The proposed method is evaluated on three challenging datasets, EgoGesture, Jester and Chalearn-IsoGD, and achieves the state-of-the-art performance on all of them. Our model ranked first on Jester’s official leader-board until the submission time. The code and the trained models are released for better communication and future works11https://github.com/lshiwjx/deform_conv3d_pytorch_op.

Event-Based Gesture Recognition With Dynamic Background Suppression Using Smartphone Computational Capabilities.

In this paper, we introduce a framework for dynamic gesture recognition with background suppression operating on the output of a moving event-based camera. The system is developed to operate in real-time using only the computational capabilities of a mobile phone. It introduces a new development around the concept of time-surfaces. It also presents a novel event-based methodology to dynamically remove backgrounds that uses the high temporal resolution properties of event-based cameras. To our knowledge, this is the first Android event-based framework for vision-based recognition of dynamic gestures running on a smartphone without off-board processing. We assess the performances by considering several scenarios in both indoors and outdoors, for static and dynamic conditions, in uncontrolled lighting conditions. We also introduce a new event-based dataset for gesture recognition with static and dynamic backgrounds (made publicly available). The set of gestures has been selected following a clinical trial to allow human-machine interaction for the visually impaired and older adults. We finally report comparisons with prior work that addressed event-based gesture recognition reporting comparable results, without the use of advanced classification techniques nor power greedy hardware.

A Dynamic Gesture Recognition Interface for Smart Home Control based on Croatian Sign Language

Deaf and hard-of-hearing people are facing many challenges in everyday life. Their communication is based on the use of a sign language, and the ability of the cultural/social environment to fully understand such a language defines whether or not it will be accessible for them. Technology is a key factor that has the potential to provide solutions to achieve a higher accessibility and therefore improve the quality of life of deaf and hard-of-hearing people. In this paper, we introduce a smart home automatization system specifically designed to provide real-time sign language recognition. The contribution of this paper implies several elements. Novel hierarchical architecture is presented, including resource-and-time-aware modules—a wake-up module and high-performance sign recognition module based on the Conv3D network. To achieve high-performance classification, multi-modal fusion of RGB and depth modality was used with the temporal alignment. Then, a small Croatian sign language database containing 25 different language signs for the use in smart home environment was created in collaboration with the deaf community. The system was deployed on a Nvidia Jetson TX2 embedded system with StereoLabs ZED M stereo camera for online testing. Obtained results demonstrate that the proposed practical solution is a viable approach for real-time smart home control.

Dynamic gesture recognition by using CNNs and star RGB: A temporal information condensation

Due to technological advances, machines are increasingly present in people’s daily lives. Thus, there has been more and more effort to develop interfaces that provide an intuitive way of interaction, such as dynamic gestures. Currently, the most common trend is to use multimodal data, as depth and skeleton information, to enable dynamic gesture recognition. However, it would be more interesting if only color information was used, since RGB cameras are usually available in almost every public place, and could be used for gesture recognition without the need of installing additional equipment. The main problem with such approach is the difficulty of representing spatio-temporal information using just color. With this in mind, we propose a technique capable of condensing a dynamic gesture, shown in a video, in just one RGB image. We call this technique star RGB. This image is then passed to a classifier formed by two Resnet CNNs, a soft-attention ensemble, and a fully connected layer, which indicates the class of the gesture present in the input video. Experiments were carried out using Montalbano, GRIT, and isoGD datasets. For Montalbano dataset, the proposed approach achieved an accuracy of 94.58%. Such result reaches the state-of-the-art when considering this dataset and only color information. For GRIT dataset, our proposal achieves more than 98% of accuracy, recall, precision, and F1-score, outperforming the authors’ approach by more than 6%. Regarding the large scale isoGD dataset, the proposal achieved 52.18% of accuracy. However, taking into account the complexity of the dataset (eight different gestures categories) and the amount of classes (249), we consider that our approach is competitive with previous ones, since we employed only color information to recognize gestures instead of all the multimodal data available, usually used by other methods.

Research on Gesture Recognition Based on Improved Canny & K-Means Algorithm and CNN

In view of the problem that the current dynamic gesture recognition system has decreased recognition rate and recognition speed due to the increase of sample size,in this study, we first use the improved Canny operator to detect the edge of the hand image. The improved algorithm includes two aspects: one is to generate HSV space image converted from RGB space and extract the V-component image; the other is to transform the traditional Gauss filter into a bilateral filter which integrates spatial distance and similarity to denoise;Then the improved K-means clustering algorithm is used to extract the feature points on the edge of the hand image. In the improved method, the pixel points at the peak of histogram are used as the initial clustering center, and the number of peak points is used as the number of classification;Then, the detection algorithm of convex hull and convex defect with geometric features is used to realize the effective fingertip tracking;Finally, CNN is used to achieve the precise sorting of fingertips.After verification, this synthesis algorithm can effectively solve the problems such as the difficulty to accurately detect the hand image contour, the poor fingertip tracking effect and the disorder of fingertip sorting caused by the similarity of background color and skin color, and the small difference between the length of fingers. With the increase of sample size, the recognition rate and speed have also improved, which can be widely used in dynamic gesture recognition and other fields.

Real‐time multi‐trajectory matching for dynamic hand gesture recognition

Focus on the field of dynamic gesture recognition, there is a problem of the action, is difficult to recognise when multiple fingertips move in a small range (rotation, grabbing), the authors proposed a method to get the recognition results with high robustness in real-time. Firstly, they proposed the concavity kernel accumulation algorithm (CKA) to cluster corners in an image. Secondly, they deem CKA as a region proposal generator and combined it with convolutional neural network to detect fingertips. Thirdly, they proposed the global nearest neighbour point matching algorithm to match fingertips from two frames. Finally, the long short-term memory is used in multi-trajectory recognition to get the results of gesture recognition. Experiments show that their method could recognise multi-trajectory gestures accurately, furthermore, it can run in real time (20 FPS) without graphics processing unit (GPU).

基于视觉的动态手势识别综述

基于视觉的动态手势识别综述

Gesture recognition and information recommendation based on machine learning and virtual reality in distance education

The dynamic and static gesture recognition in the distance education application scenario is not mature enough in theory at present, and still has a large space for development, and the application of gesture recognition in education is relatively insufficient. The purpose of this article is to combine gesture recognition with teacher classroom education and introduce a dynamic gesture recognition method. Moreover, this study introduces the data collection and preprocessing in detail and converts the data of the gesture action area into gray value images, and then uses the improved algorithm to perform classification. In addition, this study designs a control experiment to analyze the performance of the algorithm in this study and compares the accuracy of algorithm recognition from the perspective of simple background and complex background. The research results show that teaching gesture recognition in distance education can effectively improve education efficiency, with high accuracy, and can be directly applied to the system.

Fusion of 2D CNN and 3D DenseNet for Dynamic Gesture Recognition

Gesture recognition has been applied in many fields as it is a natural human–computer communication method. However, recognition of dynamic gesture is still a challenging topic because of complex disturbance information and motion information. In this paper, we propose an effective dynamic gesture recognition method by fusing the prediction results of a two-dimensional (2D) motion representation convolution neural network (CNN) model and three-dimensional (3D) dense convolutional network (DenseNet) model. Firstly, to obtain a compact and discriminative gesture motion representation, the motion history image (MHI) and pseudo-coloring technique were employed to integrate the spatiotemporal motion sequences into a frame image, before being fed into a 2D CNN model for gesture classification. Next, the proposed 3D DenseNet model was used to extract spatiotemporal features directly from Red, Green, Blue (RGB) gesture videos. Finally, the prediction results of the proposed 2D and 3D deep models were blended together to boost recognition performance. The experimental results on two public datasets demonstrate the effectiveness of our proposed method.

Mechanical arm teleoperation control system by dynamic hand gesture recognition based on kinect device

The research achieved to control the mechanical arm by using real-time dynamic gesture recognition based on Kinect. It uses the unmarked gesture segmentation algorithm based on the palm neighbourhood and the threshold detection algorithm based on the palmar contour to identify the operator's gestures and moving trajectories, and to convert it into the specific action of the mechanical arm. Through wireless network, this system sends control instructions to the mechanical arm to realise teleoperation control. The system also provides video feedback of mechanical arm operation site. It improves the teleoperation telepresence and interactivity and avoids operation error such as grasp nothing or fall. The results of the experiment indicate that the operation of the gesture control system is simple and easy, the response of mechanical arm is quick and accurate and the human–machine interaction is intuitive and friendly.

Recognition of a Single Dynamic Gesture with the Segmentation Technique HS-ab and Principle Components Analysis (PCA)

A continuous path performed by the hand in a period of time is considered for the purpose of gesture recognition. Dynamic gestures recognition is a complex topic since it spans from the conventional method of separating the hand from surrounding environment to searching for the fingers and palm. This paper proposes a strategy of hand recognition using a PC webcam, a segmentation technique (HS-ab which means HSV and CIELab color space), pre-processing of images to reduce noise and a classifier such as Principle Components Analysis (PCA) for the detection and tracking of the hand of the user. The results show that the segmentation technique HS-ab and the method PCA are robust in the execution of the system, although there are various conditions such as illumination, speed and precision of the movements. It is for this reason that a suitable extraction and classification of features allows the location of the gesture. The system was tested with the database of the training images and has a 94.74% accuracy.