Gesture Recognition Framework Research Articles

An Underwater Human–Robot Interaction Using Hand Gestures for Fuzzy Control

Autonomous underwater vehicle (AUV) plays an important role in ocean research and exploration. The underwater environment has a great influence on AUV control and human–robot interaction, since underwater environment is highly dynamic with unpredictable fluctuation of water flow, high pressure and light attenuation. The traditional control model contains a large number of parameters, which is not effective and produces errors frequently. The proposal of fuzzy control addressed this issue to a certain extent. It applies fuzzy variables to the controller, which replace the values in an interval. In addition to the controller, underwater human–robot interaction is also difficult. Divers cannot speak or show any facial expressions underwater. The buttons on the AUV also need to overcome the huge water pressure. In this paper, we proposed a method to recognize the gesture instructions and apply it to the fuzzy control of AUV. Our contribution is the gesture recognition framework for the human–robot interaction, including the gesture detection network and the algorithm for the control of AUV. The experiment result shows the efficiency of the proposed method.

Event-Based Gesture Recognition With Dynamic Background Suppression Using Smartphone Computational Capabilities.

In this paper, we introduce a framework for dynamic gesture recognition with background suppression operating on the output of a moving event-based camera. The system is developed to operate in real-time using only the computational capabilities of a mobile phone. It introduces a new development around the concept of time-surfaces. It also presents a novel event-based methodology to dynamically remove backgrounds that uses the high temporal resolution properties of event-based cameras. To our knowledge, this is the first Android event-based framework for vision-based recognition of dynamic gestures running on a smartphone without off-board processing. We assess the performances by considering several scenarios in both indoors and outdoors, for static and dynamic conditions, in uncontrolled lighting conditions. We also introduce a new event-based dataset for gesture recognition with static and dynamic backgrounds (made publicly available). The set of gestures has been selected following a clinical trial to allow human-machine interaction for the visually impaired and older adults. We finally report comparisons with prior work that addressed event-based gesture recognition reporting comparable results, without the use of advanced classification techniques nor power greedy hardware.

Static Hand Gesture Recognition With Electromagnetic Scattered Field via Complex Attention Convolutional Neural Network

We present a novel learning-based static gesture recognition framework using electromagnetic (EM) scattered field data, which can efficiently address some significant issues in traditional vision-based recognition approaches. An end-to-end complex-valued attention convolutional neural network (CNN) is devised to train the gesture recognizer, wherein the attention module is designed to learn robust region-of-interest-aware features. Extensive numerical experiments are conducted on a public static hand gesture dataset. Both full and limited aperture measurements with transverse magnetic wave illumination are investigated. It is numerically shown that: first, both complex-valued convolutional and attention module contribute to the excellent performance. The recognition accuracy is above 99.0% for full aperture and even about 95.32% under the limited one-eighth aperture, respectively, and second, the proposed method not only has good scalability to the case with limited aperture, but also performs much better than previous state-of-the-art deep networks.

Multimodal hand gesture recognition combining temporal and pose information based on CNN descriptors and histogram of cumulative magnitudes

In this paper, we present a new approach for dynamic hand gesture recognition. Our goal is to integrate spatiotemporal features extracted from multimodal data captured by the Kinect sensor. In case the skeleton data is not provided, we apply a novel skeleton estimation method to compute temporal features. Furthermore, we introduce an effective method to extract a fixed number of keyframes to reduce the processing time. To extract pose features from RGB-D data, we take advantage of two different approaches: (1) Convolutional Neural Networks and (2) Histogram of Cumulative Magnitudes. We test different integration methods to fuse the extracted spatiotemporal features to boost recognition performance in a linear SVM classifier. Extensive experiments prove the effectiveness and feasibility of the proposed framework for hand gesture recognition.

Personalised gesture recognition based on tri-axis accelerometer using Gabor filters

Human gesture is one of the most important ingredients of context information. In this paper, a novel gesture recognition framework based on tri-axis accelerometer mounted on a cell phone is proposed. Since the length of acceleration signals is variable according to different gestures and different subjects' input speed, most of recognition algorithms cannot be used. To solve this problem, we propose a novel resampling method by combining decimation and interpolation. After that, we propose 1D Gabor coefficients of acceleration signals as features. However, the dimensionality of Gabor feature space used in gesture recognition is very high. We adopt Boosting and a two-stage method PCA plus LDA to select and compress the Gabor feature. The classifier we used is fast multi-class support vector machine. The average recognition results of 17 complex gestures using the proposed Gabor feature are effective. The performance of experimental results shows that gesture-based interaction can be used as a novel human computer interaction for consumer electronics and mobile devices.

Personalised gesture recognition based on tri-axis accelerometer using Gabor filters

Human gesture is one of the most important ingredients of context information. In this paper, a novel gesture recognition framework based on tri-axis accelerometer mounted on a cell phone is propos...

Hand gesture understanding by weakly-supervised fusing shallow/deep image attributes

Accurately recognizing human hand gestures is a useful component in many modern intelligent systems, such as identification authentication, human–computer interaction, and sign language recognition. Conventional approaches are typically based on shallow visual features and relatively simple backgrounds, which cannot readily recognize partially occluded hand gestures with sophisticated backgrounds. In this work, we propose a unified hand gesture recognition framework by optimally fusing a set of shallow/deep finger-level image attributes, based on which a weakly-supervised ranking algorithm is designed to select semantically salient regions for gesture understanding. More specifically, given a rich number of hand gesture images, we employ the well-known BING object proposal generator to extract hundreds of object patches that potentially draw human visual attention. Since the hundreds of object patches are still too many for building an effective recognition system, a weakly-supervised metric is proposed to rank them by extracting multiple shallow/deep features. And visual semantics are encoded at region-level by transferring the image-level semantic tags into various human gesture image regions by a weakly-supervised learning paradigm Apparently, the top-ranking highly salient object patches are highly indicative to human visual perception of human hand gesture, Thus we extract their ImageNet-CNN features and further concatenate them. Finally, the concatenated deep feature is fed into a multi-class SVM for classifying each hand gesture image into a particular type. Comprehensive experimental validations have demonstrated the effectiveness and robustness of our proposed hybrid-feature-based hand gesture categorization.

Mobile manipulator control through gesture recognition using IMUs and Online Lazy Neighborhood Graph search

Gesture-based control potentially eliminates the need for wearisome physical controls and facilitates easy interaction between a human and a robot. At the same time, it is intuitive and enables a natural means of control. In this paper, we present and evaluate a framework for gesture recognition using four wearable Inertial Measurement Units (IMUs) to indirectly control a mobile robot. Six gestures involving different hand and arm motions are defined. A novel algorithm based on an Online Lazy Neighborhood Graph (OLNG) search is used to recognise and classify the gestures online. A software framework is developed to control a robotic platform through integrating our gesture recognition algorithm with a Robot Operating System (ROS), which is in turn used to trigger predefined robot behaviours. Experiments show that the framework is able to correctly detect and classify six different gestures in real time with average success rates of 81.61 % and 81.67 %, while keeping the false-positive rate low by designing and using only 126 training samples.

An Efficient Framework for Hand Gesture Recognition based on Histogram of Oriented Gradients and Support Vector Machine

An Efficient Framework for Hand Gesture Recognition based on Histogram of Oriented Gradients and Support Vector Machine

Grassmann manifold based dynamic hand gesture recognition using depth data

In this paper, we propose a novel Grassmann manifold based framework for dynamic hand gesture recognition from depth data. Automated dynamic hand gesture recognition is important for improving man-machine communication and understanding human behavior. It finds various applications such as human computer interaction, ambient assisted living, automated driver assisted systems. We use depth data or skeleton information to detect the fingertip and store the fingertip points to create the trajectory. In fingertip detection using depth data first we detect the hand using the depth data and used hand shape properties such as finger thickness, finger length, finger width and finger orientation angle to find the shape of the hand. If skeleton data is available we use skeleton information to detect the fingertip in each frame. Then geometrical features are extracted and a unique gesture subspaces created using SVD for each feature vector matrix of each gesture set. These gesture subspaces lie on a Grassmann manifold and capture the intra-class variations and increase the inter-class discriminatory power. We apply Grassmann manifold based discriminant analysis for recognizing each test gesture. We perform experiments on standard datasets and the results show that we have achieved recognition accuracy comparable to the state-of-the-art.

Multi-modal gesture recognition with voting-based dynamic time warping

Gesture recognition has remained a challenging problem in the fields of human robot interaction. With the development of depth sensors such as Kinect, different modalities become available for gesture recognition while its advantages have not been fully exploited. One of the critical issues for multi-modal gesture recognition is how to fuse features from different modalities. In this article, we present a unified framework for multi-modal gesture recognition based on dynamic time warping. The 3D implicit shape model is applied to characterize the space-time structure of the local features extracted from different modalities. And then, all votes from the local features are incorporated into a common probability space which is then used for building the distance matrix. Meanwhile, an upper-bounding method UB_Pro is proposed to speed up dynamic time warping. The proposed approach is evaluated on the challenging ChaLearn Isolated Gesture Dataset, showing comparable performance in comparison to the state-of-the-art approaches for multi-modal gesture recognition problem.

Framework for dynamic hand gesture recognition using Grassmann manifold for intelligent vehicles

In this study, the authors propose a novel and robust approach to control auxiliary tasks in vehicles using hand gestures. First, they create a three-dimensional video volume by appending one frame to other that captures the motion history of frames. Then, they extract features using histogram of oriented gradients on each video volume. These features are represented in the form of subspaces on Grassmann manifold. To improve the recognition accuracy, they map the data from one manifold to another manifold with the help of a Grassmann kernel. Grassmann graph embedding discriminant analysis framework is used to classify the gestures. They perform experiments on two datasets: LISA and Cambridge Hand Gesture in three different testing methods such as 1/3-subject, 2/3-subject and cross-subject. Experimental results show that their proposed model outperforms and is comparable with the state-of-the-art methods.

A Gesture Recognition Framework for Cognitive Assessment

17(Suppl.): . Purpose: Most developed countries have been confronted with an acute growth in their elderly population. This situation is accompanied by increased prevalence of several health issues decreasing their quality of life. Alzheimer's disease is one of the most common cognitive disorders associated with aging. This has resulted in an urge to take advantage of technological tools to help doctors by providing early diagnosis which is vital for treatment of the elderly. These patients are not able to correctly emulate forelimb gestures and commit spatial/temporal errors, e.g. pretend to strike a nail with a hammer. 1 We developed a gesture recognition and evaluation tool to assist doctors in early detection of cognitive disorders (Figure 1). The framework performs precise gesture recognition based on state-of-the-art computer vision algorithms and evaluates subjects' performances to obtain a diagnostic report for clinicians. Method: The gesture recognition part of the framework uses four different methods. The first method employs local descriptors such as HOG/HOF 2 using shape and motion cues of a person performing a gesture. In the second method, skeleton information of the subjects is used to benefit from the body pose information. The third method uses depth information to detect exact hand patches with body part trajectories. We also trained a deep learning based recognition model 3 which empowered the framework to perform reliable recognitions. Results and Discussion: The framework is trained to recognize correct/incorrect performances. To have an objective decision on correctness of the gesture performances, clinicians annotated the data used for training the recognition algorithms. The utilized dataset was collected at the Institute Claude Pompidou (ICP) at Nice consisting of 14 dynamic and 15 static gestures selected by clinicians. The framework achieved 90% average accuracy on classifying gestures which is a reliable rate for diagnosis. Additionally, reaction and movement time of a subject are also detected. These two motor reflex criteria which measures responsive level of a subject are important factors in cognitive diagnosis. Motion descriptors are utilized to detect those quantities. All these types of information are illustrated in the user interface which provides clinicians with a detailed assessment of individual gesture and overall performance (Figure 1). Conclusion: We developed a user-friendly tool to help doctors in the diagnosis of cognitive disorders by providing a complete assessment of gestures performed by subjects. Future Work: We will conduct extensive evaluations on a larger population of patients with different levels of cognitive disorders. To improve gesture recognition task, more accurate deep models will be investigated.

Smartwatch-based Early Gesture Detection 8 Trajectory Tracking for Interactive Gesture-Driven Applications

The paper explores the possibility of using wrist-worn devices (specifically, a smartwatch) to accurately track the hand movement and gestures for a new class of immersive, interactive gesture-driven applications. These interactive applications need two special features: (a) the ability to identify gestures from a continuous stream of sensor data early--i.e., even before the gesture is complete, and (b) the ability to precisely track the hand's trajectory, even though the underlying inertial sensor data is noisy. We develop a new approach that tackles these requirements by first building a HMM-based gesture recognition framework that does not need an explicit segmentation step, and then using a per-gesture trajectory tracking solution that tracks the hand movement only during these predefined gestures. Using an elaborate setup that allows us to realistically study the table-tennis related hand movements of users, we show that our approach works: (a) it can achieve 95% stroke recognition accuracy. Within 50% of gesture, it can achieve a recall value of 92% for 10 novice users and 93% for 15 experienced users from a continuous sensor stream; (b) it can track hand movement during such strokeplay with a median accuracy of 6.2 cm.

A Unified Framework for Multi-Modal Isolated Gesture Recognition

In this article, we focus on isolated gesture recognition and explore different modalities by involving RGB stream, depth stream, and saliency stream for inspection. Our goal is to push the boundary of this realm even further by proposing a unified framework that exploits the advantages of multi-modality fusion. Specifically, a spatial-temporal network architecture based on consensus-voting has been proposed to explicitly model the long-term structure of the video sequence and to reduce estimation variance when confronted with comprehensive inter-class variations. In addition, a three-dimensional depth-saliency convolutional network is aggregated in parallel to capture subtle motion characteristics. Extensive experiments are done to analyze the performance of each component and our proposed approach achieves the best results on two public benchmarks, ChaLearn IsoGD and RGBD-HuDaAct, outperforming the closest competitor by a margin of over 10% and 15%, respectively. Our project and codes will be released at https://davidsonic.github.io/index/acm_tomm_2017.html.

Gesture-based human–machine interfaces: a novel approach for robust hand and face tracking

Nonverbal communication forms a substantial portion of human–human interaction. In recent years, there has been increasing interest in developing gesture-based user interfaces for better human–machine interaction. Hand and face tracking is a central issue in the development of real-time gesture recognition systems. In this article, a new approach for boundary detection in blob tracking based on the mean-shift algorithm is proposed. Our approach is based on continuous sampling of the boundaries of the kernel and changing the size of the kernel using our novel Fuzzy-based algorithm. We compare our approach to the kernel density-based approach which is known as the CAM-shift algorithm in a set of different noise levels and conditions. The results show that the proposed approach is superior in stability against white noise and also provides correct boundary detection for arbitrary hand postures which is not achievable by the CAM-shift algorithm. This algorithm provides the required framework for vision-based real-time gesture recognition and hand and face tracking. It can be applied in scientific and commercial extensions of either vision-based or hybrid gesture recognition systems.

Human Behavior and Hand Gesture Classification for Smart Human-robot Interaction

This paper presents an intuitive human-robot interaction (HRI) framework for gesture and human behavior recognition. It relies on a vision-based system as interaction technology to classify gestures and a 3-axis accelerometer for behavior classification (stand, walking, etc.). An intelligent system integrates static gesture recognition recurring to artificial neural networks (ANNs) and dynamic gesture recognition using hidden Markov models (HMM). Results show a recognition rate of 95% for a library of 22 gestures and 97% for a library of 6 behaviors. Experiments show a robot controlled using gestures in a HRI process.

A Fuzzy Framework for Real-Time Gesture Spotting and Recognition

A vital requirement of any recognition system claiming to be real time is the capability to perform feature extraction in real time. In this paper, we propose an innovative fuzzy approach for real-time dynamic gesture recognition and spotting, where a compact local descriptor is designed to model moving gesture skeletons as a time series of fuzzy statistical features. Then, a set of one-vs-rest SVMs is trained on these features for gesture recognition and spotting. In this approach, the meaningful hand movements are successfully spotted while concurrently removing unintentional hand movements from an input video sequence. When evaluated on a gesture data set incorporating a relatively large and diverse collection of video data, the method proposed yields promising results that compare very favorably with those reported in the literature, while retaining real-time performance.

A framework for sign gesture recognition using improved genetic algorithm and adaptive filter

AbstractGesture based communication is the standard language utilized by the hard of hearing individuals for correspondence purpose. Despite the way that they precisely chat with each other by a method in sign language, they confront obscurity when they attempt to speak with individuals who can see sound, basically with the individuals who can’t understand sign language. Consequently, an effective method to be produced to gain and recognize the sign motion language. In our proposed work we have planned a casing work for examining and distinguishing the sign motion language. The proposed technique is handled through various modules like Noise removal using adaptive filter, segmentation using region growing algorithm and feature extraction by using an improved genetic algorithm. Finally, the proposed technique will be assessed by contrasting with the support vector machine classifier.

Corrigendum Algorithm for Gesture Recognition Based on Multiple Information Fusion and Kinect

Gesture recognition is an important and challenging task in the field of computer vision. Starting from the 3D shape of coding gestures, it puts forward a new kind of gesture recognition framework based on depth image. It extracts the space characteristics of a variety of 3D point cloud based on Kinect, including local principal components analysis on point cloud to get the histogram of main component, gradient direction histogram based on local depth difference and depth distribution histogram of local point cloud. Principal component histogram and gradient direction histogram effectively coding the local shape of gestures, depth distribution histogram compensates the loss of the shaping descriptor information. Through preliminary training of random forest classifier to filter the characteristics, and characteristics with less influence on classification results are removed, thus the computational costs are reduced. The filtered characteristics are used for training of random forest classifier again to classify gestures. Experiment is carried on two large-scale gesture data sets, for more difficult ASL dataset, the proposed method has improved the recognition rate of 3.6% then the best previous algorithm.