Gesture Samples Research Articles

Novel Algorithm for Hand Gesture Recognition Utilizing a Wrist-Worn Inertial Sensor

In this paper, we propose a dynamic time warping (DTW)-based template matching method with a novel template generation algorithm for hand gesture recognition utilizing a wrist-worn inertial sensor. Hand orientation measurements are utilized to reconstruct gesture trajectories. The DTW technique with the Riemannian distance is employed to perform similarity measurements between gesture orientation trajectories. The proposed gesture recognition algorithm comprises three stages: a data preprocessing stage, a training stage, and a recognition stage. First, a moving intrinsic average filter is introduced to suppress the effects of measurement noise and unconscious hand shaking. Next, a double-threshold segmentation scheme is applied to extract individual gesture segments. In the training stage, to cope with temporal and spatial variations in gestures, an adaptive DTW barycenter averaging algorithm combined with an intrinsic averaging method is developed to generate gesture templates. A suitable rejection threshold is determined according to intra-class DTW distances. In the recognition stage, the rejection percentage between the input gesture and each gesture template is calculated. Finally, the nearest neighbor decision rule is applied to determine the recognition result. Experiments performed on a database of 3600 gesture samples illustrate that the proposed DTW-based gesture template generation and classification algorithm outperforms existing methods.

A Novel Gesture Recognition System for Intelligent Interaction with a Nursing-Care Assistant Robot

The expansion of nursing-care assistant robots in smart infrastructure has provided more applications for homecare services, which has raised new demands for smart and natural interaction between humans and robots. This article proposed an innovative hand motion trajectory (HMT) gesture recognition system based on background velocity features. Here, a new wearable wrist-worn camera prototype for gesture’s video collection was designed, and a new method for the segmentation of continuous gestures was shown. Meanwhile, a nursing-care assistant robot prototype was designed for assisting the elderly, which is capable of carrying the elderly with omnidirectional motion and grabbing the specified object at home. In order to evaluate the performance of the gesture recognition system, 10 special gestures were defined as the move commands for interaction with the robot, and 1000 HMT gesture samples were obtained from five subjects for leave-one-subject-out (LOSO) cross-validation classification with an average recognition accuracy of up to 97.34%. Moreover, the performance and practicability of the proposed system were further demonstrated by controlling the omnidirectional movement of the nursing-care assistant robot using the predefined gesture commands.

Jointly network: a network based on CNN and RBM for gesture recognition

Hand belongs to non-rigid objects and is rich in variety, making gesture recognition more difficult. The essence of dynamic gesture recognition is the classification and recognition of single-frame still images. Therefore, this paper mainly focuses on static gesture recognition. At present, there are some problems in gesture recognition, such as accuracy, real-time or poor robustness. To solve the above problems, in this paper, the Kinect sensor is used to obtain the color and depth gesture samples, and the gesture samples are processed. On this basis, a jointly network of CNN and RBM is proposed for gesture recognition. It mainly uses superposed network of multiple RBMs to carry out unsupervised feature extraction and combined with supervised feature extraction of CNN. Finally, these two features are combined to classify them. The simulation results show that the proposed jointly network has a better performance in identifying simple background gesture samples and the recognition capability of gesture samples in complex background needs to be improved.

Augmenting User Identification with WiFi Based Gesture Recognition

Over the last few years, researchers have proposed several WiFi based gesture recognition systems that can recognize predefined gestures performed by users at runtime. As most environments are inhabited by multiple users, the true potential of WiFi based gesture recognition can be unleashed only when each user can independently define the actions that the system should take when the user performs a certain predefined gesture. To enable this, a gesture recognition system should not only be able to recognize any given predefined gesture, but should also be able to identify the user that performed it. Unfortunately, none of the prior WiFi based gesture recognition systems can identify the user performing the gesture. In this paper, we propose WiID, a WiFi and gesture based user identification system that can identify the user as soon as he/she performs a predefined gesture at runtime. WiID integrates with the WiFi based gesture recognition systems as an add-on module whose sole objective is to identify the users that perform the predefined gestures. The design of WiID is based on our novel result which states that the timeseries of the frequencies that appear in WiFi channel's frequency response while performing a given gesture are different in the samples of that gesture performed by different users, and are similar in the samples of that gesture performed by the same user. We implemented and extensively evaluated WiID in a variety of environments using a comprehensive data set comprising over 25,000 gesture samples.

Abnormal gesture recognition based on multi-model fusion strategy

In this paper, we present a novel refined fused model combining masked Res-C3D network and skeleton LSTM for abnormal gesture recognition in RGB-D videos. The key to our design is to learn discriminative representations of gesture sequences in particular abnormal gesture samples by fusing multiple features from different models. First, deep spatiotemporal features are well extracted by 3D convolutional neural networks with residual architecture (Res-C3D). As gestures are mainly derived from the arm or hand movements, a masked Res-C3D network is built to decrease the effect of background and other variations via exploiting the skeleton of the body to reserve arm regions with discarding other regions. And then, relative positions and angles of different key points are extracted and used to build a time-series model by long short-term memory network (LSTM). Based the above representations, a fusion scheme for blending classification results and remedy model disadvantage by abnormal gesture via a weight fusion layer is developed, in which the weights of each voting sub-classifier being advantage to a certain class in our ensemble model are adaptively obtained by training in place of fixed weights. Our experimental results show that the proposed method can distinguish the abnormal gesture samples effectively and achieve the state-of-the-art performance in the IsoGD dataset.

TwistIn

Smart devices contain sensitive information that has to be guarded against unauthorized access through authentication. Existing authentication methods become obsolete as they are designed either for logging-in one device at a time or are ineffective in a multi-user multi-device environment. This paper presents TwistIn, a simple gesture that takes a smartwatch as an authentication token for fast access and control of other smart devices. Our mechanism is particularly useful for devices such as smartphones, smart glasses, and small IoT objects. To log in a device, one simply need to pick it up and twist it a few times. Then, by co-analyzing the motion data from the device and the watch, our method can extend the user authentication on the watch to the device. This is a simple and tangible interaction that takes only one to two seconds to perform. Furthermore, to account for user variation in wrist bending, we decompose wrist and forearm rotations via an optimization to improve the method accuracy. We implemented TwistIn, collected thousands of gesture samples, and conducted various experiments to evaluate our prototype system and show that it achieved over 95% detection accuracy.

EgoGesture: A New Dataset and Benchmark for Egocentric Hand Gesture Recognition

Gesture is a natural interface in human-computer interaction, especially interacting with wearable devices, such as VR/AR helmet and glasses. However, in the gesture recognition community, it lacks of suitable datasets for developing egocentric (first-person view) gesture recognition methods, in particular in the deep learning era. In this paper, we introduce a new benchmark dataset named EgoGesture with sufficient size, variation, and reality to be able to train deep neural networks. This dataset contains more than 24 000 gesture samples and 3 000 000 frames for both color and depth modalities from 50 distinct subjects. We design 83 different static and dynamic gestures focused on interaction with wearable devices and collect them from six diverse indoor and outdoor scenes, respectively, with variation in background and illumination. We also consider the scenario when people perform gestures while they are walking. The performances of several representative approaches are systematically evaluated on two tasks: gesture classification in segmented data and gesture spotting and recognition in continuous data. Our empirical study also provides an in-depth analysis on input modality selection and domain adaptation between different scenes.

Data‐Driven Crowd Motion Control With Multi‐Touch Gestures

AbstractControlling a crowd using multi‐touch devices appeals to the computer games and animation industries, as such devices provide a high‐dimensional control signal that can effectively define the crowd formation and movement. However, existing works relying on pre‐defined control schemes require the users to learn a scheme that may not be intuitive. We propose a data‐driven gesture‐based crowd control system, in which the control scheme is learned from example gestures provided by different users. In particular, we build a database with pairwise samples of gestures and crowd motions. To effectively generalize the gesture style of different users, such as the use of different numbers of fingers, we propose a set of gesture features for representing a set of hand gesture trajectories. Similarly, to represent crowd motion trajectories of different numbers of characters over time, we propose a set of crowd motion features that are extracted from a Gaussian mixture model. Given a run‐time gesture, our system extracts the K nearest gestures from the database and interpolates the corresponding crowd motions in order to generate the run‐time control. Our system is accurate and efficient, making it suitable for real‐time applications such as real‐time strategy games and interactive animation controls.

Touch and You’re Trapp(ck)ed: Quantifying the Uniqueness of Touch Gestures for Tracking

Abstract We argue that touch-based gestures on touch-screen devices enable the threat of a form of persistent and ubiquitous tracking which we call touch-based tracking. Touch-based tracking goes beyond the tracking of virtual identities and has the potential for cross-device tracking as well as identifying multiple users using the same device. We demonstrate the likelihood of touch-based tracking by focusing on touch gestures widely used to interact with touch devices such as swipes and taps.. Our objective is to quantify and measure the information carried by touch-based gestures which may lead to tracking users. For this purpose, we develop an information theoretic method that measures the amount of information about users leaked by gestures when modelled as feature vectors. Our methodology allows us to evaluate the information leaked by individual features of gestures, samples of gestures, as well as samples of combinations of gestures. Through our purpose-built app, called TouchTrack, we gather gesture samples from 89 users, and demonstrate that touch gestures contain sufficient information to uniquely identify and track users. Our results show that writing samples (on a touch pad) can reveal 73.7% of information (when measured in bits), and left swipes can reveal up to 68.6% of information. Combining different combinations of gestures results in higher uniqueness, with the combination of keystrokes, swipes and writing revealing up to 98.5% of information about users. We further show that, through our methodology, we can correctly re-identify returning users with a success rate of more than 90%.

Behavior Based Human Authentication on Touch Screen Devices Using Gestures and Signatures

With the rich functionalities and enhanced computing capabilities available on mobile computing devices with touch screens, users not only store sensitive information (such as credit card numbers) but also use privacy sensitive applications (such as online banking) on these devices, which make them hot targets for hackers and thieves. To protect private information, such devices typically lock themselves after a few minutes of inactivity and prompt a password/PIN/pattern screen when reactivated. Passwords/PINs/patterns based schemes are inherently vulnerable to shoulder surfing attacks and smudge attacks. In this paper, we propose BEAT, an authentication scheme for touch screen devices that authenticates users based on their behavior of performing certain actions on the touch screens. An action is either a gesture, which is a brief interaction of a user's fingers with the touch screen such as swipe rightwards, or a signature, which is the conventional unique handwritten depiction of one's name. Unlike existing authentication schemes for touch screen devices, which use what user inputs as the authentication secret, BEAT authenticates users mainly based on how they input, using distinguishing features such as velocity, device acceleration, and stroke time. Even if attackers see what action a user performs, they cannot reproduce the behavior of the user doing those actions through shoulder surfing or smudge attacks. We implemented BEAT on Samsung Focus smart phones and Samsung Slate tablets running Windows, collected 15,009 gesture samples and 10,054 signature samples, and conducted real-time experiments to evaluate its performance. Experimental results show that, with only 25 training samples, for gestures, BEAT achieves an average equal error rate of 0.5 percent with three gestures and for signatures, it achieves an average equal error rate of 0.52 percent with single signature.

Evaluating the placement of arm-worn devices for recognizing variations of dynamic hand gestures

Dynamic hand gestures have become increasingly popular as touch-free input modality for interactive systems. There exists a variety of arm-worn devices for the recognition of hand gestures, which di er not only in their capabilities, but also in their positioning on users' arms. These di erences in positioning might in uence how well gestures are recognized, leading to di erent gesture recognition accuracies. In this paper, we investi- gate the e ect of device placement on dynamic hand gesture recognition accuracy. We consider devices being strapped to the forearm on two positions: the wrist and below the elbow. These positions represent smart watches being worn on the wrist and devices with EMG sensors for the additional detection of static hand gestures (e.g spreading the ngers) being worn right below the elbow. Our hypothesis is that wrist-worn devices will have better recognition accuracy, caused by higher acceleration values of a bigger action radius of dynamic hand gestures. We conducted a comparative study using an LG G Watch and Thalmic Labs' Myo armband, for which we recorded a total of 12960 gesture samples of eight simple dynamic gestures in three di erent variants with eight participants. We evaluated a potential di erence in gesture recognition accuracies us- ing di erent feature sets and classi ers. Although recognition accuracies for wrist-worn devices seem higher, the di erence is not statistically signi cant due to substantial vari- ations in accuracy across participants. We thus cannot conclude that di erent positions of gesture recording devices on the lower arm have signi cant in uence on correctly recognizing arm gestures.

A Human-Centered Approach to One-Shot Gesture Learning

This paper discusses the problem of one-shot gesture recognition using a human-centered approach and its potential application to fields such as human-robot interaction where the user’s intentions are indicated through spontaneous gesturing (one-shot). Casual users have limited time to learn the gestures interface, which makes one-shot recognition an attractive alternative to interface customization. In the aim of natural interaction with machines, a framework must be developed to include the ability of humans to understand gestures from a single observation. Previous approaches to one-shot gesture recognition have relied heavily on statistical and data-mining-based solutions, and have ignored the mechanisms that are used by humans to perceive and execute gestures and that can provide valuable context information. This omission has led to suboptimal solutions. The focus of this work is on the process that leads to the realization of a gesture, rather than on the gesture itself. In this case, context involves the way in which humans produce gestures—the kinematic and anthropometric characteristics. In the method presented here, the strategy is to generate a data set of realistic samples based on features extracted from a single gesture sample. These features, called the “gist of a gesture,” are considered to represent what humans remember when seeing a gesture and, later, the cognitive process involved when trying to replicate it. By adding meaningful variability to these features, a large training data set is created while preserving the fundamental structure of the original gesture. The availability of a large data set of realistic samples allows the use of training classifiers for future recognition. The performance of the method is evaluated using different lexicons, and its efficiency is compared with that of traditional N-shot learning approaches. The strength of the approach is further illustrated through human and machine recognition of gestures performed by a dual arm-robotic platform.

User-Authentication on Wearable Devices Based on Punch Gesture Biometrics

Due to commoditization and convenience, wearable technology are interwoven with our daily life. However, privacy sensitive data stored on those devices such as personal email, message can be easily stolen. Most devices require a PIN input to unlock. However, this mechanism is vulnerable to shoulder surfing attack. Thus many novel authentication approaches have been proposed to solve this problem. And biometric-based methods have been adopted by many researchers because of the efficiency and excellent performance. In this paper, we propose a new biometric-based authentication system. We focus on how the user performs a straight punch gesture subconsciously. By analysis the acceleration data from the smartwatch when user performing the gesture, we are able to profile the user. And we authenticate the user according to the biometrics of this action. This mechanism is light-weighted and do not require user to remember any secret code. We develop an authentication system on Samsung Gear Fit 2 and conducted a real-world experiment on 20 volunteers. And we collected 13000 gesture samples to evaluate our system. Results show that our system can achieve a classification accuracy of at least 95.45%. In attacking scenario, our system can achieve an equal error rate lower than 4%. The maximum number of samples required by a well-trained classifier is 25.

Natural Gesture Modeling and Recognition Approach Based on Joint Movements and Arm Orientations

Gesture recognition, as one of the most promising human–robot interaction approaches, has attracted research interests for a long time. Though many feasible methods have been proposed in this field, the current gesture modeling and recognition methods are deficient in expressiveness, naturalness, and efficiency. In this paper, we present a novel method of modeling and recognizing natural gestures in 3-D space. Inspired by the joint space modeling, which is frequently used in robotics, we decompose the gestures into information of joint movements, joint angles, and arm orientations. We calculate joint angles and detect joint movements with the orientation data measured by inertial measurement units. Numbers of gestures can be modeled by combining three types of criteria, namely, joint angle criteria, arm orientation criteria, and joint movement criteria. Our system can cope with the repetitive movements, which are common in gestures while rarely considered before. Different from the previous statistical learning or template matching approaches, our approach does not need training. User can add new gesture to the gesture set or edit existing gesture definitions conveniently. To evaluate our system, we conducted real-time gesture recognition experiments with ten subjects. Twelve gestures, including three static gestures and nine dynamic gestures, were modeled for the experiments. The average recognition accuracy of totaling 1560 gesture samples was 91.86%.

A 3‐D hand gesture signature based biometric authentication system for smartphones

AbstractThe authentication mechanism being equipped in most of the smartphones, by detecting a 4‐digit password or a simple pattern, are easy to be hacked and impersonated. In this paper, a 3‐D hand gesture signature (HGS) based biometric authentication system is designed and implemented by taking advantage of the on‐phone accelerometer to capture the 3‐D acceleration information when user makes a gesture to gain access to the phone. The captured data will be processed through a sequence of signal processing such as data smooth, gesture spotting, sequence alignment, and interpolation, and then a match rule will be used to compare the processed data and the genuine user's registered pattern to determine whether granting the access to the phone to the user. And an automatic template updating strategy based on cluster analysis is proposed to improve the stability of the system. The 3‐D HGS authentication system has been implemented on real smartphones, and the results tested for a total of 76 520 times by 19 users show very low false acceptance (0.27%) and false rejection rates (4.65%). Furthermore, comparison tests have been carried out among the 3‐D HGS and two similar authentication systems by exporting the real gesture samples from the phones to a desktop PC, the simulation results reveal the 3‐D HGS system has the best authentication accuracy. Copyright © 2016 John Wiley & Sons, Ltd.

A Gesture Recognition System for Detecting Behavioral Patterns of ADHD.

We present an application of gesture recognition using an extension of dynamic time warping (DTW) to recognize behavioral patterns of attention deficit hyperactivity disorder (ADHD). We propose an extension of DTW using one-class classifiers in order to be able to encode the variability of a gesture category, and thus, perform an alignment between a gesture sample and a gesture class. We model the set of gesture samples of a certain gesture category using either Gaussian mixture models or an approximation of convex hulls. Thus, we add a theoretical contribution to classical warping path in DTW by including local modeling of intraclass gesture variability. This methodology is applied in a clinical context, detecting a group of ADHD behavioral patterns defined by experts in psychology/psychiatry, to provide support to clinicians in the diagnose procedure. The proposed methodology is tested on a novel multimodal dataset (RGB plus depth) of ADHD children recordings with behavioral patterns. We obtain satisfying results when compared to standard state-of-the-art approaches in the DTW context.

Gesture recognition based on sparse representation

Aiming at the problem that the robustness of gesture recognition is difficult to guarantee, this paper presents a method based on multi-features and sparse representation. Hu invariant moments and HOG features of training samples are extracted in training phase. The K-SVD algorithm is used to train the initial value of dictionary formed by two features so as to obtain two sub-dictionaries. In recognition phase, sparse coefficients of corresponding training dictionary are derived by solving minimum l1-norm. Finally, the overall reconstruction error is calculated to judge the categories of test samples. In experimental simulation, five kinds of grasp gesture are collected to create gesture sample library. After selecting optimal HOG parameters and the weight of two features, the recognition effect of the method is analysed. Compared with the commonly used classification, the results show that the method has better recognition rate and robustness.

Gesture recognition based on sparse representation

Aiming at the problem that the robustness of gesture recognition is difficult to guarantee, this paper presents a method based on multi-features and sparse representation. Hu invariant moments and HOG features of training samples are extracted in training phase. The K-SVD algorithm is used to train the initial value of dictionary formed by two features so as to obtain two sub-dictionaries. In recognition phase, sparse coefficients of corresponding training dictionary are derived by solving minimum l1-norm. Finally, the overall reconstruction error is calculated to judge the categories of test samples. In experimental simulation, five kinds of grasp gesture are collected to create gesture sample library. After selecting optimal HOG parameters and the weight of two features, the recognition effect of the method is analysed. Compared with the commonly used classification, the results show that the method has better recognition rate and robustness.

Gestures à Go Go

Training a high-quality gesture recognizer requires providing a large number of examples to enable good performance on unseen, future data. However, recruiting participants, data collection, and labeling, etc., necessary for achieving this goal are usually time consuming and expensive. Thus, it is important to investigate how to empower developers to quickly collect gesture samples for improving UI usage and user experience. In response to this need, we introduce Gestures à Go Go ( g 3), a web service plus an accompanying web application for bootstrapping stroke gesture samples based on the kinematic theory of rapid human movements. The user only has to provide a gesture example once, and g 3 will create a model of that gesture. Then, by introducing local and global perturbations to the model parameters, g 3 generates from tens to thousands of synthetic human-like samples. Through a comprehensive evaluation, we show that synthesized gestures perform equally similar to gestures generated by human users. Ultimately, this work informs our understanding of designing better user interfaces that are driven by gestures.

LOCALIZATION AND RECOGNITION OF DYNAMIC HAND GESTURES BASED ON HIERARCHY OF MANIFOLD CLASSIFIERS

Abstract. Generally, the dynamic hand gestures are captured in continuous video sequences, and a gesture recognition system ought to extract the robust features automatically. This task involves the highly challenging spatio-temporal variations of dynamic hand gestures. The proposed method is based on two-level manifold classifiers including the trajectory classifiers in any time instants and the posture classifiers of sub-gestures in selected time instants. The trajectory classifiers contain skin detector, normalized skeleton representation of one or two hands, and motion history representing by motion vectors normalized through predetermined directions (8 and 16 in our case). Each dynamic gesture is separated into a set of sub-gestures in order to predict a trajectory and remove those samples of gestures, which do not satisfy to current trajectory. The posture classifiers involve the normalized skeleton representation of palm and fingers and relative finger positions using fingertips. The min-max criterion is used for trajectory recognition, and the decision tree technique was applied for posture recognition of sub-gestures. For experiments, a dataset “Multi-modal Gesture Recognition Challenge 2013: Dataset and Results” including 393 dynamic hand-gestures was chosen. The proposed method yielded 84–91% recognition accuracy, in average, for restricted set of dynamic gestures.