Gesture Trajectory Research Articles

An image-to-class dynamic time warping approach for both 3D static and trajectory hand gesture recognition

In this paper, we present an Image-to-Class Dynamic Time Warping (I2C-DTW) approach for the recognition of both 3D static hand gestures and 3D hand trajectory gestures. Our contribution is twofold. First, we propose a technique to compute the image-to-class dynamic time warping distance instead of the Image-to-Image distance. By doing so, we obtain better generalization capability using the Image-to-Class distance than the Image-to-Image distance. Second, we propose a compositional model called fingerlets for static gesture representation, and a compositional model called strokelets for trajectory gesture representation. The compositional models make it possible to compute the DTW distance between a data sample and a gesture category. We have evaluated the static gesture recognition performance on several public 3D hand gesture datasets. For better evaluating the performance on trajectory gesture recognition, we collected a 3D hand trajectory gesture dataset, called UESTC-HTG, using a Kinect device. The experiment results show that the proposed I2C-DTW approach significantly improves the recognition accuracy on both static gestures and trajectory gestures.

Recognition of global hand gestures using self co-articulation information and classifier fusion

Hand gestures are a natural way of communication in our everyday life. Gestures provide a user friendly alternative of various types of devices available in our surroundings such as keyboard, mouse, and joysticks. This paper presents a hand gesture recognition system based on new set of features along with the classifier combining techniques to enhance the performance of the existing system. The study reveals that the unwanted movement within a gesture (self co-articulation) may be used as one of the complementary feature to improve the performance of the system. The self co-articulation was detected from the gesture trajectory by exploiting velocity information along with the pause information. The proposed new set of features (such as position of the hand, ratio feature and first half trajectory features) along with the self co-articulated features were used to define the feature space. Individual models as well as hybrid models were developed using the classifiers such as artificial neural network (ANN), multiclass support vector machine (SVM), k-nearest neighbours (k-NN) and Naive Bayes model. The models based on conditional random field (CRF) and hidden conditional random field (HCRF) were used to develop the baseline system for the present study. The experimental results suggest that the proposed new set of features combined with the hybrid classifier provides an improvement in the accuracy of the system by 14.58 and 8.41 % as compared to CRF and HCRF based models respectively. An analysis of variance test has also been conducted to validate the statistical significance of results.

Bridging gaps in common ground: Speakers design their gestures for their listeners.

Communication is shaped both by what we are trying to say and by whom we are saying it to. We examined whether and how shared information influences the gestures speakers produce along with their speech. Unlike prior work examining effects of common ground on speech and gesture, we examined a situation in which some speakers have the same amount of mutually shared experience with their listener but the relevance of the information from shared experience is different for listeners in different conditions. Additionally, speakers and listeners in all conditions shared a visual perspective. Speakers and listeners solved a version of the Tower of Hanoi task together. Speakers then solved a second version of the task without the listener present with the manner of disk movement manipulated; the manner was either the same as or different from the version that had been solved with the listener present. Thus, speakers' knowledge of the relevance of shared knowledge was manipulated. We measured the content of speech along with the physical form and content of the accompanying hand gesture. Although speakers did not modulate their spoken language, speakers who knew their listeners had not previously experienced the appropriate manner of completion gestured higher in space, highlighting manner information, but without altering the physical gesture trajectory. Thus, gesture can be sensitive to the knowledge of listeners even when speech is not. Speakers' gestures can play an independent role in reflecting common ground between speakers and listeners, perhaps by simultaneously incorporating both speaker and listener perspectives. (PsycINFO Database Record

Hand Gesture Recognition Using Particle Swarm Movement

We present a gesture recognition method derived from particle swarm movement for free-air hand gesture recognition. Online gesture recognition remains a difficult problem due to uncertainty in vision-based gesture boundary detection methods. We suggest an automated process of segmenting meaningful gesture trajectories based on particle swarm movement. A subgesture detection and reasoning method is incorporated in the proposed recognizer to avoid premature gesture spotting. Evaluation of the proposed method shows promising recognition results: 97.6% on preisolated gestures, 94.9% on stream gestures with assistive boundary indicators, and 94.2% for blind gesture spotting on digit gesture vocabulary. The proposed recognizer requires fewer computation resources; thus it is a good candidate for real-time applications.

Case Study: Hand Gesture Based Digital Pen

In this paper an MEMS accelerometer mostly based on gesture recognition algorithm and its application are presented. The hardware module consists of a triaxial MEMS accelerometer, microcontroller, and RF wireless transmission module for sensing and collecting accelerations of handwriting and hand gesture trajectories. Users will use this digital pen to write down digits, alphabets& some hand gestures. The accelerations of hand motions deliberate by the accelerometer are transmit wirelessly to a personal computer for trajectory recognition. The trajectory algorithm collected of information variety assortment, signal pre-processing for reconstruct the trajectory to satisfy the collective errors caused by drift of sensors. So, by altering the location of MEMS (micro electro mechanical systems) we can proficient to demonstrate the alphabetical characters and numerical on PC.

A Dynamic Gesture Trajectory Recognition Based on Key Frame Extraction and HMM

Aiming at changing high computational complexity, underdeveloped real time, low recognition rate of dynamic gesture recognition algorithms, this paper present a real-time dynamic gesture trajectory recognition method based on key frame extraction and HMM. Key frames are selected without keeping track of all the details of one dynamic gesture, which is based on difference degree between frames. The trajectory data stream, sorted by the time-warping algorithm, is used to construct the Hidden Markov Method model of dynamic gesture. Finally, optimal transition probabilities are employed to implement dynamic gesture recognition. The result of this experiment implies that this method has high robustness and real time. The average recognition rate of dynamic gesture (0~9) is up to 87.67%, and average time efficiency is 0.46s.

RGBD Video Based Human Hand Trajectory Tracking and Gesture Recognition System

The task of human hand trajectory tracking and gesture trajectory recognition based on synchronized color and depth video is considered. Toward this end, in the facet of hand tracking, a joint observation model with the hand cues of skin saliency, motion and depth is integrated into particle filter in order to move particles to local peak in the likelihood. The proposed hand tracking method, namely, salient skin, motion, and depth based particle filter (SSMD-PF), is capable of improving the tracking accuracy considerably, in the context of the signer performing the gesture toward the camera device and in front of moving, cluttered backgrounds. In the facet of gesture recognition, a shape-order context descriptor on the basis of shape context is introduced, which can describe the gesture in spatiotemporal domain. The efficient shape-order context descriptor can reveal the shape relationship and embed gesture sequence order information into descriptor. Moreover, the shape-order context leads to a robust score for gesture invariant. Our approach is complemented with experimental results on the settings of the challenging hand-signed digits datasets and American sign language dataset, which corroborate the performance of the novel techniques.

Localization and Recognition of Digit-Writing Hand Gestures for Smart TV Systems

A novel framework is put forward in this paper for localization and recognition of digit-writing hand gestures, which forms the basis of a vision-based interactive interface that we will develop later for smart television systems. The framework uses as input a video stream capturing the hand gesture of writing a digit dynamically by the user, and outputs the digit inferred from the gesture. A fast and robust tracker is used to flnd the gesture trajectory, and prior constraints and a motion predictor are applied to improve the tracker’s performance. The trajectory is then turned into the flnal output by a recognizer, which is formed by the combination of dynamic time warping and the convolutional neural network. The efiectiveness of the proposed framework is validated by experiments.

3D hand gesture tracking and recognition for controlling an intelligent wheelchair

Hand gesture recognition is a user-friendly and intuitive means for human machine interaction. This paper proposes a novel 3D hand gesture recognition method for controlling an intelligent wheelchair based on both colour and depth information. Image depth information of human palm is obtained by a 3D Kinect vision sensor and then its position is obtained through the hand analysis module in OpenNI. The improved Centroid Distance Function is used to extract 3D hand trajectory features, while hidden Markov model HMM is applied to train samples and recognise hand gesture trajectories. Finally, the recognition results are converted into control commands through an ad hoc network and sent to an intelligent wheelchair for its motion control. Experiment results show that the proposed method has good invariance to lighting changes, hand rotation and scaling conditions and is very robust to background interference.

Rule-based trajectory segmentation for modeling hand motion trajectory

In this paper, we propose a simple but effective method of modeling hand gestures based on the angles and angular change rates of the hand trajectories. Each hand motion trajectory is composed of a unique series of straight and curved segments. In our Hidden Markov Model (HMM) implementation, these trajectories are modeled as a connected series of states analogous to the series of phonemes in speech recognition. The novelty of the work presented herein is that it provides an automated process of segmenting gesture trajectories based on a simple set of threshold values in the angular change measure. In order to represent the angular distribution of each separated state, the von Mises distribution is used. A likelihood based state segmentation was implemented in addition to the threshold based method to ensure that the gesture sets are segmented consistently. The proposed method can separate each angular state of the training data at the initialization step, thus providing a solution to mitigate the ambiguities on initializing the HMM. The effectiveness of the proposed method was demonstrated by the higher recognition rates in the experiments compared to the conventional methods.

Hidden Markov Model on a unit hypersphere space for gesture trajectory recognition

In this paper, a Mixture of von Mises-Fisher (MvMF) Probability Density Function (PDF) is incorporated into a Hidden Markov Model (HMM) in order to model spatio-temporal data in a unit-hypersphere space. The parameter estimation formulae for MvMF-HMM are derived in a closed form. As an application for the proposed MvMF-HMM, hands gesture trajectory recognition task is considered. Modeling gesture trajectory on a unit-hypersphere inherently removes bias from a subject’s arm length or distance between a subject and camera. In experiments with public datasets, InteractPlay and UCF Kinect, the proposed MvMF-HMM showed superior recognition performance compared to current state-of-the-art techniques.

다중 클래스 SVM과 트리 분류를 이용한 제스처 인식 방법

제스처 인식은 자연스러운 사용자 인터페이스를 위해 활발히 연구되는 중요한 분야이다. 본 논문에서는 키넥트 카메라로부터 입력되는 사용자의 3차원 관절(joint) 정보를 해석하여 제스처를 인식하는 방법이 제안된다. 대상으로 하는 제스처의 분포 특성에 따라 분류 트리를 설계하고 입력 패턴을 분류한다. 그리고 제스처를 리샘플링 및 정규화 하여 일정한 구간으로 나누고 각 구간의 체인코드 히스토그램을 추출한다. 트리의 각 노드별로 분류된 제스처에 다중 클래스 SVM(Multiclass Support Vector Machine)를 적용하여 학습한다. 이후 입력 데이터를 구성된 트리로 분류한 다음, 학습된 다중 클래스 SVM을 적용하여 제스처를 분류한다.

An optimized real-time hands gesture recognition based interface for individuals with upper-level spinal cord injuries

This paper presents a hand gesture-based interface to facilitate interaction with individuals with upper-level spinal cord injuries, and offers an alternative way to perform "hands-on" laboratory tasks. The presented system consists of four modules: hand detection, tracking, trajectory recognition, and actuated device control. A 3D particle filter framework based on color and depth information is proposed to provide a more efficient solution to the independent face and hands tracking problem. More specifically, an interaction model utilizing spatial and motion information was integrated into the particle filter framework to tackle the "false merge" and "false labeling" problem through hand interaction and occlusion. To obtain an optimal parameter set for the interaction model, a neighborhood search algorithm was employed. An accuracy of 98.81 % was achieved by applying the optimal parameter set to the tracking module of the system. Once the hands were tracked successfully, the acquired gesture trajectories were compared with motion models. The dynamic time warping method was used for signals' time alignment, and they were classified by a CONDENSATION algorithm with a recognition accuracy of 97.5 %. In a validation experiment, the decoded gestures were passed as commands to a mobile service robot and a robotic arm to perform simulated laboratory tasks. Control policies using the gestural control were studied and optimal policies were selected to achieve optimal performance. The computational cost of each system module demonstrated a real-time performance.

Most discriminating segment – Longest common subsequence (MDSLCS) algorithm for dynamic hand gesture classification

In this work, we consider the recognition of dynamic gestures based on representative sub-segments of a gesture, which are denoted as most discriminating segments (MDSs). The automatic extraction and recognition of such small representative segments, rather than extracting and recognizing the full gestures themselves, allows for a more discriminative classifier. A MDS is a sub-segment of a gesture that is most dissimilar to all other gesture sub-segments. Gestures are classified using a MDSLCS algorithm, which recognizes the MDSs using a modified longest common subsequence (LCS) measure. The extraction of MDSs from a data stream uses adaptive window parameters, which are driven by the successive results of multiple calls to the LCS classifier. In a preprocessing stage, gestures that have large motion variations are replaced by several forms of lesser variation. We learn these forms by adaptive clustering of a training set of gestures, where we reemploy the LCS to determine similarity between gesture trajectories. The MDSLCS classifier achieved a gesture recognition rate of 92.6% when tested using a set of pre-cut free hand digit (0–9) gestures, while hidden Markov models (HMMs) achieved an accuracy of 89.5%. When the MDSLCS was tested against a set of streamed digit gestures, an accuracy of 89.6% was obtained. At present the HMMs method is considered the state-of-the-art method for classifying motion trajectories. The MDSLCS algorithm had a higher accuracy rate for pre-cut gestures, and is also more suitable for streamed gestures. MDSLCS provides a significant advantage over HMMs by not requiring data re-sampling during run-time and performing well with small training sets.

Elemental gesture dynamics are encoded by song premotor cortical neurons

Quantitative biomechanical models can identify control parameters used during movements, and movement parameters encoded by premotor neurons. We fit a mathematical dynamical systems model including subsyringeal pressure, syringeal biomechanics, and upper vocal tract filtering to the songs of zebra finches. This reduced the dimensionality of singing dynamics, described as trajectories in pressure-tension space (motor “gestures”). We assessed model performance by characterizing the auditory response "replay" of song premotor HVC neurons to presentation of song variants in sleeping birds, and by examining HVC activity in singing birds. HVC projection neurons were excited and interneurons were suppressed with near-zero time lag, at times of gesture trajectory extrema. Thus, HVC precisely encodes vocal motor output via the timing of extreme points of movement trajectories. We propose that the sequential activity of HVC neurons represents the sequence of gestures in song as a “forward” model making predictions on expected behavior to evaluate feedback.

멀티미디어 콘텐츠 제어를 위한 사용자 경험 기반 동작 인식 기술

In this paper, a series of algorithms are proposed for controlling different kinds of multimedia contents and realizing interact between human and computer by using single input device. Human gesture recognition based on NUI is presented firstly in my paper. Since the image information we get it from camera is not sensitive for further processing, we transform it to YCbCr color space, and then morphological processing algorithm is used to delete unuseful noise. Boundary Energy and depth information is extracted for hand detection. After we receive the image of hand detection, PCA algorithm is used to recognize hand posture, difference image and moment method are used to detect hand centroid and extract trajectory of hand movement. 8 direction codes are defined for quantifying gesture trajectory, so the symbol value will be affirmed. Furthermore, HMM algorithm is used for hand gesture recognition based on the symbol value. According to series of methods we presented, we can control multimedia contents by using human gesture recognition. Through large numbers of experiments, the algorithms we presented have satisfying performance, hand detection rate is up to 94.25%, gesture recognition rate exceed 92.6%, hand posture recognition rate can achieve 85.86%, and face detection rate is up to 89.58%. According to these experiment results, we can control many kinds of multimedia contents on computer effectively, such as video player, MP3, e-book and so on.

An Accelerometer-Based Digital Pen With a Trajectory Recognition Algorithm for Handwritten Digit and Gesture Recognition

This paper presents an accelerometer-based digital pen for handwritten digit and gesture trajectory recognition applications. The digital pen consists of a triaxial accelerometer, a microcontroller, and an RF wireless transmission module for sensing and collecting accelerations of handwriting and gesture trajectories. The proposed trajectory recognition algorithm composes of the procedures of acceleration acquisition, signal preprocessing, feature generation, feature selection, and feature extraction. The algorithm is capable of translating time-series acceleration signals into important feature vectors. Users can use the pen to write digits or make hand gestures, and the accelerations of hand motions measured by the accelerometer are wirelessly transmitted to a computer for online trajectory recognition. The algorithm first extracts the time- and frequency-domain features from the acceleration signals and, then, further identifies the most important features by a hybrid method: kernel-based class separability for selecting significant features and linear discriminant analysis for reducing the dimension of features. The reduced features are sent to a trained probabilistic neural network for recognition. Our experimental results have successfully validated the effectiveness of the trajectory recognition algorithm for handwritten digit and gesture recognition using the proposed digital pen.

Dynamic Human Activity Recognition Based on Improved FNN Model

In this paper, we propose an automatic system that recognizes dynamic human gestures activity, including Arabic numbers from 0 to 9. We assume the gesture trajectory is almost in a plane that called principal gesture plane, then the Least Squares Method is used to estimate the plane and project the 3 D trajectory model onto the principal. An improved FNN model combined with HMM is proposed for dynamic gesture recognition, which combines ability of HMM model for temporal data modeling with that of fuzzy neural network. The proposed algorithm shows that satisfactory performance and high recognition rate.

강인한 손가락 끝 추출과 확장된 CAMSHIFT 알고리즘을 이용한 자연스러운 Human-Robot Interaction을 위한 손동작 인식

In this paper, we propose a robust fingertip extraction and extended Continuously Adaptive Mean Shift (CAMSHIFT) based robust hand gesture recognition for natural human-like HRI (Human-Robot Interaction). Firstly, for efficient and rapid hand detection, the hand candidate regions are segmented by the combination with robust <TEX>$YC_bC_r$</TEX> skin color model and haar-like features based adaboost. Using the extracted hand candidate regions, we estimate the palm region and fingertip position from distance transformation based voting and geometrical feature of hands. From the hand orientation and palm center position, we find the optimal fingertip position and its orientation. Then using extended CAMSHIFT, we reliably track the 2D hand gesture trajectory with extracted fingertip. Finally, we applied the conditional density propagation (CONDENSATION) to recognize the pre-defined temporal motion trajectories. Experimental results show that the proposed algorithm not only rapidly extracts the hand region with accurately extracted fingertip and its angle but also robustly tracks the hand under different illumination, size and rotation conditions. Using these results, we successfully recognize the multiple hand gestures.

Fast and Robust Method for Dynamic Gesture Recognition Using Hermite Neural Network

Due to its shortcomings such as slow convergence rate and low recognition accuracy, the traditional BP neural networks perform poorly in dynamic gesture recognition, especially for online gesture training. In this paper, a novel adaptive Hermite neural networks algorithm for dynamic gesture recognition was proposed. At first, a three-layer feed-forward neural network, of which hidden layer neurons are activated by a group of Hermite orthogonal polynomial functions was constructed. Based on its special structure, a method to determine the network weights directly was introduced, and a novel algorithm to determine the optimal number of hidden nodes adaptively was proposed. Then a rapid method of fingertips tracking was put forward to get the trajectory of dynamic gesture. At last, dynamic gesture was recognized through the trained Hermite neural network. Experiment results show that Hermite neural network can enhance the speed and precision of network training, improve the learning speed and recognition accuracy, and has good robustness and generalization ability.