Gesture Features Research Articles

Understanding of human communicative motives in domestic dogs

Chimpanzees find it easier to locate food when a human prohibits them from going to a certain location than when she indicates that location helpfully. Human children, in contrast, use the cooperative gesture more readily. The question here was whether domestic dogs are more like chimpanzees, in this regard, or more like human children. In our first study we presented 40 dogs with two communicative contexts. In the cooperative context the experimenter informed the subject where food was hidden by pointing helpfully (with a cooperative tone of voice). In the competitive context the experimenter extended her arm towards the correct location in a prohibitive manner, palm of hand out (uttering a forbidding command in a prohibitive tone of voice). Dogs were successful in the cooperative condition ( P = 0.005) but chose randomly in the competitive condition ( P = 0.221). The second study independently varied the two characteristics of the communicative gesture (the gesture itself and the tone of voice). In addition to replicating dogs’ better performance with the cooperative gestures, this study suggests that tone of voice and context had more effect than type of gesture. In the context of food acquisition, domestic dogs, like human children, seem more prepared to use human gestures when they are given cooperatively.

Grounded blends and mathematical gesture spaces: developing mathematical understandings via gestures

This paper examines how a person’s gesture space can become endowed with mathematical meaning associated with mathematical spaces and how the resulting mathematical gesture space can be used to communicate and interpret mathematical features of gestures. We use the theory of grounded blends to analyse a case study of two teachers who used gestures to construct a graphical anti-derivative while working on a professional development task in a calculus modelling activity. Results indicate that mathematical gesture spaces can encourage mathematical experimentation, lighten the cognitive load for students and can be limited by a person’s physical constraints.

Toward a Minimal Representation of Affective Gestures

This paper presents a framework for analysis of affective behavior starting with a reduced amount of visual information related to human upper-body movements. The main goal is to individuate a minimal representation of emotional displays based on nonverbal gesture features. The GEMEP (Geneva multimodal emotion portrayals) corpus was used to validate this framework. Twelve emotions expressed by 10 actors form the selected data set of emotion portrayals. Visual tracking of trajectories of head and hands were performed from a frontal and a lateral view. Postural/shape and dynamic expressive gesture features were identified and analyzed. A feature reduction procedure was carried out, resulting in a 4D model of emotion expression that effectively classified/grouped emotions according to their valence (positive, negative) and arousal (high, low). These results show that emotionally relevant information can be detected/measured/obtained from the dynamic qualities of gesture. The framework was implemented as software modules (plug-ins) extending the EyesWeb XMI Expressive Gesture Processing Library and is going to be used in user centric, networked media applications, including future mobiles, characterized by low computational resources, and limited sensor systems.

A Comparison of Dynamic Naive Bayesian Classifiers and Hidden

In this paper we present a study to assess the performance of dynamic naive Bayesian classifiers (DNBCs) versusstandard hidden Markov models (HMMs) for gesture recognition. DNBCs incorporate explicit conditional independence among gesture features given states into HMMs. We show that this factorization offers competitive classification rates and error dispersion, it requires fewer parameters and it improves training time considerably in the presence of several attributes. We propose a set of qualitative and natural set of posture and motion attributes to describe gestures. We show that these posture-motion features increase recognition rates significantly in comparison to motion features. Additionally, an adaptive skin detection approach to cope with multiple users and different lighting conditions is proposed. We performed one of the most extensive experimentation presented in the literature to date that considers gestures of a single user, multiple people and with variations on distance and rotation using a gesture database with 9441 examples of 9 different classes performed by 15 people. Results show the effectiveness of the overall approach and the reliability of DNBCs in gesture ecognition.

Emotion Fusion Recognition for Intelligent Surveillance with PSO-CSVM

The next generation of intelligent surveillance system should be able to recognize human’s spontaneous emotion state automatically. Compared to speaker recognition, sensor signals analyzing, fingerprint or iris recognition, etc, facial expression and body gesture processing are two mainly non-intrusive vision modalities, which provides potential action information for video surveillance. In our work, we care one kind of facial expression, i.e. anxiety and gesture motion only. Firstly facial expression and body gesture feature are extracted. Particle Swarm Optimization algorithm is used to select feature subset and parameters optimization. The selected features are trained or tested for cascaded Support Vector Machine to obtain a high-accuracy classifier.

Gestural cue analysis in automated semantic miscommunication annotation

The automated annotation of conversational video by semantic miscommunication labels is a challenging topic. Although miscommunications are often obvious to the speakers as well as the observers, it is difficult for machines to detect them from the low-level features. We investigate the utility of gestural cues in this paper among various non-verbal features. Compared with gesture recognition tasks in human-computer interaction, this process is difficult due to the lack of understanding on which cues contribute to miscommunications and the implicitness of gestures. Nine simple gestural features are taken from gesture data, and both simple and complex classifiers are constructed using machine learning. The experimental results suggest that there is no single gestural feature that can predict or explain the occurrence of semantic miscommunication in our setting.

Modelling gesture as speech: A linguistic approach

Gesture communication, like prosody and paralinguistic voice features, strikes the attention when there is too little of it, too much of it, or when it does not seem to fit the words or the situation. The present study follows the principle that gesture is similar to some aspects of speech, particularly prosody and parts of the lexicon. Description of visual gesture articulation is therefore treated as a conservative extension of descriptions of vocal speech gesture articulation. Well-tried models of speech forms and functions are deployed, together with accounts from gesture studies from psychology to robotics. Evidence is taken from video data of story-telling in Ega, an African language, and in German, and the adequacy of descriptive and computational models of the forms and functions of speech is discussed, with a proposal for the formal modelling of speech-like timing of gesture articulators by means of Time Types in the Linear-Feature-Timing-Realtime (LFTR) model. Finally, an integrative model for combining visual and vocal gesture articulations into a comprehensive functional model of multimodal communication is proposed: the Rank Interpretation Model (RIM).

Gesture Control of Sound Synthesis: Analysis and Classification of Percussion Gestures

In recent years, the control of virtual instruments or sound-synthesis processes by natural gestures has become an important research field, both for building new audio-visual tools and for exploring gesture-sound relationships. Such multimodal and interactive tools typically present two advantages, on the one hand they provide realistic virtual instruments whose response can be compared to existing musical instruments, and on the other hand they give the possibility to vary characteristics of natural gestures, while ensuring a certain coherence between gesture and sound parameters. In this paper, we present and evaluate a new framework for explicitly expressing the characteristics of natural percussion gestures used for modeling, controlling and finally synthesizing new percussion gestures. A preliminary analysis of pre-recorded gestures leads to the identification and evaluation of significant parameters using a classification approach. This analysis shows that a reduced-dimension representation of captured motion can be used to control a virtual character. Furthermore, the simulated gestures provide dynamical variables that can be used to control sound synthesis through a mapping-interaction process.

Laryngeal Dynamics of Pedagogical Taan Gestures in Indian Classical Singing

Vocal modulations characterize many styles of singing. Vibrato, trill, and trillo are some of the ornaments that Western classical singers use. Likewise, taan is one of the basic frequency modulations demonstrated by Hindustani Indian classical singers. The objective of this descriptive study was to discover the F₀ contour of taan; establish selected acoustic, aerodynamic, and glottographic characteristics of the taan gesture; and explore the pedagogical taan utterances demonstrated by a well-known singer and teacher. Exploratory. Fundamental frequency, alternating current (AC) glottal flow, and electroglottographic width measures were obtained for taan productions by the classical Indian singer and teacher who demonstrated taan rate variations based on his pedagogical approach. The structure of the taan gesture was found to be an F₀ lowering and rising (the "taan dip") followed by a relatively flat portion (the "taan superior surface"). Rate of the F₀ structure of the taan gestures ranged from approximately 1.65 to 3.41Hz, and the F₀ extent ranged from 1.87 to 2.21semitone (ST). As the rate of the taan gesture increased, the superior surface shortened, whereas the taan dip stayed relatively constant (ranging from 170 to 230 ms). AC flow was greater for the lowest frequencies of the dip and faster rates. The pedagogical taan gesture has a specific structure of an F₀ dip followed by a relatively flat F₀ portion that shortens as taan rate increases. The F₀ dip and extent are relatively robust across rate. The taan productions are voluntarily controlled, in contrast to vibrato productions.

Utilizing gestures to improve sentence boundary detection

An accurate estimation of sentence units (SUs) in spontaneous speech is important for (1) helping listeners to better understand speech content and (2) supporting other natural language processing tasks that require sentence information. There has been much research on automatic SU detection; however, most previous studies have only used lexical and prosodic cues, but have not used nonverbal cues, e.g., gesture. Gestures play an important role in human conversations, including providing semantic content, expressing emotional status, and regulating conversational structure. Given the close relationship between gestures and speech, gestures may provide additional contributions to automatic SU detection. In this paper, we have investigated the use of gesture cues for enhancing the SU detection. Particularly, we have focused on: (1) collecting multimodal data resources involving gestures and SU events in human conversations, (2) analyzing the collected data sets to enrich our knowledge about co-occurrence of gestures and SUs, and (3) building statistical models for detecting SUs using speech and gestural cues. Our data analyses suggest that some gesture patterns influence a word boundary's probability of being an SU. On the basis of the data analyses, a set of novel gestural features were proposed for SU detection. A combination of speech and gestural features was found to provide more accurate SU predictions than using only speech features in discriminative models. Findings in this paper support the view that human conversations are processes involving multimodal cues, and so they are more effectively modeled using information from both verbal and nonverbal channels.

Motoric characteristics of representational gestures produced by young children in a naming task

This study explores the form of representational gestures produced by forty-five hearing children (age range 2 ; 0-3 ; 1) asked to label pictures in words. Five pictures depicting objects and five pictures depicting actions which elicited more representational gestures were chosen for more detailed analysis. The range of gestures produced for each item varied from 3 to 27 for a total of 128 gestures. Gestures have been analyzed with the same parameters used to describe signs produced by deaf children: handshape, location and movement. Results show that gestures for a given picture exhibit similarities in many of the parameters across children. Some motor characteristics found in the production of hearing toddlers' gestures are similar to those described for early signs. Implications of this similarity between gestural and signed linguistic representations in young children are discussed.

Real-time motion attention and expressive gesture interfaces

This paper aims at investigating the relationship between gestures’ expressivity and the amount of attention they attract. We present a technique for quantifying behavior saliency, here understood as the capacity to capture one’s attention, by the rarity of selected motion and gestural expressive features. This rarity index is based on the real-time computation of the occurrence probability of expressive motion features numerical values. Hence, the time instants that correspond to rare unusual dynamic patterns of an expressive feature are singled out. In a multi-user scenario, the rarity index highlights the person in a group which shows the most different behavior with respect to the others. In a mono-user scenario, the rarity index highlights when the expressive content of a gesture changes. Those methods can be considered as preliminary steps toward context-aware expressive gesture analysis. This work has been partly carried out in the framework of the eNTERFACE 2008 workshop (Paris, France, August 2008) and is partially supported by the EU ICT SAME Project (www.sameproject.eu) and by the NUMEDIART Project (www.numediart.org).

Recognition of Complex Dynamic Gesture Based on HMM-FNN Model

复杂动态手势识别是利用视频手势进行人机交互的关键问题.提出一种HMM-FNN模型结构.它整合了隐马尔可夫模型对时序数据的建模能力与模糊神经网络的模糊规则构建与推理能力,并将其应用到复杂动态手势的识别中.复杂动态手势具备两大特点:运动特征的可分解性与定义描述的模糊性.针对这两种特性,复杂手势被分解为手形变化、2D平面运动与Z轴方向运动3个子部分,分别利用HMM进行建模,HMM模型对观察子序列的似然概率被作为FNN的模糊隶属度,通过模糊规则推理,最终得到手势的分类类别.HMM-FNN方法将高维手势特征分解为低;Recognition of complex dynamic gesture is a key issue for visual gesture-based human-computer interaction. In this paper, an HMM-FNN model is proposed for gesture recognition, which combines ability of HMM model for temporal data modeling with that of fuzzy neural network for fuzzy rule modeling and fuzzy inference. Complex dynamic gesture has two important properties: Its motion can be decomposed and usually being defined in a fuzzy way. By HMM-FNN, complex gesture is firstly decomposed into three components: Posture changing, movement in 2D plane and movement in Z-axis direction, each of which is modeled by HMM. The likelihood of each HMM to observation sequence is considered as membership value of FNN, and gesture is classified through fuzzy inference of FNN. In this proposed method, high-dimensional gesture feature is transformed into several low-dimensional features, as a result, computational complexity is reduced. Furthermore, human's experience or prior knowledge can be used to build and optimize model structure. Experimental results show that the proposed method is an effective method for recognition of complex dynamic gesture, and is superior to conventional HMM method.

Phrase boundary effects on the temporal kinematics of sequential tongue tip consonants

This study evaluates the effects of phrase boundaries on the intra- and intergestural kinematic characteristics of blended gestures, i.e., overlapping gestures produced with a single articulator. The sequences examined are the juncture geminate [d(#)d], the sequence [d(#)z], and, for comparison, the singleton tongue tip gesture in [d(#)b]. This allows the investigation of the process of gestural aggregation [Munhall, K. G., and Lofqvist, A. (1992). "Gestural aggregation in speech: laryngeal gestures," J. Phonetics 20, 93-110] and the manner in which it is affected by prosodic structure. Juncture geminates are predicted to be affected by prosodic boundaries in the same way as other gestures; that is, they should display prosodic lengthening and lesser overlap across a boundary. Articulatory prosodic lengthening is also investigated using a signal alignment method of the functional data analysis framework [Ramsay, J. O., and Silverman, B. W. (2005). Functional Data Analysis, 2nd ed. (Springer-Verlag, New York)]. This provides the ability to examine a time warping function that characterizes relative timing difference (i.e., lagging or advancing) of a test signal with respect to a given reference, thus offering a way of illuminating local nonlinear deformations at work in prosodic lengthening. These findings are discussed in light of the pi-gesture framework of Byrd and Saltzman [(2003) "The elastic phrase: Modeling the dynamics of boundary-adjacent lengthening," J. Phonetics 31, 149-180].

Gesture Salience as a Hidden Variable for Coreference Resolution and Keyframe Extraction

Gesture is a non-verbal modality that can contribute crucial information to the understanding of natural language. But not all gestures are informative, and non-communicative hand motions may confuse natural language processing (NLP) and impede learning. People have little diffculty ignoring irrelevant hand movements and focusing on meaningful gestures, suggesting that an automatic system could also be trained to perform this task. However, the informativeness of a gesture is context-dependent and labeling enough data to cover all cases would be expensive. We present conditional modality fusion, a conditional hidden-variable model that learns to predict which gestures are salient for coreference resolution, the task of determining whether two noun phrases refer to the same semantic entity. Moreover, our approach uses only coreference annotations, and not annotations of gesture salience itself. We show that gesture features improve performance on coreference resolution, and that by attending only to gestures that are salient, our method achieves further significant gains. In addition, we show that the model of gesture salience learned in the context of coreference accords with human intuition, by demonstrating that gestures judged to be salient by our model can be used successfully to create multimedia keyframe summaries of video. These summaries are similar to those created by human raters, and significantly outperform summaries produced by baselines from the literature.

An investigation of conductors' temporal gestures and conductor— musician synchronization, and a first experiment

This article reports an empirical investigation into the characteristics of conductors' temporal gestures and people's ability to synchronize with them. It describes a new approach to the investigation of the characteristics of conductors' gestures, the investigation of conductor—musician synchronization, and a first experiment to demonstrate the feasibility of the approach. As part of this approach, a new computer-based environment for manipulating and presenting conductors' gestures, and recording participants' responses, was developed. This involved the creation of several pieces of new software, all of which were developed under Microsoft® Windows® using Visual C++. Used in combination, these programs allowed: (1) the manipulation of the size and viewing angle of high-quality three-dimensional (3D) recordings of conductors' gestures; (2) the presentation of these gestures to participants in a controlled experimental setting; (3) the recording of participants' temporal responses to these gestures; (4) the calculation of various parameters of both the trajectory of the gestures (e.g. instantaneous speed, radius of curvature along the trajectory) and participants' responses (e.g. mean response point and associated standard deviation); and (5) the graphical display of relevant features of both the gestures and participants' responses in a clear visual form. In the experiment, participants tapped in time with simple conducting gestures while several factors that might be expected to affect synchronization accuracy were manipulated. These factors were (1) the radius of curvature with which the beat was defined; (2) the experience level of the conductor; and (3) the experience level of participants. Results indicated that only participants' previous experience affected their synchronization ability; no effects of conductors' previous experience, or radius of curvature with which the beat was defined were found. This first experiment successfully demonstrated the feasibility of the new approach for the investigation of conductors' gestures and conductor— musician synchronization, and this article concludes by suggesting a number of subsequent experiments that could be undertaken in this computer-based environment to investigate these topics further.

Spatio-Temporal Feature-Extraction Techniques for Isolated Gesture Recognition in Arabic Sign Language

This paper presents various spatio-temporal feature-extraction techniques with applications to online and offline recognitions of isolated Arabic Sign Language gestures. The temporal features of a video-based gesture are extracted through forward, backward, and bidirectional predictions. The prediction errors are thresholded and accumulated into one image that represents the motion of the sequence. The motion representation is then followed by spatial-domain feature extractions. As such, the temporal dependencies are eliminated and the whole video sequence is represented by a few coefficients. The linear separability of the extracted features is assessed, and its suitability for both parametric and nonparametric classification techniques is elaborated upon. The proposed feature-extraction scheme was complemented by simple classification techniques, namely, K nearest neighbor (KNN) and Bayesian, i.e., likelihood ratio, classifiers. Experimental results showed classification performance ranging from 97% to 100% recognition rates. To validate our proposed technique, we have conducted a series of experiments using the classical way of classifying data with temporal dependencies, namely, hidden Markov models (HMMs). Experimental results revealed that the proposed feature-extraction scheme combined with simple KNN or Bayesian classification yields comparable results to the classical HMM-based scheme. Moreover, since the proposed scheme compresses the motion information of an image sequence into a single image, it allows for using simple classification techniques where the temporal dimension is eliminated. This is actually advantageous for both computational and storage requirements of the classifier.

Gesture and aphasia: Helping hands?

Background: The study of communicative gestures is one of considerable interest for aphasia, in relation to theory, diagnosis, and treatment. Significant limitations currently permeate the general (psycho)linguistic literature on gesture production, and attention to these limitations is essential for both continued investigation and clinical application of gesture for people with aphasia. Aims: The aims of this paper are to discuss issues imperative to advancing the gesture production literature and to provide specific suggestions for applying the material herein to studies in gesture production for people with aphasia. Main Contribution: Two primary perspectives in the gesture production literature are distinct in their proposals about the function of gesture, and about where gesture arises in the communication stream. These two perspectives will be discussed, along with three elements considered to be prerequisites for advancing the research on gesture production. These include: operational definitions, coding systems, and the temporal synchrony characteristics of gesture. Conclusions: Addressing the specific elements discussed in this paper will provide essential information for both continued investigation and clinical application of gesture for people with aphasia.

Linguistic properties based on American Sign Language isolated word recognition with artificial neural networks using a sensory glove and motion tracker

Sign language (SL), which is a highly visual–spatial, linguistically complete, and natural language, is the main mode of communication among deaf people. Described in this paper are two different American Sign Language (ASL) word recognition systems developed using artificial neural networks (ANN) to translate the ASL words into English. Feature vectors of signing words taken at five time instants were used in the first system, while histograms of feature vectors of signing words were used in the second system. The systems use a sensory glove, Cyberglove™, and a Flock of Birds ® 3-D motion tracker to extract the gesture features. The finger joint angle data obtained from strain gauges in the sensory glove define the hand shape, and the data from the tracker describe the trajectory of hand movement. In both systems, the data from these devices were processed by two neural networks: a velocity network and a word recognition network. The velocity network uses hand speed to determine the duration of words. Signs are defined by feature vectors such as hand shape, hand location, orientation, movement, bounding box, and distance. The second network was used as a classifier to convert ASL signs into words based on features or histograms of these features. We trained and tested our ANN models with 60 ASL words for a different number of samples. These methods were compared with each other. Our test results show that the accuracy of recognition of these two systems is 92% and 95%, respectively.

Head gestures for perceptual interfaces: The role of context in improving recognition

Head pose and gesture offer several conversational grounding cues and are used extensively in face-to-face interaction among people. To accurately recognize visual feedback, humans often use contextual knowledge from previous and current events to anticipate when feedback is most likely to occur. In this paper we describe how contextual information can be used to predict visual feedback and improve recognition of head gestures in human–computer interfaces. Lexical, prosodic, timing, and gesture features can be used to predict a user's visual feedback during conversational dialog with a robotic or virtual agent. In non-conversational interfaces, context features based on user–interface system events can improve detection of head gestures for dialog box confirmation or document browsing. Our user study with prototype gesture-based components indicate quantitative and qualitative benefits of gesture-based confirmation over conventional alternatives. Using a discriminative approach to contextual prediction and multi-modal integration, performance of head gesture detection was improved with context features even when the topic of the test set was significantly different than the training set.