Gesture Patterns Research Articles

Optimization of Gesture Recognition Algorithm-aid of Artificial Neural Netwok

Objective:The Objective of our research is to establish an Artificial Neural Network (ANN) model for Braille coded gesture pattern recognition process and to optimize the weight of the ANN model using optimization algorithms. The ANN utilized to predict the outputs such as right gesture, left gesture, top and bottom gestures of the known input values. Methods: To optimize the weight of the ANN structure optimization techniques such as Genetic Algorithm (GA), Differential Evolution (DE), and Lion Algorithm (LA) are utilized to optimize the weights such as number of input neurons (α) and number of hidden neurons(β). Findings: The optimum results show that the attained empirical error values and predicted values are equal to zero in designed network. The Convergence graph demonstrates the fitness rate with minimum error 0.33 between different iterations for three optimization algorithms. The Lion algorithm results the optimal fitness value of Braille Coded Gesture ANN model. Applications: The lion algorithm can be utilized for gesture recognition in touch enabled devices in the area of human computer interaction.

Gesture production in the narratives of preschool children

The aim of the present study was to examine the use of gestures in the narrative productions of preschool children. Both the developmental patterns of gestures and verbal production and the relationships between them were analysed. The participants included 45 preschool children, aged 38 to 71 months. The narrative competence of each child was assessed individually using a new storybook narrative task (the Narrative Competence Task). Each child’s performance was videotaped to assess both verbal and gestural productions. The results showed that the syntactic complexity and lexical variety in the children’s narratives increased with increasing age, whereas the number of gestures significantly decreased. Moreover, the number of gestures was positively associated with the proportion of utterances without verbs and negatively correlated with the proportion of complex utterances. Therefore, we could hypothesise that gestures can be used as a support in narrative construction by those children who are less proficient in syntactic skills.

Gesture Recognition Algorithm: Braille-Coded Gesture Patterns for Touch Screens: Eyedroid

Objectives: The primary objective of this paper is to evaluate the correctness of the algorithm Gesture Recognition that is one of the parts of our novel method ‘Braille coded gesture pattern interaction method”; and to study the performance of Eyedroid system using the statistical methodologies. Methods/Statistical Analysis: The algorithm for gesture recognition was developed and implemented in C program to check the correctness of each step of algorithm. The gesture input parameters Swipe-Minimum-Distance, Swipe-Threshold-Velocity are defined with range of values. Varying gesture input range of values within domain and out of domain results the expected solution space. Our gesture recognition algorithm is developed for 2D touch surface. The Real time experiments were conducted for 12 blind fold subjects to observe the different versions Eyedroid-A(E-A), Eyedroid-B(E-B) of gesture recognition algorithm perceptions, being implemented in smart phones. The mean WPM for 10 trials was recorded for 12 subjects with E-A and E-B. The two tailed paired t test is performed for the mean dependent sample data of wpm. Findings: The existing 3D gesture recognition algorithm Derivative Dynamic Time Warping (DDTW) analyzed by Katarzyna Barczewska, Aleksandra Drozd, 2013 was proven to be efficient method for air fly environment. Our gesture recognition algorithm is developed for 2D touch surface found to be effective by the results of algorithm test. Gesture Recognition algorithm finds the mal gestures and the results proved that there is a significant difference in respect of blind subjects for two different Eyedroid versions. The observed data reveals E-B (mean WPM=14.95) is easy to use than E-A (mean WPM=12) as the number of words entered per minute by E-B is higher than E-A. Our Eyedroid with WPM is 14.95 is obtained to be better interactive system than the existing “Adaptive interaction Technique” system developed by Georgios Y fantidis Grigori Evreinov, 2005 confirms WPM is 12. Applications/Improvements: The implication is our Braille coded gesture pattern that enables visually challenged ones to enter more number of words than by the existing system. Our interaction method is deployable at all kinds of touchable devices.Keywords: Eyedroid-Braille Coded Gesture Pattern, Gesture Recognition Algorithm, t Test for Mobile Interaction

Toward the Autism Motor Signature: Gesture patterns during smart tablet gameplay identify children with autism.

Autism is a developmental disorder evident from infancy. Yet, its clinical identification requires expert diagnostic training. New evidence indicates disruption to motor timing and integration may underpin the disorder, providing a potential new computational marker for its early identification. In this study, we employed smart tablet computers with touch-sensitive screens and embedded inertial movement sensors to record the movement kinematics and gesture forces made by 37 children 3–6 years old with autism and 45 age- and gender-matched children developing typically. Machine learning analysis of the children’s motor patterns identified autism with up to 93% accuracy. Analysis revealed these patterns consisted of greater forces at contact and with a different distribution of forces within a gesture, and gesture kinematics were faster and larger, with more distal use of space. These data support the notion disruption to movement is core feature of autism, and demonstrate autism can be computationally assessed by fun, smart device gameplay.

Effect of variation in gesticulation pattern in dynamic hand gesture recognition system

This paper presents a hand gesture recognition system which addresses the effect of variations in gesture pattern during gesticulation. Different gestures can be gesticulated in various patterns which increase the difficulties in recognizing the gestures. We have proposed two new features such as left sector trajectory features and right sector trajectory features which are able to recognize gestures even with the presence of variations in the gesticulation pattern. The effectiveness of the proposed system is illustrated by different experiments with our own gesture database. A comparative study has been made with the proposed features and three state-of-art features such as orientation; combination of location, orientation, velocity; and combination of ellipse and position features. The performance of the system was evaluated using this proposed set of features for different individual classifiers such as ANN, SVM, k-NN, Naïve Bayes and ELM. Finally, the decisions of the individual classifiers were combined using major voting rule to result in classifier fusion model. Based on the experimental results it may be concluded that classifier fusion provides satisfactory results compared to other individual classifiers. An accuracy of 91.07% was achieved using the classifier fusion technique as compared to baseline CRF (79.45%) and HCRF (83.07%) models.

Twelve‐Month‐Olds Understand Social Intentions Based on Prosody and Gesture Shape

Infants infer social and pragmatic intentions underlying attention‐directing gestures, but the basis on which infants make these inferences is not well understood. Previous studies suggest that infants rely on information from preceding shared action contexts and joint perceptual scenes. Here, we tested whether 12‐month‐olds use information from act‐accompanying cues, in particular prosody and hand shape, to guide their pragmatic understanding. In Experiment 1, caregivers directed infants’ attention to an object to request it, share interest in it, or inform them about a hidden aspect. Caregivers used distinct prosodic and gestural patterns to express each pragmatic intention. Experiment 2 was identical except that experimenters provided identical lexical information across conditions and used three sets of trained prosodic and gestural patterns. In all conditions, the joint perceptual scenes and preceding shared action contexts were identical. In both experiments, infants reacted appropriately to the adults’ intentions by attending to the object mostly in the sharing interest condition, offering the object mostly in the imperative condition, and searching for the referent mostly in the informing condition. Infants’ ability to comprehend pragmatic intentions based on prosody and gesture shape expands infants’ communicative understanding from common activities to novel situations for which shared background knowledge is missing.

Efficient Query Processing in 3D Motion Capture Gesture Databases

One of the most fundamental challenges when accessing gestural patterns in 3D motion capture databases is the definition of spatiotemporal similarity. While distance-based similarity models such as the Gesture Matching Distance on gesture signatures are able to leverage the spatial and temporal characteristics of gestural patterns, their applicability to large 3D motion capture databases is limited due to their high computational complexity. To this end, we present a lower bound approximation of the Gesture Matching Distance that can be utilized in an optimal multi-step query processing architecture in order to support efficient query processing. We investigate the performance in terms of accuracy and efficiency based on 3D motion capture databases and show that our approach is able to achieve an increase in efficiency of more than one order of magnitude with a negligible loss in accuracy. In addition, we discuss different applications in the digital humanities in order to highlight the significance of similarity search approaches in the research field of gestural pattern analysis.

Gesture phase segmentation using support vector machines

An interaction between humans or between a human and a machine will be more effective if it is supported by gestures. In different levels of complexity, the communication system used in human interaction includes the use of gesture. In natural conversation, for instance, speakers use gestures for both to enhance the expressiveness of their speech and to support their own linguistic reasoning. The audience absorbs the content being transmitted also based on the speakers’ gesticulation. Thus, an analysis of gestures should add value to the purpose of the interaction. One of the concerns in the analysis of gestures is the problem arising from the segmentation of phases of a gesture (rest position, preparation, stroke, hold and retraction), which, from the standpoint of Gesture Theory, may reveal information on prosody and semantics of what is being said in a discourse. Finding an automation solution to this problem involves enabling the development of theoretical and application areas that are based on the analysis of human behavior and on the interpretation and generation of natural language. In this study, the problem of gesture phase segmentation is modeled as a problem of classification, and then support vector machine is employed to design a model able to learn the patterns of gesture that are inherent to each phase. This work presents two main highlights. The first is to address the limitations of the segmentation approach through the study of its performance in different scenarios that represent the complexity of analyzing patterns of human behavior. In this study, we reached an F-score around 0.9 for rest position and around 0.8 for stroke and preparation as segmentation results in the best cases. Moreover, it was possible to investigate how classification models are influenced by human behavior. The second highlight refers to the conduction of an analysis by considering the standpoint of specialists concerned with gesture phase segmentation in the area of Linguistics and Psycholinguistics, through which we obtained impressive results. Thus, in regard to the suitability of our approach, it is a feasible means of supporting the development of the Gesture Theory as well as the Computational Linguistics and Human Machine Interaction fields.

Imitation, Sign Language Skill and the Developmental Ease of Language Understanding (D-ELU) Model.

Imitation and language processing are closely connected. According to the Ease of Language Understanding (ELU) model (Rönnberg et al., 2013) pre-existing mental representation of lexical items facilitates language understanding. Thus, imitation of manual gestures is likely to be enhanced by experience of sign language. We tested this by eliciting imitation of manual gestures from deaf and hard-of-hearing (DHH) signing and hearing non-signing children at a similar level of language and cognitive development. We predicted that the DHH signing children would be better at imitating gestures lexicalized in their own sign language (Swedish Sign Language, SSL) than unfamiliar British Sign Language (BSL) signs, and that both groups would be better at imitating lexical signs (SSL and BSL) than non-signs. We also predicted that the hearing non-signing children would perform worse than DHH signing children with all types of gestures the first time (T1) we elicited imitation, but that the performance gap between groups would be reduced when imitation was elicited a second time (T2). Finally, we predicted that imitation performance on both occasions would be associated with linguistic skills, especially in the manual modality. A split-plot repeated measures ANOVA demonstrated that DHH signers imitated manual gestures with greater precision than non-signing children when imitation was elicited the second but not the first time. Manual gestures were easier to imitate for both groups when they were lexicalized than when they were not; but there was no difference in performance between familiar and unfamiliar gestures. For both groups, language skills at T1 predicted imitation at T2. Specifically, for DHH children, word reading skills, comprehension and phonological awareness of sign language predicted imitation at T2. For the hearing participants, language comprehension predicted imitation at T2, even after the effects of working memory capacity and motor skills were taken into account. These results demonstrate that experience of sign language enhances the ability to imitate manual gestures once representations have been established, and suggest that the inherent motor patterns of lexical manual gestures are better suited for representation than those of non-signs. This set of findings prompts a developmental version of the ELU model, D-ELU.

Do serial verb constructions describe single events?: A study of co-speech gestures in Avatime

Serial verb constructions have often been said to refer to single conceptual events. However, evidence to support this claim has been elusive. This article introduces co-speech gestures as a new way of investigating the relationship. The alignment patterns of gestures with serial verb constructions and other complex clauses were compared in Avatime (Ka-Togo, Kwa, Niger-Congo). Serial verb constructions tended to occur with single gestures overlapping the entire construction. In contrast, other complex clauses were more likely to be accompanied by distinct gestures overlapping individual verbs. This pattern of alignment suggests that serial verb constructions are in fact used to describe single events.

Patterns of non-verbal social interactions within intensive mathematics intervention contexts

This study examined the non-verbal patterns of interaction within an intensive mathematics intervention context. Specifically, the authors draw on social constructivist worldview to examine a teacher’s use of gesture in this setting. The teacher conducted a series of longitudinal teaching experiments with a small number of young, school-age children in the context of early arithmetic development. From these experiments, the authors gathered extensive video records of teaching practice and, from an inductive analysis of these records, identified three distinct patterns of teacher gesture: behavior eliciting, behavior suggesting, and behavior replicating. Awareness of their potential to influence students via gesture may prompt teachers to more closely attend to their own interactions with mathematical tools and take these teacher interactions into consideration when forming interpretations of students’ cognition.

Does language shape silent gesture?

Languages differ in how they organize events, particularly in the types of semantic elements they express and the arrangement of those elements within a sentence. Here we ask whether these cross-linguistic differences have an impact on how events are represented nonverbally; more specifically, on how events are represented in gestures produced without speech (silent gesture), compared to gestures produced with speech (co-speech gesture). We observed speech and gesture in 40 adult native speakers of English and Turkish (N=20/per language) asked to describe physical motion events (e.g., running down a path)—a domain known to elicit distinct patterns of speech and co-speech gesture in English- and Turkish-speakers. Replicating previous work (Kita & Özyürek, 2003), we found an effect of language on gesture when it was produced with speech—co-speech gestures produced by English-speakers differed from co-speech gestures produced by Turkish-speakers. However, we found no effect of language on gesture when it was produced on its own—silent gestures produced by English-speakers were identical in how motion elements were packaged and ordered to silent gestures produced by Turkish-speakers. The findings provide evidence for a natural semantic organization that humans impose on motion events when they convey those events without language.

Do gestures follow speech in bilinguals’ description of motion?

When do the gestures do and do not follow the patterns of the language one speaks? We examined this question by studying 10 Turkish-English bilingual adults (Turkish as L1) in comparison to 10 monolingual English and 10 monolingual Turkish adults as they described motion events either in speech with gesture (co-speech gesture) or only in gesture without speech (silent gesture). All speakers – monolingual and bilingual – showed cross-linguistic differences in co-speech gesture but not in silent gesture. Moreover, bilinguals followed L1 co-speech gesture patterns even when speaking L2, suggesting that acquisition of native-like gesture patterns does not co-occur with the acquisition of native-like speech patterns in bilinguals.

Trajectory‐based view‐invariant hand gesture recognition by fusing shape and orientation

Traditional studies in vision-based hand gesture recognition remain rooted in view-dependent representations, and hence users are forced to be fronto-parallel to the camera. To solve this problem, view-invariant gesture recognition aims to make the recognition result independent of viewpoint changes. However, in current works the view-invariance is achieved at the price of mixing different gesture patterns that have similar trajectory curve shape but different semantic meanings. For example, the gesture ‘push’ can be mistaken as ‘drag’ from another viewpoint. To address this shortcoming, in this study, the authors use a shape descriptor to extract the view-invariant features of a three-dimensional (3D) trajectory. As the shape features are invariant to omnidirectional viewpoint changes, the orientation features are then added into weight different rotation angles so that similar trajectory shapes are better separated. The proposed method was conducted on two different databases, including a popular Australian Sign Language database and a challenging Kinect Hand Trajectory database. Experimental results show that the proposed algorithm achieves a higher average recognition rate than the state-of-the-art approaches, and can better distinguish confusing gestures while meeting the view-invariant condition.

A Structure and Framework for Sign Language Interaction

Objective:The goal of this thesis is to design the interaction structure and framework of system to recognize sign language. Background: The sign language of meaningful individual gestures is combined to construct a sentence, so it is difficult to interpret and recognize the meaning of hand gesture for system, because of the sequence of continuous gestures. This being so, in order to interpret the meaning of individual gesture correctly, the interaction structure and framework are needed so that they can segment the indication of individual gesture. Method: We analyze 700 sign language words to structuralize the sign language gesture interaction. First of all, we analyze the transformational patterns of the hand gesture. Second, we analyze the movement of the transformational patterns of the hand gesture. Third, we analyze the type of other gestures except hands. Based on this, we design a framework for sign language interaction. Results: We elicited 8 patterns of hand gesture on the basis of the fact on whether the gesture has a change from starting point to ending point. And then, we analyzed the hand movement based on 3 elements: patterns of movement, direction, and whether hand movement is repeating or not. Moreover, we defined 11 movements of other gestures except hands and classified 8 types of interaction. The framework for sign language interaction, which was designed based on this mentioned above, applies to more than 700 individual gestures of the sign language, and can be classified as an individual gesture in spite of situation which has continuous gestures. Conclusion: This study has structuralized in 3 aspects defined to analyze the transformational patterns of the starting point and the ending point of hand shape, hand movement, and other gestures except hands for sign language interaction. Based on this, we designed the framework that can recognize the individual gestures and interpret the meaning more accurately, when meaningful individual gesture is input sequence of continuous gestures. Application: When we develop the system of sign language recognition, we can apply interaction framework to it. Structuralized gesture can be used for using database of sign language, inventing an automatic recognition system, and studying on the action gestures in other areas.

Development of compositional and contextual communicable congruence in robots by using dynamic neural network models

The current study presents neurorobotics experiments on acquisition of skills for “communicable congruence” with human via learning. A dynamic neural network model which is characterized by its multiple timescale dynamics property was utilized as a neuromorphic model for controlling a humanoid robot. In the experimental task, the humanoid robot was trained to generate specific sequential movement patterns as responding to various sequences of imperative gesture patterns demonstrated by the human subjects by following predefined compositional semantic rules. The experimental results showed that (1) the adopted MTRNN can achieve generalization by learning in the lower feature perception level by using a limited set of tutoring patterns, (2) the MTRNN can learn to extract compositional semantic rules with generalization in its higher level characterized by slow timescale dynamics, (3) the MTRNN can develop another type of cognitive capability for controlling the internal contextual processes as situated to on-going task sequences without being provided with cues for explicitly indicating task segmentation points. The analysis on the dynamic property developed in the MTRNN via learning indicated that the aforementioned cognitive mechanisms were achieved by self-organization of adequate functional hierarchy by utilizing the constraint of the multiple timescale property and the topological connectivity imposed on the network configuration. These results of the current research could contribute to developments of socially intelligent robots endowed with cognitive communicative competency similar to that of human.

Robot Comedy Lab: experimenting with the social dynamics of live performance

The success of live comedy depends on a performer's ability to “work” an audience. Ethnographic studies suggest that this involves the co-ordinated use of subtle social signals such as body orientation, gesture, gaze by both performers and audience members. Robots provide a unique opportunity to test the effects of these signals experimentally. Using a life-size humanoid robot, programmed to perform a stand-up comedy routine, we manipulated the robot's patterns of gesture and gaze and examined their effects on the real-time responses of a live audience. The strength and type of responses were captured using SHORE™computer vision analytics. The results highlight the complex, reciprocal social dynamics of performer and audience behavior. People respond more positively when the robot looks at them, negatively when it looks away and performative gestures also contribute to different patterns of audience response. This demonstrates how the responses of individual audience members depend on the specific interaction they're having with the performer. This work provides insights into how to design more effective, more socially engaging forms of robot interaction that can be used in a variety of service contexts.

Is there a universal answering strategy for rejecting negative propositions? Typological evidence on the use of prosody and gesture

Previous research has proposed that languages diverge with respect to how their speakers confirm and contradict negative questions. Taking into account the classification between truth-based and polarity-based languages, this paper is mainly concerned with the expression of REJECT (a semantic operation that signals a contradiction move with respect to the common ground, along Krifka's lines) in two languages belonging to two typologically distinct answering systems, namely Catalan (polarity-based) and Russian (a mixed system using polarity-based, truth-based, and echoic strategies). This investigation has two goals. First, to assess empirically the relevance of prosodic and gestural patterns in the interpretation of confirming and rejecting responses to negative polar questions. Second, to test the claim that in fact speakers resort to strikingly similar universal strategies at the time of expressing rejecting answers to discourse accessible negative assertions and negative polar questions, namely the use of linguistic units that encode REJECT in combination with ASSERT. The results of our investigation support the existence of a universal answering system for rejecting negative polar questions that integrates lexical and syntactic strategies with prosodic and gestural patterns, and instantiate the REJECT and ASSERT operators. We will also discuss the implications these results have for the truth-based vs. polarity-based taxonomy.

Communicating Epistemic Stance: How Speech and Gesture Patterns Reflect Epistemicity and Evidentiality

This study investigates how epistemic stance is encoded and perceived in face-to-face communication when language is regarded as comprised by speech and gesture. Two studies were conducted with this goal in mind. The first study consisted of a production task in which participants performed opinion reports. Results showed that speakers communicate epistemic stance both verbally and non-verbally, and that specific prosodic and gestural patterns are used to express different epistemic and evidential meanings. The second study consisted of a rating task in which listeners rated the degree of certainty expressed by the opinion reports. Results showed that the number of gestural high certainty markers used by a speaker was a good predictor of the perception of epistemic high certainty. We thus claim that prosodic and gestural markers can be regarded as overt manifestations of epistemicity and evidentiality, and they appear to be especially effective in the communication of epistemic stance.

Adaptive and Personalized Gesture Recognition Using Textile Capacitive Sensor Arrays

Upper extremity mobility impairment is a common sequel of Spinal Cord Injury (SCI), brain injury, strokes, and degenerative diseases such as Guillain-Barre and ALS. Existing assistive technology solutions that provide access as user input devices are intrusive and expensive, and require physical contact that can have deleterious effects such as skin friction injury for paralyzed users who have reduced skin sensitivity. To address this problem, in this paper, we present the design, implementation, and evaluation of a non-contact proximity gesture recognition system using fabric capacitive sensor arrays. The fabric sensors are lightweight, flexible, and can be easily integrated into items of quotidian use such as clothing, bed sheets, and pillow covers. Our gesture recognition algorithm builds on two known classification techniques, Hidden Markov Model and Dynamic Time Warping to convert raw capacitance values to alphanumeric gestures. Our system is personalized to the user, allowing personalized selection of gesture sets and definition of gesture patterns in accordance with their capabilities. Our system adapts to changes in sensor configuration and orientation with minimal user training and intervention. We have evaluated our system in the context of a gesture-driven home automation system on six subjects that includes an individual who has a C6 Spinal Cord injury. We show that our system can recognize gestures of varying complexity with an average accuracy of 99 percent with minimal training.