Gestural Activity Research Articles

Accidental, Assisted, Automated: An Emerging Repertoire of Infant Mobile Media Techniques

Accidental, Assisted, Automated: An Emerging Repertoire of Infant Mobile Media Techniques

You looking at me?: Interpreting social cues in schizophrenia.

Deficits in the perception of social cues are common in schizophrenia and predict functional outcome. While effective communication depends on deciphering both verbal and non-verbal features, work on non-verbal communication in the disorder is scarce. This behavioural study of 29 individuals with schizophrenia and 25 demographically matched controls used silent video-clips to examine gestural identification, its contextual modulation and related metacognitive representations. In accord with our principal hypothesis, we observed that individuals with schizophrenia exhibited a preserved ability to identify archetypal gestures and did not differentially infer communicative intent from incidental movements. However, patients were more likely than controls to perceive gestures as self-referential when confirmatory evidence was ambiguous. Furthermore, the severity of their current hallucinatory experience inversely predicted their confidence ratings associated with these self-referential judgements. These findings suggest a deficit in the contextual refinement of social-cue processing in schizophrenia that is potentially attributable to impaired monitoring of a mirror mechanism underlying intentional judgements, or to an incomplete semantic representation of gestural actions. Non-verbal communication may be improved in patients through psychotherapeutic interventions that include performance and perception of gestures in group interactions.

Apraxia of tool use: Assessment strategies

A substantial proportions of patients with brain damage and neurodegenerative diseases misuse common tools. However, this neuropsychological syndrome affecting everyday life is relatively unexplored in the field of experimental and clinical neuropsychology. Little is known about long-term evolution and specific evaluation and/or rehabilitation. This is partially due to the fact that apraxia of tool use faces the lack of an integrative theoretical framework taking into account all cognitive processes underlying gesture orientation, object selection or action sequencing. Indeed, apraxia of tool use goes far beyond the traditional, obsolete, distinction between ideational apraxia and ideomotor apraxia. This is a complex symptomatology requiring a conceptual and clinical differential analysis. After a brief overview of the theoretical principles underpinning our evaluation method, the purpose of this presentation is to describe the various types of tests, which are required to exhaustively assess tool use disorders. In that sense, we want to reemphasize the importance of preliminary neurological examination as well as the exploration of language, body schema (probably involved in imitation disorders) and motor sequences (unilateral, limb-kinetic apraxia). We will then focus on tool use assessment: types of object, basic knowledge related to them, presentation modalities, action planning. Finally, we illustrate our approach with studies in patients with left brain damage.

A Gesture Recognition Design Toolkit Everyday Gesture Library

Creating gestural recognition system is a challenging task which requires skills and updated applications. It is required for designer to be skillful and innovative in order to create interesting and acceptable gestures. Gesture Recognition Design Toolkit (GRDT) is set of tools designed to simplify the gesture creation process for non experts. From designing a new gesture, to suggesting the best fit, to test it for false positive activation, till final selection, are the step-by-step functions of Gesture Recognition Design Toolkit. Creating gestural recognition system, interface designers encounter many challenges throughout the process. Selecting the best gesture for a particular motion with low probability of false positives is one of many. This study explores ways to improve the functionality and enhance the efficiency of gestural library using machine learning techniques and data mining General Terms Multiple Action Gesture Interface Creation, Human computer interaction, Symbolic aggregate approximation, Nearest neighbor dynamic time wrapping, Hidden markov model, Dynamic time wrapping, Extract, transform and load

An adaptive hidden Markov model-based gesture recognition approach using Kinect to simplify large-scale video data processing for humanoid robot imitation

Human gesture recognition to be a new type of natural user interface (NUI) using the person's gesture action for operating the device is attracting much attention nowadays. In this study, an adaptive hidden Markov model (HMM)-based gesture recognition method with user adaptation (UA) using the Kinect camera to simplify large-scale video processing is designed to be the NUI of a humanoid robot device. The popular Kinect camera is employed for acquiring the gesture signals made by the active user, and the gesture action from the user can then be recognized and used to be as the control command for driving the humanoid robot to imitate the user's actions. The large-scale video data can be reduced by the Kinect camera where the data from the Kinect camera for representing gesture signals includes the depth measurement information, and therefore only simple 3-axis coordinate information of the joints in a human skeleton is analyzed, categorized and managed in the developed system. By the presented scheme, the humanoid robot will imitate the human active gesture according to the content of the received gesture command. The well-known HMM pattern recognition method with the support of the Kinect device is explored to classify the human's active gestures where a user adaptation scheme of MAP+GoSSRT that enhances MAP by incorporating group of states shifted by referenced transfer (GoSSRT) is proposed for adjusting HMM parameters, which will further increase the recognition accuracy of HMM gesture recognition. Human gesture recognition experiments for controlling the activity of the humanoid robot were performed on the indicated 14 classes of human active gestures. Experimental results demonstrated the superiority of the NUI by presented HMM gesture recognition with user adaptation for humanoid robot imitation applications.

Commentary: A pointer about grasping numbers.

Commentary: A pointer about grasping numbers.

Out of touch – A plugin for controlling OsiriX with gestures using the leap controller

Abstract Objectives In recent years, different systems for gesture control of medical devices have been presented. Today, low-cost gesture control systems are commercially available. In this article, we present a plugin for the OsiriX medical image viewer, which operates the viewer using finger gestures. Materials and methods We use a device called the Leap controller for gesture control input. The device has a low cost, and it uses structured light to create a depth image of the screen. The drivers provided by the manufacturer allow for easy integration into existing software projects. The presented plugin features gestures for panning, zooming, windowing and browsing of medical image datasets. Conclusion Gesture control technology is becoming increasingly robust and available. We therefore expect that gesture control solutions will be increasingly implemented in medical environments. For broad acceptance of such technologies, standard gestures for certain actions should be standardized in the future.

Middle School Teachers’ Discipline-Specific Use of Gestures and Implications for Disciplinary Literacy Instruction

Framed in theories of social semiotics, this descriptive multiple case study examined six middle school teachers’ use of gestures during one school year as they each taught two different subject areas: earth science, language arts, mathematics, and/or social studies. The data, which included field notes and photographs from 354 lessons and 151 video-recordings of lesson segments, were analyzed using constant comparative methods and multimodal concordance charts. The analyses indicated discipline-specific differences in types of gestures, frequency of gestures, and centrality of gestures to the teachers’ messages. Earth science depended on a variety of iconic and deictic gestures, the latter of which was also common in mathematics. Communications in language arts and social studies commonly included non- essential action gestures that mimicked the movements of characters and historical figures. This study modifies existing claims of the importance of gestures in teaching, suggesting that gestures can play relatively central or minor roles in communicating disciplinary concepts. It concludes with implications for disciplinary literacy instruction that more rigorously accounts for the role that gestures play in disciplinary learning.

Critical brain regions for tool-related and imitative actions: a componential analysis.

Numerous functional neuroimaging studies suggest that widespread bilateral parietal, temporal, and frontal regions are involved in tool-related and pantomimed gesture performance, but the role of these regions in specific aspects of gestural tasks remains unclear. In the largest prospective study of apraxia-related lesions to date, we performed voxel-based lesion-symptom mapping with data from 71 left hemisphere stroke participants to assess the critical neural substrates of three types of actions: gestures produced in response to viewed tools, imitation of tool-specific gestures demonstrated by the examiner, and imitation of meaningless gestures. Thus, two of the three gesture types were tool-related, and two of the three were imitative, enabling pairwise comparisons designed to highlight commonalities and differences. Gestures were scored separately for postural (hand/arm positioning) and kinematic (amplitude/timing) accuracy. Lesioned voxels in the left posterior temporal gyrus were significantly associated with lower scores on the posture component for both of the tool-related gesture tasks. Poor performance on the kinematic component of all three gesture tasks was significantly associated with lesions in left inferior parietal and frontal regions. These data enable us to propose a componential neuroanatomic model of action that delineates the specific components required for different gestural action tasks. Thus, visual posture information and kinematic capacities are differentially critical to the three types of actions studied here: the kinematic aspect is particularly critical for imitation of meaningless movement, capacity for tool-action posture representations are particularly necessary for pantomimed gestures to the sight of tools, and both capacities inform imitation of tool-related movements. These distinctions enable us to advance traditional accounts of apraxia.

Corrective feedback, gesture, and mediation in classroom language learning

This article examines the way in which a teacher’s provision of form-focused corrective feedback synchronized with gesture mediated a student’s learning in an ESL reading classroom. Specifically, the focus of the analysis is on the student’s appropriation of the teacher’s gesture as a tool that mediates his own independent functioning. It is argued that while the teacher’s corrective feedback in the verbal channel helped to scaffold the learner’s performance by modeling a correct syntactic structure and highlighting the specific locus of the correction, it was her gesture activity that became available to the student as a psychological tool. The discussion centers on the role of gesture in mediating second language development. Implications for future research and classroom language teaching are also sketched out.

Supervised Spatio-Temporal Neighborhood Topology Learning for Action Recognition

Supervised manifold learning has been successfully applied to action recognition, in which class label information could improve the recognition performance. However, the learned manifold may not be able to well preserve both the local structure and global constraint of temporal labels in action sequences. To overcome this problem, this paper proposes a new supervised manifold learning algorithm called supervised spatio-temporal neighborhood topology learning (SSTNTL) for action recognition. By analyzing the topological characteristics in the context of action recognition, we propose to construct the neighborhood topology using both supervised spatial and temporal pose correspondence information. Employing the property in locality preserving projection (LPP), SSTNTL solves the generalized eigenvalue problem to obtain the best projections that not only separates data points from different classes, but also preserves local structures and temporal pose correspondence of sequences from the same class. Experimental results demonstrate that SSTNTL outperforms the manifold embedding methods with other topologies or local discriminant information. Moreover, compared with state-of-the-art action recognition algorithms, SSTNTL gives convincing performance for both human and gesture action recognition.

Generating a lexicon without a language model: Do words for number count?

Homesigns are communication systems created by deaf individuals without access to conventional linguistic input. To investigate how homesign gestures for number function in short-term memory compared to homesign gestures for objects, actions, or attributes, we conducted memory span tasks with adult homesigners in Nicaragua, and with comparison groups of unschooled hearing Spanish speakers and deaf Nicaraguan Sign Language signers. There was no difference between groups in recall of gestures or words for objects, actions or attributes; homesign gestures therefore can function as word units in short-term memory. However, homesigners showed poorer recall of numbers than the other groups. Unlike the other groups, increasing the numerical value of the to-be-remembered quantities negatively affected recall in homesigners, but not controls. When developed without linguistic input, gestures for number do not seem to function as summaries of the cardinal values of the sets (four), but rather as indexes of items within a set (one-one-one-one).

Interação mãe-bebê com deficiência visual: estilos comunicativos e episódios interativos

Este estudo objetivou analisar a interação mãe-bebê com deficiência visual, por meio dos estilos de fala materna, dos comportamentos não verbais maternos e infantis e dos episódios interativos. Participaram da pesquisa três díades, sendo os bebês deficientes visuais, na faixa etária dos seis aos trezes meses de vida. Foi utilizada uma câmera de vídeo para apreender as interações entre as díades, em uma situação de brinquedo livre. Para cada díade foram realizadas três filmagens, cada uma com duração de vinte minutos. Os resultados sugerem que as interações entre as mães e os bebês deficientes visuais acontecem, principalmente, através da coordenação da fala com a ação gestual, na qual o toque e os movimentos corporais são utilizados com a função de facilitar a comunicação. Estudos que investigam a interação mãe-bebê com desenvolvimento atípico são relevantes, pois respondem a questões teóricas sobre o desenvolvimento e possibilitam estruturar intervenções.

Thinking with your hands: speech–gesture activity during an L2 awareness-raising task

This article reports on a study of second language (L2) French learners’ self-generated use of gesture to think through and resolve a metalinguistic awareness-raising task during small-group work with an expert mediator. Although the use of gesture in L2 communication and pedagogy has recently received increasing attention, little research has explored L2 learners' use of gesture to mediate their own thinking in order to work through L2 problems. The present study responds to Lantolf's call for research investigating learners’ use of gestures to mediate their own learning. We present three excerpts of a video-recorded small-group awareness-raising task dealing with stylistic variation in French. We argue that the learners use gesture to mediate their thinking as they learn the concept of stylistic variation and develop an understanding of various form–meaning relationships. The focus on speech–gesture activity provides an enhanced view of these learners’ thinking processes that would not have been observable if speech alone had been considered.

Getting the point: The role of gesture in managing intersubjectivity in a design activity

AbstractThis paper illustrates the complexity of pointing as it is employed in a design workshop. Using the method of interaction analysis, we argue that pointing is not merely employed to index, locate, or fix reference to an object. It also constitutes a practice for reestablishing intersubjectivity and solving interactional trouble such as misunderstandings or disagreements by virtue of enlisting something as part of the participants’ shared experience. We use this analysis to discuss implications for how such practices might be supported with computer mediation, arguing for a “bricolage” approach to systems development that emphasizes the provision of resources for users to collaboratively negotiate the accomplishment of intersubjectivity rather than systems that try to support pointing as a specific gestural action.

Clinical classification of psychogenic non-epileptic seizures based on video-EEG analysis and automatic clustering

BackgroundPsychogenic non-epileptic seizures (PNES) or attacks consist of paroxysmal behavioural changes that resemble an epileptic seizure but are not associated with electrophysiological epileptic changes. They are caused by a psychopathological...

Visual recognition of 3D emblematic gestures in an HMM framework

In human-machine interaction, gestures play an important role as input modality for natural and intuitive interfaces. While the usage of sign languages or crafted command gestures typically requires special user training, the class of gestural actions called “emblems” represents more intuitive yet expressive signs that seem well suited for the task. Following this, an approach for the visual recognition of 3D emblematic arm gestures in a realistic smart room scenario is presented. Hand and head positions are extracted in multiple unsynchronized monocular camera streams, combined to spatiotemporal 3D gesture trajectories and classified in an Hidden Markov Model (HMM) classification and detection framework. The contributions within this article are threefold: Firstly, a solution for the 3D combination of trajectories obtained from unsynchronized cameras and with varying frame rates is proposed. Secondly, the suitability of different alternative feature representations derived from a hand trajectory is assessed, and it is shown that intuitive gestures can be represented by projection on their principal plane of motion. Thirdly, it is demonstrated that a rejection model for gesture spotting and segmentation can be constructed using out-of-domain data. The approach is evaluated on a challenging realistic data set.

High contextual sensitivity of metaphorical expressions and gesture blending: A video event-related potential design

Human communication in a natural context implies the dynamic coordination of contextual clues, paralinguistic information and literal as well as figurative language use. In the present study we constructed a paradigm with four types of video clips: literal and metaphorical expressions accompanied by congruent and incongruent gesture actions. Participants were instructed to classify the gesture accompanying the expression as congruent or incongruent by pressing two different keys while electrophysiological activity was being recorded. We compared behavioral measures and event related potential (ERP) differences triggered by the gesture stroke onset. Accuracy data showed that incongruent metaphorical expressions were more difficult to classify. Reaction times were modulated by incongruent gestures, by metaphorical expressions and by a gesture–expression interaction. No behavioral differences were found between the literal and metaphorical expressions when the gesture was congruent. N400-like and LPC-like (late positive complex) components from metaphorical expressions produced greater negativity. The N400-like modulation of metaphorical expressions showed a greater difference between congruent and incongruent categories over the left anterior region, compared with the literal expressions. More importantly, the literal congruent as well as the metaphorical congruent categories did not show any difference. Accuracy, reaction times and ERPs provide convergent support for a greater contextual sensitivity of the metaphorical expressions.

Evidence for node and scope of negation in coverbal gesture

Negative structures are a characteristic of all human languages. One such structure is ‘node’ and ‘scope’ of negation. In an utterance, the node is the location of a negative form, and the scope is the stretch of language to which the negation applies. In this paper, I examine a gesture that English speakers perform when they negate and show how speakers organize the different phases of gestural action in relation to the negative structures in speech. In this gesture, speakers first bring one hand across their body (preparation phase). Then, with the palm turned down, they move their hand rapidly along the horizontal axis (stroke phase). Often, they hold their hand in space for a short period after the stroke (post-stroke hold phase), before returning it to rest (recovery phase). In several negative utterances, drawn from a corpus of audiovisual recordings of conversations in everyday settings, speakers prepared this gesture in advance of the node of negation, synchronized the stroke of the gesture with the node, and performed a post-stroke hold throughout the scope. I suggest that the grammatical concepts of node and scope can also account for the way speakers gesture when they negate. This study refines understanding of how gesture phrase structure functions and suggests a multimodal view of grammatical phenomena.

Development of Gesture-Changeable under-actuated Humanoid Robotic Finger

Robotic fingers, which are the key parts of robot hand, are divided into two main kinds: dexterous fingers and fingers. Although dexterous fingers are agile, they are too expensive. Under-actuated fingers can grasp objects self-adaptively, which makes them easy to control and low cost, on the contrary, function makes fingers feel hard to grasp things agilely enough and make many gestures. For the purpose of designing a new finger which can grasp things dexterously, perform many gestures and feel easy to control and maintain, a concept called gesture-changeable under-actuated (GCUA) function is put forward. The GCUA function combines the advantages of dexterous fingers and fingers: a pre-bending function is embedded into the finger. The GCUA finger can not only perform self-adaptive grasping function, but also actively bend the middle joint of the finger. On the basis of the concept, a GCUA finger with 2 joints is designed, which is realized by the coordination of screw-nut transmission mechanism, flexible drawstring constraint and pulley-belt mechanism. Principle analyses of its grasping and the design optimization of the GCUA finger are given. An important problem of how to stably grasp an object which is easy to glide is discussed. The force analysis on gliding object in grasping process is introduced in detail. A GCUA finger with 3 joints is developed. Many experiments of grasping different objects by of the finger were carried out. The experimental results show that the GCUA finger can effectively realize functions of pre-bending and self-adaptive grasping, the grasping processes are stable. The GCUA finger excels fingers in dexterity and gesture actions and it is easier to control and cheaper than dexterous hands, becomes the third kinds of finger.