Representational Gestures Research Articles

Feature covariance matrix-based dynamic hand gesture recognition

Over the past 2 decades, vision-based dynamic hand gesture recognition (HGR) has made significant progresses and been widely adopted in many practical applications. Although the advent of RGB-D cameras and deep learning-based methods provides more feasible solutions for HGR, it is still very challenging to satisfy the requirements of both high efficiency and accuracy for real-world HGR systems. In this paper, we propose a novel method using the feature covariance matrix for effective and efficient dynamic HGR. We extract a set of local feature vectors that represent local motion patterns to construct the feature covariance matrix efficiently, which also provides a compact representation of a dynamic hand gesture. By tracking hand keypoints in three successive frames and calculating their motion features, our method can be extended to both 2D dynamic HGR and 3D dynamic HGR. To evaluate the effectiveness of the proposed framework, we perform extensive experiments on three publicly available datasets (one 2D dataset and two 3D datasets). The experimental results demonstrate the effectiveness of our proposed method.

Bringing the Nonlinearity of the Movement System to Gestural Theories of Language Use: Multifractal Structure of Spoken English Supports the Compensation for Coarticulation in Human Speech Perception.

Coarticulation is the tendency for speech vocalization and articulation even at the phonemic level to change with context, and compensation for coarticulation (CfC) reflects the striking human ability to perceive phonemic stability despite this variability. A current controversy centers on whether CfC depends on contrast between formants of a speech-signal spectrogram—specifically, contrast between offset formants concluding context stimuli and onset formants opening the target sound—or on speech-sound variability specific to the coordinative movement of speech articulators (e.g., vocal folds, postural muscles, lips, tongues). This manuscript aims to encode that coordinative-movement context in terms of speech-signal multifractal structure and to determine whether speech's multifractal structure might explain the crucial gestural support for any proposed spectral contrast. We asked human participants to categorize individual target stimuli drawn from an 11-step [ga]-to-[da] continuum as either phonemes “GA” or “DA.” Three groups each heard a specific-type context stimulus preceding target stimuli: either real-speech [al] or [a], sine-wave tones at the third-formant offset frequency of either [al] or [aɹ], and either simulated-speech contexts [al] or [aɹ]. Here, simulating speech contexts involved randomizing the sequence of relatively homogeneous pitch periods within vowel-sound [a] of each [al] and [aɹ]. Crucially, simulated-speech contexts had the same offset and extremely similar vowel formants as and, to additional naïve participants, sounded identical to real-speech contexts. However, randomization distorted original speech-context multifractality, and effects of spectral contrast following speech only appeared after regression modeling of trial-by-trial “GA” judgments controlled for context-stimulus multifractality. Furthermore, simulated-speech contexts elicited faster responses (like tone contexts do) and weakened known biases in CfC, suggesting that spectral contrast depends on the nonlinear interactions across multiple scales that articulatory gestures express through the speech signal. Traditional mouse-tracking behaviors measured as participants moved their computer-mouse cursor to register their “GA”-or-“DA” decisions with mouse-clicks suggest that listening to speech leads the movement system to resonate with the multifractality of context stimuli. We interpret these results as shedding light on a new multifractal terrain upon which to found a better understanding in which movement systems play an important role in shaping how speech perception makes use of acoustic information.

Communicative intent modulates production and comprehension of actions and gestures: A Kinect study

Actions may be used to directly act on the world around us, or as a means of communication. Effective communication requires the addressee to recognize the act as being communicative. Humans are sensitive to ostensive communicative cues, such as direct eye gaze (Csibra & Gergely, 2009). However, there may be additional cues present in the action or gesture itself. Here we investigate features that characterize the initiation of a communicative interaction in both production and comprehension.We asked 40 participants to perform 31 pairs of object-directed actions and representational gestures in more- or less- communicative contexts. Data were collected using motion capture technology for kinematics and video recording for eye-gaze. With these data, we focused on two issues. First, if and how actions and gestures are systematically modulated when performed in a communicative context. Second, if observers exploit such kinematic information to classify an act as communicative.Our study showed that during production the communicative context modulates space–time dimensions of kinematics and elicits an increase in addressee-directed eye-gaze. Naïve participants detected communicative intent in actions and gestures preferentially using eye-gaze information, only utilizing kinematic information when eye-gaze was unavailable.Our study highlights the general communicative modulation of action and gesture kinematics during production but also shows that addressees only exploit this modulation to recognize communicative intention in the absence of eye-gaze. We discuss these findings in terms of distinctive but potentially overlapping functions of addressee directed eye-gaze and kinematic modulations within the wider context of human communication and learning.

Simple mappings, expressive movement: a qualitative investigation into the end-user mapping design of experienced mid-air musicians

ABSTRACTIn a New Interface for Musical Expression (NIME), the design of the relationship between a musician’s actions and the instrument’s sound response is critical in creating instruments that facilitate expressive music performance. A growing body of NIMEs expose this design task to the end performer themselves, leading to the possibility of new insights into NIME mapping design: what can be learned from the mapping design strategies of practicing musicians? This research contributes a qualitative study of four highly experienced users of an end-user mapping instrument to examine their mapping practice. The study reveals that the musicians focus on designing simple, robust mappings that minimize errors, embellishing these control gestures with theatrical ancillary gestures that express metaphors. However, musical expression is hindered by the unintentional triggering of musical events. From these findings, a series of heuristics are presented that can be applied in the future development of NIMEs.

Symbolic play promotes non‐verbal communicative exchange in infant–caregiver dyads

Symbolic play has long been considered a fertile context for communicative development (Bruner, 1983, Child's talk: Learning to use language, Oxford University Press, Oxford; Vygotsky, 1962, Thought and language, MIT Press, Cambridge, MA; Vygotsky, 1978, Mind in society: The development of higher psychological processes. Harvard University Press, Cambridge, MA). In the current study, we examined caregiver-infant interaction during symbolic play and compared it to interaction in a comparable but non-symbolic context (i.e., 'functional' play). Fifty-four (N=54) caregivers and their 18-month-old infants were observed engaging in 20min of play (symbolic, functional). Play interactions were coded and compared across play conditions for joint attention (JA) and gesture use. Compared with functional play, symbolic play was characterized by greater frequency and duration of JA and greater gesture use, particularly the use of iconic gestures with an object in hand. The results suggest that symbolic play provides a rich context for the exchange and negotiation of meaning, and thus may contribute to the development of important skills underlying communicative development. Statement of contribution Current knowledge Symbolic play has long been linked to communicative development. We do not know the extent to which the social context of symbolic play fosters communicative skills. Present study We test whether key communicative acts (joint attention, gesture) are more common in infant-caregiver dyads during symbolic versus functional play. Joint attention was established more often and sustained for longer periods in symbolic play than in functional play. Infants and caregivers produced more in-hand representational gestures in symbolic compared to functional play.

Vernacular Soliloquy, Theatrical Gesture, and Embodied Consciousness in The Marrow of Tradition

Vernacular Soliloquy, Theatrical Gesture, and Embodied Consciousness in The Marrow of Tradition

Gestural depiction of motion events in narrative increases symbolic distance with age

AbstractWe examined gesture representation of motion events in narratives produced by three- and nine-year-olds, and adults. Two aspects of gestural depiction were analysed: how protagonists were depicted, and how gesture space was used. We found that older groups were more likely to express protagonists as an object that a gesturing hand held and manipulated, and less likely to express protagonists with whole-body enactment gestures. Furthermore, for older groups, gesture space increasingly became less similar to narrated space. The older groups were less likely to use large gestures or gestures in the periphery of the gesture space to represent movements that were large relative to a protagonist’s body or that took place next to a protagonist. They were also less likely to produce gestures on a physical surface (e.g. table) to represent movement on a surface in narrated events. The development of gestural depiction indicates that older speakers become less immersed in the story world and start to control and manipulate story representation from an outside perspective in a bounded and stage-like gesture space. We discuss this developmental shift in terms of increasingsymbolic distancing(Werner & Kaplan, 1963).

Put your hands up! Gesturing improves preschoolers’ executive function

This study addressed the causal direction of a previously reported relation between preschoolers’ gesturing and their executive functioning on the Dimensional Change Card Sort (DCCS) sorting–switch task. Gesturing the relevant dimension for sorting was induced in a Gesture group through instructions, imitation, and prompts. In contrast, the Control group was instructed to “think hard” when sorting. Preschoolers (N = 50) performed two DCCS tasks: (a) sort by size and then spatial orientation of two objects and (b) sort by shape and then proximity of the two objects. An examination of performance over trials permitted a fine-grained depiction of patterns of younger and older children in the Gesture and Control conditions. After the relevant dimension was switched, the Gesture group had more accurate sorts than the Control group, particularly among younger children on the second task. Moreover, the amount of gesturing predicted the number of correct sorts among younger children on the second task. The overall association between gesturing and sorting was not reflected at the level of individual trials, perhaps indicating covert gestural representation on some trials or the triggering of a relevant verbal representation by the gesturing. The delayed benefit of gesturing, until the second task, in the younger children may indicate a utilization deficiency. Results are discussed in terms of theories of gesturing and thought. The findings open up a new avenue of research and theorizing about the possible role of gesturing in emerging executive function.

Effects of syllable onset on the timing of pitch accent in Belgrade Serbian

In this paper, I present the results of an acoustic study on Serbian, a pitch-accent language with sonorant-sonorant onset clusters like /mr/ and /ml/. I show that peak timing in falling accents is not affected solely by syllable onset duration, as suggested by the segmental anchoring hypothesis, but rather is determined by an interaction between syllable onset complexity and syllable onset duration, indicating a gestural representation of tone.

Abstraction Through Representation: Interview with Kevin Jerome Everson

Kevin Jerome Everson has made more than 131 shorts and nine features—so far—exploring the intersections of work and geography among African Americans and other people of African descent. Often mining events from his own life, local history, and vignettes from his actors’ lives, Everson scripts ordinary behavior and dialogue into theatrical gestures. In reediting or restaging personal stories or archival footage, the prosaic becomes specific and distinct. Everson looks for “frames that connect the necessity and the coincidence,” where “necessity” is the plot or character driving the film, “coincidence” the scenes that seem accidental or are lifted from found footage. Ratner engages Everson on his oblique approach that prizes his subjects’ dignity over his viewers’ comfort. At a time when weightless spectacle dominates imagery, Everson responds to her questions and to his craft with nuanced gravity.

Knots, Music and DNA

Musical gestures connect the symbolic layer of the score to the physical layer of sound. I focus here on the mathematical theory of musical gestures, and I propose its generalization to include braids and knots. In this way, it is possible to extend the formalism to cover more case studies, especially regarding conducting gestures. Moreover, recent developments involving comparisons and similarities between gestures of orchestral musicians can be contextualized in the frame of braided monoidal categories. Because knots and braids can be applied to both music and biology (they apply to knotted proteins, for example), I end the article with a new musical rendition of DNA.

The Sacred and the Secular in Post-Tridentine Church Music: De Rore, Lasso, and the Magnificat Da le belle contrade

Although scholars have often claimed that the Council of Trent called for the elimination of secular elements in church music, the official decree issued by that body in 1562 actually called only for the exclusion of music incorporating anything 'lascivum aut impurum'. The precise meanings of these terms and others, such as 'profanum', 'vanum', and 'turpis', which were used to describe prohibited music in the wake of Trent, nonetheless remain unclear, for church authorities never used them in reference to specific compositions. With respect to the many masses and Magnificats that composers based on secular compositions in the decades following the Council, this study proposes two axioms: first, such pieces fulfilled the expectations of the institutions that performed them; second, composers, in their choice of particular model compositions and in their treatment of those models, articulated the period's understanding of appropriate church music, using secular elements amenable to reinterpretation within the sacred realm and rejecting elements that were wanton, impure, and otherwise inimical to the sacred. In the Magnificat Da le belle contrade, for example, Lasso borrows from De Rore's madrigal in a severely circumscribed fashion, using only the music associated with the opening description of the morning star-a symbol of the Virgin-and a handful of passages whose texts reflect the meaning of specific words in the Magnificat text. Entirely absent are the eccentric and theatrical gestures of the madrigal's central lament, gestures at odds with liturgical music of the post-Tridentine Church as Lasso conceived it.

The relationship between handedness and valence: A gesture study.

People with different hand preferences assign positive and negative emotions to different sides of their bodies and produce co-speech gestures with their dominant hand when the content is positive. In this study, we investigated this side preference by handedness in both gesture comprehension and production. Participants watched faceless gesture videos with negative and positive content on eye tracker and were asked to retell the stories after each video. Results indicated no difference in looking preferences regarding being right- or left-handed. Yet, an effect of emotional valence was observed. Participants spent more time looking to the right (actor's left) when the information was positive and to the left (actor's right) when the information was negative. Participants' retelling of stories revealed a handedness effect only for different types of gestures (representational vs beat). Individuals used their dominant hands for beat gestures. For representational gestures, while the right-handers used their right hands more, the left-handers gestured using both hands equally. Overall, the lack of significant difference between handedness and emotional content in both comprehension and production levels suggests that body-specific mental representations may not extend to the conversational level.

The Relationship between Phoneme Production and Perception in Speech-Impaired and Typically-Developing Children

One of the central questions that Eric Lenneberg raised in his seminal book, Biological Foundations of Language is: What is the relationship between language comprehension and language production? This paper reviews Lenneberg’s case study of a child with congenital anarthria and then presents the results of two studies that investigate the relationship between phoneme perception and production. The first study investigates the phoneme identification skills of a child with developmental apraxia who, like the anarthric child studied by Lenneberg, had essentially no speech yet had no difficulty understanding speech. The second study investigates the extent to which 28 typically-developing children’s ability to identify phonemes is related to their ability to produce phonemes. The results of both studies support Lenneberg’s conclusion that children’s ability to perceive speech is not dependent on their ability to produce speech. Thus, Lenneberg’s original case study and the two studies presented in this paper argue against gestural theories of speech perception such as the Motor Theory.

Valcer kao jedan od prostora groteske u delima Alfreda Šnitkea

Humour, as one of the most vital phenomena of human nature and culture, has maintained the status of a special category, which evades being defined accurately. Esti Sheinberg thinks that a prerequisite for experiencing a musical piece as comic is the existence of certain incongruity among its meaningful units. She considers musical irony, satire, parody and the grotesque as correlatives of semantic ambiguities, which use two or more layers of meaning. She emphasises irony as a structural prototype for all other modes of ambiguity. Regarding irony as a phenomenon based on mutually excluding components of semantic opposites, Sheinberg makes a distinction in relation to the grotesque, which originates as a result of juxtaposition of aesthetic units, when contradictory semantic pairs are formed. The irresolvable ambiguity of the grotesque is the result of acceptance, not negation of all aspects of the representing subject. The hyperbolical nature of the grotesque in music is manifested through heightening the extreme within the framework of given parameters, while the exaggerations are noted as such in relation to the standard perceived as a human body and voice. The Dance of Death brings to the fore one of the most significant features of the grotesque - the acknowledgement and accumulation of contradiction, with no tendency to their conciliation. The selected compositions for this study are the examples of macabre waltzes in Alfred Schnittke's opus. After their analysis, a certain level of incompatibility among the meaningful elements of the piece has been located and determined, as well as the relationship between the form and content, and the level of deviation from the set norm. Taking into account the importance of the relationship between the body and movement in general and the grotesque, the significant part of the paper is devoted to Hatten's theory of musical gesture.

Sign-creation in the Seattle DeafBlind community

Abstract This article examines the social and interactional foundations of sign-creation among DeafBlind people in Seattle, Washington. Linguists studying signed languages have proposed models of sign-creation that involve the selection of an iconic gestural representation of the referent which is subjected to grammatical constraints and is thereby incorporated into the linguistic system. Drawing on 18 months of ethnographic fieldwork and more than 190 hours of video recordings of interaction and language use, I argue that a key interactional mechanism driving processes of sign-creation among DeafBlind people in Seattle is deictic integration. Deictic integration restricts the range of contextual values that the grammar can retrieve by coordinating systems of reference with patterns in activity. This process brings language into alignment with the world as it is perceived by the users of that language, making a range of potentially iconic relations available for selection in the creation of new signs.

Conflict between gesture representations extinguishes μ rhythm desynchronization during manipulable object perception: An EEG study

Recent findings showed that competition between object structural (“grasp-to-move”) and functional (“grasp-to-use”) gestures slows down the initiation of object-directed actions but also object visual processing. The present study investigates the neurophysiological correlates of the competition between gesture representations during object perception. 3D conflictual objects (distinct structural and functional gestures) and non-conflictual objects (similar structural and functional gestures) were presented in three spaces (peripersonal, boundary of peripersonal, and extrapersonal) in a virtual environment. Participants performed reach-to-grasp and semantic judgments on objects while EEG was recorded. Results revealed that the conflict between evoked gestures impacts 8–12 Hz desynchronization at both central (μ rhythm) and posterior (α rhythm) sites. Critically, μ rhythm desynchronization was suppressed when conflictual objects were presented in peripersonal space. Findings indicate that μ rhythm desynchronization is reduced by the competition between evoked gestures and suggest that neural motor resonance may also reflect action selection processes during object perception.

Imagens para não deixar de ver o gesto inumano

Seguindo a pegada deixada por Didi-Huberman e colocando em destaque a questão política da produção e reprodução das imagens, este ensaio pretende pensar as disputas tecnopolíticas por recurso a imagens. Nesse sentido, duas imagens específicas emergem no contexto das experiências transversais de formação acadêmica nas universidades brasileiras, imagens que contam com a presença de mestres e mestras de tradições orais como sujeitos capazes de legitimar modos outros de produção de saber e de fazer estético-políticas. A partir daí, coloca-se em movimento uma discussão que compreende a materialidade dos aparatos tecnológicos (via Nietzsche, Marx e Kittler), a teoria geral dos gestos em Flusser, o conceito de dispositivo para Agamben e as relações entre estética e política, segundo Benjamin e Rancière. Se parece evidente, de partida, que não se passa — — que não passaremos jamais — pelas tecnologias impunemente, o que (nos) resta como possibilidade de resistência e de potência de vida para o combate frente ao avanço aniquilador da falsa política? Pouco, talvez nada, talvez um quase nada, apenas, talvez, e ainda, imagens.

What gestures reveal about cognitive deficits in Williams Syndrome

ABSTRACTGestures have a central role in speaking and thinking about spatial information. The goal of the present study is to examine the function of gestures in Williams Syndrome (WS), a genetic disorder characterized by spatial impairments and a preservation of communication. The study’s subjects were 11 WS individuals and 22 typically-developing controls who performed a narrative task. We analyzed offline gesture production and its relation with language and spatial information. Compared to the control groups, WS individuals produced more representational gestures that anticipated, supplemented, or gesture-only communication. Gestures produced by WS participants serve a compensatory role particularly in representing spatial contents.

Representational Gestures and Tense-Aspect Improvement in L2 Grammar

The English tense-aspect system is one of the most prominent and challenging areas of English language grammar for both EFL teachers and learners. The hardships in teaching and learning this system have led many researchers to look for and come up with new approaches and models for tackling them. Likewise, this study proposes representational gestures, which utilize a bunch of movements of the hands as conceptual metaphors related to time to represent the tenses and aspects of English language, as a facilitating tool for both EFL learners and teachers. Representational gestures have been devised to make up for the shortcomings of timelines. Two groups of learners participated in this study which took part in ten 45- minute intensive grammar sessions of instruction: the first group was taught the tenses and aspects through RGs. The second group, however, underwent the conventional instruction (timelines) to learn the same grammatical points. An analysis through 2- Proportion Test revealed that the first group outperformed the second regarding tense/aspect use in a number of ways.