Representational Gestures Research Articles

Hapstick-Figure: Investigating the Design of a Haptic Representation of Human Gestures from Theater Performances for Blind and Visually-Impaired People

Theaters serve as platforms that transport audiences into diverse worlds, offering a collective enjoyment of live performances and a shared cultural experience. However, theater performances have strong visual components, such as physical props and actors’ movements and gestures, which are inaccessible to visually impaired and blind audience members and thus can exclude them from such shared social experiences. We conducted formative interviews with eight blind and visually impaired people about their experiences with barriers to theater performance gestures. We then present Hapstick-Figure, a prototype design to represent and communicate human gestures via a 3D-printed tactile surface. Next, we used Hapstick-Figure as a technology probe in a qualitative evaluation with six of our BVI participants to explore non-visual interpretation and engagement with this prototype. We outline insights into the haptic representation of theater performance gestures and reflections on designing for accessibility in this context.

Research on Mixed Reality Hand Interaction Technology in Flight Simulation Teaching

Abstract The integration of mixed reality simulation training in institutional education and teaching processes can yield significant quality benefits. Non-wearable interactive methods represent a more natural form of hand interaction within flight cockpits, covering various emerging fields such as mixed reality, computer vision, and human-computer interaction. This paper is guided by a “human-centered” approach to natural human-computer interaction and proposes a virtual hand mixed reality interaction scheme based on hand natural feature points. This study introduces a hand detection, segmentation, and recognition algorithm based on skin colour and key hand feature points. Initially, the algorithm uses the Otsu adaptive threshold algorithm in the YCbCr space for skin colour detection to accommodate varying image brightness levels. Subsequently, it employs a hand Keypoint model for rapid hand detection within skin-like areas while excluding interference from other regions. During the process of hand tracking, optimization prediction is performed using the particle filter algorithm combined with the artificial fish swarm algorithm to achieve tracking accuracy within six pixels in both horizontal and vertical directions. Gesture recognition involves extracting Fourier descriptive contour features of the hand bone structure through identification of feature points. Furthermore, this study combines the optimization capabilities of the artificial fish swarm algorithm to enhance support vector machine model parameter optimization for improved recognition rates. Testing involved 800 processed images as part of a test set for gesture recognition; only five commonly used gestures within flight cockpits were recognized and classified with an accuracy rate reaching up to 98%. Finally, this research presents virtual representations of common operational gestures including natural hand states, control stick manipulation, and throttle handling positions, button activations, and rotational movements via head-mounted displays or screens. The proposed mixed reality-based hand gesture interaction system addresses challenges related to complex background interference during hand detection in cockpit environments as well as issues arising from lighting disturbances. Additionally it resolves problems associated with insufficient particles leading to tracking failures while significantly enhancing registration effects during tracking processes. Moreover it mitigates issues related to Fourier descriptive features being influenced by background variations or changes in pose along with limitations pertaining to expressing singular forms of gestures thereby improving overall classification accuracy.

A study of gestures representing musical emotions

The creation of music has originated from the burst of inspiration of the composer. Many studies suggested that improvisation plays a more vital role than composition in the emergence of new forms of music. The production of music, especially in Western cultures, tends to follow its traditional music theory. With the emergence of Pythagorean music theory, Western musicology has been defined and developed in depth. The composer’s creation is based on music creation within the framework of the Western music theory. However, in recent years, with the rise of the topic of music gestures, some scholars have learned that music ontology as a four-dimensional hypercube is inseparable from the performer’s gesture. Therefore, some musicians also attempt to create flexible music from the perspective of music performers and based on performance gestures. This kind of music creation breaks the traditional music theory system and has more personal style characteristics, but few people can understand the thinking logic behind this kind of creative behavior. Therefore, this article focuses on the singing gesture and reversely analyzes its impact on musical creation. We use a recent mathematic definition of gestures and we discuss emotion from its etymological origin (from the Latin e-movere) as a gestural movement from inside out. This article conducts an in-depth analysis based on the singer’s gesture for the following two reasons: First, the literature on the connection between singing gesture theory and composition theory remains scarce. Second, gestures produced by a singer are the only performance form that is not enabled by other instruments.

Details in hand: how does gesturing relate to autobiographical memory?

ABSTRACT Gestures are an integral and inseparable component of speech and people frequently use gestures when retelling their autobiographical memories. This study investigates whether gestures are associated with the retrieval of episodically and phenomenologically rich memories and how this association changes during development. Thirty-five children and 46 adults were asked to report autobiographical memories. Then, they rated the recalled memories on phenomenological qualities. Episodic and non-episodic details of autobiographical memories and representational gestures produced during memory narration were coded. The use of representational gestures was positively correlated with the episodic details of adult memories; however, the same correlation was not present in child memories. The representational gesture use was not associated with the phenomenological qualities in both groups. Gesture use may be related to the retrieval of autobiographical memories, particularly in adults capable of reporting long, coherent memories.

Self-focused versus dialogic features of gesturing during simultaneous interpreting

The present study considers an implicit debate in the field of gesture studies as to whether gestures are produced primarily for the speaker or for the addressee. It considers the unique monologic setting of simultaneous interpreters working in a booth in which there is no visible audience present and where they only hear and do not see the speaker whose words they are interpreting. The hypotheses (H) are that the interpreters might produce more representational gestures, to aid in their own idea formulation (H1), and self-adapter movements, to maintain their self-focus (H2), rather than pragmatic gestures, which are known to serve interactive functions. Forty-nine interpreters were videorecorded as they interpreted two portions of popular science lectures, one from either English or German (their L2) into Russian (their L1) and one from Russian into their respective L2. The results showed that a vast majority of the gestures produced were either pragmatic in function or self adapters. H2 was thus supported, but H1 was not. The frequent use of pragmatic gestures is interpreted in terms of the internalized dialogic nature of talk and gesturing itself. Both beat gestures expressing emphasis and reduced forms of presentation gestures can facilitate the interpreters’ speaking by prompting the presentation and emphasis of ideas. Though focused on their own process of speech production, simultaneous interpreters may embody elements of the lecturer of the source text engaging with the audience, blended with their own dialogic speaking behaviors, aspects of which we may see in their gesturing.

Student visual attention during group instruction phases in collaborative geometry problem solving

AbstractTeacher gestures support mathematics learning and promote student collaboration. Aligned with speech, gestures can help students to notice the important visual information of geometry tasks. However, students’ visual attention to the teacher’s gestural cues during collaborative problem solving remains a largely unexplored field in mathematics education research. This mixed-method case study investigated relations between students’ visual attention, teacher gestures, and students’ collaborative problem-solving process on a geometry task. The data were collected with video cameras and mobile gaze trackers on four students simultaneously in two Finnish 9th-grade mathematics lessons with the same teacher. The findings show that the students attended to their own papers most of the time during the teacher's gestures, but differences in student attention between the gesture types emerged. The qualitative analysis showed that the teacher’s tracing, pointing, and representational gestures helped in directing student attention to targets relevant to the situational learning process. We conclude that teacher gestures can both convey mathematical contents and direct student attention, which intermediates mathematical thinking in problem solving.

Double Turning and Other Duplicities. A Performative Reading of John 20:11–18

Abstract In John 20:11–18, Mary Magdalene meets Jesus after his death. She turns around twice, a double gesture that has puzzled New Testament scholars. In this article, I offer a performative reading of Mary Magdalene’s turns based on Judith Butler’s theory of gesture and the literary inventory of ancient recognition scenes. I argue that the double gesture does not emphasize the difference between a physical and an inner status of recognition. Instead, it is conceived as a non-identical repetition or quotation. It points to other turnings and other duplicities. Both turns are part of a performative process that unfolds the new identities of the main characters after their separation. Mary is not portrayed as a misunderstanding disciple who needs two turns to recognize Jesus, but as part of a reciprocal process that mirrors Jesus’ double appearance and the text’s double layers of meaning.

Do bilingual adults gesture when they are disfluent?: Understanding gesture-speech interaction across first and second languages

ABSTRACT People are more disfluent in their second language (L2) than their first language (L1). Gesturing facilitates cognitive processes, including speech production. This study investigates speech disfluency and representational gesture production across Turkish-English bilinguals’ L1 (Turkish) and L2 (English) through a narrative retelling task (N = 27). Results showed that people were more disfluent and used more representational gestures in English. Controlling for L2 proficiency, people were still more disfluent in English. The more people were proficient in L2, the more they used gestures in that language. Similarly, disfluency-gesture co-occurrences were more common in English. L2 proficiency was positively correlated with the likelihood of a disfluency being accompanied by a gesture. These findings suggest that gestures may not necessarily compensate for weak language skills. Rather, people might gesture during disfluent moments if they can detect their errors, suggesting a close link between representational gestures and language competency in benefiting from gestures when disfluent.

DigCode—A generic mid-air gesture coding method on human-computer interaction

With high flexibility and rich semantic expressiveness, mid-air gesture interaction is an important part of natural human-computer interaction (HCI) and has broad application prospects. However, there is no unified representation frame for designing, recording, investigating and comparing HCI mid-air gestures. Therefore, this paper proposes an interpretable coding method, DigCode, for HCI mid-air gestures. DigCode converts the unstructured continuous actions into structured discrete string encoding. From the perspective of human cognition and expression, the research employed psychophysical methods to divide gesture actions into discrete intervals, defined the coding rules of representation in letters and numbers, and developed automated programs to enable encoding and decoding by using gesture sensors. The coding method can cover the existing representations of HCI mid-air gestures by considering human understanding and computer recognition and can be applied to HCI mid-air gesture design and gesture library construction.

Individual differences in representational gesture production are associated with cognitive and empathy skills.

Substantial individual variation exists in the frequency of gestures produced while speaking. This study investigated the associations between cognitive abilities, empathy levels, and personality traits with the frequency of representational gestures. A cartoon narration task and a social dilemma solving task were used to elicit gestures. Predictor variables were selected based on prior research on individual differences in gesture production and the cognitive and communicative functions of gestures in speech. Our findings revealed that an increased frequency of representational gestures was associated with higher empathy levels in the cartoon narration task. However, in the social dilemma solving task, a higher frequency of representational gestures was associated with lower visuospatial working memory, spatial transformation, and inhibition control abilities. Moreover, no significant relationships were found between verbal working memory, personality traits, and the frequency of representational gestures in either task. These findings suggested that predictor variables for representational gesture production vary depending on the nature of the gesture elicitation task (e.g., spatiomotoric vs. abstract topics). Future research should examine the relationship between individuals' cognitive abilities, empathy and gesture production with across a broader range of topics and in more ecologically valid contexts.

Lexical skills and gesture use: A comparison between expressive and receptive/expressive late talkers

BackgroundStudies on late talkers (LTs) highlighted their heterogeneity and the relevance of describing different communicative profiles. AimsTo examine lexical skills and gesture use in expressive (E-LTs) vs. receptive-expressive (R/E-LTs) LTs through a structured task. Methods and proceduresForty-six 30-month-old screened LTs were distinguished into E-LTs (n= 35) and R/E-LTs (n= 11) according to their receptive skills. Lexical skills and gesture use were assessed with a Picture Naming Game by coding answer accuracy (correct, incorrect, no response), modality of expression (spoken, spoken-gestural, gestural), type of gestures (deictic, representational), and spoken-gestural answers’ semantic relationship (complementary, equivalent, supplementary). Outcomes and resultsR/E-LTs showed lower scores than E-LTs for noun and predicate comprehension with fewer correct answers, and production with fewer correct and incorrect answers, and more no responses. R/E-LTs also exhibited lower scores in spoken answers, representational gestures, and equivalent spoken-gestural answers for noun production and in all spoken and gestural answers for predicate production. Conclusions and implicationsFindings highlighted more impaired receptive and expressive lexical skills and lower gesture use in R/E-LTs compared to E-LTs, underlying the relevance of assessing both lexical and gestural skills through a structured task, besides parental questionnaires and developmental scales, to describe LTs’ communicative profiles.

Phenomenology and cognitive linguistics in dialogue: A review of Ortega y Gasset's theory of emotive gesture as metaphor

AbstractThe present study pursues three objectives. First, to expose and discuss the contributions of the philosopher José Ortega y Gasset to the phenomenological study of gestures and emotive gesture. Secondly, to critically review one of the central theses defended by Ortega, according to which “every expressive phenomenon”—including, therefore, the emotive gesture—involves “a transposition, that is to say, an essential metaphor.” This thesis invites us, in the third objective, to establish a dialogue between phenomenology and cognitive linguistics (as developed by Lakoff, Langacker and Talmy), assuming an experientialist position vis‐à‐vis human reason and language, in opposition to the objectivist and formalist position of Chomsky's generative linguistics. In this interdisciplinary context, we will study the phenomenon of emotive gesture, in particular the expression of the emotion of anger, as analyzed phenomenologically by Ortega y Gasset.

Orchestration of talk and gestures for dialogic scaffolding: a case study of reading instruction / Orquestación del habla y los gestos para un andamiaje dialógico: un estudio de caso de la enseñanza de la lectura

Scholars agree that talk and scaffolding are critical to students’ learning in reading instruction. However, while many studies have focused on the role of talk, few have studied the use of gestures as scaffolding strategies. This paper answers the following questions: (i) How does the use of gestures complement a teacher’s construction of scaffolding strategies? (ii) How can the use of talk and gestures achieve dialogic scaffolding? This observation research focuses on a case study of a secondary teacher. Participants’ semiotic mediation of talk and gestures was examined closely using an analytical approach, informed by theories of scaffolding and gesture. The findings show that teacher gestures were used as learning support while students used gestures to visually represent and demonstrate their understanding. Through this interplay between talk and gesture, effective meaning-making and understanding were achieved in reading instruction.

Коартикуляция по палатальности в алтайском языке по данным УЗИ в динамическом аспекте

The paper explores the movement of the tongue dorsum in phonetic words of the Altai language. The data were collected using Articulate Instruments equipment and Articulate Assistant Advanced software, employing the ultrasound imaging technique. The focus was on several monosyllabic lexemes with a “plosive vowel nasal” structure. Every word was recorded three times in an isolated context. The audio recording, synchronized in time with the ultra-sound image, was segmented and annotated based on auditory and acoustic analysis in the Praat program. The statistical analysis was performed using the R programming language. The dorsum movement was represented as a series of static frames at 10 ms intervals. The time-mapping approach was used to calculate the coefficients capturing the dorsum uplift at different time intervals, with time dynamics described as a bundle of contours for each phonetic word. The point coefficient was determined based on the highest dorsum point, indicating the elevated section of the dorsum. The height coefficient was calculated as a vertical coordinate of the highest point. These coefficients were used to determine dorsum distance from the articulation center, identify vowel row and openness, and recognize basic and additional palatal or velar articulation in consonants. The study found that in soft-type words, the dorsum uplift was consistent throughout the word, resulting in palatalized consonants. In hard-type words with dorsal consonants, the dorsum raise was limited to the beginning of the following vowel and then lowered, indicating coarticulation movement. The paper describes this process in terms of structural phonology and gestural theory.

The importance of observing the master's hand: Action Observation Training promotes the acquisition of new musical skills.

Via mirror mechanism, motor training approaches based on the alternation of action observation and execution (i.e., Action Observation Training-AOT) promote the acquisition of motor abilities. Previous studies showed that both visual and auditory stimuli may elicit a common motor representation of music-related gestures; however, the potentialities of AOT for the acquisition of musical skills are still underexplored. Twenty-one music-naïve participants underwent two blocks of training: AOT and Key-light Observation Training (KOT). AOT consisted of the observation of a melodic sequence played on a keyboard with the right hand by an expert model, followed by participant's imitation. Observation and execution were repeated six consecutive times (T1-T6). KOT followed the same procedure, except for the visual content of the stimulus, depicting the sequential highlighting of the piano keys corresponding to the melody. The rate of correct notes (C), the trainee-model similarity of key-pressure strength (S), and the trainee-model consistency of note duration (R) were collected across T1-T6. Both AOT and KOT improved musical performance. Noteworthy, AOT showed a higher learning magnitude relative to KOT in terms of C and S. Action Observation Training promotes the acquisition of key elements of melodic sequences, encompassing not only the accurate sequencing of notes but also their expressive characteristics, such as key-pressure dynamics. The convergence of listening and observation of actions onto a shared motor representation not only explains several pedagogical approaches applied in all musical cultures worldwide, but also enhances the potential efficacy of current procedures for music training.

Do gestures reflect children’s lexical retrieval difficulties?

Abstract According to the Lexical Retrieval Hypothesis, one important function of representational gestures is to help speakers retrieve words for production. In this study, we test whether gestures help preschoolers access words for production. We tested two predictions generated from the LRH. First, since bilinguals often have greater difficulties with lexical access than monolinguals, we predicted that bilinguals will gesture more than monolinguals. Second, since low-frequency words are harder to access than high frequency words, we predicted that the more low-frequency words children used, the more they would gesture. Using children’s (aged 4 to 6 years) data from a storytelling task, we found no difference in gesture use between bilinguals and monolinguals. We did, however, find that the use of low-frequency words was a positive predictor of gesture use. While this result could support the LRH, an exploration of some examples of gesture production raises some doubts about that conclusion.

American Sign language fingerspelling recognition in the wild with spatio temporal feature extraction and multi-task learning

This study introduces a comprehensive approach to enhance the performance of fingerspelling recognition systems in dynamic environments. The methodology begins with spatial feature extraction using MobileNetV3-Small, followed by transformation through a projection layer into a latent space. The Variable-Filter-Length Temporal-Learning Convolutional Neural Network (VTCNN) is then applied to extract both short-range and long-range temporal features, providing a robust representation of dynamic gestures. The recognition system incorporates a shared encoder for both the Connectionist Temporal Classification (CTC) decoder and the attention-based decoder, capitalizing on the unique strengths of each decoder. To address weak supervision challenges, a novel strategy involving supervised contrastive learning (SupCon) during retraining is proposed. Leveraging decoding results from the CTC decoder, an image set with frame labels is constructed, contributing to more efficient differentiation between fingerspelling gestures and improving overall accuracy. The final step involves a joint CTC/attention-based decoding strategy using the beam search algorithm. This approach effectively combines decoder outputs, resulting in superior recognition performance. The synergistic interplay of proposed methods—VTCNN for temporal feature extraction, multi-task learning for leveraging decoder strengths, SupCon for feature clustering refinement, and joint decoding—culminates in a holistic and state-of-the-art fingerspelling recognition system, validated through benchmarking on the ChicagoFSWild and ChicagoFSWild+ datasets.

Gestural Temporality in Sciarrino’s Recitativo oscuro

Sciarrino’s writings describe a compositional philosophy that prizes multidimensionality and spatiotemporal discontinuity (1998, 2004). Yet his simultaneous allegiance to teleology, holism, and fractal hierarchies reveals an underlying unifying organicism with which these qualities may initially seem to conflict. I take Sciarrino’s 1999 piano concerto Recitativo oscuro as a case study for examining the composer’s gestural organicism and its various contradictions and double meanings. First, close analysis of the opening piano solo demonstrates how seemingly contradictory aesthetic priorities—organic unity and temporal multiplicity—co-exist within a single passage. Drawing from Kramer’s (1988) concept of “gestural time,” Hatten’s (2004) theory of gesture, segmentation theories, and scholarship on intensity and temporal function, I show how temporal multiplicity in the piano solo arises through conflict between gestural energetics and contextual segmentation. I then expand my analysis to the full concerto, arguing that the orchestra’s cyclic and naturalistic sonic ecosystem causes the piano gestures to take on representative functions in addition to their energetic syntax, creating what Sciarrino (2001) calls “dimensional intermittence” as they reference distant temporal realms both within and beyond the concerto. I conclude by arguing that while the intensity profile of the earliest gestures and phrases is eventually reproduced across the form of the entire work, the concerto’s form nevertheless resists reductive understanding. Instead, listeners are invited to hear form through the complex mechanics and idiosyncrasies of “gestural behavior” (Sciarrino 1998).

Gestural Simulation of Motion Verbs

Abstract We aimed to investigate how the path of a motion verb is mentally simulated and realized in gestures when it is encoded in path-joined motion verbs (e.g., “enter the house”), path-disjoined motion verbs (e.g., “go into the house”), and pathless motion verbs (e.g., “walk in the street”). We conducted 2 studies to answer this question. In Study 1, we analyzed gestures produced by presenters in a set of YouTube videos. We used several chi-square tests to find out what type of gesture (path representational, non–path representational, beat, pointing gestures, and no-gesture) co-occurred more frequently with path-joined, path-disjoined, and pathless motion verbs. In Study 2, we analyzed gestures produced by a group of participants in a story-retelling setting. Both studies showed that path representational gestures co-occurred more frequently with path-joined motion verbs than path-disjoined and pathless motion verbs. Furthermore, the probability of using a path representational gesture with a path-joined motion verb having a human subject was higher than that having a nonhuman subject. Although the language difference in Study 1 and Study 2 might be a limitation, the consistency of results of the 2 studies suggests that the findings are generalizable. We suggest that gestural simulation of a motion verb is affected by how the direction or path of motion is encoded in the verb. When the path of motion is encoded in the main part of the verb, the motion and its path or direction are simultaneously simulated. This account implies that the motor system plays an active role in the process of simulating the path.

Embodiment and gestural realization of ergative verbs.

In this study, we examined the gestural embodiment of active, passive, and active-form/passive-sense voices of ergative verbs in English. We analyzed gestures produced by presenters talking about a variety of subjects in a set of videos. We used several Chi-square tests to find out what type of gesture (representational, beat, and pointing gestures) co-occurred more frequently with active, passive, and active-form/passive-sense voices of ergative verbs. The results showed that representational gestures occurred more frequently with active than passive and active-form/passive-sense voices of ergative verbs. Furthermore, representational gestures occurred more frequently with active voices of ergative verbs having human subjects than non-human subjects. This was also the case with active-form/passive-sense sentences. Based on these results, it is suggested that form of a sentence is an influential factor in the process of embodying the situation that is described by that sentence. Active voice of an English ergative verb is more likely to be accompanied by representational gestures and is embodied more strongly than passive and active-form/passive-sense voices of that verb.