Co-speech Gestures Research Articles

When Gestures also Argue: Multimodal Viewpoint Shift as a Rhetorical Strategy in American Political Talk Shows

Drawing on Mental Spaces Theory and Conceptual Blending Theory in Cognitive Linguistics, we examine the interaction between speech and co-speech gestures in The Daily Show with Trevor Noah. Statistical analysis shows that viewpoint shift in the self-built multimodal corpus, generally following the pattern: Base Space->News Narrative Space(->Base space)->Source Viewpoint Space(+)(->Base space), is significantly related to the use of verbal markers and gestural types. Moreover, multimodal viewpoint shift in political talk shows can achieve such rhetorical functions as enhancing ironic effects, solidifying and highlighting opposing positions, and simplifying political issues. Since verbal markers and gestures can mobilize the audience’s embodied experience by primarily activating mental images and motor programs, we claim that mental simulation and perspective taking play a pivotal role in the cognitive processing of multimodal viewpoint shifts by promoting viewpoint alignment between the audience and the host.

Understanding discourse in face-to-face settings: The impact of multimodal cues and listening conditions.

In face-to-face contexts, discourse is accompanied by various cues, like gestures and mouth movements. Here, we asked whether the presence of gestures and mouth movements benefits discourse comprehension under clear and challenging listening conditions and, if so, whether this multimodal benefit depends on the communicative environment in which interlocutors are situated. In two online experiments, participants watched videoclips of a speaker telling stories, and they answered yes-no questions about the content of each story. The speaker in the videos was spontaneously gesturing (or kept her hands still) and was wearing a surgical mask (or had her lips visible). The experiments differed in the communicative environment. In Experiment 1, the speaker narrated stories in silence, whereas the listener (participants) heard them in clear or degraded speech conditions (analogous to watching the news on TV in a quiet or noisy café). In Experiment 2, the speaker narrated the stories once in silence and once while listening to background noise, and the listener heard them in clear or degraded speech condition, respectively (analogous to listening to a friend in a quiet or noisy café). Across the experiments, we found that cospeech gestures facilitated discourse comprehension regardless of the listening conditions or the presence of a mask. In contrast, mouth movements were primarily helpful in challenging listening conditions. These findings indicate that both cues matter to listeners but to a different extent. Moreover, we found that the multimodal benefit to comprehension was similar regardless of the communicative environment. Thus, this study demonstrates the importance of both cospeech gestures and mouth movements to discourse comprehension, offering insights into the dynamic interplay between these cues under different communicative environments. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Multimodal exploration of the thank God expressive construction and its implications for translation

Abstract Multimodal research in communication and translation studies is increasingly recognized, yet it remains incompletely explored. Leveraging computational linguistics with both Praat for acoustic analysis and the OpenPose and Rapid Annotator tools for visual analysis, this study delves into the intricate dynamics of the expressive construction thank God, providing a comprehensive examination of both visual and acoustic dimensions. Our objective is to uncover nuanced patterns of multimodal communication embedded within this expression and their implications for Translation and Interpreting. Through an analysis of linguistic features and co-speech gestures present in thank God, we aim to deepen our comprehension of how meaning crisscrosses modalities. Our findings underscore the necessity of a multimodal approach in language studies, emphasizing the requisite to preserve emotional and contextual nuances. The analysis unveils the phonological relevance of the duration of the construction’s second vowel, a key factor for translation. Additionally, data reveals a correlation between the emotion of relief and gestures executed with both hands closer to the chest. Overall, these findings contribute to advancing both multimodal communication research and translation studies, shedding light on the role of multimodal analysis in understanding language and translation dynamics, particularly in the context of constructions like thank God.

Enhancement of perceptual and cognitive functions in near-death experience: A perspective from embodiment theories

The phenomenon of near-death experience (NDE) is attracting a growing attention among researchers of various fields. In this study, we looked at NDE from a cognitive perspective to find out how NDE events are embodied when people recall and describe them. We examined the descriptions of a group of people talking about what they had experienced in the state of NDE. Based on the gesture-as-simulated-action theory, we assumed that co-speech gestures occurring when people were talking about their experiences were physical realizations of mentally-simulated events. The results showed that the number of iconic and metaphoric gestures occurring with expressions referring to NDE events was significantly larger than those occurred with expressions referring to ordinary events. Based on these results, we suggest that embodied memory for NDE events is stronger than embodied memory for ordinary non-NDE events. NDE events are perceived, recalled, and embodied by strong activation of sensorimotor systems in a state of perceptual and cognitive enhancement, although sensorimotor systems seem to be unresponsive to sensory stimuli during NDE. Finally, based on theories of embodied cognition, we conclude that the enhancement of cognitive functions that takes place in the state of NDE is the result of enhancement in perceptual functions.

Semantic and Visual Features Drive the Intrinsic Memorability of Co-Speech Gestures

Semantic and Visual Features Drive the Intrinsic Memorability of Co-Speech Gestures

Gesture size affects numerical estimates in quantifier comprehension.

People think and talk about numerical magnitude in terms of space, and co-speech gestures reflect this, with English speakers using expansive gestures when talking about greater quantities. Existing gestural research on the spatial conceptualization of number has largely looked at gesture production, but we do not know whether gestures can influence the interpretation of imprecise or underspecified numerical expressions such as quantifiers. Looking at the quantifier several as a test case, this study investigates the influence of manual inwards-directed (i.e., hands move closer the torso) and outwards-directed (i.e., hands move away from the torso) co-speech gestures on comprehenders' conceptualization of quantities associated with several through three preregistered experiments. Our results suggest that gesture modulates the interpretation of several such that speakers moving their hands outwards and thereby creating space between their hands lead to higher quantity estimates, compared to speakers not gesturing, or moving their hands inwards. We discuss the implications of our findings for future work in numerical cognition, multimodal communication, and pragmatics. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Personality Expression using Co-speech Gesture

We express our personality through verbal and nonverbal behavior. While verbal cues are mostly related to the semantics of what we say, nonverbal cues include our posture, gestures, and facial expressions. Appropriate expression of these behavioral elements improves conversational virtual agents’ communication capabilities and realism. Although previous studies focus on co-speech gesture generation, they do not consider the personality aspect of the synthesized animations. We show that automatically generated co-speech gestures naturally express personality traits, and heuristics-based adjustments for such animations can further improve personality expression. To this end, we present a framework for enhancing co-speech gestures with the different personalities of the Five-Factor model. Our experiments suggest that users perceive increased realism and improved personality expression when combining heuristics-based motion adjustments with co-speech gestures.

Is vision necessary for the timely acquisition of language-specific patterns in co-speech gesture and their lack in silent gesture?

Blind adults display language-specificity in their packaging and ordering of events in speech. These differences affect the representation of events in co-speech gesture-gesturing with speech-but not in silent gesture-gesturing without speech. Here we examine when in development blind children begin to show adult-like patterns in co-speech and silent gesture. We studied speech and gestures produced by 30 blind and 30 sighted children learning Turkish, equally divided into 3 age groups: 5-6, 7-8, 9-10 years. The children were asked to describe three-dimensional spatial event scenes (e.g., running out of a house) first with speech, and then without speech using only their hands. We focused on physical motion events, which, in blind adults, elicit cross-linguistic differences in speech and co-speech gesture, but cross-linguistic similarities in silent gesture. Our results showed an effect of language on gesture when it was accompanied by speech (co-speech gesture), but not when it was used without speech (silent gesture) across both blind and sighted learners. The language-specific co-speech gesture pattern for both packaging and ordering semantic elements was present at the earliest ages we tested the blind and sighted children. The silent gesture pattern appeared later for blind children than sighted children for both packaging and ordering. Our findings highlight gesture as a robust and integral aspect of the language acquisition process at the early ages and provide insight into when language does and does not have an effect on gesture, even in blind children who lack visual access to gesture. RESEARCH HIGHLIGHTS: Gestures, when produced with speech (i.e., co-speech gesture), follow language-specific patterns in event representation in both blind and sighted children. Gestures, when produced without speech (i.e., silent gesture), do not follow the language-specific patterns in event representation in both blind and sighted children. Language-specific patterns in speech and co-speech gestures are observable at the same time in blind and sighted children. The cross-linguistic similarities in silent gestures begin slightly later in blind children than in sighted children.

Kin Cognition and Communication: What Talking, Gesturing, and Drawing About Family Can Tell us About the Way We Think About This Core Social Structure.

When people talk about kinship systems, they often use co-speech gestures and other representations to elaborate. This paper investigates such polysemiotic (spoken, gestured, and drawn) descriptions of kinship relations, to see if they display recurring patterns of conventionalization that capture specific social structures. We present an exploratory hypothesis-generating study of descriptions produced by a lesser-known ethnolinguistic community to the cognitive sciences: the Paamese people of Vanuatu. Forty Paamese speakers were asked to talk about their family in semi-guided kinship interviews. Analyses of the speech, gesture, and drawings produced during these interviews revealed that lineality (i.e., mother's side vs. father's side) is lateralized in the speaker's gesture space. In other words, kinship members of the speaker's matriline are placed on the left side of the speaker's body and those of the patriline are placed on their right side, when they are mentioned in speech. Moreover, we find that the gesture produced by Paamese participants during verbal descriptions of marital relations are performed significantly more often on two diagonal directions of the sagittal axis. We show that these diagonals are also found in the few diagrams that participants drew on the ground to augment their verbo-gestural descriptions of marriage practices with drawing. We interpret this behavior as evidence of a spatial template, which Paamese speakers activate to think and communicate about family relations. We therefore argue that extending investigations of kinship structures beyond kinship terminologies alone can unveil additional key factors that shape kinship cognition and communication and hereby provide further insights into the diversity of social structures.

Co-speech head nods are used to enhance prosodic prominence at different levels of narrow focus in French.

Previous research has shown that prosodic structure can regulate the relationship between co-speech gestures and speech itself. Most co-speech studies have focused on manual gestures, but head movements have also been observed to accompany speech events by Munhall, Jones, Callan, Kuratate, and Vatikiotis-Bateson [(2004). Psychol. Sci. 15(2), 133-137], and these co-verbal gestures may be linked to prosodic prominence, as shown by Esteve-Gibert, Borrás-Comes, Asor, Swerts, and Prieto [(2017). J. Acoust. Soc. Am. 141(6), 4727-4739], Hadar, Steiner, Grant, and Rose [(1984). Hum. Mov. Sci. 3, 237-245], and House, Beskow, and Granström [(2001). Lang. Speech 26(2), 117-129]. This study examines how the timing and magnitude of head nods may be related to degrees of prosodic prominence connected to different focus conditions. Using electromagnetic articulometry, a time-varying signal of vertical head movement for 12 native French speakers was generated to examine the relationship between head nod gestures and F0 peaks. The results suggest that speakers use two different alignment strategies, which integrate both temporal and magnitudinal aspects of the gesture. Some evidence of inter-speaker preferences in the use of the two strategies was observed, although the inter-speaker variability is not categorical. Importantly, prosodic prominence itself is not the cause of the difference between the two strategies, but instead magnifies their inherent differences. In this way, the use of co-speech head nod gestures under French focus conditions can be considered as a method of prosodic enhancement.

Exploring Emotions Through Co-speech Gestures: The Caveats and New Directions

Co-speech hand gestures offer a rich avenue for research into studying emotion communication because they serve as both prominent expressive bodily cues and an integral part of language. Despite such a strategic relevance, gesture-speech integration and interaction have received less research focus on its emotional function compared to its cognitive function. This review aims to shed light on the current state of the field regarding the interplay between co-speech hand gestures and emotions, focusing specifically on the role of gestures in expressing and understanding both others’ and one's own emotions. The article concludes by addressing current limitations in the field and proposing future directions for researchers investigating gesture-emotion interaction. Our goal is to provide a roadmap to researchers in their exploration of the role of gestures in emotions, ultimately contributing to a more comprehensive understanding of how gestures and emotions intersect.

TAG2G: A Diffusion-Based Approach to Interlocutor-Aware Co-Speech Gesture Generation

Extended reality (XR) systems are about to be integrated into our daily lives and will provide support in a variety of fields such as education and coaching. Enhancing user experience demands agents that are capable of displaying realistic affective and social behaviors within these systems, and, as a prerequisite, with the capability of understanding their interaction partner and responding appropriately. Based on our literature review of recent works published in the field of co-speech gesture generation, researchers have developed complex models capable of generating gestures characterized by a high level of human-likeness and speaker appropriateness. Nevertheless, this is only true in settings where the agent has an active status (i.e., the agent acts as the speaker), or it is delivering a monologue in a non-interactive setting. However, as illustrated in multiple works and competitions like the GENEA Challenge, these models remain inadequate in generating interlocutor-aware gestures. We consider interlocutor-aware gesture generation the process of displaying gestures that take into account the conversation partner’s behavior. Moreover, in settings where the agent is the listener, generated gestures lack the level of naturalness that we expect from a face-to-face conversation. To overcome these issues, we have designed a pipeline, called TAG2G, composed of a diffusion model, which was demonstrated to be a stable and powerful tool in gesture generation, and a vector-quantized variational auto-encoder (VQVAE), widely employed to produce meaningful gesture embeddings. Refocusing from monadic to dyadic multimodal input settings (i.e., taking into account text, audio, and previous gestures of both participants of a conversation) allows us to explore and infer the complex interaction mechanisms that lie in a balanced two-sided conversation. As per our results, a multi-agent conversational input setup improves the generated gestures’ appropriateness with respect to the conversational counterparts. Conversely, when the agent is speaking, a monadic approach performs better in terms of the generated gestures’ appropriateness in relation to the speech.

Editable Co-Speech Gesture Synthesis Enhanced with Individual Representative Gestures

Co-speech gesture synthesis is a challenging task due to the complexity and uncertainty between gestures and speech. Gestures that accompany speech (i.e., Co-Speech Gesture) are an essential part of natural and efficient embodied human communication, as they work in tandem with speech to convey information more effectively. Although data-driven approaches have improved gesture synthesis, existing deep learning-based methods use deterministic modeling which could lead to averaging out predicted gestures. Additionally, these methods lack control over gesture generation such as user editing of generated results. In this paper, we propose an editable gesture synthesis method based on a learned pose script, which disentangles gestures into individual representative and rhythmic gestures to produce high-quality, diverse and realistic poses. Specifically, we first detect the time of occurrence of gestures in video sequences and transform them into pose scripts. Regression models are then built to predict the pose scripts. Next, learned pose scripts are used for gesture synthesis, while rhythmic gestures are modeled using a variational auto-encoder and a one-dimensional convolutional network. Moreover, we introduce a large-scale Chinese co-speech gesture synthesis dataset with multimodal annotations for training and evaluation, which will be publicly available to facilitate future research. The proposed method allows for the re-editing of generated results by changing the pose scripts for applications such as interactive digital humans. The experimental results show that this method generates more quality, more diverse, and realistic gestures than other existing methods.

Enhancing Reflective and Conversational User Engagement in Argumentative Dialogues with Virtual Agents

In their process of information seeking, human users tend to selectively ignore information that contradicts their pre-existing beliefs or opinions. These so-called “self-imposed filter bubbles” (SFBs) pose a significant challenge for argumentative conversational agents aiming to facilitate critical, unbiased opinion formation on controversial topics. With the ultimate goal of developing a system that helps users break their self-imposed filter bubbles (SFBs), this paper aims to investigate the role of co-speech gestures, specifically examining how these gestures significantly contribute to achieving this objective. This paper extends current research by examining methods to engage users in cooperative discussions with a virtual human-like agent, encouraging a deep reflection on arguments to disrupt SFBs. Specifically, we investigate the agent’s non-verbal behavior in the form of co-speech gestures. We analyze whether co-speech gestures, depending on the conveyed information, enhance motivation, and thus conversational user engagement, thereby encouraging users to consider information that could potentially disrupt their SFBs. The findings of a laboratory study with 56 participants highlight the importance of non-verbal agent behaviors, such as co-speech gestures, in improving users’ perceptions of the interaction and the conveyed content. This effect is particularly notable when the content aims to challenge the user’s SFB. Therefore, this research offers valuable insights into enhancing user engagement in the design of multimodal interactions with future cooperative argumentative virtual agents.

Audio2Gestures: Generating Diverse Gestures From Audio.

People may perform diverse gestures affected by various mental and physical factors when speaking the same sentences. This inherent one-to-many relationship makes co-speech gesture generation from audio particularly challenging. Conventional CNNs/RNNs assume one-to-one mapping, and thus tend to predict the average of all possible target motions, easily resulting in plain/boring motions during inference. So we propose to explicitly model the one-to-many audio-to-motion mapping by splitting the cross-modal latent code into shared code and motion-specific code. The shared code is expected to be responsible for the motion component that is more correlated to the audio while the motion-specific code is expected to capture diverse motion information that is more independent of the audio. However, splitting the latent code into two parts poses extra training difficulties. Several crucial training losses/strategies, including relaxed motion loss, bicycle constraint, and diversity loss, are designed to better train the VAE. Experiments on both 3D and 2D motion datasets verify that our method generates more realistic and diverse motions than previous state-of-the-art methods, quantitatively and qualitatively. Besides, our formulation is compatible with discrete cosine transformation (DCT) modeling and other popular backbones (i.e., RNN, Transformer). As for motion losses and quantitative motion evaluation, we find structured losses/metrics (e.g. STFT) that consider temporal and/or spatial context complement the most commonly used point-wise losses (e.g. PCK), resulting in better motion dynamics and more nuanced motion details. Finally, we demonstrate that our method can be readily used to generate motion sequences with user-specified motion clips on the timeline.

Communicative efficiency in multimodal language directed at children and adults.

The ecology of human communication is face to face. In these contexts, speakers dynamically modify their communication across vocal (e.g., speaking rate) and gestural (e.g., cospeech gestures related in meaning to the content of speech) channels while speaking. What is the function of these adjustments? Here we ask whether speakers dynamically make these adjustments to increase communicative success, and decrease cognitive effort while speaking. We assess whether speakers modulate word durations and produce iconic (i.e., imagistically evoking properties of referents) gestures depending on the predictability of each word they utter. Predictability is operationalized as surprisal and computed from computational language models trained on corpora of child-directed, or adult-directed language. Using data from a novel corpus (Ecological Language Corpus) of naturalistic interactions between adult-child (aged 3-4), and adult-adult, we show that surprisal predicts speakers' multimodal adjustments and that some of these effects are modulated by whether the comprehender is a child or an adult. Thus, communicative efficiency applies generally across vocal and gestural communicative channels not being limited to structural properties of language or vocal modality. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Gesture and Bilingualism: evidence from Polish-English preschoolers.

Communication of monolingual and bilingual children differs with respect to words they use in a conversation and choice of gestures accompanying these words. Language dominance can play a vital role in how children use gestures to back up or replace vocabulary items. In the research, we investigated the relationship between language and gestures in bilingual, Polish-English preschoolers. We assumed that children would produce more co-speech gestures in their dominant language, compared to their non-dominant language, which has not been traced thoroughly in the topical research. In the report, we provide theoretical background from gesture studies and language acquisition, we present the procedure used in the experiment, and discuss the analysis of data obtained from four bilinguals: a 53-month-old, two 67-month-old and a 76-month-old. The recordings gathered document gesticulation and speech of the children during a conversation with the researcher, which is conducted in the form of a game. We present quantitative (gesture count) and qualitative analysis of the material gathered during that game.

Evaluating Gesture Generation in a Large-scale Open Challenge: The GENEA Challenge 2022

This article reports on the second GENEA Challenge to benchmark data-driven automatic co-speech gesture generation. Participating teams used the same speech and motion dataset to build gesture-generation systems. Motion generated by all these systems was rendered to video using a standardised visualisation pipeline and evaluated in several large, crowdsourced user studies. Unlike when comparing different research articles, differences in results are here only due to differences between methods, enabling direct comparison between systems. The dataset was based on 18 hours of full-body motion capture, including fingers, of different persons engaging in a dyadic conversation. Ten teams participated in the challenge across two tiers: full-body and upper-body gesticulation. For each tier, we evaluated both the human-likeness of the gesture motion and its appropriateness for the specific speech signal. Our evaluations decouple human-likeness from gesture appropriateness, which has been a difficult problem in the field. The evaluation results show some synthetic gesture conditions being rated as significantly more human-like than 3D human motion capture. To the best of our knowledge, this has not been demonstrated before. On the other hand, all synthetic motion is found to be vastly less appropriate for the speech than the original motion-capture recordings. We also find that conventional objective metrics do not correlate well with subjective human-likeness ratings in this large evaluation. The one exception is the Fréchet gesture distance (FGD), which achieves a Kendall’s tau rank correlation of around -0.5. Based on the challenge results we formulate numerous recommendations for system building and evaluation.

The cognitive-emotional dialectics in second language development: speech and co-speech gestures of Chinese learners of English

Abstract This study examines the cognitive-emotional dialectics of Chinese learners of English in terms of both speech and co-speech gestures in their second language development within the lens of Vygotsky’s sociocultural theory. This qualitative study focuses on Chinese learners of English in different learning contexts and analyzes data from interviews and emotional narrative tasks in the English learning experience. Overall findings showed that learners’ emotion and cognition interacted in different learning contexts, and speech and co-speech gestures worked to influence the cognition-emotion dialectics in participant’s perezhivanie. This study implies the inextricable unity of emotion and cognition in second language development, and emphasizes the role of speech and co-speech gestures in shaping second language learners’ perezhivaniya.

Dual-Path Transformer-Based GAN for Co-speech Gesture Synthesis

Dual-Path Transformer-Based GAN for Co-speech Gesture Synthesis