Co-speech Gestures Research Articles

Co-speech iconic gestures and visuo-spatial working memory

Three experiments tested the role of verbal versus visuo-spatial working memory in the comprehension of co-speech iconic gestures. In Experiment 1, participants viewed congruent discourse primes in which the speaker's gestures matched the information conveyed by his speech, and incongruent ones in which the semantic content of the speaker's gestures diverged from that in his speech. Discourse primes were followed by picture probes that participants judged as being either related or unrelated to the preceding clip. Performance on this picture probe classification task was faster and more accurate after congruent than incongruent discourse primes. The effect of discourse congruency on response times was linearly related to measures of visuo-spatial, but not verbal, working memory capacity, as participants with greater visuo-spatial WM capacity benefited more from congruent gestures. In Experiments 2 and 3, participants performed the same picture probe classification task under conditions of high and low loads on concurrent visuo-spatial (Experiment 2) and verbal (Experiment 3) memory tasks. Effects of discourse congruency and verbal WM load were additive, while effects of discourse congruency and visuo-spatial WM load were interactive. Results suggest that congruent co-speech gestures facilitate multi-modal language comprehension, and indicate an important role for visuo-spatial WM in these speech–gesture integration processes.

Social eye gaze modulates processing of speech and co-speech gesture

In human face-to-face communication, language comprehension is a multi-modal, situated activity. However, little is known about how we combine information from different modalities during comprehension, and how perceived communicative intentions, often signaled through visual signals, influence this process. We explored this question by simulating a multi-party communication context in which a speaker alternated her gaze between two recipients. Participants viewed speech-only or speech+gesture object-related messages when being addressed (direct gaze) or unaddressed (gaze averted to other participant). They were then asked to choose which of two object images matched the speaker’s preceding message. Unaddressed recipients responded significantly more slowly than addressees for speech-only utterances. However, perceiving the same speech accompanied by gestures sped unaddressed recipients up to a level identical to that of addressees. That is, when unaddressed recipients’ speech processing suffers, gestures can enhance the comprehension of a speaker’s message. We discuss our findings with respect to two hypotheses attempting to account for how social eye gaze may modulate multi-modal language comprehension.

Echoes of the spoken past: how auditory cortex hears context during speech perception.

What do we hear when someone speaks and what does auditory cortex (AC) do with that sound? Given how meaningful speech is, it might be hypothesized that AC is most active when other people talk so that their productions get decoded. Here, neuroimaging meta-analyses show the opposite: AC is least active and sometimes deactivated when participants listened to meaningful speech compared to less meaningful sounds. Results are explained by an active hypothesis-and-test mechanism where speech production (SP) regions are neurally re-used to predict auditory objects associated with available context. By this model, more AC activity for less meaningful sounds occurs because predictions are less successful from context, requiring further hypotheses be tested. This also explains the large overlap of AC co-activity for less meaningful sounds with meta-analyses of SP. An experiment showed a similar pattern of results for non-verbal context. Specifically, words produced less activity in AC and SP regions when preceded by co-speech gestures that visually described those words compared to those words without gestures. Results collectively suggest that what we ‘hear’ during real-world speech perception may come more from the brain than our ears and that the function of AC is to confirm or deny internal predictions about the identity of sounds.

Body-directed gestures: Pointing to the self and beyond

The prototypical pointing gesture is directed outward at concrete objects, people, or locations in the world. But in everyday discourse pointing gestures are also commonly directed inward, toward the body. Body-directed gestures are co-produced with a variety of spoken referents, from personal pronouns (I, we) to experiential concepts (belief, instinct), exhibit wide but motivated variation in handshapes and movement patterns, and involve conceptual processes whose prevalence in co-speech gesture has gone largely unexamined. Based on a corpus of 40 one-on-one interviews from the Tavis Smiley Show, three varieties of body-directed gestures—self-points, body-points, and body-anchors—are introduced and the semiotic and morphological characteristics of each variety are investigated. Body-directed gestures present a variegated subset of pointing gestures more generally, affording a novel vantage on pointing and its relation to speech, conceptual processes in everyday real-time behavior, and the role of the body as a foundational site for anchoring meaning of all kinds.

Multimodal alignment during collaborative remembering

This article investigates the roles that interactive alignment of manual gesture, postural sway, and eye-gaze play in small groups engaged in collaborative remembering. Qualitative analyses of a video corpus demonstrate that the coordination of these behaviors may contribute to joint remembering in various ways, depending upon the cognitive and communicative affordances of these behaviors. The observation that these behaviors are different in their nature and their contributory potential to shared remembering is corroborated by the results of a quantitative analysis, which suggests that co-speech gesture, postural sway, and eye-gaze have different interactional dynamics. This supports the conclusion that in order to understand the role of multimodal alignment in the discourse of shared remembering, co-verbal behavior should not be treated as a homogeneous category. Finally, we discuss the potential of combined qualitative–quantitative approaches to inform the interplay of verbal and bodily coordination during interactive memory construction.

The multisensory perception of co-speech gestures – A review and meta-analysis of neuroimaging studies

Co-speech gestures constitute a unique form of multimodal communication because here the hand movements are temporally synchronized and semantically integrated with speech. Recent neuroimaging studies indicate that the perception of co-speech gestures might engage a core set of frontal, temporal, and parietal areas. However, no study has compared the neural processes during perception of different types of co-speech gestures, such as beat, deictic, iconic, and metaphoric co-speech gestures. The purpose of this study was to review the existing literature on the neural correlates of co-speech gesture perception and to test whether different types of co-speech gestures elicit a common pattern of brain activity in the listener. To this purpose, we conducted a meta-analysis of neuroimaging studies, which used different types of co-speech gestures to investigate the perception of multimodal (co-speech gestures) in contrast to unimodal (speech or gestures) stimuli. The results show that co-speech gesture perception consistently engages temporal regions related to auditory and movement perception as well as frontal-parietal regions associated with action understanding. The results of this study suggest that brain regions involved in multisensory processing and action understanding constitute the general core of co-speech gesture perception.

Verbal working memory predicts co-speech gesture: Evidence from individual differences

Gesture facilitates language production, but there is debate surrounding its exact role. It has been argued that gestures lighten the load on verbal working memory (VWM; Goldin-Meadow, Nusbaum, Kelly, & Wagner, 2001), but gestures have also been argued to aid in lexical retrieval (Krauss, 1998). In the current study, 50 speakers completed an individual differences battery that included measures of VWM and lexical retrieval. To elicit gesture, each speaker described short cartoon clips immediately after viewing. Measures of lexical retrieval did not predict spontaneous gesture rates, but lower VWM was associated with higher gesture rates, suggesting that gestures can facilitate language production by supporting VWM when resources are taxed. These data also suggest that individual variability in the propensity to gesture is partly linked to cognitive capacities.

Brain mechanisms for processing co-speech gesture: A cross-language study of spatial demonstratives

Abstract This electrophysiological study investigated the relationship between language and nonverbal socio-spatial context for demonstrative use in speech communication. Adult participants from an English language group and a Japanese language group were asked to make congruency judgment for simultaneous presentation of an audio demonstrative phrase in their native language and a picture that included two human figures as speaker and hearer, as well as a referent object in different spatial arrangements. The demonstratives (“this” and “that” in English, and “ko,” “so,” and “a” in Japanese) were varied for the visual scenes to produce expected and unexpected combinations to refer to an object based on its relative spatial distances to the speaker and hearer. Half of the trials included an accompanying pointing gesture in the picture, and the other half did not. Behavioral data showed robust congruency effects with longer reaction time for the incongruent trials in both subject groups irrespective of the presence or absence of the pointing gesture. Both subject groups also showed a significant N400-like congruency effect in the event-related potential responses for the gesture trials, a finding predicted from previous work (Stevens & Zhang, 2013). In the no-gesture trials, the English data alone showed a P600 congruency effect preceded by a negative deflection. These results provide evidence for shared brain mechanisms for processing demonstrative expression congruency, as well as language-specific neural sensitivity to encoding the co-expressivity of gesture and speech.

Eye’m talking to you: speakers’ gaze direction modulates co-speech gesture processing in the right MTG

Recipients process information from speech and co-speech gestures, but it is currently unknown how this processing is influenced by the presence of other important social cues, especially gaze direction, a marker of communicative intent. Such cues may modulate neural activity in regions associated either with the processing of ostensive cues, such as eye gaze, or with the processing of semantic information, provided by speech and gesture. Participants were scanned (fMRI) while taking part in triadic communication involving two recipients and a speaker. The speaker uttered sentences that were and were not accompanied by complementary iconic gestures. Crucially, the speaker alternated her gaze direction, thus creating two recipient roles: addressed (direct gaze) vs unaddressed (averted gaze) recipient. The comprehension of Speech&Gesture relative to SpeechOnly utterances recruited middle occipital, middle temporal and inferior frontal gyri, bilaterally. The calcarine sulcus and posterior cingulate cortex were sensitive to differences between direct and averted gaze. Most importantly, Speech&Gesture utterances, but not SpeechOnly utterances, produced additional activity in the right middle temporal gyrus when participants were addressed. Marking communicative intent with gaze direction modulates the processing of speech-gesture utterances in cerebral areas typically associated with the semantic processing of multi-modal communicative acts.

Narrative processing in typically developing children and children with early unilateral brain injury: seeing gesture matters.

Narrative skill in kindergarteners has been shown to be a reliable predictor of later reading comprehension and school achievement. However, we know little about how to scaffold children's narrative skill. Here we examine whether the quality of kindergarten children's narrative retellings depends on the kind of narrative elicitation they are given. We asked this question with respect to typically developing (TD) kindergarten children and children with pre- or perinatal unilateral brain injury (PL), a group that has been shown to have difficulty with narrative production. We compared children's skill in retelling stories originally presented to them in 4 different elicitation formats: (a) wordless cartoons, (b) stories told by a narrator through the auditory modality, (c) stories told by a narrator through the audiovisual modality without co-speech gestures, and (e) stories told by a narrator in the audiovisual modality with co-speech gestures. We found that children told better structured narratives in response to the audiovisual + gesture elicitation format than in response to the other 3 elicitation formats, consistent with findings that co-speech gestures can scaffold other aspects of language and memory. The audiovisual + gesture elicitation format was particularly beneficial for children who had the most difficulty telling a well-structured narrative, a group that included children with larger lesions associated with cerebrovascular infarcts.

Impact of the Temporal Dynamics of Speech and Gesture on Communication in Autism Spectrum Disorder

Autism Spectrum Disorder (ASD) is characterized by difficulties in communication and social interaction. Abnormalities in the use of gestures or flow of conversation are frequently amongst the clinical observations that contribute to a diagnosis of the disorder but the mechanisms underlying these communication difficulties remain unclear. While studies in children show a reduced use of co-speech gestures overall (e.g., Sowden et al., 2013), de Marchena and Eigsti (2010) found that in adults some gestures are used with similar frequency in ASD and typically developing (TD) populations, but that gestures produced by ASD individuals are less strongly time-locked to speech than in TD individuals. This may indicate that abnormal temporal processes contribute to impaired social skills in ASD (Allman, 2011). The objectives of this study were three-folded: 1) characterise the temporal dynamics of speech and gesture coordination in ASD in naturalistic speech; 2) quantify the use of gestures in ASD; 3) test the hypothesis that atypical temporal coordination between speech and gestures results in a lesser quality in communication. The context of a previously published study of memory in ASD (Maras et al., 2013) provided the opportunity to examine video recordings of 16 ASD and 17 TD adults attempting to recall details of a standardised event they had participated in (a first aid scenario). The current analysis was designed to quantify the participants’ communicative behaviours in three ways: 1) Segmenting the videos to identify each gesture and document its precise timing aimed to establish whether gestures are used differently in ASD and TD participants; 2) Extracting the quantity of movement as well as the pitch and volume of speech over time allowed us to characterise whether gestures were time-locked to speech similarly in ASD and TD individuals; 3) Collecting subjective ratings on the quality of communication displayed in the audio-only or audio-visual recordings helped to establish to what extent the use of gestures improved the quality of communication in ASD and TD participants. Overall, our results indicated no main group difference in the use and coordination of speech and gesture: both groups produced the same quantity of movement over time [t(33)=-0.165, p>.8], and gestures were produced within the same time window and with a similar distribution by ASD and TD individuals (η2p=.042). Similarly, no group differences were found in the subjective ratings on the quality of communication: in both groups the use of gestures improved comprehension and engagement from the listener. Interestingly, in line with previous report, we found that ASD individuals spoke on average louder than TD individuals [t(33)=-3.520, p<.005]. Notably, all measures showed a large inter-individual variability in both groups. The current data do not suggest that ASD individuals experience more difficulties than TD participants in time processes relevant to communicating personally experienced events. However, large inter-individual differences could contribute to communication difficulties in some participants. It will be important for future studies to examine the timing of communicative behaviours during reciprocal interactions, that place demands not only on coordinating speech with gesture but to coordinate one's own behaviour with that of others.

Effects of age and language on co-speech gesture production: an investigation of French, American, and Italian children's narratives

The aim of this paper is to compare speech and co-speech gestures observed during a narrative retelling task in five- and ten-year-old children from three different linguistic groups, French, American, and Italian, in order to better understand the role of age and language in the development of multimodal monologue discourse abilities. We asked 98 five- and ten-year-old children to narrate a short, wordless cartoon. Results showed a common developmental trend as well as linguistic and gesture differences between the three language groups. In all three languages, older children were found to give more detailed narratives, to insert more comments, and to gesture more and use different gestures--specifically gestures that contribute to the narrative structure--than their younger counterparts. Taken together, these findings allow a tentative model of multimodal narrative development in which major changes in later language acquisition occur despite language and culture differences.

Metaphor and Education: Reaching Business Training Goals through Multimodal Metaphor

Metaphor facilitates education by allowing for new perspectives or in aiding categorization and memorization. This study looks into the use of metaphor in a business training talk. The results show that the speaker repeatedly used words and gestures to express the journey metaphor. The project was also conceptualized as a physical object and managing a project as conducting an orchestra, both verbally and in the co-speech gesture. Other verbal war and sports metaphors were supported by either co-speech gestures or metaphoric images. Metaphors in business training settings seem to actively assist in persuading the business practitioners of the need to acquire a particular management ability.

Externalizing the Private Experience of Pain: A Role for Co-Speech Gestures in Pain Communication?

Despite the importance of effective pain communication, talking about pain represents a major challenge for patients and clinicians because pain is a private and subjective experience. Focusing primarily on acute pain, this article considers the limitations of current methods of obtaining information about the sensory characteristics of pain and suggests that spontaneously produced “co-speech hand gestures” may constitute an important source of information here. Although this is a relatively new area of research, we present recent empirical evidence that reveals that co-speech gestures contain important information about pain that can both add to and clarify speech. Following this, we discuss how these findings might eventually lead to a greater understanding of the sensory characteristics of pain, and to improvements in treatment and support for pain sufferers. We hope that this article will stimulate further research and discussion of this previously overlooked dimension of pain communication.

Handling agents and patients: representational cospeech gestures help children comprehend complex syntactic constructions.

Gesture is an important precursor of children's early language development, for example, in the transition to multiword speech and as a predictor of later language abilities. However, it is unclear whether gestural input can influence children's comprehension of complex grammatical constructions. In Study 1, 3- (M = 3 years 5 months) and 4-year-old (M = 4 years 6 months) children witnessed 2-participant actions described using the infrequent object-cleft-construction (OCC; It was the dog that the cat chased). Half saw an experimenter accompanying her descriptions with gestures representing the 2 participants and indicating the direction of action; the remaining children did not witness gesture. Children who witnessed gestures showed better comprehension of the OCC than those who did not witness gestures, both in and beyond the immediate physical context, but this benefit was restricted to the oldest 4-year-olds. In Study 2, a further group of older 4-year-old children (M = 4 years 7 months) witnessed the same 2-participant actions described by an experimenter and accompanied by gestures, but the gesture represented only the 2 participants and not the direction of the action. Again, a benefit of gesture was observed on subsequent comprehension of the OCC. We interpret these findings as demonstrating that representational cospeech gestures can help children comprehend complex linguistic structures by highlighting the roles played by the participants in the event.

How handshape type can distinguish between nouns and verbs in homesign.

All established languages, spoken or signed, make a distinction between nouns and verbs. Even a young sign language emerging within a family of deaf individuals has been found to mark the noun-verb distinction, and to use handshape type to do so. Here we ask whether handshape type is used to mark the noun-verb distinction in a gesture system invented by a deaf child who does not have access to a usable model of either spoken or signed language. The child produces homesigns that have linguistic structure, but receives from his hearing parents co-speech gestures that are structured differently from his own gestures. Thus, unlike users of established and emerging languages, the homesigner is a producer of his system but does not receive it from others. Nevertheless, we found that the child used handshape type to mark the distinction between nouns and verbs at the early stages of development. The noun-verb distinction is thus so fundamental to language that it can arise in a homesign system not shared with others. We also found that the child abandoned handshape type as a device for distinguishing nouns from verbs at just the moment when he developed a combinatorial system of handshape and motion components that marked the distinction. The way the noun-verb distinction is marked thus depends on the full array of linguistic devices available within the system.

Point of view in British Sign Language and spoken English narrative discourse: the example of “The Tortoise and the Hare”

AbstractThis paper discusses how point of view (POV) is expressed in British Sign Language (BSL) and spoken English narrative discourse. Spoken languages can mark changes in POV using strategies such as direct/indirect discourse, whereas signed languages can mark changes in POV in a unique way using “role shift”. Role shift is where the signer “becomes” a referent by taking on attributes of that referent, e.g. facial expression. In this study, two native BSL users and two native British English speakers were asked to tell the story “The Tortoise and the Hare”. The data were then compared to see how point of view is expressed and maintained in both languages. The results indicated that the spoken English users preferred the narrator's perspective, whereas the BSL users preferred a character's perspective. This suggests that spoken and signed language users may structure stories in different ways. However, some co-speech gestures and facial expressions used in the spoken English stories to denote characters' thoughts and feelings bear resemblance to the hand movements and facial expressions used by the BSL storytellers. This suggests that while approaches to storytelling may differ, both languages share some gestural resources which manifest themselves in different ways across different modalities.

Assessing language acquisition from parent-child interaction: An event-related potential study on perception of audio-visual cues in infancy

This paper promotes a theory-driven model development of parent-child interaction. In our project, we identify, test, and simulate some of the fundamental components of speech, gestures, and social-emotional behaviors and the consequences they might have on child language development. Our theoretical position is part of the connectionist tradition; language acquisition is described to be an emergent consequence of the interplay between the infant and the ambient linguistic environment, including sensory information of all modalities. It is well known that speech comprehension and production are significantly influenced by the presence of co-speech gestures. These gestures may be articulatory in nature or hand/beat co-gestures that keep the rhythm of speech. However, since the extent of this integrated relationship is difficult to determine from behavioral research solely, studies addressing neural mechanisms that underlie cognitive processes and behaviors are of importance. This paper reports an electroencephalography/event-related potential (EEG/ERP) pilot study on children’s early perception of congruent versus incongruent audio-visual pairings (e.g., acoustic information matching vs not matching the articulation shown). Ultimately, it is our hope that understanding the integrated speech-gesture relationship may provide insights into how children allocate resources while speaking and help clinicians/teachers to better identify and treat children with developmental disorders.

Gestural Viewpoint Signals Referent Accessibility

The tracking of entities in discourse is known to be a bimodal phenomenon. Speakers achieve cohesion in speech by alternating between full lexical forms, pronouns, and zero anaphora as they track referents. They also track referents in co-speech gestures. In this study, we explored how viewpoint is deployed in reference tracking, focusing on representations of animate entities in German narrative discourse. We found that gestural viewpoint systematically varies depending on discourse context. Speakers predominantly use character viewpoint in maintained contexts and observer viewpoint in reintroduced contexts. Thus, gestural viewpoint seems to function as a cohesive device in narrative discourse. The findings expand on and provide further evidence for the coordination between speech and gesture on the discourse level that is crucial to understanding the tight link between the two modalities.

The multi-modal representation of motion events in Awetí discourse

Recent research focusing on the insights gestures may offer into the mental conceptualization of motion events suggests that a number of factors may influence the way motion events are gestured in different languages. This study aims at showing how in Aweti oral discourse other than purely linguistic modes of representation such as gesture and ideophones are obligatory ‘path’ and optional ‘manner’ components of motion events. With regard to ‘manner’ it will be illustrated by corpus examples that the information state of this motion component in discourse determines whether it will additionally be gestured or not. These features suggest that the co-speech gestures in Aweti motion events have a meaning-conveying function and should be treated as a means of communication rather than as a feature which is produced with a low level of speaker awareness, merely providing insights into mental conceptualizations.