Co-speech Gestures Research Articles

Referring strategies in American Sign Language and English (with co-speech gesture): The role of modality in referring to non-nameable objects—ERRATUM

An abstract is not available for this content. As you have access to this content, full HTML content is provided on this page. A PDF of this content is also available in through the ‘Save PDF’ action button.

Indicating verbs as typologically unique constructions: Reconsidering verb ‘agreement’ in sign languages

In this paper, we present arguments for an analysis of indicating verbs, building on Liddell (2000), as a typologically unique, unimodal fusion of signs and pointing gestures used for reference tracking. This contrasts with many formalist analyses that assume that directionality in indicating verbs constitutes an agreement marking system. While exploring some of the debate in the literature about these forms, we propose a model of indicating verbs within a Construction Grammar framework that compares them to multimodal constructions in spoken languages. We explain how our model of indicating verbs appear to align with a growing body of research on co-speech gesture and is supported by some recent findings about these verbs from corpus-based studies of sign languages.

Having a different pointing of view about the future: The effect of signs on co-speech gestures about time in Mandarin–CSL bimodal bilinguals

Mandarin speakers often use gestures to represent time laterally, vertically, and sagittally. Chinese Sign Language (CSL) users also exploit signs for that purpose, and can differ from the gestures of Mandarin speakers in their choices of axes and direction of sagittal movements. The effects of sign language on co-speech gestures about time were investigated by comparing spontaneous temporal gestures of late bimodal bilinguals (Mandarin learners of CSL) and non-signing Mandarin speakers. Spontaneous gestures were elicited via a wordlist definition task. In addition to effects of temporal words on temporal gestures, results showed significant effects of sign. Compared with non-signers, late bimodal bilinguals (1) produced more sagittal but fewer lateral temporal gestures; and (2) exhibited a different temporal orientation of sagittal gestures, as they were more likely to gesture past events to their back. In conclusion, bodily experience of sign language can not only impact the nature of co-speech gestures, but also spatio-motoric thinking and abstract space-time mappings.

Co-thought gesturing supports more complex problem solving in subjects with lower visual working-memory capacity

During silent problem solving, hand gestures arise that have no communicative intent. The role of such co-thought gestures in cognition has been understudied in cognitive research as compared to co-speech gestures. We investigated whether gesticulation during silent problem solving supported subsequent performance in a Tower of Hanoi problem-solving task, in relation to visual working-memory capacity and task complexity. Seventy-six participants were assigned to either an instructed gesture condition or a condition that allowed them to gesture, but without explicit instructions to do so. This resulted in three gesture groups: (1) non-gesturing; (2) spontaneous gesturing; (3) instructed gesturing. In line with the embedded/extended cognition perspective on gesture, gesturing benefited complex problem-solving performance for participants with a lower visual working-memory capacity, but not for participants with a lower spatial working-memory capacity.

The Role of Gamma Oscillations During Integration of Metaphoric Gestures and Abstract Speech.

Metaphoric (MP) co-speech gestures are commonly used during daily communication. They communicate about abstract information by referring to gestures that are clearly concrete (e.g., raising a hand for “the level of the football game is high”). To understand MP co-speech gestures, a multisensory integration at semantic level is necessary between abstract speech and concrete gestures. While semantic gesture-speech integration has been extensively investigated using functional magnetic resonance imaging, evidence from electroencephalography (EEG) is rare. In the current study, we set out an EEG experiment, investigating the processing of MP vs. iconic (IC) co-speech gestures in different contexts, to reveal the oscillatory signature of MP gesture integration. German participants (n = 20) viewed video clips with an actor performing both types of gestures, accompanied by either comprehensible German or incomprehensible Russian (R) speech, or speaking German sentences without any gestures. Time-frequency analysis of the EEG data showed that, when gestures were accompanied by comprehensible German speech, MP gestures elicited decreased gamma band power (50–70 Hz) between 500 and 700 ms in the parietal electrodes when compared to IC gestures, and the source of this effect was localized to the right middle temporal gyrus. This difference is likely to reflect integration processes, as it was reduced in the R language and no-gesture conditions. Our findings provide the first empirical evidence suggesting the functional relationship between gamma band oscillations and higher-level semantic processes in a multisensory setting.

The connectivity signature of co-speech gesture integration: The superior temporal sulcus modulates connectivity between areas related to visual gesture and auditory speech processing

Humans integrate information communicated by speech and gestures. Functional magnetic resonance imaging (fMRI) studies suggest that the posterior superior temporal sulcus (STS) and adjacent gyri are relevant for multisensory integration. However, a connectivity model representing this essential combinatory process is still missing. Here, we used dynamic causal modeling for fMRI to analyze the effective connectivity pattern between middle temporal gyrus (MTG), occipital cortex (OC) and STS associated with auditory verbal, visual gesture-related, and integrative processing, respectively, to unveil the neural mechanisms underlying integration of intrinsically meaningful gestures (e.g., “Thumbs-up gesture”) and corresponding speech.20 participants were presented videos of an actor either performing intrinsic meaningful gestures in the context of German or Russian sentences, or speaking a German sentence without gesture, while performing a content judgment task.The connectivity analyses resulted in a winning model that included bidirectional intrinsic connectivity between all areas. Furthermore, the model included modulations of both connections to the STS (OC→STS; MTG→STS), and non-linear modulatory effects of the STS on bidirectional connections between MTG and OC. Coupling strength in the occipital pathway (OC→STS) correlated with gesture related advantages in task performance, whereas the temporal pathway (MTG→STS) correlated with performance in the speech only condition. Coupling between MTG and OC correlated negatively with subsequent memory performance for sentences of the Gesture-German condition.Our model provides a first step towards a better understanding of speech-gesture integration on network level. It corroborates the importance of the STS during audio-visual integration by showing that this region inhibits direct auditory-visual coupling.

The relationship between co-speech gesture production and macrolinguistic discourse abilities in people with focal brain injury

Brain damage is associated with linguistic deficits and might alter co-speech gesture production. Gesture production after focal brain injury has been mainly investigated with respect to intrasentential rather than discourse-level linguistic processing. In this study, we examined 1) spontaneous gesture production patterns of people with left hemisphere damage (LHD) or right hemisphere damage (RHD) in a narrative setting, 2) the neural structures associated with deviations in spontaneous gesture production in these groups, and 3) the relationship between spontaneous gesture production and discourse level linguistic processes (narrative complexity and evaluation competence). Individuals with LHD or RHD (17 people in each group) and neurotypical controls (n = 13) narrated a story from a picture book. Results showed that increase in gesture production for LHD individuals was associated with less complex narratives and lesions of individuals who produced more gestures than neurotypical individuals overlapped in frontal-temporal structures and basal ganglia. Co-speech gesture production of RHD individuals positively correlated with their evaluation competence in narrative. Lesions of RHD individuals who produced more gestures overlapped in the superior temporal gyrus and the inferior parietal lobule. Overall, LHD individuals produced more gestures than neurotypical individuals. The groups did not differ in their use of different gesture forms except that LHD individuals produced more deictic gestures per utterance than RHD individuals and controls. Our findings are consistent with the hypothesis that co-speech gesture production interacts with macro-linguistic levels of discourse and this interaction is affected by the hemispheric lateralization of discourse abilities.

Effects of linguistic context on the acceptability of co-speech gestures

We ask whether iconic co-speech gestures are judged as more natural by naive participants when their content is entailed by a preceding context, or repeated in the same utterance, or when they contribute new information (i.e., are nontrivial). Our results show, first, that the acceptability of co-speech gestures is not affected by whether they are entailed by a preceding context (they are not “hard presupposition triggers”). In contrast, our second finding is that gestures are affected by content in the same utterance, such that they are judged more acceptable when they reinforce information already present in the proposition. In addition, gestures varied widely in how acceptable they were judged even in the same information contexts, which we take to be an indication that we are just scratching the surface of a more general question about the felicity of co-speech gestures, of which the current study provides a first experimental foundation.

Multimodal Communication in Aphasia: Perception and Production of Co-speech Gestures During Face-to-Face Conversation

The role of nonverbal communication in patients with post-stroke language impairment (aphasia) is not yet fully understood. This study investigated how aphasic patients perceive and produce co-speech gestures during face-to-face interaction, and whether distinct brain lesions would predict the frequency of spontaneous co-speech gesturing. For this purpose, we recorded samples of conversations in patients with aphasia and healthy participants. Gesture perception was assessed by means of a head-mounted eye-tracking system, and the produced co-speech gestures were coded according to a linguistic classification system. The main results are that meaning-laden gestures (e.g., iconic gestures representing object shapes) are more likely to attract visual attention than meaningless hand movements, and that patients with aphasia are more likely to fixate co-speech gestures overall than healthy participants. This implies that patients with aphasia may benefit from the multimodal information provided by co-speech gestures. On the level of co-speech gesture production, we found that patients with damage to the anterior part of the arcuate fasciculus showed a higher frequency of meaning-laden gestures. This area lies in close vicinity to the premotor cortex and is considered to be important for speech production. This may suggest that the use of meaning-laden gestures depends on the integrity of patients’ speech production abilities.

Testing contrastive inferences from suprasegmental features using offline measures

Speakers add modifiers to the extent that they are informative (Grice 1975); studies using the visual world eye-tracking paradigm find that the use of pre-nominal modifiers (short, big) leads listeners to infer the existence of similar objects differing along that same scale (Grodner & Sedivy 2011; Sedivy et al. 1999). In this study, we probe these contrastive inferences using an offline questionnaire, paired with audio/video stimuli to ask whether similar inferences extend to two types of suprasegmental features: prosodic focus and depictive co-speech gestures. Our results suggest that the presence of a scalar adjective robustly leads to contrastive inferences in this offline forced choice paradigm, and that the robustness of the lexical pattern persists even when prosodic focus would indicate otherwise. Prosodic focus does, however, appear to modulate the contrastive effect of a given pre-nominal modifier. We find that the same pragmatic process fails to extend to depictive co-speech gestures, supporting a semantic analysis of these gestures as generally not-at-issue contributions.

Referring strategies in American Sign Language and English (with co-speech gesture): The role of modality in referring to non-nameable objects.

American Sign Language (ASL) and English differ in linguistic resources available to express visual-spatial information. In a referential communication task, we examined the effect of language modality on the creation and mutual acceptance of reference to non-nameable figures. In both languages, description times reduced over iterations and references to the figures' geometric properties ("shape-based reference") declined over time in favor of expressions describing the figures' resemblance to nameable objects ("analogy-based reference"). ASL signers maintained a preference for shape-based reference until the final (sixth) round, while English speakers transitioned toward analogy-based reference by Round 3. Analogy-based references were more time efficient (associated with shorter round description times). Round completion times were longer for ASL than for English, possibly due to gaze demands of the task and/or to more shape-based descriptions. Signers' referring expressions remained unaffected by figure complexity while speakers preferred analogy-based expressions for complex figures and shape-based expressions for simple figures. Like speech, co-speech gestures decreased over iterations. Gestures primarily accompanied shape-based references, but listeners rarely looked at these gestures, suggesting that they were recruited to aid the speaker rather than the addressee. Overall, different linguistic resources (classifier constructions vs. geometric vocabulary) imposed distinct demands on referring strategies in ASL and English.

How event perspective influences speech and co-speech gestures about motion

Speakers of different languages systematically differ in how they package manner and path components of a motion event rendered from a self-motion perspective in both speech and co-speech gesture. Speakers of satellite-framed languages (e.g., German) use a conflated pattern, synthesizing manner and path of motion into a single clause or gesture; while speakers of verb-framed languages (e.g., Spanish) use a separated pattern, expressing manner and path in separate clauses or gestures. In this study, we ask whether the close coupling between speech and gesture observed for self-motion also becomes evident for events rendered from a caused-motion perspective, which show greater use of conflated strategy in speech across different languages. We observed speech and co-speech gestures of adult native Spanish and German speakers (N = 15/language), as they described an animated clip depicting both self- and caused-motion events. Our findings showed that expression of motion in speech systematically varied by event perspective—with greater use of conflated strategy in caused-motion events than in self-motion events. More important, this variability became evident in co-speech gesture in both languages. Our findings show that speech and gesture form a tightly integrated system, jointly reflecting systematic differences not only between languages, but also within a particular language.

Action-Related Speech Modulates Beta Oscillations During Observation of Tool-Use Gestures.

Language and action have been thought of as closely related. Comprehending words or phrases that are related to actions commonly activates motor and premotor areas, and this comprehension process interacts with action preparation and/or execution. However, it remains unclear whether comprehending action-related language interacts with action observation. In the current study, we examined whether the observation of tool-use gesture subjects to interaction with language. In an electroencephalography (EEG) study (n = 20), participants were presented with video clips of an actor performing tool-use (TU, e.g., hammering with a fist) and emblematic (EM, e.g., the thumb up sign for 'good job') gestures accompanied by either comprehensible German (G) or incomprehensible Russian sentences (R). Participants performed a semantic judging task, evaluating whether the co-speech gestures were object- or socially-related. Behavioral results from the semantic task showed faster response for the TU versus EM gestures only in the German condition. For EEG, we found that TU elicited beta power decrease (~ 20Hz) when compared to EM gestures, however this effect was reduced when gestures were accompanied by German instead of Russian sentences. We concluded that the processing of action-related sentences might facilitate gesture observation, in the sense that motor simulation required for TU gestures, as indexed by reduced beta power, was modulated when accompanied by comprehensible German speech. Our results corroborate the functional role of the beta oscillations during perception of hand gestures, and provide novel evidence concerning language-motor interaction.

Gesture projection and cosuppositions

In dynamic theories of presupposition, a trigger pp′ (e.g. it stopped raining) with presupposition p (it rained) and at-issue component p′ (it doesn’t now rain) comes with a requirement that p should be entailed by the local context of pp′. We argue that some co-speech gestures should be analyzed within a presuppositional framework, but with a twist: an expression p co-occurring with a co-speech gesture G with content g comes with the requirement that the local context of p should guarantee that p entails g; we call such assertion-dependent presuppositions ‘cosuppositions’. We show that this analysis can be combined with earlier theories of local contexts to account for complex patterns of gesture projection in quantified and in attitudinal contexts, and we compare our account to two potential alternatives: one based on supervaluations, and one, due to Cornelia Ebert, that treats co-speech gestures as supplements. We argue that the latter is correct, but for ‘post-speech’ gestures (= gestures that come after the expressions they modify), rather than for co-speech gestures.

In which direction to move? Facilitative and interference effects of gestures on problem solver's thinking

ABSTRACTRecent research shows that co-speech gestures can influence gesturers’ thought. This line of research suggests that the influence of gestures is so strong, that it can wash out and reverse an effect of learning. We argue that these findings need a more robust and ecologically valid test, which we provide in this article. Our results support the claim that gestures not only reflect information in our mental representations, but can also influence gesturer's thought by adding action information to one's mental representation during problem solving (Tower of Hanoi). We show, however, that the effect of gestures on subsequent performance is not as strong as previously suggested. As opposed to what previous research indicates, gestures' facilitative effect through learning was not nullified by the potentially interfering effect on subsequent problem-solving performance of incompatible gestures. To conclude, using gestures during problem solving seems to provide more benefits than costs for task performance.

The relationship between handedness and valence: A gesture study.

People with different hand preferences assign positive and negative emotions to different sides of their bodies and produce co-speech gestures with their dominant hand when the content is positive. In this study, we investigated this side preference by handedness in both gesture comprehension and production. Participants watched faceless gesture videos with negative and positive content on eye tracker and were asked to retell the stories after each video. Results indicated no difference in looking preferences regarding being right- or left-handed. Yet, an effect of emotional valence was observed. Participants spent more time looking to the right (actor's left) when the information was positive and to the left (actor's right) when the information was negative. Participants' retelling of stories revealed a handedness effect only for different types of gestures (representational vs beat). Individuals used their dominant hands for beat gestures. For representational gestures, while the right-handers used their right hands more, the left-handers gestured using both hands equally. Overall, the lack of significant difference between handedness and emotional content in both comprehension and production levels suggests that body-specific mental representations may not extend to the conversational level.

Transcranial Magnetic Stimulation over Left Inferior Frontal and Posterior Temporal Cortex Disrupts Gesture-Speech Integration.

Language and action naturally occur together in the form of cospeech gestures, and there is now convincing evidence that listeners display a strong tendency to integrate semantic information from both domains during comprehension. A contentious question, however, has been which brain areas are causally involved in this integration process. In previous neuroimaging studies, left inferior frontal gyrus (IFG) and posterior middle temporal gyrus (pMTG) have emerged as candidate areas; however, it is currently not clear whether these areas are causally or merely epiphenomenally involved in gesture-speech integration. In the present series of experiments, we directly tested for a potential critical role of IFG and pMTG by observing the effect of disrupting activity in these areas using transcranial magnetic stimulation in a mixed gender sample of healthy human volunteers. The outcome measure was performance on a Stroop-like gesture task (Kelly et al., 2010a), which provides a behavioral index of gesture-speech integration. Our results provide clear evidence that disrupting activity in IFG and pMTG selectively impairs gesture-speech integration, suggesting that both areas are causally involved in the process. These findings are consistent with the idea that these areas play a joint role in gesture-speech integration, with IFG regulating strategic semantic access via top-down signals acting upon temporal storage areas.SIGNIFICANCE STATEMENT Previous neuroimaging studies suggest an involvement of inferior frontal gyrus and posterior middle temporal gyrus in gesture-speech integration, but findings have been mixed and due to methodological constraints did not allow inferences of causality. By adopting a virtual lesion approach involving transcranial magnetic stimulation, the present study provides clear evidence that both areas are causally involved in combining semantic information arising from gesture and speech. These findings support the view that, rather than being separate entities, gesture and speech are part of an integrated multimodal language system, with inferior frontal gyrus and posterior middle temporal gyrus serving as critical nodes of the cortical network underpinning this system.

Aligning speech and co-speech gesture in a constraint-based grammar

This paper concerns the form-meaning mapping of communicative actions consisting of speech and improvised co-speech gestures. Based on the findings of previous cognitive and computational approaches, we advance a new theory in which this form-meaning mapping is analysed in a constraint-based grammar. Motivated by observations in naturally occurring examples, we propose several construction rules, which use linguistic form, gesture form and their relative timing to constrain the derivation of a single speech-gesture syntax tree, from which a meaning representation can be composed via standard methods for semantic composition. The paper further reports on implementing these speech-gesture construction rules within the English Resource Grammar (Copestake and Flickinger 2000). Since gestural form often underspecifies its meaning, the logical formulae that are composed via syntax are underspecified so that current models of the semantics/pragmatics interface support the range of possible interpretations of the speech-gesture act in its context of use.

Do Children Understand Iconic Gestures About Events as Early as Iconic Gestures About Entities?

Children can understand iconic co-speech gestures that characterize entities by age 3 (Stanfield et al. in J Child Lang 40(2):1-10, 2014; e.g., "I'm drinking" [Formula: see text] tilting hand in C-shape to mouth as if holding a glass). In this study, we ask whether children understand co-speech gestures that characterize events as early as they do so for entities, and if so, whether their understanding is influenced by the patterns of gesture production in their native language. We examined this question by studying native English speaking 3- to 4 year-old children and adults as they completed an iconic co-speech gesture comprehension task involving motion events across two studies. Our results showed that children understood iconic co-speech gestures about events at age 4, marking comprehension of gestures about events one year later than gestures about entities. Our findings also showed that native gesture production patterns influenced children's comprehension of gestures characterizing such events, with better comprehension for gestures that follow language-specific patterns compared to the ones that do not follow such patterns-particularly for manner of motion. Overall, these results highlight early emerging abilities in gesture comprehension about motion events.

The effect of verbal working memory load in speech/gesture integration processing

The effect of verbal working memory load in speech/gesture integration processing