Viewpoint Gestures Research Articles

Seeing first person changes gesture but saying first person does not

Abstract Events with a motor action component (e.g., handling an object) tend to evoke gestures from the point of view of a character (character viewpoint, or CVPT) while events with a path component (moving through space) tend to evoke gestures from the point of view of an observer (observer viewpoint, or OVPT). Events that combine both components (e.g., rowing a boat across a lake) seem to evoke both types of gesture, but it is unclear why narrators use one or the other. We carry out two manipulations to explore whether gestural viewpoint can be manipulated. Participants read a series of stories and retold them in two conditions. In the image condition, story sentences were presented with images from either the actor’s perspective (actor version) or the observer’s perspective (observer version). In the linguistic condition, the same sentences were presented in either the second person (you…) or the third person point of view (h/she…). The second person led participants to use the first person (I) in retelling. Gestures produced during retelling were coded as CVPT or OVPT. Participants produced significantly more CVPT gestures after seeing images from the point of view of an actor, but the linguistic manipulation did not affect viewpoint in gesture. Neither manipulation affected overall gesture rate, or co-occurring speech. We relate these findings to frameworks in which motor action and mental imagery are linked to viewpoint in gesture.

Viewpoint and stance in gesture: How a potential taboo topic may influence gestural viewpoint in recounting films

This study investigates whether the stance narrators take toward a potential taboo topic or action influences their gestural viewpoint strategies in multimodal film retellings. The analysis focuses on (a) how gesturally expressed viewpoints interact with verbal stance markers in providing a narrow or wide scope of interpretation; and (b) how multimodal discourse practices may reveal a taboo-maintaining or a taboo-ignoring stance on the part of the narrator.Thirty bilingual native speakers of both Luganda and Ugandan English were asked to recount, once in each language, a Ugandan short film revolving around the topic of adultery. Analyses revealed that gestural viewpoints varied depending on the verbally expressed evaluation of the potential taboo film segment. Positive verbal evaluation was accompanied by imitative character viewpoint gestures, i.e., the narrator included himself in the narrated event. By contrast, the semantic features displayed via observer viewpoint gestures yielded more possible interpretations and co-occurred with paraphrasing or moral judgment. Narrator viewpoint gestures correlated with literal reference highlighting the obviousness of the described actions. The term distancing narrator viewpoint is introduced to capture gestures that create a spatial distance between the narrator and the narrated event in the context of negatively framed verbal utterances.

Discourse Reference Is Bimodal: How Information Status in Speech Interacts with Presence and Viewpoint of Gestures

ABSTRACTSpeakers use speech and gestures to represent referents in discourse. Depending on referents’ information status, in speech speakers will vary richness of expression (e.g., lexical noun phrase [NP]/pronoun), nominal definiteness (indefinite/definite), and grammatical role (subject/object). This study tested whether these three linguistic markers of information status interact with presence of gestures and gestural viewpoint (observer/character). The results show that gestures are more frequent with less accessible referents expressed with richer spoken forms but that richness of expression does not interact with viewpoint. In contrast, nominal definiteness and grammatical role interact with both presence and viewpoint of gestures. Gestures occur mainly with indefinite lexical NPs and objects. Character viewpoint gestures occur mainly with indefinite lexical NPs and objects plus predicates. The results shed light on when and how speakers use gestures in connected discourse and specifically highlight the discursive function of gestural viewpoint.

How politicians express different viewpoints in gesture and speech simultaneously

AbstractPolitical speeches are a prime example of how discourse often requires speakers to convey multiple competing viewpoints, both their own and others’. Cognitive linguists have shown how, in speech, speakers express viewpoint through individual choices at the lexical and grammatical level. Recently, cognitive linguists have also shown that speakers express viewpoint using speech-accompanying gestures. To date, the study of viewpoint expression has focused on cases where speakers deliver the same viewpoint across modalities. By examining the persuasive uses of gesture in Obama’sA More Perfect Unionspeech, I show how speakers can communicate multiple different viewpoints across gesture and speech, simultaneously. There are moments when Obama expresses his opponents’ viewpoint in speech, while framing them in terms of his own viewpoint in gesture, and vice versa. I discuss how the deviation of viewpoints across modalities provides key insights into multimodal cognition, with respect to working memory, metaphor, and persuasion. Specifically, I argue that, as an implicit medium, gesture allows speakers to inject viewpoint into the uptake of speech, below the conscious radar of recipients, and I discuss how this rhetorical capacity is evolving as a result of communication technologies.

Who Did What to Whom? Children Track Story Referents First in Gesture.

Children achieve increasingly complex language milestones initially in gesture or in gesture+speech combinations before they do so in speech, from first words to first sentences. In this study, we ask whether gesture continues to be part of the language-learning process as children begin to develop more complex language skills, namely narratives. A key aspect of narrative development is tracking story referents, specifying who did what to whom. Adults track referents primarily in speech by introducing a story character with a noun phrase and then following the same referent with a pronoun-a strategy that presents challenges for young children. We ask whether young children can track story referents initially in communications that combine gesture and speech by using character viewpoint in gesture to introduce new story characters, before they are able to do so exclusively in speech using nouns followed by pronouns. Our analysis of 4- to 6-year-old children showed that children introduced new characters in gesture+speech combinations with character viewpoint gestures at an earlier age than conveying the same referents exclusively in speech with the use of nominal phrases followed by pronouns. Results show that children rely on viewpoint in gesture to convey who did what to whom as they take their first steps into narratives.

Separating viewpoint from mode of representation in iconic co-speech gestures: insights from Danish narratives

abstractDuring narrative retelling, speakers shift between different viewpoints to reflect how they conceptualize the events that unfolded. These viewpoints can be indicated through gestural means as well as through verbal ones. Studies of co-speech gestures have inferred viewpoint from gesture form, i.e., how entities are mapped onto the (primarily manual) articulators, but the merits of this approach have not been discussed. The present study argues that viewpoint is more than gestural form. Despite connections between the two, many other factors may influence a gesture’s form. Assessing viewpoint from gesture form alone limits the applicability of gestural viewpoint as a window onto speakers’ event conceptualization and introduces unnecessary differences in the categorization of viewpoint across gestures types. The present study examines iconic co-speech gestures in Danish narratives, and makes explicit the means used to infer gestural viewpoint. The approach advocated here ensures that the notion of viewpoint can be applied in a principled way to all or most iconic gestures.

A third-person perspective on co-speech action gestures in Parkinson's disease

A combination of impaired motor and cognitive function in Parkinson's disease (PD) can impact on language and communication, with patients exhibiting a particular difficulty processing action verbs. Co-speech gestures embody a link between action and language and contribute significantly to communication in healthy people. Here, we investigated how co-speech gestures depicting actions are affected in PD, in particular with respect to the visual perspective—or the viewpoint – they depict. Gestures are closely related to mental imagery and motor simulations, but people with PD may be impaired in the way they simulate actions from a first-person perspective and may compensate for this by relying more on third-person visual features. We analysed the action-depicting gestures produced by mild-moderate PD patients and age-matched controls on an action description task and examined the relationship between gesture viewpoint, action naming, and performance on an action observation task (weight judgement). Healthy controls produced the majority of their action gestures from a first-person perspective, whereas PD patients produced a greater proportion of gestures produced from a third-person perspective. We propose that this reflects a compensatory reliance on third-person visual features in the simulation of actions in PD. Performance was also impaired in action naming and weight judgement, although this was unrelated to gesture viewpoint. Our findings provide a more comprehensive understanding of how action-language impairments in PD impact on action communication, on the cognitive underpinnings of this impairment, as well as elucidating the role of action simulation in gesture production.

Signers and Co‐speech Gesturers Adopt Similar Strategies for Portraying Viewpoint in Narratives

Gestural viewpoint research suggests that several dimensions determine which perspective a narrator takes, including properties of the event described. Events can evoke gestures from the point of view of a character (CVPT), an observer (OVPT), or both perspectives. CVPT and OVPT gestures have been compared to constructed action (CA) and classifiers (CL) in signed languages. We ask how CA and CL, as represented in ASL productions, compare to previous results for CVPT and OVPT from English-speaking co-speech gesturers. Ten ASL signers described cartoon stimuli from Parrill (2010). Events shown by Parrill to elicit a particular gestural strategy (CVPT, OVPT, both) were coded for signers' instances of CA and CL. CA was divided into three categories: CA-torso, CA-affect, and CA-handling. Signers used CA-handling the most when gesturers used CVPT exclusively. Additionally, signers used CL the most when gesturers used OVPT exclusively and CL the least when gesturers used CVPT exclusively.

A tale of two hands: children's early gesture use in narrative production predicts later narrative structure in speech.

Speakers of all ages spontaneously gesture as they talk. These gestures predict children's milestones in vocabulary and sentence structure. We ask whether gesture serves a similar role in the development of narrative skill. Children were asked to retell a story conveyed in a wordless cartoon at age five and then again at six, seven, and eight. Children's narrative structure in speech improved across these ages. At age five, many of the children expressed a character's viewpoint in gesture, and these children were more likely to tell better-structured stories at the later ages than children who did not produce character-viewpoint gestures at age five. In contrast, framing narratives from a character's perspective in speech at age five did not predict later narrative structure in speech. Gesture thus continues to act as a harbinger of change even as it assumes new roles in relation to discourse.

Gestural Viewpoint Signals Referent Accessibility

The tracking of entities in discourse is known to be a bimodal phenomenon. Speakers achieve cohesion in speech by alternating between full lexical forms, pronouns, and zero anaphora as they track referents. They also track referents in co-speech gestures. In this study, we explored how viewpoint is deployed in reference tracking, focusing on representations of animate entities in German narrative discourse. We found that gestural viewpoint systematically varies depending on discourse context. Speakers predominantly use character viewpoint in maintained contexts and observer viewpoint in reintroduced contexts. Thus, gestural viewpoint seems to function as a cohesive device in narrative discourse. The findings expand on and provide further evidence for the coordination between speech and gesture on the discourse level that is crucial to understanding the tight link between the two modalities.

Meaningful shifts

This review describes the primary strategies used to express changes in conceptual viewpoint (Parrill, 2012) in co-speech gesture and sign language. We describe the use of the face, eye gaze, body orientation and hands to represent these differences in viewpoint, focusing particularly on McNeill’s (1992) division of iconic gestures into observer versus character viewpoint gestures, and on the situations in which they occur. We also draw a parallel between the strategies used in co-speech gesture and those used in different signed languages (see Cormier, Quinto-Pozos, Sevcikova, & Schembri, 2012), and suggest possibilities for further research in this area.

An experimental investigation of how addressee feedback affects co-speech gestures accompanying speakers’ responses

There is evidence that co-speech gestures communicate information to addressees and that they are often communicatively intended. However, we still know comparatively little about the role of gestures in the actual process of communication. The present study offers a systematic investigation of speakers’ gesture use before and after addressee feedback. The findings show that when speakers responded to addressees’ feedback gesture rate remained constant when this feedback encouraged clarification, elaboration or correction. However, speakers gestured proportionally less often after feedback when providing confirmatory responses. That is, speakers may not be drawing on gesture in response to addressee feedback per se, but particularly with responses that enhance addressees’ understanding. Further, the large majority of speakers’ gestures changed in their form. They tended to be more precise, larger, or more visually prominent after feedback. Some changes in gesture viewpoint were also observed. In addition, we found that speakers used deixis in speech and gaze to increase the salience of gestures occurring in response to feedback. Speakers appear to conceive of gesture as a useful modality in redesigning utterances to make them more accessible to addressees. The findings further our understanding of recipient design and co-speech gestures in face-to-face dialogue.

Effects of input modality on speech–gesture integration

Do the gestures that accompany narrations produced after watching videos differ from those that accompany narrations produced after reading texts? Building on previous work by Hostetter and Hopkins, we ask whether participants gesture less after reading texts, whether participants produce different types of gestures after reading texts, whether the same motion event features (path and manner) are present in gesture in both cases, and whether gestural viewpoint (character or observer viewpoint) is impacted by input modality. We find that while participants produce longer narrations (and thus more gestures overall) after watching videos, gesture rate (number of gestures per word), gesture type, motion event feature encoding, and viewpoint are not affected by input modality. We comment on the implications of our findings for embodied theories of language.

Gesture viewpoint in Japanese and English

Abundant evidence across languages, structures, proficiencies, and modalities shows that properties of first languages influence performance in second languages. This paper presents an alternative perspective on the interaction between established and emerging languages within second language speakers by arguing that an L2 can influence an L1, even at relatively low proficiency levels. Analyses of the gesture viewpoint employed in English and Japanese descriptions of motion events revealed systematic between-language and within-language differences. Monolingual Japanese speakers used significantly more Character Viewpoint than monolingual English speakers, who predominantly employed Observer Viewpoint. In their L1 and their L2, however, native Japanese speakers with intermediate knowledge of English patterned more like the monolingual English speakers than their monolingual Japanese counterparts. After controlling for effects of cultural exposure, these results offer valuable insights into both the nature of cross-linguistic interactions within individuals and potential factors underlying gesture viewpoint.

An experimental investigation of some properties of individual iconic gestures that mediate their communicative power.

It has been hypothesized that the iconic hand gestures that accompany talk communicate important semantic information. This research tests whether some gestures, in the absence of speech, are more communicative than others and considers what properties of gestures might affect their communicative power. Our research found that the communicative power of gestures does vary greatly, and that this is significantly affected by the viewpoint from which a gesture is generated, with character viewpoint gestures being more communicative than observer viewpoint gestures. It has also been suggested that gesture viewpoint is connected with the transitivity of the clause that it accompanies, and it was found in our study that respondents appeared to obtain syntactic information about the associated clause from the gesture. This conclusion was based on the observation that when respondents attempted to report what information was contained in gestures, viewed in the absence of speech, there was a significantly higher proportion of transitive structures in their answers after they had watched character viewpoint gestures compared with observer viewpoint gestures. Communication about the syntax of the accompanying clause might thus be a critical, but thus far neglected, aspect of the role of gestures in everyday talk.