Character Viewpoint Gestures Research Articles

The interface between grammar and bodily enactment in ASL and English

Abstract Users of signed and spoken languages regularly engage bodily enactment (commonly referred to as constructed action [CA] for signers and character viewpoint gestures [CVPT] for speakers) for the creation of meaning, but comparatively few studies have addressed how linguistic grammar interfaces with such gestural depictive devices across language modalities. CVPT gestures have been shown to co-occur with spoken language transitive verbs, and when a reference is definite or more accessible in the discourse. In sign, CA often alternates sequentially with fully conventionalized signs. In both CVPT and CA demonstrations, syntactic and pragmatic factors appear to be important. In this work, we consider these patterns by examining short retellings of video-based elicitation stimuli (silent-movie segments) from 10 deaf users of ASL (American Sign Language) and 20 hearing speakers of English. We describe examples of signs and words that co-occur with or precede specific instances of CA and CVPT. We also examine distributions and degrees of enactment across participants in order to consider the question of gesture threshold (Hostetter and Alibali, 2008, 2019). We provide various examples of how gestural material interfaces with linguistic grammar, which has implications for syntactic theory and possible grammatical constraints on such communicative devices.

Viewpoint and stance in gesture: How a potential taboo topic may influence gestural viewpoint in recounting films

This study investigates whether the stance narrators take toward a potential taboo topic or action influences their gestural viewpoint strategies in multimodal film retellings. The analysis focuses on (a) how gesturally expressed viewpoints interact with verbal stance markers in providing a narrow or wide scope of interpretation; and (b) how multimodal discourse practices may reveal a taboo-maintaining or a taboo-ignoring stance on the part of the narrator.Thirty bilingual native speakers of both Luganda and Ugandan English were asked to recount, once in each language, a Ugandan short film revolving around the topic of adultery. Analyses revealed that gestural viewpoints varied depending on the verbally expressed evaluation of the potential taboo film segment. Positive verbal evaluation was accompanied by imitative character viewpoint gestures, i.e., the narrator included himself in the narrated event. By contrast, the semantic features displayed via observer viewpoint gestures yielded more possible interpretations and co-occurred with paraphrasing or moral judgment. Narrator viewpoint gestures correlated with literal reference highlighting the obviousness of the described actions. The term distancing narrator viewpoint is introduced to capture gestures that create a spatial distance between the narrator and the narrated event in the context of negatively framed verbal utterances.

Discourse Reference Is Bimodal: How Information Status in Speech Interacts with Presence and Viewpoint of Gestures

ABSTRACTSpeakers use speech and gestures to represent referents in discourse. Depending on referents’ information status, in speech speakers will vary richness of expression (e.g., lexical noun phrase [NP]/pronoun), nominal definiteness (indefinite/definite), and grammatical role (subject/object). This study tested whether these three linguistic markers of information status interact with presence of gestures and gestural viewpoint (observer/character). The results show that gestures are more frequent with less accessible referents expressed with richer spoken forms but that richness of expression does not interact with viewpoint. In contrast, nominal definiteness and grammatical role interact with both presence and viewpoint of gestures. Gestures occur mainly with indefinite lexical NPs and objects. Character viewpoint gestures occur mainly with indefinite lexical NPs and objects plus predicates. The results shed light on when and how speakers use gestures in connected discourse and specifically highlight the discursive function of gestural viewpoint.

Multimodal character viewpoint in quoted dialogue sequences

We investigate the multimodal production of character viewpoint in spoken American English narratives by performing complementary qualitative and quantitative analyses of two quoted dialogues, focusing on the storyteller’s use of character viewpoint gestures, character intonation, character facial expression, spatial orientation and gaze. A micro-analysis revealed that the extent of multimodal articulation depends on (i) the quoted speaker, with different multimodal articulatory patterns found for quotes by the speaker’s past self vs. a third-person character, and (ii) the position of the quoted utterance within the quoted dialogue, with mid-dialogue utterances garnering less co-articulation than initial or final utterances within the quoted dialogue. We further investigated these observations using a quantitative approach, which was based on generalized additive modeling (GAM). The GAM analysis revealed different multimodal patterns for each quoted character, as indicated by the number of co-produced multimodal articulators. These patterns were found to hold regardless of the quote’s position within the narrative. We discuss these findings with respect to previous work on multimodal quotation.This article is part of the special collection: Perspective Taking

Who Did What to Whom? Children Track Story Referents First in Gesture.

Children achieve increasingly complex language milestones initially in gesture or in gesture+speech combinations before they do so in speech, from first words to first sentences. In this study, we ask whether gesture continues to be part of the language-learning process as children begin to develop more complex language skills, namely narratives. A key aspect of narrative development is tracking story referents, specifying who did what to whom. Adults track referents primarily in speech by introducing a story character with a noun phrase and then following the same referent with a pronoun-a strategy that presents challenges for young children. We ask whether young children can track story referents initially in communications that combine gesture and speech by using character viewpoint in gesture to introduce new story characters, before they are able to do so exclusively in speech using nouns followed by pronouns. Our analysis of 4- to 6-year-old children showed that children introduced new characters in gesture+speech combinations with character viewpoint gestures at an earlier age than conveying the same referents exclusively in speech with the use of nominal phrases followed by pronouns. Results show that children rely on viewpoint in gesture to convey who did what to whom as they take their first steps into narratives.

Multimodal quotation: Role shift practices in spoken narratives

This study investigates how speakers of American English use multimodal articulation when quoting characters in personal narratives. We use the concept of role shift, adapted from signed languages, where it refers to a device used to represent one or more characters with one or more bodily articulators, to describe multimodal role shift practices. In a regression analysis, four bodily articulators were found to predict the impression of a role shift: character intonation, character facial expressions, character viewpoint gestures, and changes in body orientation; gaze was not a significant predictor. Most of the 704 quotes in our data are accompanied by activation of two or three articulators (55.3%) and very few (2.6%) occur without any of the articulators we have annotated. The extent of multimodal articulation depends on the type of quoted utterance: quotations of actual, witnessed speech events tend to garner fewer articulators than constructed (‘fictive interaction’) quotations. These findings demonstrate that speakers, like signers, use a range of bodily articulators when they take on another's role in quotation and thus underpin the importance of investigating the systematic use of the visual modality in quotation and, more generally, in ordinary interaction.

Multimodal analysis of quotation in oral narratives

AbstractWe investigate direct speech quotation in informal oral narratives by analyzing the contribution of bodily articulators (character viewpoint gestures, character facial expression, character intonation, and the meaningful use of gaze) in three quote environments, or quote sequences – single quotes, quoted monologues and quoted dialogues – and in initial vs. non-initial position within those sequences. Our analysis draws on findings from the linguistic and multimodal realization of quotation, where multiple articulators are often observed to be co-produced with single direct speech quotes (e.g. Thompson & Suzuki 2014), especially on the so-called left boundary of the quote (Sidnell 2006). We use logistic regression to model multimodal quote production across and within quote sequences, and find unique sets of multimodal articulators accompanying each quote sequence type. We do not, however, find unique sets of multimodal articulators which distinguish initial from non-initial utterances; utterance position is instead predicted by type of quote and presence of a quoting predicate. Our findings add to the growing body of research on multimodal quotation, and suggest that the multimodal production of quotation is more sensitive to the number of characters and utterances which are quoted than to the difference between introducing and maintaining a quoted characters’ perspective.

Meaningful shifts

This review describes the primary strategies used to express changes in conceptual viewpoint (Parrill, 2012) in co-speech gesture and sign language. We describe the use of the face, eye gaze, body orientation and hands to represent these differences in viewpoint, focusing particularly on McNeill’s (1992) division of iconic gestures into observer versus character viewpoint gestures, and on the situations in which they occur. We also draw a parallel between the strategies used in co-speech gesture and those used in different signed languages (see Cormier, Quinto-Pozos, Sevcikova, & Schembri, 2012), and suggest possibilities for further research in this area.

Lexicalisation and de-lexicalisation processes in sign languages: Comparing depicting constructions and viewpoint gestures

In this paper, we compare so-called “classifier” constructions in signed languages (which we refer to as “depicting constructions”) with comparable iconic gestures produced by non-signers. We show clear correspondences between entity constructions and observer viewpoint gestures on the one hand, and handling constructions and character viewpoint gestures on the other. Such correspondences help account for both lexicalisation and de-lexicalisation processes in signed languages and how these processes are influenced by viewpoint. Understanding these processes is crucial when coding and annotating natural sign language data.

Viewpoint in speech–gesture integration: Linguistic structure, discourse structure, and event structure

We examine a corpus of narrative data to determine which types of events evoke character viewpoint gestures, and which evoke observer viewpoint gestures. We consider early claims made by McNeill (1992) that character viewpoint tends to occur with transitive utterances and utterances that are causally central to the narrative. We argue that the structure of the event itself must be taken into account: there are some events that cannot plausibly evoke both types of gesture. We show that linguistic structure (transitivity), event structure (visuo-spatial and motoric properties), and discourse structure all play a role. We apply these findings to a recent model of embodied language production, the Gestures as Simulated Action framework.

Metaphor explanation attenuates the right-hand preference for depictive co-speech gestures that imitate actions

Differential activation levels of the two hemispheres due to hemispheric specialization for various linguistic processes might determine hand choice for co-speech gestures. To test this hypothesis, we compared hand choices for gesturing in 20 healthy right-handed participants during explanation of metaphorical vs. non-metaphorical meanings, on the assumption that metaphor explanation enhances the right hemisphere contribution to speech production. Hand choices were analyzed separately for: depictive gestures that imitate action (“character viewpoint gestures,” [McNeill, D. (1992). Hand and mind. What gestures reveal about thought. Chicago: University of Chicago Press.]), depictive gestures that express motion, relative locations, and shape (“observer viewpoint gestures”), and “abstract deictic gestures.” It was found that the right-hand over left-hand preference was significantly weaker in the metaphor condition than in the non-metaphor conditions for depictive gestures that imitated action. Findings suggest that the activation of the right hemisphere in the metaphor condition reduces the likelihood of left hemisphere generation of gestures that imitate action, thus attenuating the right-hand preference.

An experimental investigation of the role of different types of iconic gesture in communication

Respondents, who had either seen or not seen a sample of the iconic gestures that encoders produce when narrating a story, answered questions about the original story and it was found that the overall accuracy score for respondents who saw the iconic gestures in addition to hearing the speech was 56.8% compared to 48.6% for speech only. This was a highly reliable effect and suggests that iconic gestures are indeed communicative. Character viewpoint gestures were also significantly more communicative than observer viewpoint gestures particularly about the semantic feature relative position, but the observer viewpoint gestures were effective at communicating information, particularly about the semantic features speed and shape. There were no significant correlations between the amount of information that gestures added to speech and the amount they conveyed in its absence, which suggests that the relationship between speech and gesture is not fixed but variable. The implications of this research for our fundamental conception of iconic gestures are considered.

An experimental investigation of some properties of individual iconic gestures that mediate their communicative power.

It has been hypothesized that the iconic hand gestures that accompany talk communicate important semantic information. This research tests whether some gestures, in the absence of speech, are more communicative than others and considers what properties of gestures might affect their communicative power. Our research found that the communicative power of gestures does vary greatly, and that this is significantly affected by the viewpoint from which a gesture is generated, with character viewpoint gestures being more communicative than observer viewpoint gestures. It has also been suggested that gesture viewpoint is connected with the transitivity of the clause that it accompanies, and it was found in our study that respondents appeared to obtain syntactic information about the associated clause from the gesture. This conclusion was based on the observation that when respondents attempted to report what information was contained in gestures, viewed in the absence of speech, there was a significantly higher proportion of transitive structures in their answers after they had watched character viewpoint gestures compared with observer viewpoint gestures. Communication about the syntax of the accompanying clause might thus be a critical, but thus far neglected, aspect of the role of gestures in everyday talk.