How are speech and gesture related?

Abstract

How are Speech and Gesture Related? Hanako Yoshida (hayoshid@indiana.edu) Linda B. Smith (smith4@indiana.edu) Raedy M. Ping (rping@indiana.edu) Elizabeth L. Davis (elldavis@indiana.edu) Psychology Department, Indiana University, 1101 E. 10 th Street Bloomington, IN 47405-7007 USA Abstract People gesture when they speak. Despite considerable attention from a variety of disciplines, the precise nature of the relation between gesture, speech and thought has remained elusive. The research reported here considers two very different hypotheses about the fundamental relationship. By one account, gesture is a consequence of physiological arousal. By another account, gesture use reflects more cognitive processes and is strongly linked to mental representation. This paper seeks the mechanism underlying the link between gesture and speech by showing that both mental representation and physiological arousal are reflected in our gesture use. Introduction Communication is the activity of transmitting information about things in the world. In everyday life, we refer to things by employing a number of communicative tools. Language, gestures, facial expressions, and non-linguistic vocalizations are often considered communicative tools. Interestingly, we integrate multiple modalities in our communication and indeed, past research suggests a tight link between modalities, and particularly between speech and gesture. This study is motivated by two approaches to the study of gesture. McNeill (1992), Kita (2000) and McNeill & Duncan (2000) consider the relationship between the specific forms of gesture and speech, specifically between iconic gesture and speech. An iconic gesture is one in which the speed, motion, or shape of the gesturing hand resembles the meaning being conveyed. McNeil and colleagues consider the emergence of iconicity in gesture as natural, and due to the embodied and imagistic nature of thought. Another approach, however, considers gesture as a consequence not of meaning, but of arousal. Schwarts & Black, (1996) and Iverson & Thelen (1999) particularly suggest that gestures can be explained as an overflow of one’s excitement, which is physiologically produced by speech. The idea here is that speech is a motor program, and as the speaker is aroused—by topic or by the very act of speaking—that energy overflows and is evident in other bodily movements. This present research is concerned with one flashpoint in these accounts: the idea that gesture use fluctuates. At present, the variability in gesture use and in kind of gesturers is not well explained. Yet this very variability should be the key to underlying mechanism. We demonstrate the bi-directional relationship between gesture, thoughts and arousal by confirming that both mental representation and arousal are reflected in and influence iconic gesture use. We experimentally manipulate speech rate as a means of increasing the “motor overflow” from speech. Although speech rate is at best an indirect marker of arousal, we believe it is a good starting point for a mechanistic explanation of variability in gesture use. Iconicity Communicative tools in general range from conventional forms that are more arbitrary, to less conventional forms that are more iconic. An example of the conventional forms in traditional verbal languages would be the word “dog” which can hold the meaning only when both senders and receivers are knowledgeable about the rule indicating the label—‘a dog’ means a dog. This form can also be seen in written language when one spells out letters, d, o, g to refer to a dog. Again, receivers can make sense out of the particular combination of letters only by knowing the convention, that the order and the combination of letters are signifying a dog. People also refer iconically. For example, in verbal language use, one might imitate a sound of a dog to refer to a dog (e.g., “woof, woof!”), and in American Sign Language the word for a dog consists of a pat on one’s leg and a snap of one’s fingers as if calling a dog. People also often gesture iconically (Kita, 2000). For example, a speaker might make a circle with one’s hands to describe shape and/or the size of a plate to which the person refers. Indeed, even nonverbal primates may gesture to convey meaning. For example, Tanner and Byrne (1996) recently reported that several observed gestures of gorillas are iconic. Another example that highlights both natural emergence and ease of iconic gesture use is an observation made by Goldin-Meadow and Feldman (1977). They observed