Pitch Gestures Research Articles

Production rather than observation: comparison between the roles of embodiment and conceptual metaphor in L2 lexical tone learning

BackgroundIt remains unclear how embodiment and conceptual metaphor play a role in second language (L2) lexical tone learning. AimsThe present study aimed to examine the roles of pitch gesture production (PP), pitch feature observation (PO), and word-picture association (WA) approach in L2 lexical tone learning. SampleParticipants were 90 undergraduate students with Mandarin as their native language. MethodsParticipants learned Thai lexical tones via the three approaches and completed tone discrimination, tone identification, and word-picture matching tasks. ResultsThe PP performed better than the PO and WA in discriminating between and identifying specific tones. The PP was more accurate than the WA in word-picture matching. ConclusionsThe pitch gesture production's embodiment was superior to the pitch feature observation's conceptual metaphor in learning L2 lexical tones. However, its role was affected by lexical tones' pitch features, test tasks, and learners' tonal experience.

Observing gesture at learning enhances subsequent phonological and semantic processing of L2 words: An N400 study

This study employed the N400 event-related potential (ERP) to investigate how observing different types of gestures at learning affects the subsequent processing of L2 Mandarin words differing in lexical tone by L1 English speakers. The effects of pitch gestures conveying lexical tones (e.g., upwards diagonal movements for rising tone), semantic gestures conveying word meanings (e.g., waving goodbye for to wave), and no gesture were compared. In a lexical tone discrimination task, larger N400s for Mandarin target words mismatching vs. matching Mandarin prime words in lexical tone were observed for words learned with pitch gesture. In a meaning discrimination task, larger N400s for English target words mismatching vs. matching Mandarin prime words in meaning were observed for words learned with pitch and semantic gesture. These findings provide the first neural evidence that observing gestures during L2 word learning enhances subsequent phonological and semantic processing of learned L2 words.

Effects of prosodic structure on the temporal organization of speech and co-speech gestures

Although many studies have observed a close relationship between prosodic structure and co-speech gestures, little is understood about cross-modal gestural coordination. The present study examines the relationship between articulatory and co-speech gestures at prosodic boundaries and under prominence, focusing on non-referential manual and eyebrow beat gestures in Korean, a language in which co-speech gestures are virtually unexplored. This study hypothesizes that prosodic structure systematically governs the production of both speech and co-speech gestures and their temporal organization. Multimodal signals of a story reading were collected from eight speakers (5F, 3M). The lips, tongue, and eyebrows were point-tracked using EMA, and the vertical manual movements obtained from a video recording were auto-tracked using a geometrical centroid tracking method. Measurements taken included the duration of intervals from the timepoint of concurrent beat gesture onset and target to 1) consonant gesture onset and target, 2) vowel gesture onset and target, 3) pitch gesture (F0) onset and target, and 4) phrasal boundaries. Results reveal systematic inter-articulator coordination patterns, suggesting that beat gestures co-occurring with speech gestures are recruited to signal information grouping and highlighting. The findings are discussed with reference to the nature of prosodic representation and models of speech planning. [Work supported by NSF.]

Effectiveness, Attractiveness, and Emotional Response to Voice Pitch and Hand Gestures in Public Speaking

Some non-verbal cues, such as voice pitch and gestures, can influence the individual's emotional response to different messages. For this reason, non-verbal communication can be understood as the language of emotion, while the content is the language of ideas. But the research question here is what voice pitch and hand gesture strategies are more effective, attractive, and elicit a stronger emotional response. To answer this question, this study analyzes some pitch and hand gesture strategies in public discourses. In the experiment, 120 participants were exposed to different public presentations with three conditions for voice pitch (few, medium, and many variations) and three for hand gestures (smooth, medium, and strong intensity). Then, they rated the effectiveness and attractiveness levels and self-report arousal and valence while their emotional response was registered. Physiological arousal was measured with electrodermal activity (EDA) and valence with facial expression recognition. Participants perceived the medium pitch variations and the medium intensity of gestures as the most effective and attractive strategies with the highest self-reported arousal and valence. The combination that elicited the strongest autonomic arousal was medium pitch variations-strong gestures. Finally, the most positive emotional valence was for many pitch variations and medium gestures and the happiest for the combination many-strong.

Multimodal training using pitch gesture improves Mandarin tone recognition for children with cochlear implant

Prelinguallydeafened Mandarin-speaking children with cochlear implants (CI) encounter a significant challenge in perceiving lexical tones accurately due to their limited language experience and insufficient pitch information provided by CI devices. To facilitate their tonal perception, we examined the role of pitch gestures in the training program, which was reported to be beneficial in acquiring Mandarin lexical tones for foreign learners of Chinese. In the current study, 18 prelingually deafened preschoolers with CI were recruited in Shanghai. They were randomly assigned to two groups. The experimental group was trained with audio and pitch gestures, and the control group was trained with audio only. Three tone-identification tests were conducted before, in the middle, and after the eight training sessions. Although the two groups had identical performance before the training, the experimental group demonstrated significantly better performance than the control group after training sessions, especially in noise conditions. The results thus showed that multimodal training with pitch gestures improved the tone recognition ability of CI children more than sheer auditory training. Our findings offer more evidence to support that learning to perceive lexical tones can be facilitated by multimodal cues and also provide important implications for optimizing rehabilitation training after implantation.

Elucidating the influences of embodiment and conceptual metaphor on lexical and non-speech tone learning

In the contexts of language learning and music processing, hand gestures conveying acoustic information visually influence perception of speech and non-speech sounds (Connell et al., 2013; Morett & Chang, 2015). Currently, it is unclear whether this effect is due to these gestures' use of the human body to highlight relevant features of language (embodiment) or the cross-modal mapping between the visual motion trajectories of these gestures and corresponding auditory features (conceptual metaphor). To address this question, we examined identification of the pitch contours of lexical tones and non-speech analogs learned with pitch gesture, comparable dot motion, or no motion. Critically, pitch gesture and dot motion were either congruent or incongruent with the vertical conceptual metaphor of pitch. Consistent with our hypotheses, we found that identification accuracy increased for tones learned with congruent pitch gesture and dot motion, whereas it remained stable or decreased for tones learned with incongruent pitch gesture and dot motion. These findings provide the first evidence that both embodied and non-embodied visual stimuli congruent with the vertical conceptual metaphor of pitch enhance lexical and non-speech tone learning. Thus, they illuminate the influences of conceptual metaphor and embodiment on lexical and non-speech auditory perception, providing insight into how they can be leveraged to enhance language learning and music processing.

Dig That Lick (DTL): Analyzing Large-Scale Data for Melodic Patterns in Jazz Performances

Dig That Lick (DTL): Analyzing Large-Scale Data for Melodic Patterns in Jazz Performances

How Subtle Can It Get?

Flying drones have become common objects in our daily lives, serving a multitude of purposes. Many of these purposes involve outdoor scenarios where the user combines drone control with another activity. Traditional interaction methods rely on physical or virtual joysticks that occupy both hands, thus restricting drone usability. In this paper, we investigate one-handed human-to-drone-interaction by leveraging three modalities: force, touch, and IMU. After prototyping three different combinations of these modalities on a smartphone, we evaluate them against the current commercial standard through two user experiments. These experiments help us to find the combination of modalities that strikes a compromise between user performance, perceived task load, wrist rotation, and interaction area size. Accordingly, we select a method that achieves faster task completion times than the two-handed commercial baseline by 16.54% with the merits of subtle user behaviours inside a small-size ring-form device and implements this method within the ring-form device. The last experiment involving 12 participants shows that thanks to its small size and weight, the ring device displays better performance than the same method implemented on a mobile phone. Furthermore, users unanimously found the device useful for controlling a drone in mobile scenarios (AVG = 3.92/5), easy to use (AVG = 3.58/5) and easy to learn (AVG = 3.58/5). Our findings give significant design clues in search of subtle and effective interaction through finger augmentation devices with drone control. The users with our prototypical system and a multi-modal on-finger device can control a drone with subtle wrist rotation (pitch gestures: 43.24° amplitude and roll gestures: 46.35° amplitude) and unnoticeable thumb presses within a miniature-sized area of (1.08 * 0.61 cm2).

Manual directional gestures facilitate cross-modal perceptual learning

Action and perception interact in complex ways to shape how we learn. In the context of language acquisition, for example, hand gestures can facilitate learning novel sound-to-meaning mappings that are critical to successfully understanding a second language. However, the mechanisms by which motor and visual information influence auditory learning are still unclear. We hypothesize that the extent to which cross-modal learning occurs is directly related to the common representational format of perceptual features across motor, visual, and auditory domains (i.e., the extent to which changes in one domain trigger similar changes in another). Furthermore, to the extent that information across modalities can be mapped onto a common representation, training in one domain may lead to learning in another domain. To test this hypothesis, we taught native English speakers Mandarin tones using directional pitch gestures. Watching or performing gestures that were congruent with pitch direction (e.g., an up gesture moving up, and a down gesture moving down, in the vertical plane) significantly enhanced tone category learning, compared to auditory-only training. Moreover, when gestures were rotated (e.g., an up gesture moving away from the body, and a down gesture moving toward the body, in the horizontal plane), performing the gestures resulted in significantly better learning, compared to watching the rotated gestures. Our results suggest that when a common representational mapping can be established between motor and sensory modalities, auditory perceptual learning is likely to be enhanced.

Matching pitch gestures to visual trajectories as a function of acoustic scales

Matching pitch gestures to visual trajectories as a function of acoustic scales

OBSERVING AND PRODUCING PITCH GESTURES FACILITATES THE LEARNING OF MANDARIN CHINESE TONES AND WORDS

AbstractThis study investigates the perception and production of a specific type of metaphoric gesture that mimics melody in speech, also calledpitch gesture, in the learning of L2 suprasegmental features. In a between-subjects design, a total of 106 participants with no previous knowledge of Chinese were asked to observe (Experiment 1) and produce (Experiment 2) pitch gestures during a short multimodal training session on Chinese tones and words. In both experiments they were tested on (a) tone identification and (b) word learning. Results showed the positive effect of a training session with pitch gesture observation compared to a training session without it (Experiment 1) and the benefits of producing gestures compared to only observing them and repeating the words aloud (Experiment 2). A comparison of the results of the two experiments revealed that there was no significant difference between the simple observation of pitch gestures and the production of speech accompanied by pitch gestures in facilitating lexical tone identification and word learning. Thus, both perception and production tasks with pitch gestures can be regarded as beneficial learning strategies for the initial stages of tones acquisition in the Chinese as a Second Language classroom.

Prominence in speech and gesture favour second language novel word learning

ABSTRACTWhile recent research has shown that iconic gestures and the so-called pitch gestures (or gestures that mimic melody in speech) favour word learning in a second language, little is known about (a) the potential benefits of beat gestures (or hand gestures that accompany prosodic prominence) for second language novel word learning, and (b) the contribution of prosodic prominence (independently or in combination with gestural prominence) to this effect. This study investigates the effects of prosodic prominence (e.g. focal pitch accent) and visual prominence (e.g. beat gesture) on L2 novel vocabulary learning. In a within-subjects design, 96 Catalan-dominant native speakers were asked to learn 16 Russian words in four conditions, namely the presence or absence of prosodic prominence in speech (L+H* pitch accent) combined with the presence or absence of visual prominence (beat gesture). The results of recall and recognition tasks conducted after a training session showed that the strongest effect corresponded to target words presented with visual prominence together with prosodic prominence; by contrast, the condition involving visual prominence with no prosodic prominence triggered smaller effects than the condition involving prosodic prominence alone. Thus, beat gestures produced naturally (that is, accompanied by focal pitch accent in speech) favour second language vocabulary learning. The results have implications for second language instruction practices and multisensory integration and working memory models.

OBSERVING PITCH GESTURES FAVORS THE LEARNING OF SPANISH INTONATION BY MANDARIN SPEAKERS

AbstractRecent studies on the learning of L2 prosody have suggested that pitch gestures can enhance the learning of the L2 lexical tones. Yet it remains unclear whether the use of these gestures can aid the learning of L2 intonation, especially by tonal-language speakers. Sixty-four Mandarin speakers with basic-level Spanish were asked to learn three Spanish intonation patterns, all involving a low tone on the nuclear accent. In a pre-post test experimental design, half of the participants received intonation training without the use of pitch gestures (the control group) while the other half received the same training but with pitch gestures representing nuclear intonation contours (the experimental group). Musical (melody, pitch) abilities were also measured. The results revealed that (a) the experimental group significantly improved intonational production outcomes, and (b) even though participants with stronger musical abilities performed better, those with weaker musical abilities benefited more from observing pitch gestures.

Using Functional Data Analysis for investigating multidimensional dynamic phonetic contrasts

The study of phonetic contrasts and related phenomena, e.g. inter- and intra-speaker variability, often requires to analyse data in the form of measured time series, like f0 contours and formant trajectories. As a consequence, the investigator has to find suitable ways to reduce the raw and abundant numerical information contained in a bundle of time series into a small but sufficient set of numerical descriptors of their shape. This approach requires one to decide in advance which dynamic traits to include in the analysis and which not. For example, a rising pitch gesture may be represented by its duration and slope, hence reducing it to a straight segment, or by a richer coding specifying also whether (and how much) the rising contour is concave or convex, the latter being irrelevant in some context but crucial in others. Decisions become even more complex when a phenomenon is described by a multidimensional time series, e.g. by the first two formants.In this paper we introduce a methodology based on Functional Data Analysis (FDA) that allows the investigator to delegate most of the decisions involved in the quantitative description of multidimensional time series to the data themselves. FDA produces a data-driven parametrisation of the main shape traits present in the data that is visually interpretable, in the same way as slopes or peak heights are. These output parameters are numbers that are amenable to ordinary statistical analysis, e.g. linear (mixed effects) models. FDA is also able to capture correlations among different dimensions of a time series, e.g. between formants F1 and F2. We present FDA by means of an extended case study on diphthong – hiatus distinction in Spanish, a contrast that involves duration, formant trajectories and pitch contours.

Emphasising sound and meaning: pitch gestures enhance Mandarin lexical tone acquisition

Lexical tones – pitches differentiating between word meanings in tonal languages – are particularly difficult for atonal language speakers to learn. To test the hypotheses of embodied cognition and spoken word recognition, we examined whether – and how – gesture could facilitate English speakers' discrimination between Mandarin words differing in lexical tone. Words were learned with gestures conveying tone pitch contours (pitch gestures), gestures conveying word meanings (semantic gestures) or no gestures. The results demonstrated that pitch gestures enhanced English speakers' discrimination between the meanings of Mandarin words differing in tone, whereas semantic gestures hindered their identification of tones in learned words. These findings indicate that the visuospatial features of pitch gestures strengthen the relationship between English speakers' representations of Mandarin lexical tones and word meanings, supporting the predictions of spoken word recognition and embodied cognition.

<i>Phonetics and phonology: Interactions and interrelations</i> (review)

Reviewed by: Phonetics and phonology: Interactions and interrelations Zahir Mumin Marina Vigário, Sónia Frota, and Maria Jão Freitas, eds. 2009. Phonetics and phonology: Interactions and interrelations. In the series Current Issues in Linguistic Theory 306. Amsterdam/Philadelphia: John Benjamins. Pp. vi + 290. US $158.00 (hardcover). This book consists of a collection of 13 papers presented at the Third Phonetics and Phonology in Iberia Conference in 2007 at the University of Minho in Portugal. The book is divided into three parts, which represent the overall major issues examined in the fields of phonetics and phonology: "Between phonetics and phonology" (Part I), "Segmental and prosodic interactions" (Part II), and "Interactions between segments and features" (Part III). The primary purpose of this book is to examine the speech of children and adults who speak different world languages, such as English, Spanish, Dutch, Italian, Portuguese, Catalan, and Romanian, in order to determine the extent to which linguistic theories such as Articulatory Phonology, Optimality Theory, and the Parallel Structures Model are applicable to the results of the descriptive data analyzed in each study. The secondary purpose is to show that there is still an ongoing debate on the possibility of combining the fields of phonetics and phonology into one field because of their inevitable interrelatedness which often constitutes the examination of segments, prosody, features, and speech patterns. The first paper, by Riera et al., examines the speech of two native US English speakers to determine if there is production of a schwa-like element in vowel V + /r/ sequences, and compare this element with the canonical schwa. This study uses a one-way and two-way Analysis of Variance (ANOVA) to examine the formant and duration differences between this schwa-like element and the canonical schwa. The results show that this schwa-like element is highly variable (unlike the canonical schwa) and is dynamically conditioned by the phonetic context and speaking rate. The study argues overall that epenthetic schwa and centering diphthongs are inaccurate terms used to describe this schwa-like element because this element constantly undergoes the process of coarticulation. The results of the study do not support Baker and Goldstein's (1990) argument that there is an epenthetic schwa (without a clear process of coarticulation) that is similar to a canonical schwa.1 The results of Riera et al.'s study also contradict Roach et al.'s (2006) argument in favour of centering diphthongs, which are formed according to the respective V + /r/ phonetic context. The second paper, by Ortega-Llebaria and Prieto, analyses native Spanish-speakers' perception of stress contrast in unaccented contexts. The study uses duration, spectral tilt, and intensity as independent variables to determine the extent to which 20 research participants [End Page 133] perceive the aforementioned contrast in the Spanish vowels [a] and [i], using the paroxytone words mimi and mama, and oxytone words mimí and mamá. The results of Ortega-Llebaria and Prieto's study show that Spanish speakers rely on duration and intensity as cues to stress in unaccented contexts and that spectral tilt is not a significant factor. These results supplement the findings of Sluijter et al. (1997), which show that English speakers use vowel reduction patterns to perceive stress in unaccented contexts, whereas Dutch speakers employ duration, spectral tilt, and intensity. The third paper, also by Prieto and Ortega-Llebaria, examines the complexity of the nuclear pitch accent in broad focus and narrow contrastive focus to find out if the durational properties of the penultimate stress syllables and word-final stress syllables in Catalan and Spanish show a lengthening effect on the target syllables. Although Grabe (1998) argues that, cross-linguistically, speakers (depending on their dialect) use truncation and compression of rising and falling accents to condition the durational properties of stress, Prieto and Ortega-Llebaria argue that this is not dialectal but a phonetic realisation strategy. The target word pairs used for their study were mama/mamá and mimi/mimí in sentence-final position and the results show that, in Catalan and Spanish, duration increases in narrow-focus words in penultimate-and final-syllable position and when there is a complex pitch gesture. The fourth paper, by Manolescu et al...

A Gesture-based Typology of Sliding-tones in Guqin Music

A good part of the expressive character of traditional Chinese guqin music can be accounted for in terms of sliding-tones, of which vibrato and portamento form major constituents. In this paper, a representative performance of a guqin piece formed the starting point for identifying an alphabet of basic movements on top of which it was possible to define different vibrato and portamento gesture types. The methodology is based on a pitch analysis of the musical audio and a subsequent segmentation and categorization of the pitch patterns which draws upon the player's knowledge of guqin playing. The typology focuses on the morphological aspects of pitch and control gestures, and it allows a description of sliding-tones in terms of concatenations of basic movements of the finger on the string. The close relationship between control gestures and pitch patterns justifies the use of the term “gesture” for both sonic and control patterns, although sonic gestures do not always completely overlap with control gestures. The present paper is a first attempt at an identification and classification of the gestures that make up guqin vibrato and portamento. The approach offers opportunities for future testing, using measurement tools and experimental techniques. The applications of this typology in education and sound synthesis are straightforward.

Traversing distances: Pitch organization, gesture and imagery in the late works of T l ru Takemitsu

This study focuses on pitch organization in several of T l ru Takemitsu's last works, showing how projection of octatonic and other referential set complexes in Takemitsu's music contribute to the formation of large-scale gestures, which become agents of metaphorical imagery and affective meaning. In Takemitsu's music, associations of harmonic sonority, timbral color, and metaphorical imagery are embodied in an archetypal musical gesture. An aesthetic aspect of the octatonic and other referential set complexes in Takemitsu's late music can be found in their association with the hopeful humanistic ideal Takemitsu conveys in his music. As block-form spatializations link the temporal and that which is beyond time, referential pitch structures provide a way of grounding Takemitsu's "dream" imagery of infinite human potential in a particular manifestation of mediated reality. Relating imagery of pitch organization in Takemitsu's music to sources Takemitsu found in the works of Debussy and Messiaen, the study aims to contribute to greater understudying of Takemitsu's temporal structures, aesthetic aims, and the influences which shaped his development.

Pitch and Manual Gestures

Gestures and speech parallel or complement each other semantically and pragmatically. Previous studies have postulated a similarly close correlation between intonation and gesture. This study microanalyzed the gestures of three subjects frame by frame and matched the movements with direction of pitch changes for each syllable in 90 intonation groups. The speech was analyzed using spectrographs. Coordinating direction of pitch and manual gesture movements is an option available to speakers, but it is not biologically mandated.

Interactive prosody training workstation.

Few devices exist to aid in the training of pitch, intensity, and rhythm of speech. The Interactive prosody training workstation (PW) employs state‐of‐the‐art technology to assist users in achieving clinician‐ or client‐programmable targets in each of these features. Two training interfaces are currently implemented. The simpler displays F0 and/or intensity in real time as a fluctuating one‐ or two‐dimensional display. Smoothing can be adjusted to accommodate varying levels of voice variations. With the more advanced interface, model utterances are presented using stored LPC‐coded speech. The user’s response is then compared to the model, using any of a wide variety of scoring methods. The user’s response may be played, in comparison to the model, as often as desired. A demonstration tape will be played showing one hearing impaired individual with a cochlear implant using the PW first to find his appropriate F0 register and then practicing pitch gestures appropriate for speech. Strong carryover of learning to later sessions is demonstrated. [Work supported in part by an SBIR grant from NIH.]