Pitch Contour Research Articles

Rise and fall: Effects of tone and intonation on spoken word recognition in early childhood

A crucial component of word learning is the ability to recognize words in spite of the varying forms they assume. This may be particularly challenging in tone languages as learners have to develop tone representations in the face of intonational variation in order to accurately recognize words. The effects of intonational variation on word recognition of tone-marked words in Mandarin Chinese were investigated in toddlers and preschoolers using a cross-sectional design. Participants were presented with known words where intonation (question/statement) and tone (rising/falling) were independently manipulated. Results demonstrated that word recognition in toddlers was heavily influenced by changes in the pitch contour of a tone due to intonational variation. In contrast, preschool were able to recognize tone-marked words regardless of simultaneous intonational variation, demonstrating a comparatively robust representation of lexical tone. Results chart an evolution in integrating pitch cues to tone and intonation over the first few years of life.

Interactions between musicality and native language for enhanced sound feature processing

Interactions between musicality and native language for enhanced sound feature processing

Neural Segregation of Concurrent Speech: Effects of Background Noise and Reverberation on Auditory Scene Analysis in the Ventral Cochlear Nucleus.

Concurrent complex sounds (e.g., two voices speaking at once) are perceptually disentangled into separate "auditory objects". This neural processing often occurs in the presence of acoustic-signal distortions from noise and reverberation (e.g., in a busy restaurant). A difference in periodicity between sounds is a strong segregation cue under quiet, anechoic conditions. However, noise and reverberation exert differential effects on speech intelligibility under "cocktail-party" listening conditions. Previous neurophysiological studies have concentrated on understanding auditory scene analysis under ideal listening conditions. Here, we examine the effects of noise and reverberation on periodicity-based neural segregation of concurrent vowels /a/ and /i/, in the responses of single units in the guinea-pig ventral cochlear nucleus (VCN): the first processing station of the auditory brain stem. In line with human psychoacoustic data, we find reverberation significantly impairs segregation when vowels have an intonated pitch contour, but not when they are spoken on a monotone. In contrast, noise impairs segregation independent of intonation pattern. These results are informative for models of speech processing under ecologically valid listening conditions, where noise and reverberation abound.

The limits of metrical segmentation: intonation modulates infants' extraction of embedded trochees.

We tested German nine-month-olds' reliance on pitch and metrical stress for segmentation. In a headturn-preference paradigm, infants were familiarized with trisyllabic words (weak-strong-weak (WSW) stress pattern) in sentence-contexts. The words were presented in one of three naturally occurring intonation conditions: one in which high pitch was aligned with the stressed syllable and two misalignment conditions (with high pitch preceding vs. following the stressed syllable). Infants were tested on the SW unit of the WSW carriers. Experiment 1 showed recognition only when the stressed syllable was high-pitched. Intonation of test items (similar vs. dissimilar to familiarization) had no influence (Experiment 2). Thus, German nine-month-olds perceive stressed syllables as word onsets only when high-pitched, although they already generalize over different pitch contours. Different mechanisms underlying this pattern of results are discussed.

Automatic prosodic tone choice classification with Brazil’s intonation model

This paper examines the performance of automatically classifying five tone choices (i.e., falling, rising, rising-falling, falling-rising, and neutral) of Brazil's intonation model. We tested two machine learning classifiers (neural network and boosting ensemble) in two configurations (multi-class and pairwise coupling) and a rule-based classifier. Three sets of acoustic features built from the TILT and Bezier pitch contour models and a new four-point pitch contour model we introduced here were investigated. Tone choices are one of the key elements of Brazil's prosodic intonation model. We found the rule-based classifier, which was built on our four-point model, achieved better results than the others with an accuracy of 75.1 % and a Cohen's kappa coefficient of 0.73. This research proves that it is possible to classify tone choices with an accuracy reaching close to the percentage of agreement between two human analysts. The findings further concluded that our four-point model was better for classifying Brazil's tone choices than both of the TILT or Bezier models.

Representation and Processing of Lexical Tone and Tonal Variants: Evidence from the Mismatch Negativity.

Pronunciation variation is ubiquitous in the speech signal. Different models of lexical representation have been put forward to deal with speech variability, which differ in the level as well as the nature of mental representation. We present the first mismatch negativity (MMN) study investigating the effect of allophonic variation on the mental representation and neural processing of lexical tones. Native speakers of Standard Chinese (SC) participated in an oddball electroencephalography (EEG) experiment. All stimuli have the same segments (ma) but different lexical tones: level [T1], rising [T2], and dipping [T3]. In connected speech with a T3T3 sequence, the first T3 may undergo allophonic change and is produced with a rising pitch contour (T3V), similar to the lexical T2 pitch contour. Four oddball conditions were constructed (T1/T3, T3/T1, T2/T3, T3/T2; standard/deviant). All four conditions elicited MMN effects, with the T1–T3 pair eliciting comparable MMNs, but the T2–T3 pair asymmetrical MMN effects. There were significantly greater and earlier MMN effects in the T2/T3 condition than that in the reversed T3/T2 condition. Furthermore, the T3/T2 condition showed more rightward MMN effects than the T2/T3 condition and the T1–T3 pair. Such asymmetries suggest co-activation of long-term memory representations of both T3 and T3V when T3 serves as the standard. The acoustic similarity between the activated T3V (by the standard T3) and the incoming deviant stimulus T2 induces acoustic processing of the tonal contrast in the T3/T2 condition, similar to that of within-category lexical tone processing, which is in contrast to the processing of between-category lexical tones observed in the T2/T3, T1/T3, and T3/T1 conditions.

The interactional function of prosody in repair initiation: Pitch height and timing of va ‘what’ in Helsinki Swedish

This paper analyzes the relation between the prosodic form and interactional function of va ‘what’ in Helsinki Swedish interactions. The method adopted was sequential and phonetic analysis. The results show that prosodic features can display whether a va is used to initiate normal repair or affective repair. Normal repair indicates a trouble in hearing or understanding, while affective repair simultaneously initiates repair and displays an affective stance. The relevant prosodic features are the pitch span and range, as well as the temporal placement of turns. Contrary to claims by previous studies, pitch contour does not distinguish between trouble types in Helsinki Swedish. Furthermore, when participants ascribe functions to va-tokens, the other significant factors are sequential context, ongoing activity, and situation. As the results demonstrate that one form can be used for several functions and several forms can have one function, this study suggests that there is a one-to-many and many-to-one relationship between the prosodic form and interactional function of the va-turns in Helsinki Swedish.

Dorsal and Ventral Pathways for Prosody

Our vocal tone--the prosody--contributes a lot to the meaning of speech beyond the actual words. Indeed, the hesitant tone of a "yes" may be more telling than its affirmative lexical meaning. The human brain contains dorsal and ventral processing streams in the left hemisphere that underlie core linguistic abilities such as phonology, syntax, and semantics. Whether or not prosody--a reportedly right-hemispheric faculty--involves analogous processing streams is a matter of debate. Functional connectivity studies on prosody leave no doubt about the existence of such streams, but opinions diverge on whether information travels along dorsal or ventral pathways. Here we show, with a novel paradigm using audio morphing combined with multimodal neuroimaging and brain stimulation, that prosody perception takes dual routes along dorsal and ventral pathways in the right hemisphere. In experiment 1, categorization of speech stimuli that gradually varied in their prosodic pitch contour (between statement and question) involved (1) an auditory ventral pathway along the superior temporal lobe and (2) auditory-motor dorsal pathways connecting posterior temporal and inferior frontal/premotor areas. In experiment 2, inhibitory stimulation of right premotor cortex as a key node of the dorsal stream decreased participants' performance in prosody categorization, arguing for a motor involvement in prosody perception. These data draw a dual-stream picture of prosodic processing that parallels the established left-hemispheric multi-stream architecture of language, but with relative rightward asymmetry.

Children's identification of questions from rising terminal pitch.

Young children are slow to master conventional intonation patterns in their yes/no questions, which may stem from imperfect understanding of the links between terminal pitch contours and pragmatic intentions. In Experiment 1, five- to ten-year-old children and adults were required to judge utterances as questions or statements on the basis of intonation alone. Children eight years of age or younger performed above chance levels but less accurately than adult listeners. To ascertain whether the verbal content of utterances interfered with young children's attention to the relevant acoustic cues, low-pass filtered versions of the same utterances were presented to children and adults in Experiment 2. Low-pass filtering reduced performance comparably for all age groups, perhaps because such filtering reduced the salience of critical pitch cues. Young children's difficulty in differentiating declarative questions from statements is not attributable to basic perceptual difficulties but rather to absent or unstable intonation categories.

Improving the Flexibility of Dynamic Prosody Modification Using Instants of Significant Excitation

Modification of suprasegmental features such as pitch and duration of original speech by fixed scaling factors is referred to as static prosody modification. In dynamic prosody modification, the prosodic scaling factors (time-varying modification factors) are defined for all the pitch cycles present in the original speech. The present work is focused on improving the naturalness of the prosody modified speech by reducing the generation of piecewise constant segments in the modified pitch contour. The prosody modification is performed by anchoring around the accurate instants of significant excitation estimated from the original speech. The division of longer pitch intervals into many equal intervals over long speech segments introduces step-like discontinuities in the form of piecewise constant segments in the modified pitch contours. The effectiveness of proposed dynamic modification method is initially confirmed from the smooth modified pitch contour plot obtained for finer static prosody scaling factors, waveforms, spectrogram plots and comparison subjective evaluations. Also, the average $$F_0$$F0 jitter computed from the pitch segments of each glottal activity region in the modified speech is proposed as an objective measure for the prosody modification. The naturalness of the prosody modified speech using the proposed method is objectively and subjectively compared with that of the existing zero frequency filtered signal-based dynamic prosody modification. Also, the proposed algorithm effectively preserves the dynamics of the prosodic patterns in singing voices where in the $$F_0$$F0 parameters rapidly and continuously fluctuate within a higher $$F_0$$F0 range.

L + H* and H* pitch accents in Mandarin Chinese

In this study, we examine acoustic correlates of L + H* and H* pitch accents in Mandarin Chinese. In English, L + H* denotes “exclusiveness” and H* conveys “new” information. For example, L + H* A. Katie did not pet the cat, (Kellie did). H* B. Katie did not pet the cat (she pet the dog). Ten Mandarin speakers were asked to produce the English equivalent L + H* and H* pitch accents in Mandarin as in C and D: L + H* C. 咪妮 没有 摸 猫， 妞蒙 摸 了 猫 MiNi meiyou mo mao, NiuMeng mo le mao. MiNi did not pet the cat, it is NiuMeng who petted the cat. H* D. 咪妮 没有 摸 猫，咪妮也 没有 摸 狗 MiNi meiyou mo mao, MiNi ye meiyou mo gou. MiNi did not pet the cat, and she did not pet the dog either. MiNi and NiuMeng were produced with all combinations of four Mandarin tones. Preliminary findings from one speaker suggest an interaction between tones and pitch accents such that pitch contour and pitch height of both types of pitch accents vary as a function of tones.

Using automatic speech recognition to identify dementia in early stages

Early non-invasive diagnosis of Alzheimer’s disease (AD) and other forms of dementia is a challenging task. Early detection of the symptoms of the disorder could help families and medical professionals prepare for the difficulties ahead, as well as possibly provide a recruitment tool for clinical trials. One possible approach to a non-invasive diagnosis is based on analysis of speech patterns. Subjects are asked to describe a picture and their description (typically 1 to 3 minute speech sample) is recorded. For this study, a database of 70 people were recorded, 24 with a clinical diagnosis of probable or possible Alzheimer’s disease. When these data were combined with 140 other recorded samples, a classifier built with manually transcribed versions of the speech was found to be quite accurate for determining whether or not a speech sample was obtained from an Alzheimer’s patient. A classifier built using automatically determined prosodic features (pitch and energy contours) was also reasonably accurate, with several subsets of pitch and energy features especially effective for classification, as assessed by cross validation. The manually transcribed text has now been replaced by automatically transcribed text using automatic speech recognition (ASR) technology. The main objective of this paper is to report on the relative effectiveness of several ASR approaches, including public domain ones, for this task.

Prosodic features of stance acts

While textual aspects of stance (attitudes/opinions) have been well studied in conversation analysis and computational models, acoustic-phonetic properties have received less attention. Recent work (2014 and 2015) has found that variations in prosodic measures (speech rate, vowel duration, pitch, and intensity) are correlated with stance presence and strength in unscripted speech, and stances with different discourse functions may be distinguishable by the shapes of their pitch and intensity contours. Building on these early findings, this presentation investigates prosodic properties of various stance-act types in spontaneous conversation (e.g., opinion-offering and soliciting, (dis)agreement, persuasion, rapport-building). The dataset contains over 32,000 stressed vowels from content words spoken by 40 speakers drawn from an audio corpus of dyads engaged in collaborative tasks. Speaker-normalized vowel duration, pitch, and intensity are automatically extracted from time-aligned transcriptions that have been hand-annotated for stance strength, polarity, and act type. Results show that changes in the prosodic measures combine to distinguish several notable stance-act types, including: weak-positive agreement, rapport-building agreement, reluctance to accept a stance, stance-softening, and backchanneling. Pitch and intensity contours over vowel duration are particularly illustrative, suggesting a future avenue in examining contours over whole stance acts.

The intonation of wh-in situ questions in Northern Peninsular Spanish

This project investigates the intonational characteristics of wh-in situ questions (As in: ?Compraste que? ‘You bought what?) in Northern Peninsular Spanish. Although these questions reportedly occur with various pragmatic readings in several Spanish dialects (Uribe-Etxebarria 2002; Chernova 2013; Reglero and Ticio 2013), their intonation remains to be investigated. Data from 22 speakers from Northern Peninsular Spanish was collected in a reading and an elicitation task. Each task included 30 contextualized wh-in situ questions representing information-seeking, echo-repetition, and echo-surprise readings. Target sentences were interspersed among declaratives and yes–no questions. Preliminary results from the elicitation task are discussed for two female participants. Prevalent pitch contours are analyzed following Spanish ToBi conventions (Face and Prieto 2007). In addition, we report the global tonal range for each type of wh-in situ, as well as the tonal range average from the nuclear tone to the bounda...

Syllabic tone articulation influences the identification and use of words during Chinese sentence reading: Evidence from ERP and eye movement recordings.

In two experiments, we examined the contribution of articulation-specific features to visual word recognition during the reading of Chinese. In spoken Standard Chinese, a syllable with a full tone can be tone-neutralized through sound weakening and pitch contour change, and there are two types of two-character compound words with respect to their articulation variation. One type requires articulation of a full tone for each constituent character, and the other requires a full- and a neutral-tone articulation for the first and second characters, respectively. Words of these two types with identical first characters were selected and embedded in sentences. Native speakers of Standard Chinese were recruited to read the sentences. In Experiment 1, the individual words of a sentence were presented serially at a fixed pace while event-related potentials were recorded. This resulted in less-negative N100 and anterior N250 amplitudes and in more-negative N400 amplitudes when targets contained a neutral tone. Complete sentences were visible in Experiment 2, and eye movements were recorded while participants read. Analyses of oculomotor activity revealed shorter viewing durations and fewer refixations on-and fewer regressive saccades to-target words when their second syllable was articulated with a neutral rather than a full tone. Together, the results indicate that readers represent articulation-specific word properties, that these representations are routinely activated early during the silent reading of Chinese sentences, and that the representations are also used during later stages of word processing.

List intonation in pre-schoolers with normal and disordered language development

The principal aim of this study was to evaluate pre-schoolers’ expressive intonation in light of current debates about the underlying nature of language impairment (LI). Children with LI typically have deficits in grammar, a component of language that is phonologically represented on the segmental level. The hypothesis is that children with LI do not have deficits of this type when grammar is conveyed by intonation, a pitch-based component of language that is phonologically represented on the suprasegmental level. This study focused on the richly diversified suprasegmental patterns of sentences in which the speaker produces a series of items in a list. To address the hypothesis, list intonation in the speech of 4-year-olds with and without LI was acoustically analysed. Lists produced by children with LI were comparable to those produced by children with normal language development (NL). The results do not support the claim that LI stems from a poor understanding of grammatical principles. Rather, LI reflects an underlying impairment of segmental information processing. The discussion focuses on two characteristics of pitch contours which may account for the resilience of intonation in children with LI. Namely, steady state versus transient signals and universal symbol meanings versus arbitrary relationships between form and function.

Vowel-Based Non-uniform Prosody Modification for Emotion Conversion

The objective of this work is to develop a rule-based emotion conversion method for a better emotional perception. In this work, performance of emotion conversion using the linear modification model is improved by using vowel-based non-uniform prosody modification. In the present approach, attempts were made to integrate features like position and identity for addressing the non-uniformity in prosody generated due to the emotional state of the speaker. We mainly concentrate on the parameters such as strength, duration and pitch contour of vowels at different parts of the sentence. The influence of emotions on the above parameters is exploited to convert the speech from neutral emotion to the target emotion. Non-uniform prosody modification factors for emotion conversion are based on the position of vowels in the word, and the position of the word in the sentence. This study is carried out by using Indian Institute of Technology-Simulated Emotion speech corpus. Evaluation of the proposed algorithm is carried out by a subjective listening test. From the listening tests, it is observed that the performance of the proposed approach is better than the existing approaches.

Sensitivity of the cortical pitch onset response to height, time-variance, and directionality of dynamic pitch

Event-related brain potentials (ERPs) demonstrate that human auditory cortical responses are sensitive to changes in static pitch as indexed by the pitch onset response (POR), a negativity generated at the initiation of acoustic periodicity. Yet, it is still unclear if this brain signature is sensitive to dynamic, time-varying properties of pitch more characteristic of those found in naturalistic speech and music. Neuroelectric PORs were recorded in response to contrastive pitch patterns differing in their pitch height, time-variance, and directionality (i.e., rise vs. fall). Broadband noise followed by contiguous iterated rippled noise (producing salient pitch sweeps) was used to temporally separate neural activity coding the onset of acoustic energy from the onset of time-varying pitch. Analysis of PORs revealed distinct modulations in response latency that distinguished static from time-varying pitch contours (steady-state<dynamic) and pitch height (high<low). However, PORs were insensitive to the direction of pitch sweeps (rise=fall). Our findings suggest that the POR signature provides a useful neural index of auditory cortical pitch processing for some, but not all pitch-evoking stimuli.

Experience-dependent enhancement of pitch-specific responses in the auditory cortex is limited to acceleration rates in normal voice range

The aim of this study is to determine how pitch acceleration rates within and outside the normal pitch range may influence latency and amplitude of cortical pitch-specific responses (CPR) as a function of language experience (Chinese, English). Responses were elicited from a set of four pitch stimuli chosen to represent a range of acceleration rates (two each inside and outside the normal voice range) imposed on the high rising Mandarin Tone 2. Pitch-relevant neural activity, as reflected in the latency and amplitude of scalp-recorded CPR components, varied depending on language-experience and pitch acceleration of dynamic, time-varying pitch contours. Peak latencies of CPR components were shorter in the Chinese than the English group across stimuli. Chinese participants showed greater amplitude than English for CPR components at both frontocentral and temporal electrode sites in response to pitch contours with acceleration rates inside the normal voice pitch range as compared to pitch contours with acceleration rates that exceed the normal range. As indexed by CPR amplitude at the temporal sites, a rightward asymmetry was observed for the Chinese group only. Only over the right temporal site was amplitude greater in the Chinese group relative to the English. These findings may suggest that the neural mechanism(s) underlying processing of pitch in the right auditory cortex reflect experience-dependent modulation of sensitivity to acceleration in just those rising pitch contours that fall within the bounds of one’s native language. More broadly, enhancement of native pitch stimuli and stronger rightward asymmetry of CPR components in the Chinese group is consistent with the notion that long-term experience shapes adaptive, distributed hierarchical pitch processing in the auditory cortex, and reflects an interaction with higher order, extrasensory processes beyond the sensory memory trace.

Flexibility in Bilingual Infants' Word Learning.

The present experiments tested bilingual infants' developmental narrowing for the interpretation of sounds that form words. These studies addressed how language specialization proceeds when the environment provides varied and divergent input. Experiment 1 (N = 32) demonstrated that bilingual 14- and 19-month-olds learned a pair of object labels consisting of the same syllable produced with distinct pitch contours (rising and falling). Infants' native languages did not use pitch contour to differentiate words. In Experiment 2 (N = 16), 22-month-old bilinguals failed to learn the labels. These results conflict with the developmental trajectory of monolinguals, who fail to learn pitch contour contrasts as labels at 17-19 months (Hay, Graf Estes, Wang, & Saffran, 2015). Bilingual infants exhibited a prolonged period of flexibility in their interpretation of potential word forms.