F0 Trajectories Research Articles

Unequal effects of speech and nonspeech contexts on the perceptual normalization of Cantonese level tones

Context is important for recovering language information from talker-induced variability in acoustic signals. In tone perception, previous studies reported similar effects of speech and nonspeech contexts in Mandarin, supporting a general perceptual mechanism underlying tone normalization. However, no supportive evidence was obtained in Cantonese, also a tone language. Moreover, no study has compared speech and nonspeech contexts in the multi-talker condition, which is essential for exploring the normalization mechanism of inter-talker variability in speaking F0. The other question is whether a talker's full F0 range and mean F0 equally facilitate normalization. To answer these questions, this study examines the effects of four context conditions (speech/nonspeech × F0 contour/mean F0) in the multi-talker condition in Cantonese. Results show that raising and lowering the F0 of speech contexts change the perception of identical stimuli from mid level tone to low and high level tone, whereas nonspeech contexts only mildly increase the identification preference. It supports the speech-specific mechanism of tone normalization. Moreover, speech context with flattened F0 trajectory, which neutralizes cues of a talker's full F0 range, fails to facilitate normalization in some conditions, implying that a talker's mean F0 is less efficient for minimizing talker-induced lexical ambiguity in tone perception.

Very low bit-rate F0 coding for phonetic vocoders using MSD-HMM with quantized F0 symbols

This paper presents a technique of very low bit-rate F0 coding for phonetic vocoders based on a hidden Markov model (HMM) using phone-level quantized F0 symbols. In the proposed technique, an input F0 sequence is converted into an F0 symbol sequence at the phone level using scalar quantization. The quantized F0 symbols represent the rough shape of the original F0 contour and are used as the prosodic context for the HMM in the decoding process. To model the F0 that has voiced and unvoiced regions, we use multi-space probability distribution HMM (MSD-HMM). Synthetic speech is generated from the context-dependent labels and pre-trained MSD-HMMs by using the HMM-based parameter generation algorithm. By taking into account the preceding and succeeding contexts as well as the current one in the modeling and synthesis, we can generate a smooth F0 trajectory similar to that of the original with only a small number of quantization bits. The experimental results reveal that the proposed F0 coding outperforms the conventional segment-based F0 coding technique using MSD-VQ. We also demonstrate that the decoded speech of the proposed vocoder has acceptable quality even when the F0 bit-rate is less than 50 bps.

Dynamic differences in the production of diphthongs by French–English bilingual children

This study examines the cross-linguistic phonetic interactions in the production of diphthongs by French–English bilingual children. Tautosyllabic vowel-glide combinations in English and French have different phonological statuses. This combination corresponds to a single segment (i.e., a diphthong) in English, but two separate segments (i.e., vowel+glide) in French. Using a picture-naming experiment, the study aims to investigate (1) whether English diphthongs (e.g., /aI/ as in bye) and French tautosyllabic vowel-glide combinations (e.g. /aj/ as in baille ‘yawn’) have different phonetic implementations and, if so, (2) whether bilingual children maintain two separate categories. Diphthongs were recorded for six monolingual speakers of French and English, and four 6-7 year-old bilingual French-English speakers living in the US. To best capture the dynamic properties of diphthongs, the curves corresponding to F1 and F2 trajectories were submitted to statistical comparisons using the Smoothing Spline ANOVA. ...

Continuous F0 Modeling for HMM Based Statistical Parametric Speech Synthesis

The modeling of fundamental frequency, or F0, in HMM-based speech synthesis is a critical factor in delivering speech which is both natural and accurately conveys all of the many nuances of the message. However, F0 modeling is difficult because F0 values are normally considered to depend on a binary voicing decision such that they are continuous in voiced regions and undefined in unvoiced regions. F0 is therefore a discontinuous function of time. Multi-space probability distribution HMM (MSDHMM) is a widely used solution to this problem. The MSDHMM essentially uses a joint distribution of discrete voicing labels and the discontinuous F0 observations. However, due to the discontinuity assumption, the MSDHMM provides a rather weak F0 trajectory model. In this paper, F0 is viewed as being a continuous function of time and this is achieved by assuming that F0 can be observed within unvoiced regions as well as voiced regions. This provides a continuous F0 data stream which can be modeled by standard HMMs. Voicing labels are modeled either implicitly or explicitly in order to perform voicing classification and a globally tied distribution (GTD) technique is used to achieve robust F0 estimation. Both objective measures and subjective listening tests demonstrate that continuous F0 modeling yields better synthesized F0 trajectories and significant improvements to the naturalness of synthesized speech compared to using the MSDHMM model.

Perceived imitation of regional dialects.

This study examined the effect of regional dialect on perceived spontaneous phonetic imitation. Participants shadowed CVC target words produced by talkers from two American English dialect regions, the North and the Midland. Imitation was measured acoustically in terms of vowel quality, vowel duration, midpoint f0, f0 trajectory, and onset and coda consonant duration. The productions and targets from the shadowing task were then used as stimulus materials in an AXB discrimination task to measure perceived imitation. Listeners selected whether the shadowed production of each word was more similar to the target (X) than a baseline recording of that same word. The AXB experiment included three conditions to distinguish between imitation of talker-specific and dialect-specific properties, in which X was the original target from the shadowing experiment, X was produced by another talker from the same region, or X was produced by a talker from the other region, respectively. Preliminary results suggest that listeners used talker-specific, rather than dialect-specific, properties when judging imitation. Among the acoustic measures, perceived imitation was correlated with vowel duration. These results suggest that talkers in a shadowing task imitate talker-specific variability more than general properties of regional dialect.

Developmental changes in the fundamental frequency (f0) of infants’ cries: a study of children with Autism Spectrum Disorder

Episodes of crying with higher fundamental frequency (f0) are perceived as more aversive and distressful than lower frequency cries. Besides, previous studies have speculated that in children with Autism Spectrum Disorder (ASD) higher f0 may account for evoking mental states of uneasiness in the caregiver. Moreover no evidence on developmental trajectory of f0 in ASD in infancy is present in literature. This research investigated: developmental trajectory of f0 of episodes of crying in children with ASD as opposed to typically developing (TD) children and children with other developmental delay (DD). Two studies were employed. In Study 1, using a standardised protocol for acoustic analysis a group of 30 TD children was followed longitudinally (five, 13 and 20 months) and f0 of their episodes of crying was analysed. In Study 2, acoustic analysis was applied to episodes of crying selected from retrospective home videos of 30 children with ASD, TD and DD at five and 18 months. Decrease of f0 in TD children between the first and second year both using a standard procedure and the retrospective procedure was found. Besides while in TD and DD children a change of the f0 trajectory was present, no changes were evident in ASD. Higher f0 may account in ASD for a caregiver bias in their interpretation of crying, evoking mental states of uneasiness and it may interfere with the adequacy of the parental response. Limitations and future goal of the research are described.

The effect of increased loudness on anticipatory coarticulation of steady-state formants in a vowel-consonant-vowel context for patients with Parkinson’s disease.

The current investigation examines the effect of increased loudness on anticipatory coarticulation of the vowels [i, ɑ, u] on the preceding schwa vowel for patients with Parkinson’s disease. Four speakers, three males and one female with speech severity levels between 8% and 49%, are involved in pre-treatment, treatment, and post-treatment conditions. During treatment, participants use an auditory device which plays speech-babble into one ear while they are speaking. The speech-babble competition results in an automatic loudness increase (Lombard effect) in the talker wearing the device. Participants produced target nonsense CVC words embedded in carrier sentences at comfortable loudness levels and at an average of 3 dB above that level. The steady-state F1, F2, and F3 formant frequencies of the preceding [ə] were analyzed to determine whether increased loudness results in increased anticipatory coarticulation from the [i, ɑ, u] vowels in the CVC words, as treatment progresses. Results using steady-state formant frequency measures from this study will be compared with previous studies that used F2 trajectory measures [Tjaden and Sussman (2006); Weismer et al. (1995); Tjaden (2003)]. (Work supported by NIH 1R01DC009409-01)

The effect of lexical frequency and Lombard reflex on tone hyperarticulation

Among the many factors that affect phonetic variation, two of them, low word frequency and speech in noise (Lombard speech), are linked to stronger or fuller realizations of segments. Noise and low usage frequency are both factors that may lead to perceptual difficulty suggesting that the strengthening due to these two factors may be related. To understand how these two factors relate to each other and to see if they play a similar role with respect to suprasegmental phenomena like tone, we investigated the effect of ambient noise and word usage frequency on tone production in Cantonese. We recorded Cantonese monosyllabic words of high and low usage frequency in both normal and Lombard speech styles, controlling for segmental and other factors and measured f0. We found that both ambient noise and lexical frequency influence tonal production. All six tones are produced with higher f0 in the presence of noise. Usage frequency affects f0 realization as well, but only for words with mid-range (mid-level or mid-rising) tones. Low-frequency words with mid-range tones are produced with higher f0 than frequent words. Extending earlier work showing greater vowel-space dispersion in less frequent words, we also found that the f0 trajectories of less frequent words are more dispersed than that of their more frequent counterparts, especially for mid–low tones. This dispersion does not occur for Lombard speech. Our results suggest that different aspects of speech production may account for the strengthening of speech in noise and in low-frequency words.

Subglottal coupling as a quantal basis for the feature [back

A model of acoustic coupling between the oral and subglottal cavities is described, which predicts attenuation of vowel formant prominences and discontinuities in formant trajectories near resonances of the subglottal system. The hypothesis that these effects on F2 near the second subglottal resonance (SubF2) are quantal effects for the feature [back] is examined using acoustic and subglottal data from English-speaking adults. Experimental studies of F2 and SubF2 in English vowel production [Chi and Sonderegger, JASA 122, 1735–1745 (2007)] are reviewed and show that attenuation of second formant prominence and discontinuities in F2 trajectories near SubF2 consistently occur in back-front diphthongs, in accordance with the acoustic model, while for monophthongs front and back vowel F2 values pattern above and below SubF2, as expected under the quantal hypothesis. An additional analysis of the data is presented, showing that breathiness, attenuation, and discontinuity are positively correlated across back-front diphthongs, as predicted by the acoustic model. Lastly, a cross-linguistic survey of F2 data from 45 languages from the literature is presented, showing the cross-linguistic plausibility of the hypothesis that [+back] and [−back] vowel F2 values are separated by SubF2.

Development of subglottal quantal effects in young children.

Quantal articulatory acoustic relations between formants and subglottal resonances have been shown to define several vowel and consonant distinctive features, including the vowel features (back) and (low). In particular, the F2 frequency of front vowels is higher than the second subglottal resonance (Sg2) but is lower than Sg2 for back vowels. Likewise for low vowels, F1 is higher than Sg1. Measurements of F1 and F2 for productions of the vowels /a/ and /ae/ were made on a number of occasions for several children in the age range 2.5–3.7 years. Measurements of Sg1 and Sg2 for each child were obtained from the locations of discontinuities in the F1 and F2 trajectories in diphthongs and in vowel-consonant transitions. At the earlier ages for these children, there was variability in the utterances, and the F1 and F2 values deviated from the expected relation. At the later age of 3 years and above, considerable agreement with the expected values of F1 and F2 in relation to the subglottal resonances was obtained. The transition from the expected quantal relation appears to occur in the age range between 2 and 3 years. [Work supported in part by NIH Grant R01-DC00075.]

F0 contour estimation based on time‐varying complex speech analysis

Robust F0 (Fundamental frequency) estimation plays an important role in speech processing such as speech coding and tonal speech recognition. We have already proposed robust F0 estimation algorithm based on time‐varying complex AR (TV‐CAR) speech analysis for analytic signal, in which the weighted autocorrelation function is calculated using the complex residual and then the F0 is searched as the peak sample for each frame [IEICE Trans. on Fundamentals, Vol.E91‐A.No.3] [IEICE Trans. on Fundamentals, Vol.E90‐A, No.8]. Although the algorithm can estimate more robust F0 estimation for IRS filtered speech corrupted by additive noise, the algorithm cannot perform better for non‐IRS filtered speech or slightly contaminated IRS‐filtered speech. In addition, the frame‐based F0 estimation cannot extract the F0 trajectories in the time‐domain. In order to cope with the drawbacks, this paper proposes quite simple F0 contour estimation algorithm based on the TV‐CAR speech analysis, in which the F0 contour is estimated by peak‐picking for the estimated time‐varying spectrum that is the same manner as formant frequency estimation. The experimental results demonstrate that the proposed method leads to more accurate continuous F0 estimation than the conventional one for high‐pitched speech due to the nature of analytic signal for non‐IRS filtered high‐pitched female speech.

Coarticulation in non-native speakers of English and French: An acoustic study

This study investigates whether coarticulation, often in the past assumed to be universal and “non-linguistic”, is learned by language learners. The study concerns C-to-V coarticulation, V-to-C coarticulation, and formant transition dynamics in CV syllables (C=coronal stop consonant, V=high back vowel) produced by English and French native speakers and by learners of English and French. As indices of the C-to-V and V-to-C coarticulation degree, absolute undershoot measurements were calculated by taking differences between the target F2 and the F2 in context. The F2 trajectory shapes were indexed by slope change values, which produced qualitatively different values between the trajectories characteristic of English and French. The findings suggest that the degree of coarticulation differed significantly across languages and that non-native learners showed signs of acquiring the patterns of coarticulation characteristic of the L2. The results also provide some evidence for acquisition of L2 patterns of coarticulation being a gradual process linked to amount of experience with L2. Early errors in non-native coarticulation appear to be the result of L1-to-L2 interference at the level of coarticulatory parameters, but learners gradually acquire native-like patterns of coarticulation. These observations of interference and gradualness of acquisition with experience are in accord with the general findings regarding the non-native acquisition of other phonetic details. Furthermore, the results suggest that the acquisition of coarticulation is not an automatic consequence of the acquisition of the phonological inventory. Learners do not appear to exploit any predictable relationship between the inventory size and the amount of coarticulation. They need to learn language-specific coarticulation, just as they need to learn other language-specific aspects of sound-patterning.

A method for multiple F0 detection of continuous audio with dissimilar timbres

This paper discusses a method for multiple fundamental frequency (F0) tracking performed on a polyphonic mixture of musical instruments with dissimilar timbres recorded on a single monaural track. For each time frame, given a number of possible F0 candidates, F0 trajectories are extracted using an HMM-like method where observations are the input spectra and each F0 candidate is represented by one of the hidden states. The F0 trajectories are optimized according to a harmonic overlap score for each F0 candidate and a continuity score that is based on a distance between the F0 candidate’s harmonic amplitudes and frequencies calculated for all possible candidate pairs at successive adjacent frames. This step leads to the formation of note tracks. In a second step, instrument identification is performed on harmonic spectra associated with these note tracks to group similar timbres together. The method is demonstrated with an acoustic audio recording of a woodwind quintet. Applications include audio-to-MIDI conversion, instrument/voice separation, and automatic music transcription. F0 tracks can also be used as high level features for music information retrieval systems.

Frequency and amplitude derivatives as syllable‐level F0 features

Most descriptions of tone and intonation focus on the alignment of F0 rises, falls, and inflection points relative to the phonetic unfolding of associated syllables. In this context, it is suggested that the relationship between proportional changes in F0 (treated modularly to eliminate the effects of period doubling and halving) and proportional changes in amplitude (within the frequency bands typically dominated by energy from voicing) provides an interesting characterization of F0 trajectories in relation to syllabic structure. Using data from conversational and broadcast speech in English and Chinese, parameters of this type (in combination with other acoustic measures) are used in identifying relevant features of tone and intonation.

The role of acoustic familiarity in the perception of L2 contrasts

Difficulties in L2 perception are often attributed to assimilation of the L2 contrast by an L1 category or to interference by L1‐relevant acoustic cues [Iverson et al. ‘‘Phonetic training with acoustic cue manipulations: A comparison of methods for teaching English /r/‐/p/ to Japanese adults,’’ J. Acoust. Soc. Am. 118, 3267–3278 (2005)]. These approaches assume that cue weights change slowly over developmental time and are relatively constant for short time scales. An alternative view is that cue extraction and weighting are dependent on the overall familiarity of the current sound, with more familiar sounds facilitating more robust cue extraction and better discrimination [e.g., McFadden and Callaway, JEP:HPP, 25, 543 (1999)]. This hypothesis was tested with the perception of English sounds by Japanese adults. The Japanese listeners are insensitive to a cue (F3 onset) when it is part of (putatively unfamiliar) English /ra/ and /la/ sounds. The effect of familiarity was evaluated by fixing the range of F3 onsets and manipulating the rest of each sound in two ways: (1) by changing the transition durations to be more suitable for a moraic rhythm and (2) by changing the F1, F2, and amplitude trajectory to be more like the Japanese /r/. Discrimination improved markedly with the latter manipulation, suggesting that cue sensitivity is conditioned upon overall acoustic familiarity. [Work supported by NIH MH64445.]

Rate and loudness manipulations in dysarthria: acoustic and perceptual findings.

Both rate reduction and increased loudness reportedly are associated with an increase in the size of the articulatory-acoustic working space and improved acoustic distinctiveness for speakers with dysarthria. Improved intelligibility also has been reported. Few studies have directly compared rate and loudness effects for speakers with dysarthria, however, although rate reduction and increasing vocal loudness are common treatment techniques. In the current study, 15 speakers with dysarthria secondary to multiple sclerosis, 12 speakers with dysarthria secondary to Parkinson's disease (PD), and 15 healthy controls read a passage in habitual, loud, and slow conditions. Rate and loudness variations were elicited using magnitude production. Acoustic measures included articulatory rate, sound pressure level, vowel space area, first moment difference measures, and F2 trajectory characteristics for diphthongs. Ten listeners scaled intelligibility for reading passages produced by the speakers with dysarthria. Relationships between intelligibility estimates and acoustic measures were determined by regression analysis. All speaker groups reduced articulatory rate for the slow condition and increased vocal intensity for the loud condition, relative to the habitual condition. Vowel acoustic distinctiveness, as indexed by vowel space area, was maximized in the slow condition, but stop consonant acoustic distinctiveness, as indexed by first moment difference measures, was maximized in the loud condition. F2 slope measures for diphthongs were not consistently affected by rate or loudness. Scaled intelligibility for speakers with PD also improved in the loud condition relative to both the habitual and slow conditions. Intelligibility estimates for speakers with dysarthria, however, were not strongly related to acoustic measures of supraglottal behavior. Findings are compared with previous studies, and hypotheses for future treatment studies are discussed.

Automatic Recognition of Pitch Movements Using Multilayer Perceptron and Time-Delay Recursive Neural Network

This letter demonstrates hidden Markov model (HMM), multilayer perceptron (MLP), and time-delay recursive neural network (TDRNN) architectures for the purpose of recognizing pitch accents given observation of the F0 and energy trajectories. At an insertion error rate of 25%, the deletion error rates of the MLP, TDRNN, and HMM are 13.2%, 7.9%, and 32.7%, respectively, despite the fact that both MLP and TDRNN have 70% fewer trainable parameters than the HMM. Error analysis suggests that low-pitch accents may require long-term context to correctly recognize, while high-pitch accents may be recognizable based on local pitch contour.

Effects of speaking rate on second formant trajectories of selected vocalic nuclei.

The effect of speaking rate variations on second formant (F2) trajectories was investigated for a continuum of rates. F2 trajectories for the schwa preceding a voiced bilabial stop, and one of three target vocalic nuclei following the stop, were generated for utterances of the form "Put a bV here, where V was /i/,/ae/ or /oI/. Discrete spectral measures at the vowel-consonant and consonant-vowel interfaces, as well as vowel target values, were examined as potential parameters of rate variation; several different whole-trajectory analyses were also explored. Results suggested that a discrete measure at the vowel consonant (schwa-consonant) interface, the F2off value, was in many cases a good index of rate variation, provided the rates were not unusually slow (vowel durations less than 200 ms). The relationship of the spectral measure at the consonant-vowel interface, F2 onset, as well as that of the "target" for this vowel, was less clearly related to rate variation. Whole-trajectory analyses indicated that the rate effect cannot be captured by linear compressions and expansions of some prototype trajectory. Moreover, the effect of rate manipulation on formant trajectories interacts with speaker and vocalic nucleus type, making it difficult to specify general rules for these effects. However, there is evidence that a small number of speaker strategies may emerge from a careful qualitative and quantitative analysis of whole formant trajectories. Results are discussed in terms of models of speech production and a group of speech disorders that is usually associated with anomalies of speaking rate, and hence of formant frequency trajectories.

Acoustic effects of speaking rate changes in articulatory models

Speaking rate effects on articulatory phasing, velocity, and magnitude have been reported in the speech-kinematic literature. Yet little is known about the acoustic consequences of manipulating these dimensions. Using an articulatory modeling method developed by Milenkovic, simulations of the acoustic effects of changes in articulatory phasing, velocity, and magnitude have been explored. Articulatory models have been optimized for speakers from the University of Wisconsin X-Ray Microbeam Speech Production Database. Preliminary work has revealed that models of rate change which use only a single dimension (e.g., phasing) may reveal less plausible acoustic solutions than models involving multiple dimensions. Using sample productions of ‘‘a boy’’ to determine articulatory parameter time histories, the acoustic consequences of rate change simulations have been described by the F2 value at onset (F2on), the maximum F2 value (F2tar), and the shape of the F2 trajectory throughout the diphthong. Speaking rate models that combine velocity and phase changes to articulatory parameters result in a reduction in the magnitude of the F2on effect, near elimination of any effect on F2tar, and a relative increase in the homogeneity of the F2 trajectory shapes across speaking rates when compared with phase-only models. [Work supported by NIH (NIDCD) P60 DC1409-10.]

Systematic downgrading for investigating ‘‘naturalness’’ in synthesized singing using STRAIGHT: A high quality VOCODER

Conceptual simplicity of the classical channel VOCODER provides a powerful means for systematic investigations on perceptual effects of speech related physical parameters when combined with modern computational power and signal processing theories. A modern version of channel VOCODER, STRAIGHT [Kawahara et al., Speech Commun. 27, 187–207 (1999)], which is also an extension to pitch-synchronous analysis and synthesis, generates naturally sounding resynthesized speech from the analyzed smooth time-frequency surface and source parameters such as F0. This high-quality resynthesis enables close investigations on naturalness deterioration as a function of feature modifications in the decomposed parameter domain; for example, detailed shape of a F0 trajectory, underlying parameters to determine F0 trajectory dynamics, group delay alignment of excitation pulses and aperiodicity/periodicity ratio of the excitation source and so on. One of potential advantages of this strategy is based on the fact that our perceptual function is highly nonlinear. The other source of advantage is virtually an independent parameter set which allows precise control of parameter deviations from the original analysis results. An overview of recent findings and modification demonstrations will be presented. [Work supported by CREST grant of Japanese Science and Technology Corporation.]