F0 Contour Research Articles

F0 Perturbation Due to Articulatory Movements: Filtering, Characterization and Applications

This paper addresses the issue of local disturbances in the fundamental frequency contour of speech, caused by the articulation of voiced/unvoiced consonant phonemes. Depending on the intended use of the F0 contour, these disturbances are usually eliminated by a filtering, smoothing or stylization procedure. These procedures that seek to preserve only the F0 points perceptually relevant, are generally applied roughly at a global level, which may not completely eliminate micro intonation in some cases or distort macro intonation in others. In this work we propose a local filtering algorithm based on a fine level analysis of the microprosodic morphologies. The performance of the algorithm is validated by a perceptual experiment. Assuming the algorithm allows partial/total disturbance elimination, we perform a statistical description of the perturbation morphologies. Statistics were collected from a corpus of 741 sentences designed to study Argentine Spanish prosody. The corpus was recorded by four professional announcers native speakers from Buenos Aires city. The results show that perturbation morphologies are affected by: consonant phoneme identity; global F0 contour shape; and speaker identity. As an application case, we use the proposed filtering algorithm as a pre-processing stage in our automatic prominent syllable detection system, with a statistically significant improvement in its performance.

Boundary-conditioned anticipatory tonal coarticulation in Standard Mandarin

The goal of this study is to examine (1) whether and how the F0 contour of a sequence of neutral tones in Standard Mandarin is affected by the following full tone due to anticipatory tonal coarticulation, (2) whether and how the degree of anticipatory tonal coarticulation varies as a function of the strength of an intervening boundary, and (3) whether boundary-conditioned anticipatory tonal coarticulatory variations can be accounted for by boundary-induced durational changes. By using generalized additive mixed models to analyze time-varying F0 contours of neutral-tone sequence, we are able to formally detect anticipatory tonal coarticulation effects across different boundary conditions while simultaneously taking into account individual variations of 20 speakers. Results show that (a) F0 contours of neutral-tone sequence can be affected by both assimilatory and dissimilatory anticipatory effects and anticipation could spread to non-adjacent neutral tones, that (b) anticipatory tonal coarticulation becomes weaker when crossing a strong boundary than when crossing a weak boundary, and that (c) boundary-induced anticipatory tonal coarticulation reduction cannot be simply accounted for by boundary-induced lengthening effect. Based on the results, we argue that the anticipatory tonal coarticulation patterns reported in this study are more consistent with the soft template mark-up language (Stem-ML) model than with the parallel encoding and target approximation (PENTA) model, and that boundary-induced anticipatory tonal coarticulation reduction is primarily the result of active speech planning.

Sentence Context Differentially Modulates Contributions of Fundamental Frequency Contours to Word Recognition in Chinese-Speaking Children With and Without Dyslexia.

Previous work has shown that children with dyslexia are impaired in speech recognition in adverse listening conditions. Our study further examined how semantic context and fundamental frequency (F0) contours contribute to word recognition against interfering speech in dyslexic and non-dyslexic children. Thirty-two children with dyslexia and 35 chronological-age-matched control children were tested on the recognition of words in normal sentences versus wordlist sentences with natural versus flat F0 contours against single-talker interference. The dyslexic children had overall poorer recognition performance than non-dyslexic children. Furthermore, semantic context differentially modulated the effect of F0 contours on the recognition performances of the two groups. Specifically, compared with flat F0 contours, natural F0 contours increased the recognition accuracy of dyslexic children less than non-dyslexic children in the wordlist condition. By contrast, natural F0 contours increased the recognition accuracy of both groups to a similar extent in the sentence condition. These results indicate that access to semantic context improves the effect of natural F0 contours on word recognition in adverse listening conditions by dyslexic children who are more impaired in the use of natural F0 contours during isolated and unrelated word recognition. Our findings have practical implications for communication with dyslexic children when listening conditions are unfavorable.

The effect of fundamental frequency contour similarity on multi-talker listening in older and younger adults.

Older adults with hearing loss have greater difficulty recognizing target speech in multi-talker environments than young adults with normal hearing, especially when target and masker speech streams are perceptually similar. A difference in fundamental frequency (f0) contour depth is an effective stream segregation cue for young adults with normal hearing. This study examined whether older adults with varying degrees of sensorineural hearing loss are able to utilize differences in target/masker f0 contour depth to improve speech recognition in multi-talker listening. Speech recognition thresholds (SRTs) were measured for speech mixtures composed of target/masker streams with flat, normal, and exaggerated speaking styles, in which f0 contour depth systematically varied. Computational modeling estimated differences in energetic masking across listening conditions. Young adults had lower SRTs than older adults; a result that was partially explained by differences in audibility predicted by the model. However, audibility differences did not explain why young adults experienced a benefit from mismatched target/masker f0 contour depth, while in most conditions, older adults did not. Reduced ability to use segregation cues (differences in target/masker f0 contour depth), and deficits grouping speech with variable f0 contours likely contribute to difficulties experienced by older adults in challenging acoustic environments.

F0 Modeling for Isarn Speech Synthesis using Deep Neural Networks and Syllable-level Feature Representation

The generation of the fundamental frequency (F0) plays an important role in speech synthesis, which directly influences the naturalness of synthetic speech. In conventional parametric speech synthesis, F0 is predicted frame-by-frame. This method is insufficient to represent F0 contours in larger units, especially tone contours of syllables in tonal languages that deviate as a result of long-term context dependency. This work proposes a syllable-level F0 model that represents F0 contours within syllables, using syllable-level F0 parameters that comprise the sampling F0 points and dynamic features. A Deep Neural Network (DNN) was used to represent the relationships between syllable-level contextual features and syllable-level F0 parameters. The proposed model was examined using an Isarn speech synthesis system with both large and small training sets. For all training sets, the results of objective and subjective tests indicate that the proposed approach outperforms the baseline systems based on hidden Markov models and DNNS that predict F0 values at the frame level

Early cortical processing of pitch height and the role of adaptation and musicality

Pitch is an important perceptual feature; however, it is poorly understood how its cortical correlates are shaped by absolute vs relative fundamental frequency (f0), and by neural adaptation. In this study, we assessed transient and sustained auditory evoked fields (AEFs) at the onset, progression, and offset of short pitch height sequences, taking into account the listener's musicality. We show that neuromagnetic activity reflects absolute f0 at pitch onset and offset, and relative f0 at transitions within pitch sequences; further, sequences with fixed f0 lead to larger response suppression than sequences with variable f0 contour, and to enhanced offset activity. Musical listeners exhibit stronger f0-related AEFs and larger differences between their responses to fixed vs variable sequences, both within sequences and at pitch offset. The results resemble prominent psychoacoustic phenomena in the perception of pitch contours; moreover, they suggest a strong influence of adaptive mechanisms on cortical pitch processing which, in turn, might be modulated by a listener's musical expertise.

Effects of Fundamental Frequency Contours on Sentence Recognition in Mandarin-Speaking Children With Cochlear Implants.

Purpose Fundamental frequency (F0) is the primary acoustic cue for lexical tone perception in tonal languages but is processed in a limited way in cochlear implant (CI) systems. The aim of this study was to evaluate the importance of F0 contours in sentence recognition in Mandarin-speaking children with CIs and find out whether it is similar to/different from that in age-matched normal-hearing (NH) peers. Method Age-appropriate sentences, with F0 contours manipulated to be either natural or flattened, were randomly presented to preschool children with CIs and their age-matched peers with NH under three test conditions: in quiet, in white noise, and with competing sentences at 0 dB signal-to-noise ratio. Results The neutralization of F0 contours resulted in a significant reduction in sentence recognition. While this was seen only in noise conditions among NH children, it was observed throughout all test conditions among children with CIs. Moreover, the F0 contour-induced accuracy reduction ratios (i.e., the reduction in sentence recognition resulting from the neutralization of F0 contours compared to the normal F0 condition) were significantly greater in children with CIs than in NH children in all test conditions. Conclusions F0 contours play a major role in sentence recognition in both quiet and noise among pediatric implantees, and the contribution of the F0 contour is even more salient than that in age-matched NH children. These results also suggest that there may be differences between children with CIs and NH children in how F0 contours are processed.

Vocal discrimination of African lions and its potential for collar-free tracking

ABSTRACT Previous research has shown that African lions (Panthera leo) have the ability to discriminate between conspecific vocalisations, but little is known about how individual identity is conveyed in the spectral structure of roars. Using acoustic – accelerometer biologgers that allow vocalisations to be reliably associated with individual identity, we test for vocal individuality in the fundamental frequency (f0) of roars from 5 male lions, firstly by comparing simple f0 summary features and secondly by modelling the temporal pattern of the f0 contour. We then assess the application of this method for discriminating between individuals using passive acoustic monitoring. Results indicate that f0 summary features only allow for vocal discrimination with 70.7% accuracy. By comparison, vocal discrimination can be achieved with an accuracy of 91.5% based on individual differences in the temporal pattern of the f0 sequence. We further demonstrate that passively recorded lion roars can be localised and differentiated with similar accuracy. The existence of individually unique f0 contours in lion roars and their relatively lower attenuation indicates a likely mechanism enabling individual lions to identify conspecifics over long distances. These differences can be exploited by researchers to track individuals across the landscape and thereby supplement conventional lion monitoring approaches.

Tonal Contour Generation for Isarn Speech Synthesis Using Deep Learning and Sampling-Based F0 Representation

The modeling of fundamental frequency (F0) in speech synthesis is a critical factor affecting the intelligibility and naturalness of synthesized speech. In this paper, we focus on improving the modeling of F0 for Isarn speech synthesis. We propose the F0 model for this based on a recurrent neural network (RNN). Sampled values of F0 are used at the syllable level of continuous Isarn speech combined with their dynamic features to represent supra-segmental properties of the F0 contour. Different architectures of the deep RNNs and different combinations of linguistic features are analyzed to obtain conditions for the best performance. To assess the proposed method, we compared it with several RNN-based baselines. The results of objective and subjective tests indicate that the proposed model significantly outperformed the baseline RNN model that predicts values of F0 at the frame level, and the baseline RNN model that represents the F0 contours of syllables by using discrete cosine transform.

Visual and auditory cues of assertions and questions in brazilian portuguese and Mexican Spanishy

The aim of this paper is to compare the multimodal production of questions in two different language varieties: Brazilian Portuguese and Mexican Spanish. Descriptions of the auditory and visual cues of two speech acts, assertions and questions, are presented based on Brazilian and Mexican corpora. The sentence “Como você sabe” was produced as an yes-no (echo) question and an assertion by ten speakers (five male) from Rio de Janeiro and the sentence “Apaga la tele” was produced as a yes-no question and an assertion by five speakers (three male) from Mexico City. The results show that, whereas the Brazilian Portuguese and Mexican Spanish assertions are produced with different F0 contours and different facial expressions, questions in both languages are produced with specific F0 contours but similar facial expressions. The outcome of this comparative study suggests that lowering the eyebrows, tightening the lid and wrinkling the nose can be considered question markers in both language varieties.

Context-related variation in the peripartum vocalisations and phonatory behaviours of Holstein-Friesian dairy cows

Context-related information can be reflected within mammalian vocalisations and could in turn be studied to benefit production animals. Whilst previous research has revealed the contextual, and even emotional content of goat, pig and horse vocalisations, cattle vocalisations remain relatively less explored. In this study we recorded the vocalisations and accompanying phonatory behaviours of 19 Holstein-Friesian dairy cows (n = 10 primiparous, n = 9 multiparous) during the peripartum contexts of parturition with dystocia and fence-line calf separation. Findings revealed that vocal structure was context dependent, with parturition calls longer in duration, higher in fundamental frequency (F0) min and F0 start, containing more nonlinear phenomena and more minor F0 modulations. By contrast, calls during calf separation were distinguishable by their F0 contour, having a higher F0 mean and F0 max, and greater F0 var, F0 cv and F0 absolute slope. During parturition, primiparous cows emitted calls longer in duration, lower in F0 mean, and containing more F0 modulations than their multiparous conspecifics. Parturient cows expressed a greater number of open mouth calls, whereas mixed calls were more common in the calf separation context. Additionally, 82 % of the primiparous and 10 % of the multiparous cow vocalisations involved tongue exposure from the mouth during parturition, which may represent a visual indicator of compromised welfare. Overall, our results show that the context-related variation is conveyed through an integration of vocal and visual sensory modalities, with the combination of an open mouth configuration, tongue protrusion, and spectrographic properties such as an overall lower F0 and longer duration during parturition likely enhancing vocal transmission over longer distances to inform more receivers about the urgency of this context. Knowledge of these vocal and behavioural cues could be adopted on farm to determine the welfare of peripartum cattle and are instrumental for future cattle bioacoustics research.

The role of F0 and phonation cues in Cantonese low tone perception.

For languages that primarily exploit F0 to signal tonal contrast, the role of phonation cues in tonal perception remains controversial. This study revisits the use of F0 and phonation cues in Cantonese low tone perception (tone 4, 21/tone 6, 22) using synthesized stimuli. In line with previous studies, F0 contour and height were found to be the most salient cues, with F0 height being more important. The effects of non-modal phonation (creaky and breathy voice) were relatively small. Non-modal phonation enhanced low tone perception only in the low F0 range. The results are consistent with the differential integration hypothesis that the perceptual role of phonation is dependent on F0 and that phonation cues integrate with F0 differently depending on F0 height.

Production-perception relationship of Mandarin tones as revealed by critical perceptual cues.

The relationship of lexical tone production and perception has not been well studied. Using Mandarin tone, this research tests the hypothesis that a production-perception link is revealed by critical perceptual cues. The critical status of perceptual tonal cues was determined by perceptual cue weights, showing fundamental frequency (F0) contour as being more critical than height. Then, tone production features were examined for critical F0 contour (slope, curvature, turning-point location) and non-critical F0 height (mean, onset) cues. A production-perception correlation was found for F0 contour but not height cues, suggesting that critical perceptual cues dictate the relationship between production and perception.

Word segmentation in Persian continuous speech using F0 contour

Word segmentation in Persian continuous speech using F0 contour

Partial perceptual compensation for nasal coarticulation is robust to fundamental frequency variation.

Listeners show better-than-chance discrimination of nasalized and oral vowels occurring in appropriate consonantal contexts. Yet, the methods for investigating partial perceptual compensation for nasal coarticulation often include nasal and oral vowels containing naturally different pitch contours. Listeners may therefore be discriminating between these vowels based on pitch differences and not nasalization. The current study investigates the effect of pitch variation on the discrimination of nasalized and oral vowels in C_N and C_C items. The f0 contour of vowels within paired discrimination trials was varied. The results indicate that pitch variation does not influence patterns of partial perceptual compensation for coarticulation.

Statistical Modeling of Application Completeness of Two Tone Sandhi Rules / 两组变调规则应用程度的统计建模

This study examines the application of two Mandarin tone sandhi rules on real and wug words varying in degrees of phonological and semantic dependency. Using two statistical methods, we examined the surface f0 contours and underlying pitch targets. For the third tone sandhi, a lexical effect was discovered on the relationship between “word-likeness” of stimuli and completeness of rule application. The degree of application for the half-third sandhi tone, however, was less consistent. This study offers new insights in the debate between categorical and gradient views of sandhi rules. We propose three hypothesized situations and argue that the Mandarin tone sandhi rule application involves computation of sandhi forms, though it becomes more incomplete on wug words containing more illegitimate morphemes. Finally, between the two rules, the application of the third tone sandhi rule is less phonetically motivated and more biased in wug words, exhibiting differences between real words and wug words.

Approximations to the Voice of a Cochlear Implant: Explorations With Single-Sided Deaf Listeners.

Fourteen single-sided deaf listeners fit with an MED-EL cochlear implant (CI) judged the similarity of clean signals presented to their CI and modified signals presented to their normal-hearing ear. The signals to the normal-hearing ear were created by (a) filtering, (b) spectral smearing, (c) changing overall fundamental frequency (F0), (d) F0 contour flattening, (e) changing formant frequencies, (f) altering resonances and ring times to create a metallic sound quality, (g) using a noise vocoder, or (h) using a sine vocoder. The operations could be used singly or in any combination. On a scale of 1 to 10 where 10 was a complete match to the sound of the CI, the mean match score was 8.8. Over half of the matches were 9.0 or higher. The most common alterations to a clean signal were band-pass or low-pass filtering, spectral peak smearing, and F0 contour flattening. On average, 3.4 operations were used to create a match. Upshifts in formant frequencies were implemented most often for electrode insertion angles less than approximately 500°. A relatively small set of operations can produce signals that approximate the sound of the MED-EL CI. There are large individual differences in the combination of operations needed. The sound files in Supplemental Material approximate the sound of the MED-EL CI for patients fit with 28-mm electrode arrays.

A quantitative analysis of tone sandhi in Standard Mandarin and Nanjing Mandarin based on surface pitch contours and underlying pitch targets

Abstract Two statistical modelling methods are first used to quantify the third tone sandhi in Standard Mandarin in which the first falling-rising T3 becomes a rising T2 in the T3 + T3 disyllabic context. Growth curve analysis suggests non-neutralization of the surface F0 contours of the sandhi tone and its corresponding citation tone (T2), whereas a quantitative F0 target approximation model reveals neutralization of their underlying pitch targets, congruent with the stable and categorical tone shift properties of tone sandhi processes. The same statistical procedure is successfully extended to examine tone sandhi rules in Nanjing Mandarin. Our analysis leads to a proposal to change tonal values based on the Chao’s number system of some Nanjing Mandarin monosyllabic tones. The transformation method from acoustic data to Chao’s number applies well to our data and proves superior to those previously described in the literature.

Speech emotion recognition using discriminative dimension reduction by employing a modified quantum-behaved particle swarm optimization algorithm

In recent years, Speech Emotion Recognition (SER) has received considerable attention in affective computing field. In this paper, an improved system for SER is proposed. In the feature extraction step, a hybrid high-dimensional rich feature vector is extracted from both speech signal and glottal-waveform signal using techniques such as MFCC, PLPC, and MVDR. The prosodic features derived from fundamental frequency (f0) contour are also added to this feature vector. The proposed system is based on a holistic approach that employs a modified quantum-behaved particle swarm optimization (QPSO) algorithm (called pQPSO) to estimate both the optimal projection matrix for feature-vector dimension reduction and Gaussian Mixture Model (GMM) classifier parameters. Since the problem parameters are in a limited range and the standard QPSO algorithm performs a search in an infinite range, in this paper, the QPSO is modified in such a way that it uses a truncated probability distribution and makes the search more efficient. The system works in real-time and is evaluated on three standard emotional speech databases Berlin database of emotional speech (EMO-DB), Surrey Audio-Visual Expressed Emotion (SAVEE) and Interactive Emotional Dyadic Motion Capture (IEMOCAP). The proposed method improves the accuracy of the SER system compared to classical methods such as FA, PCA, PPCA, LDA, standard QPSO, wQPSO, and deep neural network, and also outperforms many state-of-the-art recent approaches that use the same datasets.

Duration and amplitude of tone production in children with cochlear implants

The purpose of the present study was two-fold: (1) to examine whether Mandarin-speaking children with CIs showed distinctive durational and amplitude features for the four lexical tones in their tone production; (2) to compare the duration and amplitude patterns of Mandarin lexical tones in monosyllables produced in citation form between CI children and age-matched normal-hearing (NH) children. The participants included 14 prelingually deafened Mandarin-speaking children with CIs and 14 NH children, all aged between 2.9 and 8.3 years old. Each participant produced five CV syllables (fa, fu, pi, xu, ke) in four tones through a tone drill activity. The vowel duration and rms amplitude values at nine equidistant time locations over the vowel duration were obtained. The results revealed that the CI children can produce distinctive duration and amplitude features for the four lexical tones. Their durational pattern and amplitude contours were highly similar to the NH children on tone 1, 2, and 4 but differed from the NH children on tone 3. In addition, NH children showed positively correlated amplitude contour and F0 contour but the CI children demonstrated inconsistent amplitude and F0 contours for tone 3. This finding suggested that the amplitude contour of a tone and the F0 contour of the same tone may not always be highly correlated.