F0 Trajectories Research Articles

Bayesian Singing Transcription Based on a Hierarchical Generative Model of Keys, Musical Notes, and F0 Trajectories

This article describes automatic singing transcription (AST) that estimates a human-readable musical score of a sung melody represented with quantized pitches and durations from a given music audio signal. To achieve the goal, we propose a statistical method for estimating the musical score by quantizing a trajectory of vocal fundamental frequencies (F0s) in the time and frequency directions. Since vocal F0 trajectories considerably deviate from the pitches and onset times of musical notes specified in musical scores, the local keys and rhythms of musical notes should be taken into account. In this article we propose a Bayesian hierarchical hidden semi-Markov model (HHSMM) that integrates a musical score model describing the local keys and rhythms of musical notes with an F0 trajectory model describing the temporal and frequency deviations of an F0 trajectory. Given an F0 trajectory, a sequence of musical notes, that of local keys, and the temporal and frequency deviations can be estimated jointly by using a Markov chain Monte Carlo (MCMC) method. We investigated the effect of each component of the proposed model and showed that the musical score model improves the performance of AST.

Tonal languages speech synthesis using an indirect pitch markers and the quantitative target approximation methods

Synthesizing tones plays an important role in text-to-speech systems of tonal languages. To accomplish this, the two important steps are to determine the pitch markers of voice utterances and synthesize F0 trajectories for lexical tones. In this paper, we propose two efficient algorithms, one of them is to locate the pitch markers at the peaks of the cumulative signal of each voiced part of the input utterance and the other is to generate F0 trajectories of tones with quantitative target approximation (qTA) parameters of Xu model. The experimentation has shown that the proposed algorithms present pitch markers with high accuracy which has enabled us to generate tones with complex shapes.

The acoustics of feeling: Emotional prosody in the StoryCorps corpus

Effective communication depends not only on lexical content but also on how language is produced and perceived. Prosodic elements such as intensity, pitch accent (i.e., the pattern of low and high tones used in a stressed word), and intonation, have been suggested to aid in conveying emotional affect (e.g., happiness) in acted speech. For example, the word “yes” produced with a falling tone, high intensity, and large pitch range may indicate a high-arousal emotion like anger, whereas the same word produced with a flatter intonation, and lower intensity may indicate a low-arousal emotion like gloominess. Understanding the emotion with which words are produced facilitates appropriate communication because it tells the interlocutor how to best respond. Although this is intuitively true, the role of prosody in conveying emotional meaning is understudied, particularly in natural speech. This project thus utilizes StoryCorps, an extensive corpus of naturalistic interviews (publicly available at www.storycorps.org), to acoustically analyze pitch-accent usage in speech conveying different emotional affects. Portions of the StoryCorps interviews that convey overt emotional effect are selected and transcribed, and f0 trajectory, pitch range, duration, and intensity are tracked across stressed vowels to explore whether there are distinctive pitch-accent patterns used to convey different emotions.

Prosodic encoding of focus in Hijazi Arabic

This paper presents findings of the first systematic acoustic analysis of focus prosody in Hijazi Arabic (HA), an under-researched Arabic dialect. A question-answer paradigm was used to elicit information and contrastive focus at different sentence locations in comparison with their neutral focus counterparts. Systematic acoustic analyses were performed to compare all the focus conditions, in terms of both continuous F0 trajectories and specific acoustic measurements. Results show that focused words have significantly expanded excursion size, higher maximum F0 and longer duration. Post-focus words have significantly lowered F0 (except in the case of penultimate focus). Pre-focus words, in contrast, lack systematic changes. These patterns are consistent with previously reported prosodic patterns of focus in other Arabic dialects. They are also consistent with a number of others languages that have also been applied similar systematic acoustic analyses. Thus Arabic appears to belong to a group of languages that all exhibit post-focus compression (PFC), as opposed to languages that lack PFC. In addition, the results also show evidence of prosodic differences between contrastive focus and information focus. This difference, however, is interpreted as due to a methodological feature that allowed elicitation of incredulity related to contrastive focus, rather than as a language-specific property. It is also argued that the possible involvement of incredulity in focus marking needs further research.

Non-native production of Mandarin T2/T3 pairs in disyllabic real and pseudo words

Mandarin rising (T2) and falling rising (T3) tones have been found to be the most confusing tone pair for native and non-native production due to similar F0 trajectory and equivalent lexical function known as T3 sandhi. This ongoing preliminary study investigates non-native production of T2 and T3 combinations in disyllabic real and pseudo words, aiming to examine whether T3 sandhi is realized solely at the real word level. Another goal of this study is to explore the effects of two distinctive orthographic systems in Mandarin (i.e., the sound-based pinyin and the meaning-based Chinese characters) on tone production. Ten native English speakers with 8-12 months experience in learning Mandarin participated in the production task. The stimuli were combinations of T2-T2, T2-T3, T3-T3, and T3-T2 grouped into four semantic and orthographic contexts: real and pseudo words in pinyin only, and those in both pinyin and Chinese characters. In the perception task, five native listeners of Mandarin identified the tone category for the first and second syllables. The results are expected to indicate the effects of tonal context, word meaning, and/or orthographic input on T2/T3 productions, and will be compared with earlier findings (e.g., Mok et al., 2018; Zhang et al., 2015).

Spectral change and duration as cues in Australian English listeners' front vowel categorization.

Australian English /iː/, /ɪ/, and /ɪə/ exhibit almost identical average first (F1) and second (F2) formant frequencies and differ in duration and vowel inherent spectral change (VISC). The cues of duration, F1 × F2 trajectory direction (TD) and trajectory length (TL) were assessed in listeners' categorization of /iː/ and /ɪə/ compared to /ɪ/. Duration was important for distinguishing both /iː/ and /ɪə/ from /ɪ/. TD and TL were important for categorizing /iː/ versus /ɪ/, whereas only TL was important for /ɪə/ versus /ɪ/. Finally, listeners' use of duration and VISC was not mutually affected for either vowel compared to /ɪ/.

Making predictable unpredictable with style – Behavioral and electrophysiological evidence for the critical role of prosodic expectations in the perception of prominence in speech

Perceptual prominence of linguistic units such as words has been earlier connected to the concepts of predictability and attentional orientation. One hypothesis is that low-probability prosodic or lexical content is perceived as prominent due to the surprisal and high information value associated with the stimulus. However, the existing behavioral studies have used stimulus manipulations that follow or violate typical linguistic patterns present in the listeners’ native language, i.e., assuming that the listeners have already established a model for acceptable prosodic patterns in the language. In the present study, we investigated whether prosodic expectations and the resulting subjective impression of prominence is affected by brief statistical adaptation to suprasegmental acoustic features in speech, also in the case where the prosodic patterns do not necessarily follow language-typical marking for prominence. We first exposed listeners to five minutes of speech with uneven distributions of falling and rising fundamental frequency (F0) trajectories on sentence-final words, and then tested their judgments of prominence on a set of new utterances. The results show that the probability of the F0 trajectory affects the perception of prominence, a less frequent F0 trajectory making a word more prominent independently of the absolute direction of F0 change. In the second part of the study, we conducted EEG-measurements on a set of new subjects listening to similar utterances with predominantly rising or falling F0 on sentence-final words. Analysis of the resulting event-related potentials (ERP) reveals a significant difference in N200 and N400 ERP-component amplitudes between standard and deviant prosody, again independently of the F0 direction and the underlying lexical content. Since N400 has earlier been associated with semantic processing of stimuli, this suggests that listeners implicitly track probabilities at the suprasegmental level and that predictability of a prosodic pattern during a word has an impact to the semantic processing of the word. Overall, the study suggests that prosodic markers for prominence are at least partially driven by the statistical structure of recently perceived speech, and therefore prominence perception could be based on statistical learning mechanisms similar to those observed in early word learning, but in this case operating at the level of suprasegmental acoustic features.

Investigating very deep highway networks for parametric speech synthesis

Deep neural networks are powerful tools for classification and regression tasks. While a network with more than 100 hidden layers has been reported for image classification, how such a non-recurrent neural network with more than 10 hidden layers will perform for speech synthesis is as yet unknown. This work investigates the performance of deep networks on statistical parametric speech synthesis, particularly the question of whether different acoustic features can be better generated by a deeper network. To answer this question, this work examines a multi-stream highway network that separately generates spectral and F0 acoustic features based on the highway architecture. Experiments on the Blizzard Challenge 2011 corpus show that the accuracy of the generated spectral features consistently improves as the depth of the network increases from 2 to 40, but the F0 trajectory can be generated equally well by either a deep or a shallow network. Additional experiments on a single-stream highway and normal feedforward network, both of which generate spectral and F0 features from a single network, show that these networks must be deep enough to generate both kinds of acoustic features well. The difference in the performance of multi- and single-stream highway networks is further analyzed on the basis of the networks’ activation and sensitivity to input features. In general, the highway network with more than 10 hidden layers, either multi- or single-stream, performs better on the experimental corpus than does a shallow network.

The sound of trustworthiness: Acoustic-based modulation of perceived voice personality

When we hear a new voice we automatically form a "first impression" of the voice owner's personality; a single word is sufficient to yield ratings highly consistent across listeners. Past studies have shown correlations between personality ratings and acoustical parameters of voice, suggesting a potential acoustical basis for voice personality impressions, but its nature and extent remain unclear. Here we used data-driven voice computational modelling to investigate the link between acoustics and perceived trustworthiness in the single word "hello". Two prototypical voice stimuli were generated based on the acoustical features of voices rated low or high in perceived trustworthiness, respectively, as well as a continuum of stimuli inter- and extrapolated between these two prototypes. Five hundred listeners provided trustworthiness ratings on the stimuli via an online interface. We observed an extremely tight relationship between trustworthiness ratings and position along the trustworthiness continuum (r = 0.99). Not only were trustworthiness ratings higher for the high- than the low-prototypes, but the difference could be modulated quasi-linearly by reducing or exaggerating the acoustical difference between the prototypes, resulting in a strong caricaturing effect. The f0 trajectory, or intonation, appeared a parameter of particular relevance: hellos rated high in trustworthiness were characterized by a high starting f0 then a marked decrease at mid-utterance to finish on a strong rise. These results demonstrate a strong acoustical basis for voice personality impressions, opening the door to multiple potential applications.

A model of Mandarin Chinese question intonation

Echo questions in Mandarin Chinese provide an interesting case study of the interaction between lexical tone and intonation because it appears that echo questions are distinguished from declaratives by modifying the F0 trajectory of the sentence-final lexical tone in ways that depend on the identity of that tone. The most general effect is that the offset of the final tone is higher in a question than in a declarative rendition of the same sentence. If the final tone is high or falling, F0 is raised throughout the tone, but if the final tone if low or rising, the F0 minimum is not raised in questions. We propose a quantitative analysis according to which question intonation consists of a high boundary tone realized simultaneously with the offset of the final lexical tone. Compromise between the boundary tone and targets for the lexical tone results in raising of the offset of the tone. Additional effects result from interactions with other targets pertaining to the shape of the lexical tone. For example, simply raising the offset of a falling tone would result in failure to realize a sufficiently steep fall, so the onset of the fall is raised as well.

A Modeling Study of the Effects of Vocal Tract Movement Duration and Magnitude on the F2 Trajectory in CV Words

This study used a computational vocal tract model to investigate the relationship of diphthong duration and vocal tract movement magnitude to measures of the F2 trajectory in CV words. Three words (bough, boy, and buy) were simulated on the basis of an adult female vocal tract model, in which the model parameters were estimated from audio recordings of a female talker. Model parameters were then modified to generate 35 simulations of each word corresponding to 7 different durations and 5 movement magnitude settings. In addition, these simulations were repeated with vocal tract lengths representative of an adult male and an approximately 6-year-old child. On the basis of univariate analysis, measures of frequency predicted changes in magnitude, and temporal measures predicted changes in speaking rate consistent with the hypothesis. The combined effects of duration and magnitude showed that F2 was more sensitive to changes in magnitude at shorter word durations compared with longer word durations. This finding held across words and vocal tract length. Results suggest that there is an interaction between duration and magnitude that affects the slope of the F2 trajectory. The next step is to relate kinematics to F2 trajectory output using real speakers.

Investigation of using the highway network to predict the F0 trajectory for text-to-speech synthesis

Generation of the F0 trajectory is critical for Text-to-Speech (TTS) synthesis. In most of the recently proposed TTS systems based on neural networks, F0, together with the spectral features of speech, is predicted by a single neural network. However, our recent work, which utilizes the highway network with trainable gates to bypass the non-linear transformation layers, has shown that more accurate F0 can be generated by separating the network for the F0 and the spectral streams. It is also observed that most of the layers for the F0 stream are inactive. In this work, the inactive network for the F0 stream will be analyzed, and empirical evidence will show that a shallow network as well as a deep network for the F0 stream can perform. Additionally, this work will analyze the sensitivity values of the hidden units in the highway network to identify the uninformative input features for predicting the F0 trajectory, which include the automatically inferred ToBI tags.

Study on relationship between subjective reproducibility of individuality and distance measure for vibrato of singing voice

In this research, a distance measure between two vibratos had been developed to evaluate singing style. This paper verifies whether the measure properly represents subjective distance by the following two subjective experiments. In these experiments, synthetic sounds whose vibratory amplitude computed from a target singer’s singing is variable were used, and listeners were asked to answer reproducibility of individuality of the target singer for the synthetic sound. In the first experiment, listeners were asked to remember vibrato of the target singer by listening to the singing. As a result, it was found that not only synthetic sounds showing small distance to the target singer in the distance measure but also those not showing small distance produced higher subjective reproducibility, as long as the sound whose vibratory amplitude is large, i.e., includes deformation in the F0 trajectory. To confirm whether this tendency is caused by the experimental procedure, the second experiment that listeners were compared with the reference synthetic sound was conducted. As a result, it was found that the results were mostly the same as those of the first experiment, and the vibrato rate affects the subjective reproducibility more strongly, as the magnitude of vibratory amplitude becomes large.

Formant trajectories in the realization of 3 Malayalam rhotics

Malayalam has a typologically rare set of three rhotic consonants—an alveolar trill, an alveolar tap, and a retroflex/postalveolar approximant. The first one is described as velarized, while the other two are somewhat palatalized. Previous acoustic work by Punnoose et al. [Phonetica 70, 274-297 (2013)] has established that the contrast can be differentiated by formant values around the constriction, as well as at the midpoint of adjacent vowels. Less clear, however, is the temporal extent of these formant differences and their overall dynamics. In this study, we analyze three rhotics produced by 10 male Malayalam speakers. Measurements of F1-F4 were taken at 10 points throughout the VCV interval and subjected to a series of Smoothing Spline ANOVAs. The results revealed differences in F2 (tap, approximant > trill) and F3 (tap, trill > approximant) among the rhotics extended through most of the preceding and following vowel duration, while differences in F1 (trill > tap, approximant) were less extensive. While similar to the tap in its F2 trajectory, the approximant showed a more abrupt fall during the following vowel, seemingly indicative of a rapid tongue posture change at the release. [Work supported by SSHRC.]

The Korean Prevocalic Palatal Glide: A Comparison with the Russian Glide and Palatalization

Phonetic studies of the Korean prevocalic glides have often suggested that they are shorter in duration than those of languages like English, and lack a prolonged steady state. In addition, the formant frequencies of the Korean labiovelar glide are reported to be greatly influenced by the following vowel. In this study the Korean prevocalic palatal glide is investigated vis-à-vis the two phonologically similar configurations of another language - the glide /j/ and the secondary palatalization of Russian, with regard to the inherent duration of the glide component, F2 trajectory, vowel-to-glide coarticulation and glide-to-vowel coarticulation. It is revealed that the Korean palatal glide is closer to the Russian palatalization in duration and F2 trajectory, indicating a lack of steady state, and to the Russian segmental glide in the vowel-to-glide coarticulation degree. When the glide-to-vowel coarticulation is considered, the Korean palatal glide is distinguished from both Russian categories. The results suggest that both the Korean palatal glide and the Russian palatalization involve significant articulatory overlap, the former with the vowel and the latter with the consonant. Phonological implications of such a difference in coarticulation pattern are discussed, as well as the comparison between the Korean labiovelar and palatal glides.

Modeling F0 trajectories in hierarchically structured deep neural networks

This paper investigates F0 modeling of speech in deep neural networks (DNN) for statistical parametric speech synthesis (SPSS). Recently, DNN has been applied to the acoustic modeling of SPSS and has shown good performance in characterizing complex dependencies between contextual features and acoustic observations. However, the additive nature and long-term suprasegmental property of F0 features have not been fully exploited in the existing DNN-based SPSS. Two different model structures, cascade DNN and parallel DNN are proposed to embody the hierarchical and additive properties of the F0 in DNN-based prosody modeling. In the cascade structure, the DNN-predicted F0 contours of higher levels are used as input to the DNN of the current level. In the parallel structure, F0 components corresponding to different prosody levels are separately generated by DNNs and added together to obtain the final F0 contour. An optimized discrete cosine transform (DCT) is used to extract long-term F0 features at syllable, word, and phrase levels. The experimental results show that our approach yields better subjective performance than either the conventional HMM or DNN approaches. Among all compared systems, the parallel DNN achieves the best objective and subjective performance.

Perception of Sentence Stress in Speech Correlates With the Temporal Unpredictability of Prosodic Features.

Numerous studies have examined the acoustic correlates of sentential stress and its underlying linguistic functionality. However, the mechanism that connects stress cues to the listener's attentional processing has remained unclear. Also, the learnability versus innateness of stress perception has not been widely discussed. In this work, we introduce a novel perspective to the study of sentential stress and put forward the hypothesis that perceived sentence stress in speech is related to the unpredictability of prosodic features, thereby capturing the attention of the listener. As predictability is based on the statistical structure of the speech input, the hypothesis also suggests that stress perception is a result of general statistical learning mechanisms. To study this idea, computational simulations are performed where temporal prosodic trajectories are modeled with an n-gram model. Probabilities of the feature trajectories are subsequently evaluated on a set of novel utterances and compared to human perception of stress. The results show that the low-probability regions of F0 and energy trajectories are strongly correlated with stress perception, giving support to the idea that attention and unpredictability of sensory stimulus are mutually connected.

Variability in analyst decisions during the computation of numerical likelihood ratios

This study assesses the extent to which likelihood ratios (LRs) are affected by analyst decisions regarding the number of reference speakers, number of tokens per speaker, and degree of linguistic match between the suspect-offender and the reference data. Using F1 and F2 trajectories from spontaneous /u?/ vowels, LRs were computed against a reference set of up to 120 speakers, and between 2 and 13 tokens per speaker. LR scores proved robust when using more than 30 speakers, although validity improved as the sample size increased. Regional dialect mismatch was found to overestimate same-speaker scores by an average of one order of log10 magnitude. For different-speaker pairs, dialect mismatch generally resulted in more false hits. The results indicate that small amounts of reference data should generally be avoided, and that an awareness of the complexity of sociophonetic variation is essential in the definition of the relevant population for voice comparison.

The relevant population in forensic voice comparison: Effects of varying delimitations of social class and age

In forensic voice comparison, the expert is typically instructed to compare the voices in a pair of offender and suspect samples. To appropriately evaluate the strength of such evidence, it is necessary to consider both the similarity between the samples and their typicality in the wider, relevant population. This paper considers the effects of different definitions of the relevant population when computing numerical likelihood ratios (LR), with specific regard to socio-economic class and age. Input data consist of cubic polynomial estimations of F1, F2 and F3 trajectories for /eɪ/ in New Zealand English. Calibrated LRs are computed for a sociolinguistically homogeneous sets of test data using three systems comprising of training and reference data which, with regard to the social class or age of the test speakers, are Matched, Mismatched or Mixed. The distributions of LRs were found to be relatively stable across systems, although LRs for individual comparisons may be substantially affected. As expected, the Mismatched systems produced the worst validity, while the Matched systems produced the best validity. The implications of these results for voice comparison casework are considered in light of the paradox that one cannot know for certain the sociolinguistic community to which the offender belongs.

Speech Compensation for Time-Scale-Modified Auditory Feedback

PURPOSE The purpose of this study was to examine speech compensation in response to time-scale-modified auditory feedback during the transition of the semivowel for a target utterance of /ija/. METHOD Each utterance session consisted of 10 control trials in the normal feedback condition followed by 20 perturbed trials in the modified auditory feedback condition and 10 return trials in the normal feedback condition. The authors examined speech compensation and the aftereffect in terms of 3 acoustic features: the maximum velocities on the (a) F1 and (b) F2 trajectories (VF1 and VF2) and (c) the F1-F2 onset time difference (TD) during the transition. They also conducted a syllable perception test on the feedback speech. RESULTS Speech compensation was observed in VF1, VF2, and TD. The magnitudes of speech compensation in VF1 and TD monotonically increased as the amount of the time-scale perturbation increased. The amount of speech compensation increased as the phonemic perception change increased. CONCLUSIONS Speech compensation for time-scale-modified auditory feedback is carried out primarily by changing VF1 and secondarily by adjusting VF2 and TD. Furthermore, it is activated primarily by detecting the speed change in altered feedback speech and secondarily by detecting the phonemic categorical change.