Prosody Modeling Research Articles

Unfolding Prosody Guides the Development of Word Segmentation

Prosody is known to scaffold the learning of language, and thus understanding prosodic development is vital for language acquisition. The present study explored the unfolding prosody model of prosodic development (proposed in Frota’s et al. study in 2016) beyond early production data, to examine whether it predicted the development of early segmentation abilities. European Portuguese-learning infants aged between 5 and 17 months were tested in a series of word segmentation experiments. Developing prosodic structure was evidenced in word segmentation as proposed by the unfolding model: (i) a simple monosyllabic word shape crucially placed at a major prosodic edge was segmented first, before more complex word shapes under similar prosodic conditions; (ii) the segmentation of more complex words was easier at a major prosodic edge than in phrase-medial position; and (iii) the segmentation of complex words with an iambic pattern preceded the segmentation of words with a trochaic pattern. These findings demonstrated that word segmentation evolved with unfolding prosody, suggesting that the prosodic units developed in the unfolding process are used both as speech production planning units and to extract word-forms from continuous speech. Therefore, our study contributes to a better understanding of the mechanisms underlying word segmentation, and to a better understanding of early prosodic development, a cornerstone of language acquisition.

Predicting children’s perceived reading proficiency with prosody modeling

Reading is a foundational skill and the focus of school-level education efforts across countries. The assessment of linguistic competence from oral reading has long been the subject of scientific studies linking the reader’s comprehension of the text to various measures of oral reading fluency. Given the time and resource intensive nature of such assessment, it is of interest to automate the prediction of reading fluency from audio recordings using the same pedagogical rubrics. Given recent findings about the importance of prosody to the communicative purpose of reading aloud, we discuss new approaches to modeling it reliably for the automatic assessment task. We present a new data set of children’s oral reading screened for minimum word decoding skill and rated for comprehensibility by two human experts. We develop a system for the automatic prediction of rater scores that also facilitates insights about the complementarity and inter-dependence of computed lexical accuracy, rate and prosodic features as corroborated by multiple performance measures. With achieved values of correlation and agreement that surpass the corresponding inter-rater measures, we also show how text-dependent prosodic features, informed by speech rate and speaking style, contribute prominently to system performance.

ZSE-VITS: A Zero-Shot Expressive Voice Cloning Method Based on VITS

Voice cloning aims to synthesize the voice with a new speaker’s timbre from a small amount of the new speaker’s speech. Current voice cloning methods, which focus on modeling speaker timbre, can synthesize speech with similar speaker timbres. However, the prosody of these methods is flat, lacking expressiveness and the ability to control the expressiveness of cloned speech. To solve this problem, we propose a novel method ZSE-VITS (zero-shot expressive VITS) based on the end-to-end speech synthesis model VITS. Specifically, we use VITS as the backbone network and add the speaker recognition model TitaNet as the speaker encoder to realize zero-shot voice cloning. We use explicit prosody information to avoid effects from the speaker information and adjust speech prosody using the prosody information prediction and prosody fusion methods directly. We widen the pitch distribution of the train datasets using pitch augmentation to improve the generalization ability of the prosody model, and we fine-tune the prosody predictor alone in the emotion corpus to learn prosody prediction of various styles. The objective and subjective evaluations of the open datasets show that our method can generate more expressive speech and adjust prosody information artificially without affecting the similarity of speaker timbre.

Prosody Modelling With Pre-Trained Cross-Utterance Representations for Improved Speech Synthesis

Prosody Modelling With Pre-Trained Cross-Utterance Representations for Improved Speech Synthesis

A Perceiver-Centered Approach for Representing and Annotating Prosodic Functions in Performed Music.

Musical prosody is characterized by the acoustic variations that make music expressive. However, few systematic and scalable studies exist on the function it serves or on effective tools to carry out such studies. To address this gap, we introduce a novel approach to capturing information about prosodic functions through a citizen science paradigm. In typical bottom-up approaches to studying musical prosody, acoustic properties in performed music and basic musical structures such as accents and phrases are mapped to prosodic functions, namely segmentation and prominence. In contrast, our top-down, human-centered method puts listener annotations of musical prosodic functions first, to analyze the connection between these functions, the underlying musical structures, and acoustic properties. The method is applied primarily to the exploring of segmentation and prominence in performed solo piano music. These prosodic functions are marked by means of four annotation types—boundaries, regions, note groups, and comments—in the CosmoNote web-based citizen science platform, which presents the music signal or MIDI data and related acoustic features in information layers that can be toggled on and off. Various annotation strategies are discussed and appraised: intuitive vs. analytical; real-time vs. retrospective; and, audio-based vs. visual. The end-to-end process of the data collection is described, from the providing of prosodic examples to the structuring and formatting of the annotation data for analysis, to techniques for preventing precision errors. The aim is to obtain reliable and coherent annotations that can be applied to theoretical and data-driven models of musical prosody. The outcomes include a growing library of prosodic examples with the goal of achieving an annotation convention for studying musical prosody in performed music.

Phone-Level Prosody Modelling With GMM-Based MDN for Diverse and Controllable Speech Synthesis

Generating natural speech with a diverse and smooth prosody pattern is a challenging task. Although random sampling with phone-level prosody distribution has been investigated to generate different prosody patterns, the diversity of the generated speech is still very limited and far from what can be achieved by humans. This is largely due to the use of uni-modal distribution, such as single Gaussian, in the prior works of phone-level prosody modelling. In this work, we propose a novel approach that models phone-level prosodies with a GMM-based mixture density network(MDN) and then extend it for multi-speaker TTS using speaker adaptation transforms of Gaussian means and variances. Furthermore, we show that we can clone the prosodies from a reference speech by sampling prosodies from the Gaussian components that produce the reference prosodies. Our experiments on LJSpeech and LibriTTS dataset show that the proposed method with GMM-based MDN not only achieves significantly better diversity than using a single Gaussian in both single-speaker and multi-speaker TTS, but also provides better naturalness. The prosody cloning experiments demonstrate that the prosody similarity of the proposed method with GMM-based MDN is comparable to recent proposed fine-grained VAE while the target speaker similarity is better.

Expressive TTS Training With Frame and Style Reconstruction Loss

We propose a novel training strategy for Tacotron-based text-to-speech (TTS) system to improve the expressiveness of speech. One of the key challenges in prosody modeling is the lack of reference that makes explicit modeling difficult. The proposed technique doesn't require prosody annotations from training data. It doesn't attempt to model prosody explicitly either, but rather encodes the association between input text and its prosody styles using a Tacotron-based TTS framework. Our proposed idea marks a departure from the style token paradigm where prosody is explicitly modeled by a bank of prosody embeddings. The proposed training strategy adopts a combination of two objective functions: 1) frame level reconstruction loss, that is calculated between the synthesized and target spectral features; 2) utterance level style reconstruction loss, that is calculated between the deep style features of synthesized and target speech. The proposed style reconstruction loss is formulated as a perceptual loss to ensure that utterance level speech style is taken into consideration during training. Experiments show that the proposed training strategy achieves remarkable performance and outperforms a state-of-the-art baseline in both naturalness and expressiveness. To our best knowledge, this is the first study to incorporate utterance level perceptual quality as a loss function into Tacotron training for improved expressiveness.

Empathetic Speech Synthesis and Testing for Healthcare Robots

One of the major factors that affect the acceptance of robots in Human-Robot Interaction applications is the type of voice with which they interact with humans. The robot’s voice can be used to express empathy, which is an affective response of the robot to the human user. In this study, the aim is to find out if social robots with empathetic voice are acceptable for users in healthcare applications. A pilot study using an empathetic voice spoken by a voice actor was conducted. Only prosody in speech is used to express empathy here, without any visual cues. Also, the emotions needed for an empathetic voice are identified. It was found that the emotions needed are not only the stronger primary emotions, but also the nuanced secondary emotions. These emotions are then synthesised using prosody modelling. A second study, replicating the pilot test is conducted using the synthesised voices to investigate if empathy is perceived from the synthetic voice as well. This paper reports the modelling and synthesises of an empathetic voice, and experimentally shows that people prefer empathetic voice for healthcare robots. The results can be further used to develop empathetic social robots, that can improve people’s acceptance of social robots.

Prosody indexes both competence and performance

Prosody is an important feature of language that conveys a wide range of information. However, prosody is widely considered to be a difficult domain of study within the language sciences. One consequence of this is that existing grammatical theories of prosody fail to explain prosodic choices that seem to arise from nonlinguistic cognitive demands, such as communicative context, top-down expectations, and recent articulatory and acoustic experience. We provide an account of some of these phenomena and argue that linguistic theories that do not incorporate these factors into models of prosody are likely to mischaracterize its role in language. This article is categorized under: Linguistics > Language in Mind and Brain Psychology > Language Linguistics > Linguistic Theory.

Analysis of Inter-Pausal Units in Indian Languages and Its Application to Text-to-Speech Synthesis

Lack of punctuation in Indian language text makes the analysis of phrases difficult. In this paper, inter-pausal units IPUs in read sentences are considered as phrases and are analyzed. A key observation from this analysis is that the length of the IPUs in read sentences follow uniformly across all languages a Gamma distribution. Additionally, an analysis of the scale and shape parameters suggest that these parameters are governed by the location of the IPU in an utterance. This information is used in text-to-speech TTS systems for four Indian languages leading to an improvement in naturalness. A novel IPU-based TTS system is proposed for better prosody modeling as well. A given text is parsed into IPUs, and an appropriate TTS system for different IPUs is used for synthesis. It is observed that there is a significant improvement in the naturalness of synthesized speech compared to that of a single TTS system being used for the entire sentence.

Hierarchical prosody modeling for Mandarin spontaneous speech

In this paper, a hierarchical prosody model (HPM)-based method for Mandarin spontaneous speech is proposed. First, an HPM is designed for describing relations among acoustic features of utterances, linguistic features of texts, and prosodic tags representing the underlying hierarchical prosodic structures of utterances. Subsequently, a sequential optimization algorithm is employed to train the HPM based on a large conversational speech corpus, the Mandarin Conversational Dialogue Corpus (MCDC), which features orthographic transcriptions and prosodic event annotations. In this unsupervised training method, all utterances of the MCDC are labeled with two types of prosodic tags, namely, break and prosodic states, automatically and simultaneously. After training, the HPM parameters are examined to identify critical prosodic properties of Mandarin spontaneous speech, which are then compared with their counterparts in the read-speech HPM. The prosodic tags on the studied utterances enable mapping of various prosodic events onto the hierarchical prosodic structures of the utterances. Prosodic analyses of some disfluent events are conducted using the prosodic tags affixed to the MCDC. Finally, an application of the HPM to assist in Mandarin spontaneous-speech recognition is discussed. Significant relative error rate reductions of 9.0%, 9.2%, 15.6%, and 7.3% are obtained for base-syllable, character, tone, and word recognition, respectively.

A comparison of visual and audio scaffolds in L2 English reading

This is a study of Japanese learners’ perspectives on visual and audio scaffolds in the teaching of second language English reading. We compare the approaches of ER, according to Day & Bamford’s (1998) ten principles (outlined below) in class, and Assisted Repeated Reading (ARR) of a work of fiction to a class. ARR refers to the reading aloud of a text multiple times to students as they follow along silently (see Taguchi & Gorsuch, 2002). Two classes of students in required English classes undertook both ER and ARR. Both the ER and ARR enhanced the students’ comprehension of the texts; in the case of ER, most students chose picture books, which aided their comprehension. In the case of ARR, the embodied presence of the reader, and the modeling of prosody facilitated comprehension.

Automatic assessment of children’s oral reading using speech recognition and prosody modeling

This work targets building an oral reading “tutor” that provides automatic and reliable feedback to children learning to read. The work uses state-of-the-art in speech recognition technology coupled with prosody modeling. The system is tested on available datasets of children’s readings in English as second language. The expected challenges relate to dealing with children’s speech demonstrating a variety of skill levels. Both word decoding accuracies and prosody attributes like phrasing and prominence are considered for assessment. The relation between different acoustic features computed from the speech signal and the perceived quality will be investigated. The goal is to have a system that can provide feedback and evaluation that is highly correlated with that of human judges such as language teachers.

A hierarchical linguistic information-based model of English prosody: L2 data analysis and implications for computer-assisted language learning

The paper presents a prosody model of native English (L1) continuous speech as corrective prosodic feedback for non-native learners. The model incorporates both hierarchical discourse association and information structure to (1) pinpoint the prosodic features of multi-phrase continuous speech, and (2) simulate native-like expressive speech using corpus of North American and Taiwan L2 English. The bottom-up, additive, data-driven model aims to generate L1-like expressive continuous speech with built-in phonetic and phonological specifications at the lexical level, syntactic/semantic specifications at the next higher phrase and sentence levels, and completed with patterned paragraph associations and prosodic projections of information allocation at higher levels. The hierarchical model successfully allows us to identify L1-L2 differences by prosodic modules/patterns as novel additional features “discourse structure” and “information density” reliably nail down L1-L2 prosodic differences related to phrase association as well as information placement. Our L1 prosodic model with the proposed predictors and optimized model trained from L1 speech corpus showed increase of prediction over existing methods. As a corrective feedback for L2 learners, these predicted L1 prosodic features were compared with a baseline model by objective evaluation (RMS error and correlation) then superimposed onto the L2 speech tokens. Resynthesized L2 tokens were subsequently compared with the original L2 tokens for degrees of perceived accent using subjective evaluation (native-listener perception test). We believe the proposed model can be an effective alternative for implementing computer-assisted language learning (CALL) systems that helps generate L1-like prosody from text, and at the same time serves as corrective feedback for L2 learners.

Emotional voice conversion using neural networks with arbitrary scales F0 based on wavelet transform

An artificial neural network is an important model for training features of voice conversion (VC) tasks. Typically, neural networks (NNs) are very effective in processing nonlinear features, such as Mel Cepstral Coefficients (MCC), which represent the spectrum features. However, a simple representation of fundamental frequency (F0) is not enough for NNs to deal with emotional voice VC. This is because the time sequence of F0 for an emotional voice changes drastically. Therefore, in our previous method, we used the continuous wavelet transform (CWT) to decompose F0 into 30 discrete scales, each separated by one third of an octave, which can be trained by NNs for prosody modeling in emotional VC. In this study, we propose the arbitrary scales CWT (AS-CWT) method to systematically capture F0 features of different temporal scales, which can represent different prosodic levels ranging from micro-prosody to sentence levels. Meanwhile, the proposed method uses deep belief networks (DBNs) to pre-train the NNs that then convert spectral features. By utilizing these approaches, the proposed method can change the spectrum and the F0 for an emotional voice simultaneously as well as outperform other state-of-the-art methods in terms of emotional VC.

Improvement of Prosody Modeling Using Semantic Role Labeling, Hybrid Feature Selection and BPSO–PSO–Optimized RNN

A modified Elman-type recurrent neural network (RNN) is used in this study for prosody generation in natural speech synthesis application. To improve the performance of generator, three modifications are applied to the system as compared to a traditional neural model: (a) using semantic role labeling to enrich the word-level features, (b) using hybrid of genetic algorithm and ant colony optimization algorithm for feature selection to reduce the number of input features, and (c) using hybrid of particle swarm optimization (PSO) and binary PSO algorithms for optimization of weights and structure of RNN-based prosody generator model, respectively. Experimental results show the superior performance of proposed model in predicting prosodic information with lower root mean squared error and a lighter model as compared to a traditional neural model.

PROSODY PREDICTION FOR TAMIL TEXT-TO-SPEECH SYNTHESIZER USING SENTIMENT ANALYSIS

A speech synthesizer which sounds similar to a human voice is preferred over a robotic voice, and hence to increase the naturalness of a speech synthesizer an efficacious prosody model is imperative. Hence, this paper is focused on developing a prosody prediction model using sentiment analysis for a Tamil speech synthesizer. Two variations of prosody prediction models using SentiWordNet are experimented: one without a stemmer and the other with a stemmer. The prosody prediction model with a stemmer performs much more efficiently than the one without a stemmer as it tackles the highly agglutinative and inflectional words in Tamil language in a better way and is exemplified clearly, in this paper. The performance of the prosody prediction model with a stemmer has a higher classification accuracy of 77% on the test set in comparison to the 57% accuracy by the prosody model without a stemmer.

Quantitative intonation modeling of interrogative sentences for Mandarin speech synthesis

Previous intonational research on Mandarin has mainly focused on the prosody modeling of statements or the prosody analysis of interrogative sentences. To support related speech technologies, e.g., Text-to-Speech, the quantitative modeling of intonation of interrogative sentences with a large-scale corpus still deserves attention. This paper summarizes our work on the quantitative prosody modeling of interrogative sentence in Mandarin. A large-scale natural speech corpus was used in this study. By extracting the pitch contours and fitting the intonation curves, we found that F0 declination and final lowering both existed in interrogative sentences, while they were claimed to be absent in Mandarin in some previous studies. In addition, the declination function could be modeled linearly, and the bearing unit of final lowering in Mandarin was found to be the last prosodic word in the utterance, regardless of its length, rather than a fixed duration range. It was argued in this study that the difference between this finding and the commonly believed rising intonation of the interrogative sentences resulted from the nonlinear relationship between prosody production and perception. The underlying mechanism for the existence of F0 declination and final lowering in interrogative sentences is also discussed.

Modeling F0 trajectories in hierarchically structured deep neural networks

This paper investigates F0 modeling of speech in deep neural networks (DNN) for statistical parametric speech synthesis (SPSS). Recently, DNN has been applied to the acoustic modeling of SPSS and has shown good performance in characterizing complex dependencies between contextual features and acoustic observations. However, the additive nature and long-term suprasegmental property of F0 features have not been fully exploited in the existing DNN-based SPSS. Two different model structures, cascade DNN and parallel DNN are proposed to embody the hierarchical and additive properties of the F0 in DNN-based prosody modeling. In the cascade structure, the DNN-predicted F0 contours of higher levels are used as input to the DNN of the current level. In the parallel structure, F0 components corresponding to different prosody levels are separately generated by DNNs and added together to obtain the final F0 contour. An optimized discrete cosine transform (DCT) is used to extract long-term F0 features at syllable, word, and phrase levels. The experimental results show that our approach yields better subjective performance than either the conventional HMM or DNN approaches. Among all compared systems, the parallel DNN achieves the best objective and subjective performance.

Prosody modeling for syllable based text-to-speech synthesis using feedforward neural networks

Prosody plays an important role in improving the quality of text-to-speech synthesis (TTS) system. In this paper, features related to the linguistic and the production constraints are proposed for modeling the prosodic parameters such as duration, intonation and intensities of the syllables. The linguistic constraints are represented by positional, contextual and phonological features, and the production constraints are represented by articulatory features. Neural network models are explored to capture the implicit duration, F0 and intensity knowledge using above mentioned features. The prediction performance of the proposed neural network models is evaluated using objective measures such as average prediction error (μ), standard deviation (σ) and linear correlation coefficient (γX,Y). The prediction accuracy of the proposed neural network models is compared with other state-of-the-art prosody models used in TTS systems. The prediction accuracy of the proposed prosody models is also verified by conducting listening tests, after integrating the proposed prosody models to the baseline TTS system.