Unit Selection Speech Synthesis Research Articles

Automatic statistical evaluation of quality of unit selection speech synthesis with different prosody manipulations

Abstract Quality of speech synthesis is a crucial issue in comparison of various text-to-speech (TTS) systems. We proposed a system for automatic evaluation of speech quality by statistical analysis of temporal features (time duration, phrasing, and time structuring of an analysed sentence) together with standard spectral and prosodic features. This system was successfully tested on sentences produced by a unit selection speech synthesizer with a male as well as a female voice using two different approaches to prosody manipulation. Experiments have shown that for correct, sharp, and stable results all three types of speech features (spectral, prosodic, and temporal) are necessary. Furthermore, the number of used statistical parameters has a significant impact on the correctness and precision of the evaluated results. It was also demonstrated that the stability of the whole evaluation process is improved by enlarging the used speech material. Finally, the functionality of the proposed system was verified by comparison of the results with those of the standard listening test.

Learning and Modeling Unit Embeddings Using Deep Neural Networks for Unit-Selection-Based Mandarin Speech Synthesis

A method of learning and modeling unit embeddings using deep neutral networks (DNNs) is presented in this article for unit-selection-based Mandarin speech synthesis. Here, a unit embedding is defined as a fixed-length embedding vector for a phone-sized unit candidate in a corpus. Modeling phone-sized embedding vectors instead of frame-sized acoustic features can better measure the long-term dependencies among consecutive units in an utterance. First, a DNN with an embedding layer is built to learn the embedding vectors of all unit candidates in the corpus from scratch. In order to enable the extracted embedding vectors to carry both acoustic and linguistic information of unit candidates, a multitarget learning strategy is designed for the DNN. Its optional prediction targets include frame-level acoustic features, unit durations, monophone and tone identifiers, and context classes. Then, another two DNNs are constructed to map linguistic features toward the extracted embedding vectors. One of them employs the unit vectors of preceding phones besides the linguistic features of current phone as its input. At synthesis time, the distances between the unit vectors predicted by these two DNNs and the ones derived from unit candidates are used as a part of the target cost and a part of the concatenation cost, respectively. Our experiments on a Mandarin speech synthesis corpus demonstrate that learning and modeling unit embeddings improve the naturalness of hidden Markov model (HMM)-based unit selection speech synthesis. Furthermore, integrating multiple targets for learning unit embeddings achieves better performance than using only acoustic targets according to our subjective evaluation results.

Unit Selection Speech Synthesis Using Frame-Sized Speech Segments and Neural Network Based Acoustic Models

This paper proposes to select frame-sized speech segments for waveform concatenation speech synthesis using neural network based acoustic models. First, a deep neural network (DNN) based frame selection method is presented. In this method, three DNNs are adopted to calculate target costs and concatenation costs respectively for selecting candidate frames of 5ms length. One DNN is built in the same way as the DNN-based statistical parametric speech synthesis, which predicts target acoustic features given linguistic context inputs. The distance between the acoustic features of a candidate unit and the predicted ones for a target unit is calculated as the target cost. The other two DNNs are constructed to predict the acoustic features at current frame using its context features and the acoustic features of preceding frames. At synthesis time, these two DNNs are employed to calculate the concatenation cost for each candidate frame given its preceding ones. Furthermore, recurrent neural networks (RNNs) with long short-term memory (LSTM) cells are adopted to replace DNNs for acoustic modeling in order to make better use of the sequential information. A strategy of using multi-frame instead of single frame as the basic unit for selection is also presented to reduce the concatenation points within synthetic speech. Experimental results show that our proposed method can achieve better naturalness than the hidden Markov model (HMM)-based frame selection method and the HMM-based parametric speech synthesis method.

Unit-Selection Speech Synthesis Method Using Words as Search Units

Unit-selection speech-synthesis systems have been proposed. In most of the unit-selection speech-synthesis systems, search units are rather short such as syllables, phonemes and diphones. However, when applied to large speech databases, shorter units produce more voice-waveform candidates and a larger speech database cannot be used without narrow pruning for practical use. Narrow pruning impairs the quality of the synthesized speech. Here the author examined the possibility of using words as search units. Subjective evaluations indicated that 70% of the speech synthesized by the proposed method sounded more natural than that synthesized by a conventional method. The five-point mean opinion score of the synthesized speech was 3.5, and 21% was judged to sound as natural as human speech. These results demonstrate the effectiveness of unit-selection speech synthesis using words as search units.

HMM-based unit selection speech synthesis using log likelihood ratios derived from perceptual data

This paper presents a hidden Markov model (HMM) based unit selection speech synthesis method using log likelihood ratios (LLR) derived from perceptual data. The perceptual data is collected by judging the naturalness of each synthetic prosodic word manually. Two acoustic models which represent the natural speech and the unnatural synthetic speech are trained respectively. At synthesis time, the LLRs are derived from the estimated acoustic models and integrated into the unit selection criterion as target cost functions. The experimental results show that our proposed method can synthesize more natural speech than the conventional method using likelihood functions. Due to the inadequacy of the acoustic model estimated for the unnatural synthetic speech, utilizing the LLR-based target cost functions to rescore the pre-selection results or the N-best sequences can achieve better performance than substituting them for the original target cost functions directly.

A Text to Speech System for Fon Language Using Multisyn Algorithm

This paper describes the design of a unit selection speech synthesis system for Fon language, a tonal language spoken in Benin. It shows the approach that is used to deal with a text written in Fon language, with steps such as the use of letter-to-sound rules, phonetic and post lexical rules for automatic phonetic transcription of input text. We provide an overview of the implementation of the modules of the system in Java programming language and the various stages of building a synthetic voice in Fon using Festival, a text-to-speech system. The unit selection algorithm used is Multisyn. The results of tests are also discussed.

Admissible Stopping in Viterbi Beam Search for Unit Selection Speech Synthesis

Admissible Stopping in Viterbi Beam Search for Unit Selection Speech Synthesis

OPTIMIZATION OF COST FUNCTION WEIGHTS FOR UNIT SELECTION SPEECH SYNTHESIS USING SPEECH RECOGNITION

A well known problem in unit selection speech synthesis is designing the join and target function sub-costs and optimizing their corresponding weights so that they reflect the human listeners' preferences. To achieve this, we propose a procedure where an objective criterion for optimal speech unit selection is used. The objective criterion for tuning the cost function weights is based on automatic speech recognition results. In order to demonstrate the effectiveness of the proposed method listening tests with 31 naive listeners were performed. The experimental results have shown that the proposed method improves speech quality and intelligibility. In order to evaluate the quality of synthesized speech, the unit selection speech synthesis system is compared with two other Croatian speech synthesis systems with voices built using the same recorded speech corpus. One of these voices was built with the Festival speech synthesis system using the statistical parametric method and the other is a diphone concatenation based text-to-speech system. The comparison is based on subjective tests using MOS (mean opinion score) evaluation. The system using the proposed method used for cost function weights optimization performs better than other compared systems according to the subjective tests.

Multiple f 0 contour parallel Viterbi search for unit selection speech synthesis

Unit selection speech systems generate synthetic speech by concatenation of acoustic units that are chosen by means of a Viterbi search on a previously recorded corpus of the same speaker. Given the finite nature of the corpus, there is usually a trade-off between the acoustic quality of the signal, and the quality of the prosody. Proposed is a parallel Viterbi search considering multiple intonation contours that improves at the same time both acoustic quality and intonation, while maintaining the computational efficiency of the process.

Efficient and reliable perceptual weight tuning for unit-selection text-to-speech synthesis based on active interactive genetic algorithms: A proof-of-concept

Unit-selection speech synthesis is one of the current corpus-based text-to-speech synthesis techniques. The quality of the generated speech depends on the accuracy of the unit selection process, which in turn relies on the cost function definition. This function should map the user perceptual preferences when selecting synthesis units, which is still an open research issue. This paper proposes a complete methodology for the tuning of the cost function weights by fusing the human judgments with the cost function, through efficient and reliable interactive weight tuning. To that effect, active interactive genetic algorithms (aiGAs) are used to guide the subjective weight adjustments. The application of aiGAs to this process allows mitigating user fatigue and frustration by improving user consistency. However, it is still unfeasible to subjectively adjust the weights of the whole corpus units (diphones and triphones in this work). This makes it mandatory to perform unit clustering before conducting the tuning process. The aiGA-based weight tuning proposal is evaluated in a small speech corpus as a proof-of-concept and results in more natural synthetic speech when compared to previous objective and subjective-based approaches.

One-Class Classification for Spectral Join Cost Calculation in Unit Selection Speech Synthesis

This letter introduces one-class classification as a framework for the spectral join cost calculation in unit selection speech synthesis. Instead of quantifying the spectral cost by a single distance measure, a data-driven approach is adopted which exploits the natural similarity of consecutive speech frames in the speech database. A pair of consecutive frames is jointly represented as a vector of spectral distance measures which provide training data for the one-class classifier. At synthesis runtime, speech units are selected based on the scores derived from the classifier. Experimental results provide evidence on the effectiveness of the proposed method which clearly outperforms the conventional approaches currently employed.

The Bonn Open Synthesis System 3

The Bonn Open Synthesis System (BOSS) is an open-source software distribution for unit selection speech synthesis that aims to be easily extensible to new target languages and different applications. To achieve this flexibility, many aspects of the software have been changed in recent years, including the addition of a refined interface to synthesis modules and a more strict separation of language-specific and language-independent code. This article wants to give an overview of the architecture from a technical perspective and explain how it can be adapted for a particular purpose and voice. This is preceded by a short introduction to the unit selection paradigm in general and a section on the specifics of the approach taken by BOSS. A particular focus will be placed on the extensions made for the integration of Polish during which some of the flexibilisation measures were conducted. Further information on the application to Polish but with an emphasis on the linguistic, phonetic and acoustic aspects as well as the speech corpus used can be found in the second part of this two-part article, “Polish unit selection speech synthesis with BOSS”, also published in this issue of the Journal.

Polish unit selection speech synthesis with BOSS: extensions and speech corpora

This article presents research and development aimed at creating a Polish speech database for speech synthesis and adapting BOSS (The Bonn Open Synthesis System) to the Polish language. First of all, the linguistic background for the design of Polish spoken resources for unit selection is presented, together with the presentation of the applied transcription and annotation methods. The next section details the assumptions and the structure of the Polish corpus and its segmental and prosodic annotation. Then, the linguistic features used in duration modelling and the selection of adequate speech units of two Polish modules in BOSS are reported: the duration prediction module (the description is accompanied by a concise overview of Polish duration modelling for speech technology purposes) and the cost functions module. Finally, the results of two kinds of perception tests are discussed: the first is a preference test aimed at the evaluation of synthesized speech obtained using three variants of speech signal segmentation (automatic, semi-automatic and manual) and the second is a mean opinion score test carried out to provide a preliminary assessment of the synthesized speech quality attained with the Polish version of the BOSS synthesizer. The closing chapter summarizes future perspectives and challenges for the Polish TTS (text-to-speech) and further developments of BOSS for Polish.

Implementation of Polish speech synthesis for the BOSS system

The Bonn Open Synthesis System (BOSS) is an open-source software for the unit selection speech synthesis that has been used for the generation of high-quality German and Dutch speech. This article presents ongoing research and development aimed at adapting BOSS to the Polish language. In the first section, the origins and workings of the unit selection method for speech synthesis are explained. Section two details the structure of the Polish corpus and its segmental and prosodic annotation. The subsequent sections focus on the implementation of Polish TTS modules in the BOSS architecture (duration prediction and cost function) and the steps involved in preparing a new speech corpus for BOSS.

Analysis of statistical parametric and unit selection speech synthesis systems applied to emotional speech

We have applied two state-of-the-art speech synthesis techniques (unit selection and HMM-based synthesis) to the synthesis of emotional speech. A series of carefully designed perceptual tests to evaluate speech quality, emotion identification rates and emotional strength were used for the six emotions which we recorded – happiness, sadness, anger, surprise, fear, disgust. For the HMM-based method, we evaluated spectral and source components separately and identified which components contribute to which emotion. Our analysis shows that, although the HMM method produces significantly better neutral speech, the two methods produce emotional speech of similar quality, except for emotions having context-dependent prosodic patterns. Whilst synthetic speech produced using the unit selection method has better emotional strength scores than the HMM-based method, the HMM-based method has the ability to manipulate the emotional strength. For emotions that are characterized by both spectral and prosodic components, synthetic speech using unit selection methods was more accurately identified by listeners. For emotions mainly characterized by prosodic components, HMM-based synthetic speech was more accurately identified. This finding differs from previous results regarding listener judgements of speaker similarity for neutral speech. We conclude that unit selection methods require improvements to prosodic modeling and that HMM-based methods require improvements to spectral modeling for emotional speech. Certain emotions cannot be reproduced well by either method.

Trainable unit selection speech synthesis under statistical framework

This paper proposes a trainable unit selection speech synthesis method based on statistical modeling framework. At training stage, acoustic features are extracted from the training database and statistical models are estimated for each feature. During synthesis, the optimal candidate unit sequence is searched out from the database following the maximum likelihood criterion derived from the trained models. Finally, the waveforms of the optimal candidate units are concatenated to produce synthetic speech. Experiment results show that this method can improve the automation of system construction and naturalness of synthetic speech effectively compared with the conventional unit selection synthesis method. Furthermore, this paper presents a minimum unit selection error model training criterion according to the characteristics of unit selection speech synthesis and adopts discriminative training for model parameter estimation. This criterion can finally achieve the full automation of system construction and improve the naturalness of synthetic speech further.

Multisyn: Open-domain unit selection for the Festival speech synthesis system

We present the implementation and evaluation of an open-domain unit selection speech synthesis engine designed to be flexible enough to encourage further unit selection research and allow rapid voice development by users with minimal speech synthesis knowledge and experience. We address the issues of automatically processing speech data into a usable voice using automatic segmentation techniques and how the knowledge obtained at labelling time can be exploited at synthesis time. We describe target cost and join cost implementation for such a system and describe the outcome of building voices with a number of different sized datasets. We show that, in a competitive evaluation, voices built using this technology compare favourably to other systems.

Subjective evaluation of join cost and smoothing methods for unit selection speech synthesis

In unit selection-based concatenative speech synthesis, join cost (also known as concatenation cost), which measures how well two units can be joined together, is one of the main criteria for selecting appropriate units from the inventory. Usually, some form of local parameter smoothing is also needed to disguise the remaining discontinuities. This paper presents a subjective evaluation of three join cost functions and three smoothing methods. We also describe the design and performance of a listening test. The three join cost functions were taken from our previous study, where we proposed join cost functions derived from spectral distances, which have good correlations with perceptual scores obtained for a range of concatenation discontinuities. This evaluation allows us to further validate their ability to predict concatenation discontinuities. The units for synthesis stimuli are obtained from a state-of-the-art unit selection text-to-speech system: rVoice from Rhetorical Systems Ltd. In this paper, we report listeners' preferences for each join cost in combination with each smoothing method

Optimal Utterance Selection for Unit Selection Speech Synthesis Databases

This paper describes techniques to find an optimal data set for building high quality unit-selection speech synthesis inventories. As the quality of unit-selection speech synthesis is dependent on the coverage of the database used in the selection, it is important to select the right data to record. In this paper we describe some simple techniques as well as a more complex acoustic modeling technique based on the database speaker's acoustic characteristics. Result of a simple evaluation procedure are presented justifying the technique.