Speech Synthesis System Research Articles

Acoustic modeling for Kazakh speech synthesis

We present a new framework of generative adversarial network for training of acoustic model for speech synthesis. The proposed generative adversarial network consists of a generator and a pair of agent discriminators, where the generator predicts the acoustic features from the linguistic representation. Training and testing were carried out on the Kazakh speech corpus, which consisted of 5.6 hours of speech recording. According to the experiment results the 3.46 mean opinion score was obtained which shows an acceptable quality of speech synthesis. This approach of the acoustic model development can be applied in speech synthesis systems of the other languages.

A Review of Deep Learning Based Speech Synthesis

Speech synthesis, also known as text-to-speech (TTS), has attracted increasingly more attention. Recent advances on speech synthesis are overwhelmingly contributed by deep learning or even end-to-end techniques which have been utilized to enhance a wide range of application scenarios such as intelligent speech interaction, chatbot or conversational artificial intelligence (AI). For speech synthesis, deep learning based techniques can leverage a large scale of <text, speech> pairs to learn effective feature representations to bridge the gap between text and speech, thus better characterizing the properties of events. To better understand the research dynamics in the speech synthesis field, this paper firstly introduces the traditional speech synthesis methods and highlights the importance of the acoustic modeling from the composition of the statistical parametric speech synthesis (SPSS) system. It then gives an overview of the advances on deep learning based speech synthesis, including the end-to-end approaches which have achieved start-of-the-art performance in recent years. Finally, it discusses the problems of the deep learning methods for speech synthesis, and also points out some appealing research directions that can bring the speech synthesis research into a new frontier.

Building a controllable expressive speech synthesis system with multiple emotion strengths

Emotion is considered to be an essential element in the performance of human-computer interactions. In expressive synthesis speech, it is important to generate emotional speech that reflects subtle and complex emotional states. However, there has been limited research on how to effectively synthesize emotional speech using different levels of emotion strength with intuitive control, which is difficult to be modeled effectively. In this paper, we explore an expressive speech synthesis model that can be used to produce speech with multiple emotion strengths. Unlike previous studies that encoded emotions into discrete codes, we propose an embedding vector to continuously control the emotion strength, which is a data-driven method to synthesize speech with a fine control over the emotions. Compared with the models using the retraining technique or a one-hot vector, our proposed model using an embedding vector can explicitly learn the high-level emotion strength from the low-level acoustic features. As a result, we can control the emotion strength of synthetic speech in a relatively predictable and globally consistent way. The objective and subjective evaluation tests show that our proposed model achieves state-of-the-art performance in terms of model flexibility and controllability.

An Effective Style Token Weight Control Technique for End-to-End Emotional Speech Synthesis

In this letter, we propose a high-quality emotional speech synthesis system, using emotional vector space, i.e., the weighted sum of global style tokens (GSTs). Our previous research verified the feasibility of GST-based emotional speech synthesis in an end-to-end text-to-speech synthesis framework. However, selecting appropriate reference audio (RA) signals to extract emotion embedding vectors to the specific types of target emotions remains problematic. To ameliorate the selection problem, we propose an effective way of generating emotion embedding vectors by utilizing the trained GSTs. By assuming that the trained GSTs represent an emotional vector space, we first investigate the distribution of all the training samples depending on the type of each emotion. We then regard the centroid of the distribution as an emotion-specific weighting value, which effectively controls the expressiveness of synthesized speech, even without using the RA for guidance, as it did before. Finally, we confirm that the proposed controlled weight-based method is superior to the conventional emotion label-based methods in terms of perceptual quality and emotion classification accuracy.

The virtual doctor: An interactive clinical-decision-support system based on deep learning for non-invasive prediction of diabetes.

Artificial intelligence (AI) will pave the way to a new era in medicine. However, currently available AI systems do not interact with a patient, e.g., for anamnesis, and thus are only used by the physicians for predictions in diagnosis or prognosis. However, these systems are widely used, e.g., in diabetes or cancer prediction. In the current study, we developed an AI that is able to interact with a patient (virtual doctor) by using a speech recognition and speech synthesis system and thus can autonomously interact with the patient, which is particularly important for, e.g., rural areas, where the availability of primary medical care is strongly limited by low population densities. As a proof-of-concept, the system is able to predict type 2 diabetes mellitus (T2DM) based on non-invasive sensors and deep neural networks. Moreover, the system provides an easy-to-interpret probability estimation for T2DM for a given patient. Besides the development of the AI, we further analyzed the acceptance of young people for AI in healthcare to estimate the impact of such a system in the future.

Importance of Signal Processing Cues in Transcription Correction for Low-Resource Indian Languages

Accurate phonetic transcriptions are crucial for building robust acoustic models for speech recognition as well as speech synthesis applications. Phonetic transcriptions are not usually provided with speech corpora. A lexicon is used to generate phone-level transcriptions of speech corpora with sentence-level transcriptions. When lexical entries are not available, letter-to-sound (LTS) rules are used. Whether it is a lexicon or LTS, the rules for pronunciation are generic and may not match the spoken utterance. This can lead to transcription errors. The objective of this study is to address the issue of mismatch between the transcription and its acoustic realisation. In particular, the issue of vowel deletions is studied. Group-delay-based segmentation is used to determine insertion/deletion of vowels in the speech utterance. The transcriptions are corrected in the training data based on this. The corrected data are used in automatic speech recognition (ASR) and text to speech synthesis (TTS) systems. ASR and TTS systems built with the corrected transcriptions show improvements in the performance.

Adaptive Refinements of Pitch Tracking and HNR Estimation within a Vocoder for Statistical Parametric Speech Synthesis

Recent studies in text-to-speech synthesis have shown the benefit of using a continuous pitch estimate; one that interpolates fundamental frequency (F0) even when voicing is not present. However, continuous F0 is still sensitive to additive noise in speech signals and suffers from short-term errors (when it changes rather quickly over time). To alleviate these issues, three adaptive techniques have been developed in this article for achieving a robust and accurate F0: (1) we weight the pitch estimates with state noise covariance using adaptive Kalman-filter framework, (2) we iteratively apply a time axis warping on the input frame signal, (3) we optimize all F0 candidates using an instantaneous-frequency-based approach. Additionally, the second goal of this study is to introduce an extension of a novel continuous-based speech synthesis system (i.e., in which all parameters are continuous). We propose adding a new excitation parameter named Harmonic-to-Noise Ratio (HNR) to the voiced and unvoiced components to indicate the degree of voicing in the excitation and to reduce the influence of buzziness caused by the vocoder. Results based on objective and perceptual tests demonstrate that the voice built with the proposed framework gives state-of-the-art speech synthesis performance while outperforming the previous baseline.

Text Spotting in the Wild with Embedded Device

Abstract Detecting and recognizing text in natural scenes (e.g. streets, restaurants, shops, etc.) could be a part of an artificial intelligence system, especially with regard to the speech synthesis system. Properly detected text is passed to a recognition stage and then to the speech synthesis system, which translates text to speech. Research is carried out for the ‘Toucan Eye’ project — embedded device with artificial intelligence system able to help people with impaired sight. Due to constrained resources and abilities of embedded devices, criteria for text spotting must be met. First criterion is quality of detected and recognized regions with text and the second is time spent on both operations. Particular stages of the system and chosen methods of text spotting under aforementioned constraints are presented.

Es-Tacotron2: Multi-Task Tacotron 2 with Pre-Trained Estimated Network for Reducing the Over-Smoothness Problem

Text-to-speech synthesis is a computational technique for producing synthetic, human-like speech by a computer. In recent years, speech synthesis techniques have developed, and have been employed in many applications, such as automatic translation applications and car navigation systems. End-to-end text-to-speech synthesis has gained considerable research interest, because compared to traditional models the end-to-end model is easier to design and more robust. Tacotron 2 is an integrated state-of-the-art end-to-end speech synthesis system that can directly predict closed-to-natural human speech from raw text. However, there remains a gap between synthesized speech and natural speech. Suffering from an over-smoothness problem, Tacotron 2 produced ’averaged’ speech, making the synthesized speech sounds unnatural and inflexible. In this work, we first propose an estimated network (Es-Network), which captures general features from a raw mel spectrogram in an unsupervised manner. Then, we design Es-Tacotron2 by employing the Es-Network to calculate the estimated mel spectrogram residual, and setting it as an additional prediction task of Tacotron 2, to allow the model focus more on predicting the individual features of mel spectrogram. The experience shows that compared to the original Tacotron 2 model, Es-Tacotron2 can produce more variable decoder output and synthesize more natural and expressive speech.

Phoneme concatenation method considering half vowel sound for the Myanmar speech synthesis system

Phoneme concatenation method considering half vowel sound for the Myanmar speech synthesis system

Development of Vietnamese Speech Synthesis System using Deep Neural Networks

In this paper, we present our first Vietnamese speech synthesis system based on deep neural networks. To improve the training data collected from the Internet, a cleaning method is proposed. The experimental results indicate that by using deeper architectures we can achieve better performance for the TTS than using shallow architectures such as hidden Markov model. We also present the effect of using different amounts of data to train the TTS systems. In the VLSP TTS challenge 2018, our proposed DNN-based speech synthesis system won the first place in all three subjects including naturalness, intelligibility, and MOS.

Applying Linguistic G2P Knowledge on a Statistical Grapheme-to-phoneme Conversion in Khmer

A Grapheme-to-Phoneme (G2P) convertor generates the pronunciation given a word. G2P is an important module in a speech synthesis system and an automatic speech recognition system. Two main G2P approaches are: knowledge-based and data-driven. The knowledge based G2P is built based on linguist knowledge. The data-driven approach such as the statistical approach on the other hand does not need expert knowledge, but it requires data to learn the rules. In this research, we propose an approach that combines linguistic knowledge into a statistical-based G2P convertor for Khmer. We examined a simple way of adding linguistic knowledge into the statistical G2P convertor by simply inserting vowel tags into a Khmer word. Three types of vowel tags were used. The main strength of this approach is it combines the strength of linguistic knowledge and statistical-based approach, to build a robust G2P model. The information allows better modeling and prediction of the phoneme sequence, thus improving the phoneme error rate (PER) and word error rate (WER). The PER and WER of our proposed Khmer G2P improve from 23.2% and 69.6% to 11.1% and 51.4% respectively.

Text to Speech Synthesis System in Yoruba Language

Previous and existing studies convert texts to speech (English Language) without consideration for indigenous languages. This project work is an effort to design a speech extension that can convert the highlighted text displayed by a web browser to speech considering the indigenous Yoruba language through a designed software tool specifically attached to Mozilla internet as a plug-in. This software tool allows the user to highlight the text on the web page with the mouse. The selected text provides an input to the speech program that runs in the back end. The objective of this project work is to design a program that will help people with weak eyesight, those that lack in pronunciation skills and those learning the Yoruba language as an additional language skill; it is also designed to solve problems that exist when using the manual system which include difficulty in reading and misinterpretation of information. The methodology used to convert the text to speech are the API (Google-translate, speech’.org and ttsyoruba.com) and the programming language used for the web is JavaScript. A text to speech extension is an application that converts the highlighted texts into spoken words by analyzing and processing the text in the Yoruba language. The use of the text to speech is to read out the highlighted text to the user in the Yoruba language which can be saved as an audio file.

A New Unit Selection Optimisation Algorithm for Corpus-Based TTS Systems Using the RBF-Based Data Compression Technique

A major drawback of corpus-based speech synthesis systems is the use of large acoustic inventories, and currently one of the main challenges is the optimal representation of concatenation costs associated with units in the acoustic inventory. These concatenation costs are used to evaluate spectral mismatches between the acoustic units to be concatenated. The combinatorics of costs grows exponentially with the size of the acoustic inventories and can result in hundreds of millions or even billions of concatenation costs to be processed. Therefore, in this paper, we represent a novel unit selection optimization algorithm, which minimizes the size of concatenation costs through the vector quantization-based compression technique and tuple structures. Furthermore, the proposed optimization algorithm is designed to be used as an objective measure to optimize the performance of the unit selection cost function regarding the quality of the speech output, and to evaluate the effect of the vector quantization-based compression technique on its performance. The results obtained show that even when data compression is above 50%, the effect on the quality of the synthesized speech is negligible.

Nepali Text to Speech Synthesis System using FreeTTS

This paper confers the tools and methodology used in developing a Nepali Text to Speech Synthesis System using FreeTTS and is entirely developed in Java and uses FreeTTS synthesize1: Vocalized form of human communication is Speech. Here the Nepali Language is Synthetized based on formant approach and the use of one of the popular generic frameworks FreeTTS that is available in public domain for the development of a TTS system. The Text To Speech Architecture has been developed putting more emphasis on the Natural Language Processing (NLP) component rather than Digital Signal Processing (DSP) component. Nepali language being mostly used language in Nepal and some parts of India and abroad, a text-to-speech (TTS} synthesizer for this language will prove to be a convenient tool and communication technology (JCT) based system to aid to those majorities of people who are illiterate and also to those who are physical impairments like visually handicapped and vocally disabled persons. This ability to convert text to voice may reduce the dependency, frustration, and sense of helplessness of these people. The system can be extended to include more features such as emotions, improved tokenization, interactive options and the use of minimal database.

A Weighted Speaker-Specific Confusion Transducer-Based Augmentative and Alternative Speech Communication Aid for Dysarthric Speakers.

An augmentative and alternative speech communication (AASC) aid comprises a speech recognition system and a speech synthesis system. The main challenge in developing such an aid for dysarthric speakers lies in handling errors in the text derived from the recognition system. These errors (substitution, deletion, and insertion) may be due to inability of a dysarthric speaker to utter certain phones (articulatory error) or due to inaccuracy of the models trained (modeling error). Most existing AASC approaches only focus on the articulatory errors and the ones that do address both errors, and do not differentiate between them. However, this paper performs a three-level cascaded analysis to identify and distinguish between these errors, as differentiating these errors will aid in appropriately handling them. Furthermore, analyses in the paper are independent of the syntax of utterances. Based on these analyses, weighted phone confusion transducers are formulated and used to correct erroneous text from the recognition system. The corrected text is finally synthesized by a text-to-speech synthesis system. The proposed AASC is observed to significantly reduce a word error rate of severe dysarthric speakers from 100% to 41.52%, moderate from 61.85% to 18.08%, and mild from 12.23% to 8.55%.

Language-independent acoustic cloning of HTS voices

Speaker adaptation techniques can be classified as intra-lingual or cross-lingual depending on whether or not the source model and the target speaker employ the same language. Most of the work in this field has been focused on the first case, while the second one has been less explored. In this paper we address the cross-lingual paradigm in the framework of a HMM-based speech synthesis system by further developing a formerly proposed approach. This method is able to clone a given speaker into a different language by combining the linguistic structure and the acoustic characteristics of two HTS models. In this work, we discuss the extension of the adaptation procedure to some other source model parameters that were kept unmodified in the initial version, and compare the performance of both versions by means of subjective and objective tests. These results are also contrasted with those obtained by a KLD-based technique proposed in the literature for a similar purpose. While no significant preference for any of the versions of our method is observed, our approach clearly outperforms the KLD-based technique.

Hidden-Markov-model based statistical parametric speech synthesis for Marathi with optimal number of hidden states

Hidden Markov Model and Deep Neural Networks based Statistical Parametric Speech Synthesis systems, gain a significant attention from researchers because of their flexibility in generating speech waveforms in diverse voice qualities as well as in styles. This paper describes HMM-based speech synthesis system (SPSS) for the Marathi language. In proposed synthesis method, speech parameter trajectories used for synthesis are generated from the trained hidden Markov models (HMM). We have recorded our database of 5300 phonetically balanced Marathi sentences to train the context-dependent HMM with five, seven and nine hidden states. The subjective quality measures (MOS and PWP) shows that the HMMs with seven hidden states are capable of giving an adequate quality of synthesized speech as compared to five state and with less time complexity than seven state HMMs. The contextual features used for experimentation are inclusive of a position of an observed phoneme in a respective syllable, word, and sentence.

Isarn Dialect Speech Synthesis using HMM with syllable-context features

This paper describes the Isarn speech synthesis system, which is a regional dialect spoken in the Northeast of Thailand. In this study, we focus to improve the prosody generation of the system by using the additional context features. In order to develop the system, the speech parameters (Mel-ceptrum and fundamental frequencies of phoneme within different phonetic contexts) were modelled using Hidden Markov Models (HMM). Synthetic speech was generated by converting the input text into context-dependent phonemes. Speech parameters were generated from the trained HMM, according to the context-dependent phonemes, and were then synthesized through a speech vocoder. In this study, systems were trained using three different feature sets: basic contextual features, tonal, and syllable-context features. Objective and subjective tests were conducted to determine the performance of the proposed system. The results indicated that the addition of the syllable-context features significantly improved the naturalness of synthesized speech.

Evaluation of speech unit modelling for HMM-based speech synthesis for Arabic

This paper investigates the use of hidden Markov models (HMM) for Modern Standard Arabic speech synthesis. HMM-based speech synthesis systems require a description of each speech unit with a set of contextual features that specifies phonetic, phonological and linguistic aspects. To apply this method to Arabic language, a study of its particularities was conducted to extract suitable contextual features. Two phenomena are highlighted: vowel quantity and gemination. This work focuses on how to model geminated consonants (resp. long vowels), either considering them as fully-fledged phonemes or as the same phonemes as their simple (resp. short) counterparts but with a different duration. Four modelling approaches have been proposed for this purpose. Results of subjective and objective evaluations show that there is no important difference between differentiating modelling units associated to geminated consonants (resp. long vowels) from modelling units associated to simple consonants (resp. short vowels) and merging them as long as gemination and vowel quantity information is included in the set of features.