TTS Synthesis Research Articles

DeepDet: YAMNet with BottleNeck Attention Module (BAM) for TTS synthesis detection

Spoofed speeches are becoming a big threat to society due to advancements in artificial intelligence techniques. Therefore, there must be an automated spoofing detector that can be integrated into automatic speaker verification (ASV) systems. In this study, we recommend a novel and robust model, named DeepDet, based on deep-layered architecture, to categorize speech into two classes: spoofed and bonafide. DeepDet is an improved model based on Yet Another Mobile Network (YAMNet) employing a customized MobileNet combined with a bottleneck attention module (BAM). First, we convert audio into mel-spectrograms that consist of time–frequency representations on mel-scale. Second, we trained our deep layered model using the extracted mel-spectrograms on a Logical Access (LA) set, including synthesized speeches and voice conversions of the ASVspoof-2019 dataset. In the end, we classified the audios, utilizing our trained binary classifier. More precisely, we utilized the power of layered architecture and guided attention that can discern the spoofed speech from bonafide samples. Our proposed improved model employs depth-wise linearly separate convolutions, which makes our model lighter weight than existing techniques. Furthermore, we implemented extensive experiments to assess the performance of the suggested model using the ASVspoof 2019 corpus. We attained an equal error rate (EER) of 0.042% on Logical Access (LA), whereas 0.43% on Physical Access (PA) attacks. Therefore, the performance of the proposed model is significant on the ASVspoof 2019 dataset and indicates the effectiveness of the DeepDet over existing spoofing detectors. Additionally, our proposed model is robust enough that can identify the unseen spoofed audios and classifies the several attacks accurately.

An end-to-end Tacotron model versus pre trained Tacotron model for Arabic text-to-speech synthesis

Text-to-Speech (TTS) systems turn normal text into spoken language, which is important for accessibility and user interaction. Many of these systems make speech from phonetic or phonemic transcriptions, but another way is to make speech by connecting together pre-recorded units from a database. The size of the units varies, from diphones to whole phrases. Even though this method covers a lot of ground, it sometimes needs more clarity, especially when high-quality output requires storing whole words or phrases in certain situations. Synthesizers can also use the way humans talk and the way their vocal tracts work to make voices. The Arabic language is hard to develop TTS methods for, as a result of its complicated morphology, semantic nuances, and many different dialects. These dialects often have a lot of differences from standard Arabic and don't follow formal rules for spelling. This means that traditional Arabic that hasn't been edited often has spelling and grammar mistakes. In this study, we show and test a Tacotron model that was made just for Arabic TTS synthesis from beginning to end. This model uses the richness of acoustic information in audio files, such as frequency and pitch, to make naturalistic speech that sounds a lot like what humans say. We also compare the performance of this model with that of a pre-trained Tacotron model applied to Arabic text. This gives us important information about how well Arabic TTS systems work and where they could be improved.

MASS: Multi-task anthropomorphic speech synthesis framework

Text-to-Speech (TTS) synthesis plays an important role in human-computer interaction. Currently, most TTS technologies focus on the naturalness of speech, namely, making the speeches sound like humans. However, the key tasks of the expression of emotion and the speaker identity are ignored, which limits the application scenarios of TTS synthesis technology. To make the synthesized speech more realistic and expand the application scenarios, we propose a multi-task anthropomorphic speech synthesis framework (MASS), which can synthesize speeches from text with specified emotion and speaker identity. The MASS framework consists of a base TTS module and two novel voice conversion modules: the emotional voice conversion module and the speaker voice conversion module. We propose deep emotion voice conversion model (DEVC) and deep speaker voice conversion model (DSVC) based on convolution residual networks. It solves the problem of feature loss during voice conversion. The model trainings are independent of parallel datasets, and are capable of many-to-many voice conversion. In the emotional voice conversion, speaker voice conversion experiments, as well as the multi-task speech synthesis experiments, experimental results show DEVC and DSVC convert speech effectively. The quantitative and qualitative evaluation results of multi-task speech synthesis experiments show MASS can effectively synthesis speech with specified text, emotion and speaker identity.

Voice Spoofing Countermeasure for Logical Access Attacks Detection

Voice-driven devices (VDDs) like Google Home and Amazon Alexa, which are well-known connected devices in consumer IoT, have applications in various domains i.e., home appliances automation, next-generation vehicles, voice banking, and so on. However, these VDDs that are based on automatic speaker verification systems (ASVs) are vulnerable to voice based logical access (LA) attacks like Text-to-Speech (TTS) synthesis and converted voice signals. Intruders can exploit these attacks to bypass the security of such systems and gain access of victim’s bank account or home control. Thus, there exists a need to develop an effective voice spoofing countermeasure that can reliably be used to protect these VDDs against such malicious attacks. This work presents a novel audio features descriptor named as extended local ternary pattern (ELTP) to capture the vocal tract dynamically induced attributes of bonafide speech and algorithmic artifacts in synthetic and converted speeches. We fused our novel ELTP features with the linear frequency cepstral coefficients (LFCC) to further strengthen the capability of our features for capturing the traits of bonafide and spoofed signals. We employ the proposed ELTP-LFCC features to train the deep bidirectional Long Short-Term Memory (DBiLSTM) network for classification of the bonafide and spoof signal (i.e., TTS synthesis, converted speech). Performance of our spoofing countermeasure is measured on the large-scale and diverse ASVspoof 2019 logical access dataset. Experimental results demonstrate that the proposed audio spoofing countermeasure can reliably be used to detect the LA spoofing attacks.

Acoustic model-based subword tokenization and prosodic-context extraction without language knowledge for text-to-speech synthesis

This paper presents text tokenization and context extraction without using language knowledge for text-to-speech (TTS) synthesis. To generate prosody, statistical parametric TTS synthesis typically requires the professional knowledge of the target language. Therefore, languages suitable for TTS synthesis are limited to only rich-resource languages. To achieve TTS synthesis without using language knowledge, we propose acoustic model-based subword tokenization and unsupervised extraction of prosodic contexts. The subword tokenization can determine language units suitable for prosody generation. The context extraction can retrieve contexts from pairs of subwords and prosody. The proposed methods function without language knowledge and can improve F0 prediction accuracy. Experimental evaluation demonstrates that 1) the training of proposed subword tokenization, which uses the expectation-maximization algorithm and deep neural networks, is empirically stable, 2) the proposed subword tokenization tokenizes text into subwords that are close to language-specific units, and 3) the proposed methods outperform the conventional methods using language model-based tokenization in terms of synthetic speech quality.

A Flexible Architecture for Urdu Phonemes-Based Concatenative Speech Synthesis

TTS (Text-to-Speech) synthesis systems are extensively used across the world to intensify the accessibility of information and to make it possible for the handicapped to be involved directly with computers to get the benefits from this high technology revolution. Various TTS synthesis techniques have been used with their own advantages and limitations. There is not a concatenative synthesis strategy based architecture for Urdu TTS synthesis system for handling the homographs and to avoid the unnatural robot sounding speech produced due the use of di-phones. In this paper, we propose a flexible architecture for Urdu TTS synthesis system that uses concatenative synthesis strategy because this approach has the ability to join together the small corpus of speech to generate natural and intelligible sound. The main aspiration of this research is to disambiguate the homographs in the Urdu language and to avoid the unnatural robot sounding speech. Finally, the effectiveness of the system is tested in terms of intelligibility and acceptability on word and sentence level. The intelligibility rate is near to 80% and 65% while acceptability rate for the naturalness is 95% (75% natural, 20% acceptable).

Optimal state duration assignment in hidden Markov model‐based text‐to‐speech synthesis system

In state-of-the-art text-to-speech (TTS) systems the state durations for each phoneme are generated so as to maximise the state sequence probability given the constraint that the sum of all state durations should be equal to the phoneme duration. Such maximisation sometimes results in negative state durations when the specified phoneme duration is less than the sum of the means of all the states of the phoneme. Such discrepancy implicitly results in the violation of the equality constraint. This has implications for speech research problems, in which each phoneme duration is specified. One such problem is the use of the TTS synthesis system for singing voice synthesis research. An algorithm for state duration assignment is derived so as to maximise the probability of the state sequence with the constraints that the sum of state durations should be equal to the total duration of the phoneme and all the state durations must be greater than or equal to 1. Experimental results show that the proposed algorithm always produces state durations greater than or equal to 1 while satisfying the equality constraint.

Text-to-speech synthesis with an Indian language perspective

Speech synthesis deals with artificial production of speech, and a Text-to-Speech TTS system in this aspect converts natural language text into a corresponding spoken waveform or speech. There have been sufficient successes today in the area of speech and natural language processing that suggest that these technologies will continue to be a major area of research and development in creating intelligent systems. In this paper we provide an overview of the TTS synthesis technology along with details of the phases involved. The thrust has been given to explore the usefulness of this technique in designing a TTS system for Indian languages. This paper also focuses on some of the open research issues where work in this area may further be done.

Design and Development of a Prosody Generator for Arabic TTS Systems

Prosody modeling has become the backbone of TTS synthesis systems. Amongst all the prosodic modeling approaches, phonetic methods aiming to predict duration and F0 contour are being very praised, thanks to the development of regression tools, such as neural networks (NN). Besides, parametric representations like Fujisaki model for F0 contour generation help to reduce the problem into the approximation of parameters only. But, prior to the prediction process, text analysis should be carried out first, to select and encode the necessary input features. In our purpose to promote Arabic TTS synthesis, an Integrated Model of Arabic Prosody for Speech Synthesis (IMAPSS) tool has been designed to integrate our developed models for text analysis, NN-based phonemic duration prediction and Fujisaki-inspired F0 contour. Hence, the yielding parameters provide a command file to be read by speech synthesis systems, like MBROLA. General Terms Signal processing, Speech synthesis, Prosody, Neural Networks.

Design and Development of a Prosody Generator for Arabic TTS Systems

Prosody modeling has become the backbone of TTS synthesis systems. Amongst all the prosodic modeling approaches, phonetic methods aiming to predict duration and F0 contour are being very praised, thanks to the development of regression tools, such as neural networks (NN). Besides, parametric representations like Fujisaki model for F0 contour generation help to reduce the problem into the approximation of parameters only. But, prior to the prediction process, text analysis should be carried out first, to select and encode the necessary input features. In our purpose to promote Arabic TTS synthesis, an Integrated Model of Arabic Prosody for Speech Synthesis (IMAPSS) tool has been designed to integrate our developed models for text analysis, NN-based phonemic duration prediction and Fujisaki-inspired F0 contour. Hence, the yielding parameters provide a command file to be read by speech synthesis systems, like MBROLA. General Terms Signal processing, Speech synthesis, Prosody, Neural Networks.

Reliable Pitch Marking of Affective Speech at Peaks or Valleys Using Restricted Dynamic Programming

The affective communication channel plays a key role in multimodal human-computer interaction. In this context, the generation of realistic talking-heads expressing emotions both in appearance and speech is of great interest. The synthetic speech of talking-heads is generally obtained from a text-to-speech (TTS) synthesizer. One of the dominant techniques for achieving high-quality synthetic speech is unit-selection TTS (US-TTS) synthesis. Affective US-TTS systems are driven by affective annotated speech databases. Since affective speech involves higher acoustic variability than neutral speech, achieving trustworthy speech labeling is a more challenging task. To that effect, this paper introduces a methodology for achieving reliable pitch marking on affective speech. The proposal adjusts the pitch marks at the signal peaks or valleys after applying a three-stage restricted dynamic programming algorithm. The methodology can be applied as a post-processing of any pitch determination and pitch marking algorithm (with any local criterion for locating pitch marks), or their merging. The experiments show that the proposed methodology significantly improves the results of the input state-of-the-art markers on affective speech.

A Unified Framework for Multilingual Text-to-Speech Synthesis with SSML Specification as Interface

This paper describes the design of a unified framework for a multilingual text-to-speech (TTS) synthesis engine – Crystal. The unified framework defines the common TTS modules for different languages and/or dialects. The interfaces between consecutive modules conform to the speech synthesis markup language (SSML) specification for standardization, interoperability, multilinguality, and extensibility. Detailed module divisions and implementation technologies for the unified framework are introduced, together with possible extensions for the algorithm research and evaluation of the TTS synthesis. Implementation of a mixed-language TTS system for Chinese Putonghua, Chinese Cantonese, and English demonstrates the feasibility of the proposed unified framework.

A statistical approach to lexical stress assignment for TTS synthesis

Lexical stress is primarily important to generate a correct pronunciation of words in many languages; hence its correct placement is a major task in prosody prediction and generation for high-quality TTS (text-to-speech) synthesis systems. This paper proposes a statistical approach to lexical stress assignment for TTS synthesis in Romanian. The method is essentially based on n-gram language models at character level, and uses a modified Katz backoff smoothing technique to solve the problem of data sparseness during training. Monosyllabic words are considered as not carrying stress, and are separated by an automatic syllabification algorithm. A maximum accuracy of 99.11% was obtained on a test corpus of about 47,000 words.

Is text-to-speech synthesis ready for use in computer-assisted language learning?

Text-to-speech (TTS) synthesis, the generation of speech from text input, offers another means of providing spoken language input to learners in Computer-Assisted Language Learning (CALL) environments. Indeed, many potential benefits (ease of creation and editing of speech models, generation of speech models and feedback on demand, etc.) and uses (talking dictionaries, talking texts, dictation, pronunciation training, dialogue partner, etc.) of TTS synthesis in CALL have been put forward. Yet, the use of TTS synthesis in CALL is not widely accepted and only a few applications have found their way onto the market. One potential reason for this is that TTS synthesis has not been adequately evaluated for this purpose. Previous evaluations of TTS synthesis for use in CALL, have only addressed the comprehensibility of TTS synthesis. Yet, CALL places demands on the comprehensibility, naturalness, accuracy, register and expressiveness of the output of TTS synthesis. In this paper, the aforementioned aspects of the quality of the output of four state-of-the-art French TTS synthesis systems are evaluated with respect to their use in the three different roles that TTS synthesis systems may assume within CALL applications, namely: (1) reading machine, (2) pronunciation model and (3) conversational partner [Handley, Z., Hamel, M.-J., 2005. Establishing a methodology for benchmarking speech synthesis for computer-assisted language learning (CALL). Language Learning and Technology Journal 9(3), 99–119. Retrieved from: http://llt.msu.edu/vol9num3/handley/default.html.]. The results of this evaluation suggest that the best TTS synthesis systems are ready for use in applications in which they ‘add value’ to CALL, i.e. exploit the unique capacity of TTS synthesis to generate speech models on demand. An example of such an application is a dialogue partner. In order to fully meet the requirements of CALL, further attention needs to be paid to accuracy and naturalness, in particular at the prosodic level, and expressiveness.

Humanoid Audio–Visual Avatar With Emotive Text-to-Speech Synthesis

Emotive audio-visual avatars are virtual computer agents which have the potential of improving the quality of human-machine interaction and human-human communication significantly. However, the understanding of human communication has not yet advanced to the point where it is possible to make realistic avatars that demonstrate interactions with natural-sounding emotive speech and realistic-looking emotional facial expressions. In this paper, We propose the various technical approaches of a novel multimodal framework leading to a text-driven emotive audio-visual avatar. Our primary work is focused on emotive speech synthesis, realistic emotional facial expression animation, and the co-articulation between speech gestures (i.e., lip movements) and facial expressions. A general framework of emotive text-to-speech (TTS) synthesis using a diphone synthesizer is designed and integrated into a generic 3-D avatar face model. Under the guidance of this framework, we therefore developed a realistic 3-D avatar prototype. A rule-based emotive TTS synthesis system module based on the Festival-MBROLA architecture has been designed to demonstrate the effectiveness of the framework design. Subjective listening experiments were carried out to evaluate the expressiveness of the synthetic talking avatar.

MLP-based phone boundary refining for a TTS database

The automatic labeling of a large speech corpus plays an important role in the development of a high-quality Text-To-Speech (TTS) synthesis system. This paper describes a method for the automatic labeling of speech signals, which mainly involves the construction of a large database for a TTS synthesis system. The main objective of the work involves the refinement of an initial estimation of phone boundaries which are provided by an alignment, based on a Hidden Markov Model. A multilayer perceptron (MLP) was employed to refine the phone boundaries. To increase the accuracy of phoneme segmentation, several specialized MLPs were individually trained based on phonetic transition. The optimum partitioning of the entire phonetic transition space and the corresponding MLPs were constructed from the standpoint of minimizing the overall deviation from the hand-labeling position. The experimental results showed that more than 93% of all phone boundaries have a boundary deviation from a reference position smaller than 20 ms. We also confirmed that the database constructed using the proposed method produced results that were perceptually comparable to a hand-labeled database, based on subjective listening tests.

A new speech synthesis algorithm for high quality TTS systems: a mixed phase vocoder

AbstractCurrently, time domain pitch synchronous overlap‐add (TD‐PSOLA) is the most popular synthesis algorithm for text‐to‐speech (TTS) systems. The algorithm produces very high quality synthetic speech, particularly when a pitch modification factor is small. However, as the pitch modification factor becomes larger, the quality degradation due to a slight pitch epoch detection error becomes severe. On the other hand, the vocoder framework has very flexible prosody manipulation. It can obtain a uniform voice quality over a wide range of prosody modification. Unfortunately, the synthesized speech quality from the vocoder is far from natural human speech, often showing buzzy quality. To remedy buzzy quality from the vocoder and make more natural synthetic speech, we propose a new speech synthesis algorithm for high quality TTS systems that is based on the homomorphic vocoder framework. The impulse response of vocal tract is obtained by mixing the minimum phase in lower frequency band and original phase in higher frequency band—thus, the name is a mixed phase vocoder. Informal subjective listening tests reveal that the mixed phase vocoder is a good candidate for TTS synthesis with high intelligibility and naturalness. Copyright © 2004 AEI.

SPEAKER (GOVOREC): A Complete Slovenian Text-to Speech System

While text-to-speech (TTS) systems for major world languages are quite advanced, smaller languages, like our Slovenian language, lack quality TTS synthesis. At the “Jožef Stefan” Institute a system called SPEAKER (GOVOREC) has been developed. It is capable of automatic conversion of any Slovenian text into speech. The different phases of the synthesis task are performed by several sequentially operating independent modules: text analysis, prosody generation and segmental concatenation. The first module is comprised of text normalization and grapheme-to-phoneme conversion tasks. In order to generate rules for our synthesis scheme, data were collected by analysing the readings of ten speakers, five males and five females. A two-level approach has been used for duration modeling, and a so-called superpositional approach for pitch modeling. A speech waveform is synthesized using unit selection-based methods and a concatenative TD-PSOLA or HNM+ technique. The system was first implemented in the EMA employment agent, which provides information about available jobs in Slovenia and is now used by members of the Slovenian Foundation for the Blind and Vision-Impaired. Then, it was given free of charge to all people with disabilities. The system was awarded with the first prize for innovation in the field of life improvements for people with disabilities (given by the Government Office for the Disabled and Chronically Sick of the Republic of Slovenia). SPEAKER is freely accessible for non-commercial purposes through the Internet. Currently, several leading Slovenian telecommunication companies are testing the system for providing information (e-mail, short messaging service—SMS, weather reports, traffic information) through mobile phones.

Basic Research and Implementation Decisions for a Text-to-Speech Synthesis System in Romanian

Speech synthesis is one of the most language-dependent domains of speech technology. In particular, the natural language processing stage of a text-to-speech (TTS) system contains the largest part of the linguistic knowledge for a given language. In this respect, one can state that building a high-quality TTS system for a new language involves many theoretical and technical challenges. Especially, extensive studies must exist (or be done) at the linguistic level, in order to endow the system with the most relevant language information; this requirement represents an essential condition to obtain a true naturalness of the synthesized speech, starting from unrestricted input text. This paper presents fundamental research and the related implementation issues in developing a complete TTS system in Romanian, emphasizing the language particularities and their influence on improving the language processing stage efficiency. The first section describes our standpoint on TTS synthesis as well as the overall architecture of our TTS system. The next sections formulate several important tasks of the natural language processing stage (input text preprocessing, letter-to-phone conversion, acoustic database preparation) and discuss the design philosophy of the corresponding modules, implementation decisions and evaluation experiments. A distinct section is devoted to an acoustic-phonetic study that assisted the phone-set selection and acoustic database generation. The paper ends with conclusions and a description of the work that is currently in progress at other levels of the TTS system.

Concatenative text-to-speech synthesis based on waveform interpolation (a time frequency approach)

The time domain pitch synchronous overlap and add (TD-PSOLA) is the technique most used in comercial concatenative text-to-speech (TTS) synthesis systems. However, it is well known that TD-PSOLA presents several drawbacks. In order to overcome some drawbacks of the TD-PSOLA, this work presents a method based on time frequency interpolation (TFI) [Yair Shoham]. The method introduced here is a pitch-synchronous time-frequency approach of the waveform interpolation technique (WI) [Bastian Kleijn]. The goal of this work is to show that the TFI technique presents some important advantages to concatenative TTS synthesis. It allows pitch scale modification (PSM) independent of time scale modification (TSM) in a quite straightforward manner, and with high quality. TSM and PSM can be done in a continuous way, without any limitation of pitch period resolution. Moreover, the TFI technique allows simple, flexible, and efficient procedures to smooth diphone (or any other kind of unit) boundaries. The proposed system was evaluated using diphones and prosodies generated by the Festival system [Alan Black, Paul Taylor]. Subjective tests were performed, between the proposed TFI system and the standard TD-PSOLA system, highlighting the superior quality of the proposed system in comparison with TD-PSOLA.