Speech Synthesis System Research Articles

Адаптація фреймворку WORLD для пофреймового аналізу мовлення в реальному часі

WORLD is a vocoder-based speech synthesis system developed by M. Morise et al. and implemented in C++. It was demonstrated to have improved performance and accuracy when compared to other algorithms. However, it turned out to not perform well in certain scenarios, particularly, when applying the framework to very short waveforms on a frame-by-frame basis. This paper reviews the issues of the C++ implementation of WORLD and pro-poses modified versions of its constituting algorithms that attempt to mitigate those issues. The resulting framework is tested on both synthetic signals and on real recorded speech.

Implement Speech Recognition and Synthesis System Using Python

Python script implements a speech recognition and synthesis system using the Speech Recognition and pyttsx3 libraries. The system allows users to speak into their microphone, recognizes the speech using Google's Web Speech API, and then speaks out the recognized text. The main program loop continuously listens for speech input, recognizes it, and then synthesizes the recognized text into speech. The loop terminates when the recognized speech is "exit". This implementation demonstrates the integration of speech recognition and synthesis capabilities in Python, showcasing their potential applications in various domains such as human-computer interaction, accessibility, and automation.

SR-TTS: a rhyme-based end-to-end speech synthesis system.

Deep learning has significantly advanced text-to-speech (TTS) systems. These neural network-based systems have enhanced speech synthesis quality and are increasingly vital in applications like human-computer interaction. However, conventional TTS models still face challenges, as the synthesized speeches often lack naturalness and expressiveness. Additionally, the slow inference speed, reflecting low efficiency, contributes to the reduced voice quality. This paper introduces SynthRhythm-TTS (SR-TTS), an optimized Transformer-based structure designed to enhance synthesized speech. SR-TTS not only improves phonological quality and naturalness but also accelerates the speech generation process, thereby increasing inference efficiency. SR-TTS contains an encoder, a rhythm coordinator, and a decoder. In particular, a pre-duration predictor within the cadence coordinator and a self-attention-based feature predictor work together to enhance the naturalness and articulatory accuracy of speech. In addition, the introduction of causal convolution enhances the consistency of the time series. The cross-linguistic capability of SR-TTS is validated by training it on both English and Chinese corpora. Human evaluation shows that SR-TTS outperforms existing techniques in terms of speech quality and naturalness of expression. This technology is particularly suitable for applications that require high-quality natural speech, such as intelligent assistants, speech synthesized podcasts, and human-computer interaction.

Two efforts towards natural speech synthesis: Incorporating disfluency and speaking style change based on the interlocutor

During the period 2000–2005, a Japanese female speaker recorded her everyday conversations with many different interlocutors using a head-set microphone. As a result, 600 hours of natural Japanese speech data were obtained. This study describes a DNN-based speech synthesis system which was trained on 300 hours of the data, focusing on two unique efforts to make it more expressive in a human-like way: (1) allowing for disfluencies, and (2) accounting for the category of interlocutor. Incorporating some frequently observed disfluent patterns in general Japanese speech such as fillers, phrase-final rising intonation, and word-internal prolongation or suspension, is believed to be effective in practical application as certain disfluencies are connected to a speaker’s attitude in Japanese communication. For example, having word-internal prolongations can show hesitation or politeness, and word-internal suspending can show the speaker’s surprised attitude. Interlocutors in the original data were categorized into four groups: family, friend, child and others. This information was used in the training process, and as a result, the synthesizer can generate different speaking styles according to the interlocutor setting. Being able to generate disfluent speech and change the speaking style depending on who you are talking to can make the synthesizer ever more expressive.

Speech synthesis system based on big data and evaluation of Japanese language feeling

Speech synthesis system based on big data and evaluation of Japanese language feeling

Speaker anonymization using generative adversarial networks

The advent use of smart devices has enabled the emergence of many applications that facilitate user interaction through speech. However, speech reveals private and sensitive information about the user’s identity, posing several security risks. For example, a speaker’s speech can be acquired and used in speech synthesis systems to generate fake speech recordings that can be used to attack that speaker’s verification system. One solution is to anonymize the speaker’s identity from speech before using it. Existing anonymization schemes rely on using a pool of real speakers’ identities for anonymization, which may result in associating a speaker’s speech with an existing speaker. Hence, this paper investigates the use of Generative Adversarial Networks (GAN) to generate a pool of fake identities that are used for anonymization. Several GAN types were considered for this purpose, and the Conditional Tabular GAN (CTGAN) showed the best performance among all GAN types according to different metrics that measure the naturalness of the anonymized speech and its linguistic content.

Computational Approaches to Phonology: Advances in Speech Recognition and Synthesis

This article explores the field of computational approaches to phonology, with a particular focus on recent increases in popularity and speech synthesis. Phonology, as a department of linguistics, deals with the observation of sounds and their work in language. Recent improvements in this area have resulted in major improvements in accuracy and efficiency, towards the integration of deep recognition strategies together with recurrent neural networks (RNNs) and convolutional neural networks (CNNs). the models have tested the ability to capture complex phonetic patterns and decorate the overall performance of computerized speech reputation structures. These advances have seen packages in multiple domains, such as virtual assistants, text-to-speech structures, and language acquisition tools. This article looks at the main computational procedures used in phonology, along with characteristic-based representations, rule-based modes, and statistical methods. It explores demanding situations related to phonological evaluation, which include phoneme segmentation, pronunciation variants, and language-specific problems. In addition, this paper discusses mixing phonological knowledge with device learning strategies, which enables the development of robust and correct speech recognition and synthesis systems. As usual, this research represents remarkable advances from computational methods to phonology, particularly in the geographical area of speech recognition and synthesis. the combination of technology and linguistic insights has increased the accuracy, naturalness, and efficiency in processing spoken language. These enhancements open up new opportunities to enhance human-pc interaction, language recognition, and many other packages that rely on speech-based technologies.

Deep learning-based speaker-adaptive postfiltering with limited adaptation data for embedded text-to-speech synthesis systems

End-to-end (e2e) speech synthesis systems have become popular with the recent introduction of text-to-spectrogram conversion systems, such as Tacotron, that use encoder–decoder-based neural architectures. Even though those sequence-to-sequence systems can produce mel-spectrograms from the letters without a text processing frontend, they require substantial amounts of well-manipulated, labeled audio data that have high SNR and minimum amounts of artifacts. These data requirements make it difficult to build end-to-end systems from scratch, especially for low-resource languages. Moreover, most of the e2e systems are not designed for devices with tiny memory and CPU resources. Here, we investigate using a traditional deep neural network (DNN) for acoustic modeling together with a postfilter that improves the speech features produced by the network. The proposed architectures were trained with the relatively noisy, multi-speaker, Wall Street Journal (WSJ) database and tested with unseen speakers. The thin postfilter layer was adapted with minimal data to the target speaker for testing. We investigated several postfilter architectures and compared them with both objective and subjective tests. Fully-connected and transformer-based architectures performed the best in subjective tests. The novel adversarial transformer-based architecture with adaptive discriminator loss performed the best in the objective tests. Moreover, it was faster than the other architectures both in training and inference. Thus, our proposed lightweight transformer-based postfilter architecture significantly improved speech quality and efficiently adapted to new speakers with few shots of data and a hundred training iterations, making it computationally efficient and suitable for scalability.

Exploring Prosodic Features Modelling for Secondary Emotions Needed for Empathetic Speech Synthesis

A low-resource emotional speech synthesis system for empathetic speech synthesis based on modelling prosody features is presented here. Secondary emotions, identified to be needed for empathetic speech, are modelled and synthesised in this investigation. As secondary emotions are subtle in nature, they are difficult to model compared to primary emotions. This study is one of the few to model secondary emotions in speech as they have not been extensively studied so far. Current speech synthesis research uses large databases and deep learning techniques to develop emotion models. There are many secondary emotions, and hence, developing large databases for each of the secondary emotions is expensive. Hence, this research presents a proof of concept using handcrafted feature extraction and modelling of these features using a low-resource-intensive machine learning approach, thus creating synthetic speech with secondary emotions. Here, a quantitative-model-based transformation is used to shape the emotional speech's fundamental frequency contour. Speech rate and mean intensity are modelled via rule-based approaches. Using these models, an emotional text-to-speech synthesis system to synthesise five secondary emotions-anxious, apologetic, confident, enthusiastic and worried-is developed. A perception test to evaluate the synthesised emotional speech is also conducted. The participants could identify the correct emotion in a forced response test with a hit rate greater than 65%.

An Emotion Speech Synthesis Method Based on VITS

People and things can be connected through the Internet of Things (IoT), and speech synthesis is one of the key technologies. At this stage, end-to-end speech synthesis systems are capable of synthesizing relatively realistic human voices, but the current commonly used parallel text-to-speech suffers from loss of useful information during the two-stage delivery process, and the control features of the synthesized speech are monotonous, with insufficient expression of features, including emotion, leading to emotional speech synthesis becoming a challenging task. In this paper, we propose a new system named Emo-VITS, which is based on the highly expressive speech synthesis module VITS, to realize the emotion control of text-to-speech synthesis. We designed the emotion network to extract the global and local features of the reference audio, and then fused the global and local features through the emotion feature fusion module based on the attention mechanism, so as to achieve more accurate and comprehensive emotion speech synthesis. The experimental results show that the Emo-VITS system’s error rate went up a little bit compared with the network without emotionality and does not affect the semantic understanding. However, this system is superior to other networks in naturalness, sound quality, and emotional similarity.

A Multi-Layer Transcription Model – concept outline

This paper discusses the assumptions of a Multi-Layer Transcription Model (hereinafter: MLTM). The solution presented is an advanced grapheme-to-phoneme (G2P) conversion method that can be implemented in technical applications, such as automatic speech recognition and synthesis systems. The features of MLTM also facilitate the application of text-to-transcription conversion in linguistic research. The model presented here is the basis for multi-step processing of the orthographic representation of words with those being transcribed gradually. The consecutive stages of the procedure include, among other things, identification of multi-character phonemes, voicing status change, and consonant clusters simplification. The multi-layer model described in this paper makes it possible to assign individual phonetic processes (for example assimilation), as well as other types of transformation, to particular layers. As a result, the set of rules becomes more transparent. Moreover, the rules related to any process can be modified independently of the rules connected with other forms of transformation, provided that the latter have been assigned to a different layer. These properties of the multi-layer transcription model in question provide crucial advantages for the solutions based on it, such as their flexibility and transparency. There are no assumptions in the model about the applicable number of layers, their functions, or the number of rules defined in each layer. A special mechanism used for the implementation of the MLTM concept enables projection of individual characters onto either a phonemic or a phonetic transcript (obtained after processing in the final layer of the MLTM-based system has been completed). The solution presented in this text has been implemented for the Polish language, however, it is not impossible to use the same model for other languages.

Emotional Vietnamese Speech Synthesis Using Style-Transfer Learning

In recent years, speech synthesis systems have allowed for the production of very high-quality voices. Therefore, research in this domain is now turning to the problem of integrating emotions into speech. However, the method of constructing a speech synthesizer for each emotion has some limitations. First, this method often requires an emotional-speech data set with many sentences. Such data sets are very time-intensive and labor-intensive to complete. Second, training each of these models requires computers with large computational capabilities and a lot of effort and time for model tuning. In addition, each model for each emotion failed to take advantage of data sets of other emotions. In this paper, we propose a new method to synthesize emotional speech in which the latent expressions of emotions are learned from a small data set of professional actors through a Flowtron model. In addition, we provide a new method to build a speech corpus that is scalable and whose quality is easy to control. Next, to produce a high-quality speech synthesis model, we used this data set to train the Tacotron 2 model. We used it as a pre-trained model to train the Flowtron model. We applied this method to synthesize Vietnamese speech with sadness and happiness. Mean opinion score (MOS) assessment results show that MOS is 3.61 for sadness and 3.95 for happiness. In conclusion, the proposed method proves to be more effective for a high degree of automation and fast emotional sentence generation, using a small emotional-speech data set.

Comparison of four classifiers for speech-music discrimination: a first case study for costa rican radio broadcasting

During the past decades, a vast amount of audio data has be- come available in most languages and regions of the world. The efficient organization and manipulation of this data are important for tasks such as data classification, searching for information, diarization among many others, but also can be relevant for building corpora for training models for automatic speech recognition or building speech synthesis systems. Several of those tasks require extensive testing and data for specific languages and accents, especially when the development of communication systems with machines is a goal. In this work, we explore the application of several classifiers for the task of discriminating speech and music in Costa Rican radio broadcast. This discrimination is a first task in the exploration of a large corpus, to determine whether or not the available information is useful for particular research areas. The main contribution of this exploratory work is the general procedure and selection of algorithms for the Costa Rican radio corpus, which can lead to the extensive use of this source of data in many own applications and systems.

How to generate Muong speech directly from Vietnamese text: Cross-lingual speech synthesis for close language pair

The paper introduces a method for automatic translation of Vietnamese text into Muong speech in two dialects, Muong Bi - Hoa Binh and Muong Tan Son - Phu Tho, which are all unwritten dialects of the Muong language. Due to the very close relationship between the Vietnamese and Muong languages, the translation system was built to look like a cross-lingual speech synthesis system, in which the input is the text of one language (i.e., the Vietnamese) and the output is the speech of another language (i.e., the two Muong dialects). The system used the modern sequence-to-sequence TTS neural models Tacotron2 and WaveGlow. The evaluation results showed a high quality of translation (with a fluency score of 4.61/5.0 and an adequacy score of 4.79/5.0) and also synthesized speech quality (with naturalness on the MOS scale of 4.68/5.0 and intelligibility of 94.60%). The received results show that the applicability of the proposed system to other minority languages is promising, especially in the case of unwritten languages.

Cross-lingual multi-speaker speech synthesis with limited bilingual training data

Modeling voices for multiple speakers and multiple languages with one speech synthesis system has been a challenge for a long time, especially in low-resource cases. This paper presents two approaches to achieve cross-lingual multi-speaker text-to-speech (TTS) and code-switching synthesis under two training scenarios: (1) cross-lingual synthesis with sufficient data, (2) cross-lingual synthesis with limited data per speaker. Accordingly, a novel TTS synthesis model and a non-autoregressive multi-speaker voice conversion model are proposed. The TTS model designed for sufficient-data cases has a Tacotron-based structure that uses shared phonemic representations associated with numeric language ID codes. As for the data-limited scenario, we adopt a framework cascading several speech modules to achieve our goal. In particular, we proposed a non-autoregressive many-to-many voice conversion module to address multi-speaker synthesis for data-insufficient cases. Experimental results on speaker similarity show that our proposed voice conversion module can maintain the voice characteristics well in data-limited cases. Both approaches use limited bilingual data and demonstrate impressive performance in cross-lingual synthesis, which can deliver fluent foreign speech and even code-switching speech for monolingual speakers.

TTS - VLSP 2021: The Thunder Text-To-Speech System

This paper describes our speech synthesis system participating in the Vietnamese Text-To-Speech track of the 2021 VLSP evaluation campaign. The goal of this challenge is to build a synthetic voice from a provided spontaneous speech corpus in Vietnamese. In this paper, we propose our implementation of FastSpeech2 model on spontaneous speech. We used a special strategy with spontaneous datasets using the TTS system. We present our utilization in generating mel-spectrograms from given texts and then synthesize speech from generated mel-spectrograms using a separately trained vocoder. In evaluation, our team achieved 3.943 mean score in MOS in-domain test, 3.3 in MOS out-domain test, and 85.00% SUS, which indicates the effectiveness of the proposed system.

Measuring the Quality of Low-Resourced Statistical Parametric Speech Synthesis Trained with Noise-Degraded Data Supported by the University of Costa Rica

After the successful implementation of speech synthesis in several languages, the study of robustness became an important topic so as to increase the possibility of building voices from non-standard sources, e.g. historical recordings, children's speech, and data freely available on the Internet. In this work, a measure of the influence of noise in the source speech of the statistical parametric speech synthesis system based on HMM is performed, for a case of a low-resourced database. For this purpose, three types of additive noise were considered at five signal-to-noise ratio levels to affect the source speech data. Using objective measures to assess the perceptual quality of the results and the propagation of the noise through all the processes of building speech synthesis, the results show a severe drop in the quality of artificial speech, even for the cases of lower levels of noise. Such degradation seems to be independent of the noise type, and is at lower proportion to the noise level. This results are of importance for any practical implementation of speech synthesis from degraded data in similar conditions, and shows that applying denoising processes became mandatory in order to keep the possibility of building intelligible voices.

A Situational Analysis of Current Speech-Synthesis Systems for Child Voices: A Scoping Review of Qualitative and Quantitative Evidence

(1) Background: Speech synthesis has customarily focused on adult speech, but with the rapid development of speech-synthesis technology, it is now possible to create child voices with a limited amount of child-speech data. This scoping review summarises the evidence base related to developing synthesised speech for children. (2) Method: The included studies were those that were (1) published between 2006 and 2021 and (2) included child participants or voices of children aged between 2–16 years old. (3) Results: 58 studies were identified. They were discussed based on the languages used, the speech-synthesis systems and/or methods used, the speech data used, the intelligibility of the speech and the ages of the voices. Based on the reviewed studies, relative to adult-speech synthesis, developing child-speech synthesis is notably more challenging. Child speech often presents with acoustic variability and articulatory errors. To account for this, researchers have most often attempted to adapt adult-speech models, using a variety of different adaptation techniques. (4) Conclusions: Adapting adult speech has proven successful in child-speech synthesis. It appears that the resulting quality can be improved by training a large amount of pre-selected speech data, aided by a neural-network classifier, to better match the children’s speech. We encourage future research surrounding individualised synthetic speech for children with CCN, with special attention to children who make use of low-resource languages.

Neural network based autonomous control of a speech synthesis system

• Control of composite tasks by adaptive heuristically enhanced gradient approximation. • Training an artificial neural network (ANN) to control the parameters of a speech synthesizer. • The training error is estimated from the “quality” of the produced speech and not at the outputs of the ANN. • Currently used ANN training algorithms cannot cope with the above requirement. • The presented methodology may be applied to other applications too, e.g., coordination of robotic manipulators. This work is inspired by the ability of neural systems to control the behavior of specialized actuator mechanisms in living organisms by monitoring the end-effect of their actions. We consider as an example of such an actuator mechanism the human vocal tract where neurons learn to activate its muscles that move the velum, jaw, tongue and the lips, in order to exhibit desired phonetic activity. As a technical approximation to such a setup, we use an artificial neural network (ANN) and a speech synthesizer and we study the capability of the ANN to estimate the synthesizer’s parameters targeting desired speech activity. In this setup, we assume that the training error is obtained by measuring the “perceived distance” between the original (target) and the synthesized speech signals. Thus, the training error needs to be measured after processing the output of the speech synthesizer, instead of measuring it directly at the outputs of the ANN. This operational requirement on error measurement restricts the application of widely used ANN training algorithms that are based on back propagation of gradients but can be met by our earlier proposed “Heuristically Enhanced Gradient Approximation” (HEGA) algorithm. We also propose enhancements to HEGA that further optimize its performance in this demanding application.

Development of a System and Interface for Speech Synthesis in Ukrainian for Websites

The paper describes the system of content synthesis and sounding on websites in Ukrainian, designed to simplify the consumption of content for the visually impaired, which features easy integration into the most popular content management system on sites, namely CMC Wordpress. Currently, people with visual impairments are severely limited in their use of Internet products because most web resources are not tailored to their needs. Modern information technology allows such people to receive information along with healthy ones thanks to solutions developed by scientists and engineers from different countries. Text information can be delivered to the visually impaired with a magnifying glass or by enlarging the font by software, blind - by sounding the text using computer programs or displaying texts on the Braille screen of the monitor. This is an effective solution, but the choice of methods for reproducing such information must be fully provided by people with disabilities themselves, which is a significant problem due to the significant time spent on information consumption. To facilitate the perception of visually impaired people when using websites, an international standard for webmasters has been developed - a guide to web content accessibility called the W3 Web Content Accessibility Guidelines (WCAG) Consortium 2.0. The standard describes in detail the requirements of visually impaired people that are recommended to be met in order for them to view the website without any problems. The basic provisions of WCAG 2.0 will describe the parameters and algorithms for scaling, clustering and separation of information by programs for the visually impaired and provide recommendations for writing website code. However, to implement such recommendations, webmasters need to learn new programming principles and algorithms and use additional development tools. This is often difficult, requiring additional training, which entails non-compliance by webmasters with such requirements.