Quality Of Synthesized Speech Research Articles

SR-TTS: a rhyme-based end-to-end speech synthesis system.

Deep learning has significantly advanced text-to-speech (TTS) systems. These neural network-based systems have enhanced speech synthesis quality and are increasingly vital in applications like human-computer interaction. However, conventional TTS models still face challenges, as the synthesized speeches often lack naturalness and expressiveness. Additionally, the slow inference speed, reflecting low efficiency, contributes to the reduced voice quality. This paper introduces SynthRhythm-TTS (SR-TTS), an optimized Transformer-based structure designed to enhance synthesized speech. SR-TTS not only improves phonological quality and naturalness but also accelerates the speech generation process, thereby increasing inference efficiency. SR-TTS contains an encoder, a rhythm coordinator, and a decoder. In particular, a pre-duration predictor within the cadence coordinator and a self-attention-based feature predictor work together to enhance the naturalness and articulatory accuracy of speech. In addition, the introduction of causal convolution enhances the consistency of the time series. The cross-linguistic capability of SR-TTS is validated by training it on both English and Chinese corpora. Human evaluation shows that SR-TTS outperforms existing techniques in terms of speech quality and naturalness of expression. This technology is particularly suitable for applications that require high-quality natural speech, such as intelligent assistants, speech synthesized podcasts, and human-computer interaction.

Research on a Mongolian Text to Speech Model Based on Ghost and ILPCnet

The core challenge of speech synthesis technology is how to convert text information into an audible audio form to meet the needs of users. In recent years, the quality of speech synthesis based on end-to-end speech synthesis models has been significantly improved. However, due to the characteristics of the Mongolian language and the lack of an audio corpus, the Mongolian speech synthesis model has achieved few results, and there are still some problems with the performance and synthesis quality. First, the phoneme information of Mongolian was further improved and a Bang-based pre-training model was constructed to reduce the error rate of Mongolian phonetic synthesized words. Second, a Mongolian speech synthesis model based on Ghost and ILPCnet was proposed, named the Ghost-ILPCnet model, which was improved based on the Para-WaveNet acoustic model, replacing ordinary convolution blocks with stacked Ghost modules to generate Mongolian acoustic features in parallel and improve the speed of speech generation. At the same time, the improved vocoder ILPCnet had a high synthesis quality and low complexity compared to other vocoders. Finally, a large number of data experiments were conducted on the proposed model to verify its effectiveness. The experimental results show that the Ghost-ILPCnet model has a simple structure, fewer model generation parameters, fewer hardware requirements, and can be trained in parallel. The average subjective opinion score of its synthesized speech reached 4.48 and the real-time rate reached 0.0041. It ensures the naturalness and clarity of synthesized speech, speeds up the synthesis speed, and effectively improves the performance of the Mongolian speech synthesis model.

Voice Cloning Using Artificial Intelligence and Machine Learning: A Review

This paper represents a thorough method for integrating emotions, texttospeech conversion, and state of the art voice cloning. The paper focuses on novel background noise adaptation, emotional voice synthesis, and multi-speaker voice cloning for better speech synthesis. The synthesis of emotive voices, multi-speaker voice cloning, and creative methods for modifying background noise to improve speech synthesis quality are among the topics covered in this study. Additionally, the study explores the domain of emotional artificial intelligence by adding a variety of emotions to artificial voices, improving user engagement through sympathetic reactions. The study also looks at how background noise can be altered to change it from a disturbing to a silent, non-disruptive state. The texttospeech systems usability in noisy conditions is greatly enhanced by this improvement. By integrating these components, the project makes a substantial contribution to text to speech, emotional AI, and voice cloning, creating new avenues for human-computer connection.

The Evaluation of Performance Related to Noise Robustness of VITS for Speech Synthesis

In recent years, the utilization of voice interfaces has gained significant popularity, with speech synthesis technology playing a pivotal role in their functionality. However, speech synthesis technology is susceptible to noise interference in practical applications, which may lead to a decrease in the quality of speech synthesis. In this paper, the noise robustness of the Variational Inference with adversarial learning for end-to-end Text-to-Speech (VITS) model was investigated, which has shown promising results in speech synthesis tasks. This study conducted experiments using six different texts and evaluated the speech synthesis results using three metrics: Mean Opinion Score (MOS), Disfluency Prediction (DIS), and Colorfulness Prediction (COL). The experiments consist of a control group and six experimental groups, which include two types of noise, Additive White Gaussian Noise (AWGN) and real-world noise, at three different signal-to-noise ratios (SNRs). The results demonstrated that both types of noise can significantly reduce the MOS scores of the synthesized speech, with a more severe decrease at lower SNRs. In terms of DIS and COL scores, the VITS model exhibits superior performance with real-world noise compared to AWGN noise, especially at lower SNRs. Moreover, even at an SNR of 3, the VITS model can still generate intelligible speech, which demonstrates its high noise robustness. The findings have important implications for the design of robust speech synthesis models in noisy environments. Future studies may focus on exploring more advanced noise-robust models or investigating the application of these models in practical voice interfaces.

APNet: An All-Frame-Level Neural Vocoder Incorporating Direct Prediction of Amplitude and Phase Spectra

This paper presents a novel neural vocoder named APNet which reconstructs speech waveforms from acoustic features by predicting amplitude and phase spectra directly. The APNet vocoder is composed of an amplitude spectrum predictor (ASP) and a phase spectrum predictor (PSP). The ASP is a residual convolution network which predicts frame-level log amplitude spectra from acoustic features. The PSP also adopts a residual convolution network using acoustic features as input, then passes the output of this network through two parallel linear convolution layers respectively, and finally integrates into a phase calculation formula to estimate frame-level phase spectra. Finally, the outputs of ASP and PSP are combined to reconstruct speech waveforms by inverse short-time Fourier transform (ISTFT). All operations of the ASP and PSP are performed at the frame level. We train the ASP and PSP jointly and define multilevel loss functions based on amplitude mean square error, phase anti-wrapping error, short-time spectral inconsistency error and time domain reconstruction error. Experimental results show that our proposed APNet vocoder achieves an approximately 8x faster inference speed than HiFi-GAN v1 on a CPU due to the all-frame-level operations, while its synthesized speech quality is comparable to HiFi-GAN v1. The synthesized speech quality of the APNet vocoder is also better than that of several equally efficient models. Ablation experiments also confirm that the proposed parallel phase estimation architecture is essential to phase modeling and the proposed loss functions are helpful for improving the synthesized speech quality.

Common brain activity features discretization for predicting perceived speech quality

The synthesized speech quality evaluation is one of the important steps to ensure the generated speech audio sounds good to humans. There are two main approaches to perform the evaluation; subjective and objective. Subjective approaches use human as the assessor, which is the most natural approach. However, it is time-consuming and expensive. Hence, it has generally been replaced by the quicker and cheaper objective approaches. Nevertheless, since objective approaches only analyze the audio features, the predicted quality might not correlated to what humans would perceive. Recent studies shows that brain activity contains some information that can be useful to enhance the prediction performance. This work proposed a method to extract the common features among participants’ brain activity to predict the perceived speech audio quality. The result shows that the proposed approach significantly reduces the prediction error.

Autoregressive Speech-To-Text Alignment is a Critical Component of Neural Text-To-Speech (TTS) Models

Autoregressive speech-to-text alignment is a critical component of neural text-to-speech (TTS) models. Commonly, autoregressive TTS models rely on an attention mechanism to train these alignments online--but they are often brittle and fail to generalize in long utterances or out-of-domain text, leading to missing or repeating words. Non-autoregressive endto end TTS models usually rely on durations extracted from external sources. Our work exploits the alignment mechanism proposed in RAD -, which can be applied to various neural TTS architectures. In our experiments, the proposed alignment learning framework improves all tested TTS architectures—both autoregressive (Flowtron and Tacotron 2) and non-autoregressive (FastPitch, FastSpeech 2, RAD-TTS). Specifically, it improves alignment convergence speed of existing attention-based mechanisms; simplifies the training pipeline; and makes models more robust to errors on long utterances. Most importantly, it also improved the perceived speech synthesis quality when subject to expert human evaluation.

Meta-Learning for Mandarin-Tibetan Cross-Lingual Speech Synthesis

The paper proposes a meta-learning-based Mandarin-Tibetan cross-lingual text-to-speech (TTS) to realize both Mandarin and Tibetan speech synthesis under a unique framework. First, we build two kinds of Tacotron2-based Mandarin-Tibetan cross-lingual baseline TTS. One is a shared encoder Mandarin-Tibetan cross-lingual TTS, and another is a separate encoder Mandarin-Tibetan cross-lingual TTS. Both baseline TTS use the speaker classifier with a gradient reversal layer to disentangle speaker-specific information from the text encoder. At the same time, we design a prosody generator to extract prosodic information from sentences to explore syntactic and semantic information adequately. To further improve the synthesized speech quality of the Tacotron2-based Mandarin-Tibetan cross-lingual TTS, we propose a meta-learning-based Mandarin-Tibetan cross-lingual TTS. Based on the separate encoder Mandarin-Tibetan cross-lingual TTS, we use an additional dynamic network to predict the parameters of the language-dependent text encoder that could realize better cross-lingual knowledge sharing in the sequence-to-sequence TTS. Lastly, we synthesize Mandarin or Tibetan speech through the unique acoustic model. The baseline experimental results show that the separate encoder Mandarin-Tibetan cross-lingual TTS could handle the input of different languages better than the shared encoder Mandarin-Tibetan cross-lingual TTS. The experimental results further show that the proposed meta-learning-based Mandarin-Tibetan cross-lingual speech synthesis method could effectively improve the voice quality of synthesized speech in terms of naturalness and speaker similarity.

How to generate Muong speech directly from Vietnamese text: Cross-lingual speech synthesis for close language pair

The paper introduces a method for automatic translation of Vietnamese text into Muong speech in two dialects, Muong Bi - Hoa Binh and Muong Tan Son - Phu Tho, which are all unwritten dialects of the Muong language. Due to the very close relationship between the Vietnamese and Muong languages, the translation system was built to look like a cross-lingual speech synthesis system, in which the input is the text of one language (i.e., the Vietnamese) and the output is the speech of another language (i.e., the two Muong dialects). The system used the modern sequence-to-sequence TTS neural models Tacotron2 and WaveGlow. The evaluation results showed a high quality of translation (with a fluency score of 4.61/5.0 and an adequacy score of 4.79/5.0) and also synthesized speech quality (with naturalness on the MOS scale of 4.68/5.0 and intelligibility of 94.60%). The received results show that the applicability of the proposed system to other minority languages is promising, especially in the case of unwritten languages.

A two-level Item Response Theory model to evaluate speech synthesis and recognition

Automatic speech recognition (ASR) systems should be tested ideally using diverse speech test data. A promising alternative to produce such test data is to synthesize speeches from diverse sentences and speakers. However, despite the great amount of test data that can be produced, not all speeches are equally relevant. This paper proposes a two-level Item Response Theory (IRT) model to simultaneously evaluate ASR systems, speakers and sentences. In the first level, the transcription rates obtained by a pool of ASR systems on a set of synthesized speeches are recorded and then analyzed to estimate: each speech’s difficulty and each ASR system’s ability. In the second level, each speech’s difficulty is decomposed as a function of two factors: the sentence’s difficulty and the speaker’s quality. Thus, the speech’s difficulty is high when generated from a difficult sentence and a bad speaker, while an ASR is good when it is robust to hard speeches. Performed experiments revealed useful insights on how the quality of speech synthesis and recognition can be affected by distinct factors (e.g., sentence difficulty and speaker ability).

Learning to Model Prosodic and Spectral Features for Non-parallel Emotive Speech Conversion

Emotion conversion in speech has attracted recent attention owing to its importance in human-machine interaction and the current high quality of speech synthesis. Most existing approaches rely on parallel data, which is not available in many real-time applications. We propose a non-parallel emotion conversion approach based on the cycle generative adversarial network (cycleGAN) framework. We introduce new variants of cycleGAN that use recurrent neural networks and multi-kernel convolutional neural networks for modeling prosodic features along with spectral features for emotion conversion in speech. Subjective evaluation results show the effectiveness of our approach in converting natural speech, and also unseen synthesized speech samples to different target emotive states.

A Cross-Lingual Speech Synthesis System of Malay and Indonesian Based on HMM–DNN

English as a second language is widely used in countries such as Malaysia and Indonesia, and it is common for English words to appear in Malay and Indonesian sentences. Malay and Indonesian have high homology and relatively few electronic language resources. We combine the corpus datasets of these two similar languages to design and implement a HMM–DNN-based cross-lingual speech synthesis system for Malay (including English words) and Indonesian (including English words). The methods used include: sharing synthesis units between Malay, Indonesian, and English, designing unified context attributes and question set in the process of acoustic model training, speaker-adaptive training with speech corpus of these three languages, and synthesizing speech using speaker-dependent Malay and Indonesian acoustic models. Experimental results show that the speech synthesis quality of the system is better than the traditional Hidden Markov model-based cross-lingual speech synthesis system.

Improvement of the post-processing algorithm in AMR-NB speech codec

Abstract Narrow band adaptive multi-rate (AMR-NB) is a speech codec that supports multiple bit-rate modes and is widely used in mobile communication systems. However, the synthesized speech quality of its lower bit-rate modes is not very satisfying. This paper aims to improve the synthesized speech quality of the lower bit-rate modes of AMR-NB (e.g., 4.75kb/s) by optimizing its post-processing algorithm. The optimization mainly focuses on four modules including the formant filter with adjusted correction factor, the fixed codebook gain smoothing, the adaptive gain control and the anti-sparse processing. The experimental results show that the AMR-NB codec with the optimized post-processing algorithm achieves an obvious improvement of the synthesized speech quality, and the lower bit-rate mode achieves the bigger improvement.

Acoustic modeling for Kazakh speech synthesis

We present a new framework of generative adversarial network for training of acoustic model for speech synthesis. The proposed generative adversarial network consists of a generator and a pair of agent discriminators, where the generator predicts the acoustic features from the linguistic representation. Training and testing were carried out on the Kazakh speech corpus, which consisted of 5.6 hours of speech recording. According to the experiment results the 3.46 mean opinion score was obtained which shows an acceptable quality of speech synthesis. This approach of the acoustic model development can be applied in speech synthesis systems of the other languages.

A Comparison Between STRAIGHT, Glottal, and Sinusoidal Vocoding in Statistical Parametric Speech Synthesis

A vocoder is used to express a speech waveform with a controllable parametric representation that can be converted back into a speech waveform. Vocoders representing their main categories (mixed excitation, glottal, and sinusoidal vocoders) were compared in this study with formal and crowd-sourced listening tests. The vocoder quality was measured within the context of analysis–synthesis as well as text-to-speech (TTS) synthesis in a modern statistical parametric speech synthesis framework. Furthermore, the TTS experiments were divided into synthesis with vocoder-specific features and synthesis with a shared envelope model, where the waveform generation method of the vocoders is mainly responsible for the quality differences. Finally, all of the tests included four distinct voices as a way to investigate the effect of different speakers on the synthesized speech quality. The obtained results suggest that the choice of the voice has a profound impact on the overall quality of the vocoder-generated speech, and the best vocoder for each voice can vary case by case. The single best-rated TTS system was obtained with the glottal vocoder GlottDNN using a male voice with low expressiveness. However, the results indicate that the sinusoidal vocoder PML (pulse model in log-domain) has the best overall performance across the performed tests. Finally, when controlling for the spectral models of the vocoders, the observed differences are similar to the baseline results. This indicates that the waveform generation method of a vocoder is essential for quality improvements.

Interactive Hesitation Synthesis: Modelling and Evaluation

Conversational spoken dialogue systems that interact with the user rather than merely reading the text can be equipped with hesitations to manage dialogue flow and user attention. Based on a series of empirical studies, we elaborated a hesitation synthesis strategy for dialogue systems, which inserts hesitations of a scalable extent wherever needed in the ongoing utterance. Previously, evaluations of hesitation systems have shown that synthesis quality is affected negatively by hesitations, but that they result in improvements of interaction quality. We argue that due to its conversational nature, hesitation synthesis needs interactive evaluation rather than traditional mean opinion score (MOS)-based questionnaires. To validate this claim, we dually evaluate our system’s speech synthesis component, on the one hand, linked to the dialogue system evaluation, and on the other hand, in a traditional MOS way. We are thus able to analyze and discuss differences that arise due to the evaluation methodology. Our results suggest that MOS scales are not sufficient to assess speech synthesis quality, leading to implications for future research that are discussed in this paper. Furthermore, our results indicate that synthetic hesitations are able to increase task performance and that an elaborated hesitation strategy is necessary to avoid likability issues.

Low complexity forward error correction for CELP-type speech coding over erasure channel transmission

One of the well-known problems of Code-Excited Linear Prediction (CELP)-type codec is its vulnerability to a frame erasure. When a frame is erased, the inter-frame dependency introduced by the Long Term Prediction causes a desynchronization of the Adaptive Codebook (ACB) which introduces in its turn an error propagation through the correctly received frames. In this paper, we propose a media-specific Forward Error Correction (FEC) method using a Pitch-Pulse Codebook (PPCB)-based approach to model the ACB contribution for voiced frame (frame onset) determined under Zero Crossing Rate constraint. The PPCB uses a single pulse optimized by Multipulse Maximum Likelihood Quantization algorithm to model the pitch-like contribution at the encoder side while the quantized version of that pulse will be sent as FEC information to resynchronize the ACB at the decoder side after a frame erasure. Through this approach a noticeable improvement of the synthesis speech quality is achieved under adverse channel conditions with the advantage of low computational complexity while the legacy bit-rate of the codec is kept unchanged.

Global Variance in Speech Synthesis With Linear Dynamical Models

Linear Dynamical Models (LDMs) have been used in speech synthesis recently as an alternative to hidden Markov models (HMMs). Among the advantages of LDMs are the ability to capture the dynamics of speech and the achievement of synthesized speech quality similar to HMM-based speech systems on a smaller footprint. However, such as in the HMM case, LDMs produce over-smoothed trajectories of speech parameters, resulting in muffled quality of synthetic speech. Inspired by a similar problem found in HMM-based speech synthesis, where the naturalness of the synthesized speech is greatly improved when the global variance (GV) is compensated, this paper proposes a novel speech parameter generation algorithm that considers GV in LDM-based speech synthesis. Experimental results show that the application of GV during parameter generation significantly improves speech quality.

Performance evaluation of a ACF-AMDF based pitch detection scheme in real-time

In this paper, a complete speech analysis-synthesis system for a pitch detection algorithm based on short-time autocorrelation function (ACF) and average magnitude difference function (AMDF) has been implemented in real-time using TMS320C6713 DSK. Performance of this system has been compared with the analysis-synthesis system that use autocorrelation, cepstrum and wavelet based pitch detection method in terms of synthesized speech quality, Diagnostic Rhyme Test, execution time and memory consumption. Results show that this method of pitch detection scheme is the best in terms of speech quality, execution time and memory consumption. Moreover the synthesized speech using ACF-AMDF method of pitch detection is more intelligible compared to other pitch detection algorithm.

Text-to-speech synthesis system with Arabic diacritic recognition system

Text-to-speech synthesis system has been widely studied for many languages. However, speech synthesis for Arabic language has not sufficient progresses and it is still in its first stage. Statistical parametric synthesis based on hidden Markov models was the most commonly applied approach for Arabic language. Recently, synthesized speech quality based on deep neural networks was found as intelligible as human voice. This paper describes a Text-To-Speech (TTS) synthesis system for modern standard Arabic language based on statistical parametric approach and Mel-cepstral coefficients. Deep neural networks achieved state-of-the-art performance in a wide range of tasks, including speech synthesis. Our TTS system includes a diacritization system which is very important for Arabic TTS application. Our diacritization system is also based on deep neural networks. In addition to the use deep techniques, different methods were also proposed to model the acoustic parameters in order to address the problem of acoustic models accuracy. They are based on linguistic and acoustic characteristics (e.g. letter position based diacritization system, unit types based synthesis system, diacritic marks based synthesis system) and based on deep learning techniques (stacked generalization techniques). Experimental results show that our diacritization system can generate a diacritized text with high accuracy. As regards the speech synthesis system, the experimental results and subjective evaluation show that our proposed method for synthesis system can generate intelligible and natural speech.