Speech Synthesis Techniques Research Articles

Vowel perception research from formant thresholds to sentence intelligibility

Historically, research on the contribution of vowels to speech understanding lagged that of consonants. Speech synthesis techniques were then developed that established that the primary acoustic features of vowels are formant frequency, fundamental frequency, speech dynamics and naturalness. For speech perception research, precise control of the features in vowel stimuli was required. Starting in the 1980s, the Klatt formant synthesizer was the first important tool, and by 2000 Kawahara’s STRAIGHT synthesizer generated nearly natural speech. As a baseline for vowel perception, my research sought to determine psychophysical thresholds for F1 and F2 under ideal conditions. This talk reports contributions from my vowel perception studies on three questions. First, how do formant thresholds change with speech context (isolated vowels up to sentences), across age and with hearing impairment? Second, do vowels or consonants contribute more to intelligibility in noise-interrupted sentences? Third, given formant dynamics, how is intelligibility affected as consonant-vowel boundary conditions are manipulated across age and hearing impairment? Significant results include: (1) Vowels carry more information about sentence intelligibility than consonants for both young and older listeners; (2) Even though older listeners’ performance is reduced compared to young, hearing impairment has a greater negative impact than age-related cognitive decline.

Chinese personalised text‐to‐speech synthesis for robot human–machine interaction

AbstractSpeech interaction is an important means of robot interaction. With the rapid development of deep learning, end‐to‐end speech synthesis methods based on this technique have gradually become mainstream. Chinese deep learning‐based speech synthesis techniques suffer from problems such as unstable synthesised speech, poor naturalness and poor personalised speech synthesis, which do not satisfy some practical application scenarios. Hence, an F‐MelGAN model is adopted to improve the performance of Chinese speech synthesis. A post‐processing network is used to refine the Mel‐spectrum predicted by the decoder and alleviate the Mel‐spectrum distortion phenomenon. A phoneme‐level and sentence‐level combined module is proposed to model the personalised style of speakers. A combination of an acoustic conditioning network, speaker encoder network GCNet and feedback‐constrained training is proposed to solve the problem of poor personalised speech synthesis and achieve personalised speech customisation in Chinese. Experimental results show that the whole model can generate high‐quality speech with high speaker similarity for both speakers that appear in the training process and speakers that never appear in the training process.

Deepfake Speech Recognition and Detection

Deepfake technology, especially deep voice, which has been derived from artificial intelligence in recent years, is potentially harmful, and the public is not yet wary. However, many speech synthesis models measure the degree of true restitution by Mean Opinion Rating (MOS), a subjective assessment of naturalness and quality of speech by human subjects, but in future it will be difficult to distinguish the interlocutor’s identity through the screen. For this reason, this study addresses the threat posed by this new technology by combining representational learning and 0transfer learning in two sub-systems: a recognition system and a voice print system. The recognition system is responsible for the detection of which voice is a fake voice generated by speech conversion or speech synthesis techniques, while the acoustic system is responsible for the verification of the speaker’s identity through acoustic features. In the speech recognition system, we use the representation learning method and the transfer classification method. We use X-vector data for training, and then fine-tune the model using four types of marker data to learn the representation vectors of real and fake voice, and use support vector machine to classify real and fake voice in the back-end to reduce the negative effect of the new technique.

ADAPT-TTS: HIGH-QUALITY ZERO-SHOT MULTI-SPEAKER TEXT-TO-SPEECH ADAPTIVE-BASED FOR VIETNAMESE

Current adaptive-based speech synthesis techniques are based on two main streams: 1. Fine-tuning the model using small amounts of adaptive data, and 2. Conditionally training the entire model through a speaker embedding of the target speaker. However, both of these methods require adaptive data to appear during training, which makes the training cost to generate new voices quite expensively. In addition, the traditional TTS model uses a simple loss function to reproduce the acoustic features. However, this optimization is based on incorrect distribution assumptions leading to noisy composite audio results. We introduce the Adapt-TTS model that allows high-quality audio synthesis from a small adaptive sample without training to solve these problems. Key recommendations: 1. The Extracting Mel-vector (EMV) architecture allows for a better representation of speaker characteristics and speech style; 2. An improved zero-shot model with a denoising diffusion model (Mel-spectrogram denoiser) component allows for new voice synthesis without training with better quality (less noise). The evaluation results have proven the model's effectiveness when only needing a single utterance (1-3 seconds) of the reference speaker, the synthesis system gave high-quality synthesis results and achieved high similarity.

A real-time voice cloning system with multiple algorithms for speech quality improvement.

With the development of computer technology, speech synthesis techniques are becoming increasingly sophisticated. Speech cloning can be performed as a subtask of speech synthesis technology by using deep learning techniques to extract acoustic information from human voices and combine it with text to output a natural human voice. However, traditional speech cloning technology still has certain limitations; excessively large text inputs cannot be adequately processed, and the synthesized audio may include noise artifacts like breaks and unclear phrases. In this study, we add a text determination module to a synthesizer module to process words the model has not included. The original model uses fuzzy pronunciation for such words, which is not only meaningless but also affects the entire sentence. Thus, we improve the model by splitting the letters and pronouncing them separately. Finally, we also improved the preprocessing and waveform conversion modules of the synthesizer. We replace the pre-net module of the synthesizer and use an upgraded noise reduction algorithm combined with the SV2TTS framework to achieve a system with superior speech synthesis performance. Here, we focus on improving the performance of the synthesizer module to achieve higher-quality speech synthesis audio output.

Modelling of Speech Parameters of Punjabi by Pre-trained Deep Neural Network Using Stacked Denoising Autoencoders

Statistical parametric speech synthesis techniques such as deep neural network (DNN) and hidden Markov model (HMM) have grown in popularity since last decade over the concatenative speech synthesis approaches by modelling excitation and spectral parameters of speech to synthesize the waveforms from the written text. Due to inappropriate acoustic modelling, speech synthesized using HMM-based speech synthesis sounds muffled. DNN tried to improve the acoustic model by replacing decision trees in HMM with powerful regression model. Further, the performance of a deep neural network is greatly enhanced using pre-learning either restricted Boltzmann machines (RBM) or autoencoders. RBMs are capable to map multi-modal property of speech but result in spectral distortion of synthesized speech waveforms as non-consideration of reconstruction error. This article proposed the model of deep neural network, which is pre-trained using stacked denoising autoencoders to map speech parameters of the Punjabi language. Denoising autoencoders work by adding noise in the training data and then reconstructing the original measurements to reduce the reconstruction error. The synthesized voice using the proposed model showed the VARN of 0.82, F0 RMSE (Hz) 9.03, and V/UV error rate of 4.04% have been observed.

Advancements in Arabic Text-to-Speech Systems: A 22-Year Literature Review

Although there are several speech synthesis models available for different languages tailored to specific domain requirements and applications, there is currently no readily available information on the latest trends in Arabic language speech synthesis. This can make it challenging for beginners to research and develop text-to-speech (TTS) systems for Arabic languages. To address this issue, this article provides a comprehensive overview of several scholars’ contributions to the field of Arabic TTS, along with an examination of the unique features of the Arabic language and the corresponding challenges in creating TTS systems. Reporting only on papers discussing Arabic TTS, this systematic review evaluated the available literature published between 2000 and 2022. We conducted a systematic review in six databases using preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines to identify studies that addressed Arabic Text-to-Speech systems. Of a total of 3719 articles identified, only 36 (0.96%) articles met our search criteria. Bibliometric analyses of these studies were conducted and reported. The results highlight the main types of speech synthesis techniques used in TTS systems: concatenative, formant, deep neural network (DNN), hybrid models, and multiagent. The corpora used to develop these systems, as well as the diacritization techniques incorporated, evaluation techniques, and the results of the performance of the systems are reported. Subjective evaluation using the mean opinion score is most applied to measure the accuracy of systems. This study also identifies gaps in the literature and makes recommendations for future research directions.

Data augmentation based non-parallel voice conversion with frame-level speaker disentangler

Non-parallel data voice conversion is a popular and challenging research area. The main task is to build acoustic mappings from the source speaker to the target speaker in different units (e.g., frame, phoneme, cluster, sentence). With the help of the recent high-quality speech synthesis techniques, it is possible to directly produce parallel speech using non-parallel data. This paper proposes ParaGen: a data augmentation based technique for non-parallel data voice conversion. The system consists of a speaker disentangler based text-to-speech model and a simple frame-to-frame spectrogram conversion model. The text-to-speech model takes the text and reference audio as input to produce the speech in the target speaker identity with the time-aligned local speaking style from the reference audio. The spectrogram conversion model directly converts the source spectrogram to the target speaker framewisely. The local speaking style is extracted by an acoustic encoder while the speaker identity is eliminated by a conditional convolutional disentangler. The local style encodings are time-aligned with the text encodings by the attention mechanism. The attention contexts are decoded by a conditional recurrent decoder. The experiment shows that the speaker identity of the source speech is converted to the target speaker while the local speaking style (e.g., prosody) is preserved after the augmentation. The method is compared to the augmentation model with typical statistical parameter speech synthesis (SPSS) with pre-aligned phoneme duration. The result shows that the converted speech has better speech naturalness than the SPSS system, while the speaker similarities of the converted speech are close.

An innovative protocol for the artificial speech-directed, contactless administration of laboratory-based comprehensive cognitive assessments: PAAD-2 trial management during the COVID-19 pandemic

The COVID-19 pandemic resulted in suspending in-person human subject research across most institutions in the US. Our extensive cognitive assessment for a phase-2 clinical trial, Physical Activity and Alzheimer's Disease–2 (PAAD-2), was also paused in March 2020. It was important to identify strategies to mitigate the risk of COVID-19 transmission during our testing, which initially required substantial human speech and close person-to-person contact for test directions and instant feedback on paper/pencil tests. Given current understanding of the COVID-19 transmission, we dramatically adjusted the testing protocol to minimize the production of speech droplets and allow social distancing while maintaining the integrity of testing. We adopted state-of-the-art speech synthesis and computerization techniques to create an avatar to speak on behalf of the experimenter for all verbal instructions/feedback, used a document camera to observe the paper/pencil tests from the required distances, and automated the testing sequence and timing. This paper aims 1) to describe an innovative laboratory-based cognitive testing protocol for a completely contact-free, computer-speaking, and semi-automated administration; and 2) to evaluate the integrity of the modified protocol (n = 37) compared with the original protocol (n = 32). We have successfully operated the modified protocol since July 2020 with no evidence of COVID-19 transmission during testing, and data support that the modified protocol is robust and captures data identical to the original protocol. This transition of data collection methods has been critical during the pandemic and will be useful in future studies to mitigate the risk of contagious disease transmission and standardize laboratory-based psychological tests.Trial registration: ClinicalTrials.gov NCT03876314. Registered March 15, 2019

Robustness of Speech Spoofing Detectors Against Adversarial Post-Processing of Voice Conversion

With the development of speech synthesis and voice conversion techniques, the quality of artificially generated speech has been significantly improved and detecting such spoofing speech becomes crucial to practical applications, such as automatic speaker verification (ASV). State-of-the-art neural-network-based spoofing detection models can distinguish most artificial utterances from natural ones effectively in the latest ASVspoof 2019 evaluation. Motivated by recent progresses of adversarial example generation, this paper studies the robustness of neural-network-based speech spoofing detectors against adversarial attacks. To this end, an adversarial post-processing network (APN) is proposed which generates adversarial examples against a white-box anti-spoofing model by post-processing the speech waveforms produced by a baseline voice conversion system. Experimental results demonstrate the adversarial ability of our proposed APNs against the white-box anti-spoofing models which were used as the adversarial targets of APNs at the training stage. For example, the equal error rate (EER) of a fused detection model based on light convolution neural networks (LCNNs) increased from 0.278% to 12.743% under the white-box condition without degrading the subjective quality of converted speech. Furthermore, the trained APNs can also perform against the detectors with either unseen structures or unseen features by raising their EERs in our experiments. All these results indicate the threat of adversarial speech generation to the performance of state-of-the-art spoofing detection models.

A Multimodal User Interface for an Assistive Robotic Shopping Cart

This paper presents the research and development of the prototype of the assistive mobile information robot (AMIR). The main features of the presented prototype are voice and gesture-based interfaces with Russian speech and sign language recognition and synthesis techniques and a high degree of robot autonomy. AMIR prototype’s aim is to be used as a robotic cart for shopping in grocery stores and/or supermarkets. Among the main topics covered in this paper are the presentation of the interface (three modalities), the single-handed gesture recognition system (based on a collected database of Russian sign language elements), as well as the technical description of the robotic platform (architecture, navigation algorithm). The use of multimodal interfaces, namely the speech and gesture modalities, make human-robot interaction natural and intuitive, as well as sign language recognition allows hearing-impaired people to use this robotic cart. AMIR prototype has promising perspectives for real usage in supermarkets, both due to its assistive capabilities and its multimodal user interface.

Syllable based approach for text to speech synthesis of Assamese language: A review

In this review article authors are trying to put light on text to speech synthesis of Assamese language using unit selection concatenative speech synthesis technique. Assamese is one of the North East Indian languages spoken by millions of people. This article tries to highlight some major difficulties when developing the synthesizer. The speech units used to concatenate for building the synthesizer is syllable. Assamese is a syllable centric language and syllable based concatenation gives more natural sound. A discussion is done with phoneme and diphone units as well. There is also a short overview on development process of the Assamese synthesizer on festival framework as a part of review. Another challenging task the researchers dealt with is building speech corpus for a low resource language like Assamese.

Gated Recurrent Attention for Multi-Style Speech Synthesis

End-to-end neural network-based speech synthesis techniques have been developed to represent and synthesize speech in various prosodic style. Although the end-to-end techniques enable the transfer of a style with a single vector of style representation, it has been reported that the speaker similarity observed from the speech synthesized with unseen speaker-style is low. One of the reasons for this problem is that the attention mechanism in the end-to-end model is overfitted to the training data. To learn and synthesize voices of various styles, an attention mechanism that can preserve longer-term context and control the context is required. In this paper, we propose a novel attention model which employs gates to control the recurrences in the attention. To verify the proposed attention’s style modeling capability, perceptual listening tests were conducted. The experiments show that the proposed attention outperforms the location-sensitive attention in both similarity and naturalness.

A survey on speech synthesis techniques in Indian languages

The text to speech technology has achieved significant progress during the past decade and is an active area of research and development in providing different human–computer interactive systems. Even though a number of speech synthesis models are available for different languages focusing on the domain requirements with many motive applications, a source of information on current trends in Indian language speech synthesis is unavailable till date making it difficult for the beginners to initiate research for the development of TTS systems for the low-resourced languages. This paper provides a review of the contributions made by different researchers in the field of Indian language speech synthesis along with a study on the Indian language characteristics and the associated challenges in designing TTS systems. A set of available applications and tools results out of different projects undertaken by different organizations along with a set of possible future developments are also discussed to provide a single reference to an important strand of research in speech synthesis which may benefit anyone interested to initiate research in this area.

Linguistic-Based and user-Based Recommending Posts using Two-Level Clustering Methods

Online social networks have produced bunches of online social groups where individuals can collaborate and shuffle their thoughts. In spite of, the real problems that conflicts with the user security and convenience are confidentiality break, groups without inception, confusion created from various groups of which user is a member of and difficulty in moderating groups. This can be moderated to an extent by an automated filtering method required to categorizing group members based on their response patterns. This paper proposes clustering of group posts on stylistic, thematic, emotional, sentimental and psycholinguistic methods and members of the group are categorized based on their responses to the posts belonging to different clustering methods. The categorization affords security just like the conflict associated with irrelevant notifications received from more than one groups, via recommending the users, posts which might be probable to be of interest to them. It also helps to identify the group members meant closer to spreading posts that violate group policies. The categorization post shows increased performance where there are large numbers of members in a social group by performing linguistic clustering. The contribution work is to implement location-aware personalized posts recommendation using users’ behavioral patterns and their geographic location. Another, important work is to implement text-to-speech system converting English text into speech using speech synthesis technique. The system gives rating to the users who shares posts depending on clustering. Also system provides read later functionality to the user side. The system has been tested on Twitter API group data where a significant solution to an unaddressed problem associated with social networking groups is offered.

Text to Speech Synthesis System for Punjabi language using Statistical Parametric Speech Synthesis Technique

Statistical Parametric Speech Synthesis has been most growing technique rather than the traditional approaches that we are used to synthesizing the speech. The shortcoming of traditional approaches will be overcome with latest statistical techniques. The main advantages of SPSS from traditional synthesis technique are that it has more flexibility to change the characteristics of voice and support more multiple languages i.e. multilingual, has good coverage of acoustic ` and robustness. It generates high quality of speech from small training database. Deep Neural network and Hidden Morkov model are basic statistical parametric speech synthesis techniques. Gaussian mixture model, sinusoidal model are also under this categories. Features were extracted in two type spectral features like spectral bandwidth, spectral centroid etc. and excitation features like F0 frequencies etc. We are using 722 Punjabi phonemes. Using sound forge software we extracted the 200 wave file from 1 hour pre-recording wave file related to those phonemes. Each and every phonemes feature was extracted and saved in database. We were extracting 28 features of each phoneme. TTS text-to-speech system generates sounds or speech as a output when provided the text of Punjabi language. There were already many TTS are developed on different Indian languages. The system that we are trying to build is based only on Punjabi language.

The effect of speech rhythm and speaking rate on assessment of pronunciation in a second language

AbstractThe study explores the effect of deviations from native speech rhythm and rate norms on the assessement of pronunciation mastery of a second language (L2) when the native language of the learner is either rhythmically similar to or different from the target language. Using the concatenative speech synthesis technique, different versions of the same sentence were created in order to produce segmentally and intonationally identical utterances that differed only in rhythmic patterns and/or speaking rate. Speech rhythm and tempo patterns modeled those from the speech of French or German native learners of English at different proficiency levels. Native British English speakers rated the original sentences and the synthesized utterances for accentedness. The analysis shows that (a) differences in speech rhythm and speaking tempo influence the perception of accentedness; (b) idiosyncratic differences in speech rhythm and speech rate are sufficient to differentiate between the proficiency levels of L2 learners; (c) the relative salience of rhythm and rate on perceived accentedness in L2 speech is modulated by the native language of the learners; and (d) intonation facilitates the perception of finer differences in speech rhythm between otherwise identical utterances. These results emphasize the importance of prosodic timing patterns for the perception of speech delivered by L2 learners.

Es-Tacotron2: Multi-Task Tacotron 2 with Pre-Trained Estimated Network for Reducing the Over-Smoothness Problem

Text-to-speech synthesis is a computational technique for producing synthetic, human-like speech by a computer. In recent years, speech synthesis techniques have developed, and have been employed in many applications, such as automatic translation applications and car navigation systems. End-to-end text-to-speech synthesis has gained considerable research interest, because compared to traditional models the end-to-end model is easier to design and more robust. Tacotron 2 is an integrated state-of-the-art end-to-end speech synthesis system that can directly predict closed-to-natural human speech from raw text. However, there remains a gap between synthesized speech and natural speech. Suffering from an over-smoothness problem, Tacotron 2 produced ’averaged’ speech, making the synthesized speech sounds unnatural and inflexible. In this work, we first propose an estimated network (Es-Network), which captures general features from a raw mel spectrogram in an unsupervised manner. Then, we design Es-Tacotron2 by employing the Es-Network to calculate the estimated mel spectrogram residual, and setting it as an additional prediction task of Tacotron 2, to allow the model focus more on predicting the individual features of mel spectrogram. The experience shows that compared to the original Tacotron 2 model, Es-Tacotron2 can produce more variable decoder output and synthesize more natural and expressive speech.

HMM speech synthesis based on MDCT representation

Hidden Markov model (HMM) based text-to-speech (TTS) has become one of the most promising approaches, as it has proven to be a particularly flexible and robust framework to generate synthetic speech. However, several factors such as mel-cepstral vocoder and over-smoothing are responsible for causing quality degradation of synthetic speech. This paper presents an HMM speech synthesis technique based on the modified discrete cosine transform (MDCT) representation to cope with these two issues. To this end, we use an analysis/synthesis technique based on MDCT that guarantees a perfect reconstruction of the signal frame from feature vectors and allows for a 50% overlap between frames without increasing the data vector, in contrast to the conventional mel-cepstral spectral parameters that do not ensure direct speech waveform reconstruction. Experimental results show that a sound of good quality, conveniently evaluated using both objective and subjective tests, is obtained.

A Novel Data Independent Approach for Conversion of Hand Punched Kannada Braille Script to Text and Speech

In society, there exists a gap in communication between the sighted community and the visually challenged people due to different scripts followed to read and write. To bridge this gap there is a need for a system that supports automatic conversion of Braille script to text and speech in the corresponding language. Optical Braille Recognition (OBR) system converts the hand-punched Braille characters into their equivalent natural language characters. The Text-to-Speech (TTS) system converts the recognized characters into audible speech using speech synthesis techniques. Existing literature reveals that OBR and TTS systems have been well established independently for English. There is a scope for development of OBR and TTS systems for regional languages. In spite of Kannada being one of the most widely spoken regional languages in India, minimal work has been done towards Kannada OBR and TTS. There is no system that directly converts Braille script to speech, therefore, this development of Kannada Braille to text and speech system is one of a kind. The acquired image is processed and feature extraction is performed using [Formula: see text]-means algorithm and heuristics to convert the Braille characters to Kannada script. The concatenation based speech synthesis technique employing phoneme as the basic unit is used to convert Kannada TTS using Festival TTS framework. Performance evaluation of the proposed system is done using Kannada Braille database developed independently, and the results obtained are found to be satisfactory when compared to existing methods in the literature.