Speech Synthesis Research Articles

Audio Anti-Spoofing Based on Audio Feature Fusion

The rapid development of speech synthesis technology has significantly improved the naturalness and human-likeness of synthetic speech. As the technical barriers for speech synthesis are rapidly lowering, the number of illegal activities such as fraud and extortion is increasing, posing a significant threat to authentication systems, such as automatic speaker verification. This paper proposes an end-to-end speech synthesis detection model based on audio feature fusion in response to the constantly evolving synthesis techniques and to improve the accuracy of detecting synthetic speech. The model uses a pre-trained wav2vec2 model to extract features from raw waveforms and utilizes an audio feature fusion module for back-end classification. The audio feature fusion module aims to improve the model accuracy by adequately utilizing the audio features extracted from the front end and fusing the information from timeframes and feature dimensions. Data augmentation techniques are also used to enhance the performance generalization of the model. The model is trained on the training and development sets of the logical access (LA) dataset of the ASVspoof 2019 Challenge, an international standard, and is tested on the logical access (LA) and deep-fake (DF) evaluation datasets of the ASVspoof 2021 Challenge. The equal error rate (EER) on ASVspoof 2021 LA and ASVspoof 2021 DF are 1.18% and 2.62%, respectively, achieving the best results on the DF dataset.

SeDepTTS: Enhancing the Naturalness via Semantic Dependency and Local Convolution for Text-to-Speech Synthesis

Self-attention-based networks have obtained impressive performance in parallel training and global context modeling. However, it is weak in local dependency capturing, especially for data with strong local correlations such as utterances. Therefore, we will mine linguistic information of the original text based on a semantic dependency and the semantic relationship between nodes is regarded as prior knowledge to revise the distribution of self-attention. On the other hand, given the strong correlation between input characters, we introduce a one-dimensional (1-D) convolution neural network (CNN) producing query(Q) and value(V) in the self-attention mechanism for a better fusion of local contextual information. Then, we migrate this variant of the self-attention networks to speech synthesis tasks and propose a non-autoregressive (NAR) neural Text-to-Speech (TTS): SeDepTTS. Experimental results show that our model yields good performance in speech synthesis. Specifically, the proposed method yields significant improvement for the processing of pause, stress, and intonation in speech.

VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing

Video dubbing aims to translate the original speech in a film or television program into the speech in a target language, which can be achieved with a cascaded system consisting of speech recognition, machine translation and speech synthesis. To ensure the translated speech to be well aligned with the corresponding video, the length/duration of the translated speech should be as close as possible to that of the original speech, which requires strict length control. Previous works usually control the number of words or characters generated by the machine translation model to be similar to the source sentence, without considering the isochronicity of speech as the speech duration of words/characters in different languages varies. In this paper, we propose VideoDubber, a machine translation system tailored for the task of video dubbing, which directly considers the speech duration of each token in translation, to match the length of source and target speech. Specifically, we control the speech length of generated sentence by guiding the prediction of each word with the duration information, including the speech duration of itself as well as how much duration is left for the remaining words. We design experiments on four language directions (German -> English, Spanish -> English, Chinese English), and the results show that VideoDubber achieves better length control ability on the generated speech than baseline methods. To make up the lack of real-world datasets, we also construct a real-world test set collected from films to provide comprehensive evaluations on the video dubbing task.

Avocodo: Generative Adversarial Network for Artifact-Free Vocoder

Neural vocoders based on the generative adversarial neural network (GAN) have been widely used due to their fast inference speed and lightweight networks while generating high-quality speech waveforms. Since the perceptually important speech components are primarily concentrated in the low-frequency bands, most GAN-based vocoders perform multi-scale analysis that evaluates downsampled speech waveforms. This multi-scale analysis helps the generator improve speech intelligibility. However, in preliminary experiments, we discovered that the multi-scale analysis which focuses on the low-frequency bands causes unintended artifacts, e.g., aliasing and imaging artifacts, which degrade the synthesized speech waveform quality. Therefore, in this paper, we investigate the relationship between these artifacts and GAN-based vocoders and propose a GAN-based vocoder, called Avocodo, that allows the synthesis of high-fidelity speech with reduced artifacts. We introduce two kinds of discriminators to evaluate speech waveforms in various perspectives: a collaborative multi-band discriminator and a sub-band discriminator. We also utilize a pseudo quadrature mirror filter bank to obtain downsampled multi-band speech waveforms while avoiding aliasing. According to experimental results, Avocodo outperforms baseline GAN-based vocoders, both objectively and subjectively, while reproducing speech with fewer artifacts.

Towards Voice Reconstruction from EEG during Imagined Speech

Translating imagined speech from human brain activity into voice is a challenging and absorbing research issue that can provide new means of human communication via brain signals. Efforts to reconstruct speech from brain activity have shown their potential using invasive measures of spoken speech data, but have faced challenges in reconstructing imagined speech. In this paper, we propose NeuroTalk, which converts non-invasive brain signals of imagined speech into the user's own voice. Our model was trained with spoken speech EEG which was generalized to adapt to the domain of imagined speech, thus allowing natural correspondence between the imagined speech and the voice as a ground truth. In our framework, an automatic speech recognition decoder contributed to decomposing the phonemes of the generated speech, demonstrating the potential of voice reconstruction from unseen words. Our results imply the potential of speech synthesis from human EEG signals, not only from spoken speech but also from the brain signals of imagined speech.

What Does Your Face Sound Like? 3D Face Shape towards Voice

Face-based speech synthesis provides a practical solution to generate voices from human faces. However, directly using 2D face images leads to the problems of uninterpretability and entanglement. In this paper, to address the issues, we introduce 3D face shape which (1) has an anatomical relationship between voice characteristics, partaking in the "bone conduction" of human timbre production, and (2) is naturally independent of irrelevant factors by excluding the blending process. We devise a three-stage framework to generate speech from 3D face shapes. Fully considering timbre production in anatomical and acquired terms, our framework incorporates three additional relevant attributes including face texture, facial features, and demographics. Experiments and subjective tests demonstrate our method can generate utterances matching faces well, with good audio quality and voice diversity. We also explore and visualize how the voice changes with the face. Case studies show that our method upgrades the face-voice inference to personalized custom-made voice creating, revealing a promising prospect in virtual human and dubbing applications.

A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech

Recent Text-to-Speech (TTS) systems trained on reading or acted corpora have achieved near human-level naturalness. The diversity of human speech, however, often goes beyond the coverage of these corpora. We believe the ability to handle such diversity is crucial for AI systems to achieve human-level communication. Our work explores the use of more abundant real-world data for building speech synthesizers. We train TTS systems using real-world speech from YouTube and podcasts. We observe the mismatch between training and inference alignments in mel-spectrogram based autoregressive models, leading to unintelligible synthesis, and demonstrate that learned discrete codes within multiple code groups effectively resolves this issue. We introduce our MQTTS system whose architecture is designed for multiple code generation and monotonic alignment, along with the use of a clean silence prompt to improve synthesis quality. We conduct ablation analyses to identify the efficacy of our methods. We show that MQTTS outperforms existing TTS systems in several objective and subjective measures.

Computational Approaches to Phonology: Advances in Speech Recognition and Synthesis

This article explores the field of computational approaches to phonology, with a particular focus on recent increases in popularity and speech synthesis. Phonology, as a department of linguistics, deals with the observation of sounds and their work in language. Recent improvements in this area have resulted in major improvements in accuracy and efficiency, towards the integration of deep recognition strategies together with recurrent neural networks (RNNs) and convolutional neural networks (CNNs). the models have tested the ability to capture complex phonetic patterns and decorate the overall performance of computerized speech reputation structures. These advances have seen packages in multiple domains, such as virtual assistants, text-to-speech structures, and language acquisition tools. This article looks at the main computational procedures used in phonology, along with characteristic-based representations, rule-based modes, and statistical methods. It explores demanding situations related to phonological evaluation, which include phoneme segmentation, pronunciation variants, and language-specific problems. In addition, this paper discusses mixing phonological knowledge with device learning strategies, which enables the development of robust and correct speech recognition and synthesis systems. As usual, this research represents remarkable advances from computational methods to phonology, particularly in the geographical area of speech recognition and synthesis. the combination of technology and linguistic insights has increased the accuracy, naturalness, and efficiency in processing spoken language. These enhancements open up new opportunities to enhance human-pc interaction, language recognition, and many other packages that rely on speech-based technologies.

ADAPT-TTS: HIGH-QUALITY ZERO-SHOT MULTI-SPEAKER TEXT-TO-SPEECH ADAPTIVE-BASED FOR VIETNAMESE

Current adaptive-based speech synthesis techniques are based on two main streams: 1. Fine-tuning the model using small amounts of adaptive data, and 2. Conditionally training the entire model through a speaker embedding of the target speaker. However, both of these methods require adaptive data to appear during training, which makes the training cost to generate new voices quite expensively. In addition, the traditional TTS model uses a simple loss function to reproduce the acoustic features. However, this optimization is based on incorrect distribution assumptions leading to noisy composite audio results. We introduce the Adapt-TTS model that allows high-quality audio synthesis from a small adaptive sample without training to solve these problems. Key recommendations: 1. The Extracting Mel-vector (EMV) architecture allows for a better representation of speaker characteristics and speech style; 2. An improved zero-shot model with a denoising diffusion model (Mel-spectrogram denoiser) component allows for new voice synthesis without training with better quality (less noise). The evaluation results have proven the model's effectiveness when only needing a single utterance (1-3 seconds) of the reference speaker, the synthesis system gave high-quality synthesis results and achieved high similarity.

Speech Synthesis in the “Mother Tongue”: Designing, Training, and Evaluating a Text-to-Speech System for Yiddish

Abstract Although few linguistic corpora are available in Yiddish, there are numerous sources of so-called “found data” that can be adapted for language research, pedagogy, and resource development. We describe the steps taken to create the first speech synthesis (text-to-speech) program in Yiddish. A state-of-the-art TTS model, FastSpeech 2, was trained on a hand-corrected data set consisting of literary texts paired with audio narrations by native speakers of the Polish and Lithuanian dialects. A quantitative evaluation by listeners found that the system produced speech that was both intelligible and natural-sounding. To demonstrate the system’s applications for language pedagogy, we offer a qualitative evaluation of Yiddish phonological features that are present or absent in a sample of synthesized recordings. We hope that the success of speech synthesis in Yiddish will inspire future projects to enable technological support for other minority languages in which transcribed recordings are available.

LiVoAuth: Liveness Detection in Voiceprint Authentication With Random Challenges and Detection Modes

Voiceprint authentication provides great con- venience to users in many application scenarios. However, it easily suffers from spoofing attacks including speech synthesis, speech conversion and speech replay. Liveness detection is an effective way to resist these attacks. But existing methods suffer from many disadvantages, such as extra deployment costs due to precise data collection, environmental disturbance, high computational overhead and operational complexity. A uniform platform that can offer Voiceprint Authentication as a Service (VAaS) over the cloud is also lacked. Hence, it is imperative to design an economic and effective method for liveness detection in voiceprint authentication. In this paper, we propose a novel liveness detection method named LiVoAuth for voiceprint authentication. It applies a randomly generated vector se- quence as Liveness Detection Mode (LDM), corresponding to a random challenge code used for authentication. We implement LiVoAuth and conduct a series of user studies to evaluate its performance in terms of accuracy, stability, efficiency, security, and user acceptance. Experimental re- sults demonstrate its advantages compared with cutting- edge methods.

Real-Time Driver Drowsiness Detection Using Images

Abstract: The suggested technology intends to reduce the amount of accidents caused by driver sleepiness and exhaustion, hence increasing transportation safety. In recent years, this has been a prevalent cause of accidents. Several expressions and body motions, including exhaustion in the eyes and yawning, are seen as symptoms of sleepiness and fatigue in drivers. These characteristics indicate that the driver's condition is poor. The EAR (Eye Aspect Ratio) calculates the distance ratio between horizontal and vertical eye landmarks, which is essential for detecting sleepiness. A YAWN value is calculated for yawn detection utilising the distance between the lower and upper lips, and the distance is compared to a threshold value. We installed an eSpeak module (text to speech synthesiser) to provide suitable voice notifications when the driver becomes tired or yawns. The suggested method is intended to reduce the number of accidents and to contribute to technology in order to avoid fatalities caused by road accidents.

Analysis of the survey of voice synthesis technology

The original purpose of speech synthesis was to complete the task of converting from text to speech (TTS). With the application of deep learning models in this field of speech synthesis, the results of speech synthesis have gradually reached the level of human voice, which makes it widely used for voice assistant, navigation, reading, intelligent customer service, and many other aspects. In order to help readers expand more research ideas and understand the development process of speech synthesis technology and the ethical choices faced, this article introduces as much as possible in the development process of speech synthesis technology, several frameworks for optimization, as well as interpreting their respective advantages and disadvantages, summarize their approximate results obtained in the MOS test method and analyze the ethical issues that may be brought about by voice cloning technology.

1991 to 2019: The rise of machine interpreting research

Conceived as a scientometric study, this paper searches for understanding the research status of machine interpreting on the IEEE (Institute of Electrical and Electronics Engineers) database from 1991 to 2019. Documents were analyzed considering a series of measures such as most prominent academic institutions and countries that investigate machine interpreting, citation, co-authorship, keywords co-occurrence, reference coupling, and textual-based analysis retrieved from the documents’ titles and abstracts. Through VOSviewer software and its tools for data collecting and visualization, machine interpreting research in the analyzed corpus focuses on three main concerns: machine translation, speech synthesis, and Japanese language.

Semi-Supervised Learning for Robust Emotional Speech Synthesis with Limited Data

Emotional speech synthesis is an important branch of human–computer interaction technology that aims to generate emotionally expressive and comprehensible speech based on the input text. With the rapid development of speech synthesis technology based on deep learning, the research of affective speech synthesis has gradually attracted the attention of scholars. However, due to the lack of quality emotional speech synthesis corpus, emotional speech synthesis research under low-resource conditions is prone to overfitting, exposure error, catastrophic forgetting and other problems leading to unsatisfactory generated speech results. In this paper, we proposed an emotional speech synthesis method that integrates migration learning, semi-supervised training and robust attention mechanism to achieve better adaptation to the emotional style of the speech data during fine-tuning. By adopting an appropriate fine-tuning strategy, trade-off parameter configuration and pseudo-labels in the form of loss functions, we efficiently guided the learning of the regularized synthesis of emotional speech. The proposed SMAL-ET2 method outperforms the baseline methods in both subjective and objective evaluations. It is demonstrated that our training strategy with stepwise monotonic attention and semi-supervised loss method can alleviate the overfitting phenomenon and improve the generalization ability of the text-to-speech model. Our method can also enable the model to successfully synthesize different categories of emotional speech with better naturalness and emotion similarity.

Research on a Guide Dog Robot for Expressing Visual Environment by Voice

This paper proposed a guide dog robot that can express visual environment by voice. The guide dog robot uses NIC, a combination of CNN and LSTM methods, to recognize the visual scene and generate captions. The generated sentences are then converted to speech using speech synthesis to guide the visually impaired. The guide dog robot can not only guide the visually impaired person safely to their destination, but also providing explanation services of street scenes, enabling the visually impaired person to enjoy visual scenes. The system is consisted of visual scene recognition, caption, and speech synthesis. The effectiveness of the system is confirmed through experiments using the proposed guide dog robot.

A Self-Attentional ResNet-LightGBM Model for IoT-Enabled Voice Liveness Detection

Voice user interface (VUI) brings high efficiency and convenience for the applications of Internet of Things (IoTs), meanwhile, it can also cause increasingly serious security issues. The word-level voice liveness detection is proved to be the promising solution to thwart spoofing attacks. However, The complex acoustic feature, diversified attacks, and different interaction distance can severely affects the improvement of detection accuracy. To alleviate this issue, we develop a novel pop noise based word-level voice liveness detection framework. Firstly, a new voice frame selection method is proposed for determining optimal frames, including short time Fourier transform, low frequency average energy computation and sequencing. Then, the acoustic features of the selected frames are calculated by Gammatone frequency cepstral coefficient (GFCC). Finally, based on these features, a newly-built joint voice detector, fusing the self-attentional residual network (ResNet) and light gradient boosting machine (LightGBM), can achieve accurate voice classification. On the popular voice spoofing attack datasets, experimental results show that our proposal significantly outperforming the baseline and the state of the arts models, and it is gender dependent. Moreover, our proposal has good generalization ability for far-field replay voice attack, speech synthesis and voice conversion attacks, and partial fake voice attack. Finally, its effectiveness is verified by the ablation study.

Differentiation of the Functional Systems of Speech and Language and the Study of the Differences of the Neural Networks That Support Them

The paper deals with the problem of differentiation of human speech and language systems. Based on the modern ideas in the field of speech psychology, speech study, intonology, the concept of units (forms) of speech as non-linguistic phenomena is formed. These phenomena function as translators of illocutionary meanings in human communication, which mostly are communicative and social goals, as well as the quality of social relations. To support the concept of “Speech vs. Language”, the results of an fMRI study conducted on a sample collection of adults are presented. The differences between the neural networks of the human brain that provide the perception of natural forms of oral speech having the same pragmasemantic content – the actual speech and the minimal structure of speech-language signals are established. Due to the data obtained, the prospects of research related to the analysis of functional connectivity in the structures of two types of networks, as well as with the sequential complication of the speech and language components of the statement and the presentation of multimodal multilevel natural speech-communicative stimuli are discussed. The analysis of the differences between the processes and neural networks of speech and language is important for the development of speech synthesis technologies, diagnostic and communicative artificial intelligence.

The Validation of Automated Social Skills Training in Members of the General Population Over 4 Weeks: Comparative Study.

Social skills training by human trainers is a well-established method of teaching appropriate social and communication skills and strengthening social self-efficacy. Specifically, human social skills training is a fundamental approach to teaching and learning the rules of social interaction. However, it is cost-ineffective and offers low accessibility, since the number of professional trainers is limited. A conversational agent is a system that can communicate with a human being in a natural language. We proposed to overcome the limitations of current social skills training with conversational agents. Our system is capable of speech recognition, response selection, and speech synthesis and can also generate nonverbal behaviors. We developed a system that incorporated automated social skills training that completely adheres to the training model of Bellack et al through a conversational agent. This study aimed to validate the training effect of a conversational agent-based social skills training system in members of the general population during a 4-week training session. We compare 2 groups (with and without training) and hypothesize that the trained group's social skills will improve. Furthermore, this study sought to clarify the effect size for future larger-scale evaluations, including a much larger group of different social pathological phenomena. For the experiment, 26 healthy Japanese participants were separated into 2 groups, where we hypothesized that group 1 (system trained) will make greater improvement than group 2 (nontrained). System training was done as a 4-week intervention where the participants visit the examination room every week. Each training session included social skills training with a conversational agent for 3 basic skills. We evaluated the training effect using questionnaires in pre- and posttraining evaluations. In addition to the questionnaires, we conducted a performance test that required the social cognition and expression of participants in new role-play scenarios. Blind ratings by third-party trainers were made by watching recorded role-play videos. A nonparametric Wilcoxson Rank Sum test was performed for each variable. Improvement between pre- and posttraining evaluations was used to compare the 2 groups. Moreover, we compared the statistical significance from the questionnaires and ratings between the 2 groups. Of the 26 recruited participants, 18 completed this experiment: 9 in group 1 and 9 in group 2. Those in group 1 achieved significant improvement in generalized self-efficacy (P=.02; effect size r=0.53). We also found a significant decrease in state anxiety presence (P=.04; r=0.49), measured by the State-Trait Anxiety Inventory (STAI). For ratings by third-party trainers, speech clarity was significantly strengthened in group 1 (P=.03; r=0.30). Our findings reveal the usefulness of the automated social skills training after a 4-week training period. This study confirms a large effect size between groups on generalized self-efficacy, state anxiety presence, and speech clarity.

ANALYSIS OF ACOUSTIC INFORMATION PROTECTION METHODS IN CRITICAL APPLICATIONS

Subject of the study: the process of acoustic information protection in computer systems of critical applications to ensure the required level of system security. The aim of the article is to analyze the methods of acoustic information protection in computer systems of critical application by means of masking to ensure the impossibility of unauthorized access to the system. The article solves the following tasks: to analyze the software and hardware masking of speech; to study the masking of speech messages in order to introduce unrecognizability; to study the features of speech message compression; to investigate methods of covert transmission of acoustic information. The results of the work, which were obtained using mathematical methods of information transformation in computer systems, are potentially possible methods of masking speech messages to ensure the impossibility of unauthorized access to the system. The analysis of the functioning of the presented methods has led to the following conclusions. One of the perspective directions of acoustic information protection in communication channels and dedicated premises can be considered the creation and development of computerized speech masking systems along with or in conjunction with traditional technologies of semantic protection of acoustic information, namely, speech signal classification based on cryptographic algorithms. The main requirements for today's systems that provide protection of acoustic information in critical computer systems are speed and efficiency of various speech signal processing procedures using standard inexpensive technical means of computer telephony, namely: a personal computer, sound card, telephone line interface device and/or modem. These requirements can be met by applying digital methods of dynamic spectral analysis, i.e. synthesis of speech and audio signals. The choice of specific methods and means of speech masking as one of the types of semantic protection of acoustic information will depend on the practical requirements for the speech protection system and the technical characteristics of the acoustic information transmission channel. Further research is desirable to analyze the possible use of methods for synthesizing large ensembles of quasi-orthogonal discrete signals with improved ensemble, structural and correlation properties to ensure higher security indicators of acoustic channels in computer systems of critical applications.