Voice Conversion Research Articles

A hybrid CNN-LSTM model with adaptive instance normalization for one shot singing voice conversion

<abstract><p>Singing voice conversion methods encounter challenges in achieving a delicate balance between synthesis quality and singer similarity. Traditional voice conversion techniques primarily emphasize singer similarity, often leading to robotic-sounding singing voices. Deep learning-based singing voice conversion techniques, however, focus on disentangling singer-dependent and singer-independent features. While this approach can enhance the quality of synthesized singing voices, many voice conversion systems still grapple with the issue of singer-dependent feature leakage into content embeddings. In the proposed singing voice conversion technique, an encoder decoder framework was implemented using a hybrid model of convolutional neural network (CNN) accompanied by long short term memory (LSTM). This paper investigated the use of activation guidance and adaptive instance normalization techniques for one shot singing voice conversion. The instance normalization (IN) layers within the auto-encoder effectively separated singer and content representations. During conversion, singer representations were transferred using adaptive instance normalization (AdaIN) layers. This singing voice system with the help of activation function prevented the transfer of singer information while conveying the singing content. Additionally, the fusion of LSTM with CNN can enhance voice conversion models by capturing both local and contextual features. The one-shot capability simplified the architecture, utilizing a single encoder and decoder. Impressively, the proposed hybrid CNN-LSTM model achieved remarkable performance without compromising either quality or similarity. The objective and subjective evaluation assessments showed that the proposed hybrid CNN-LSTM model outperformed the baseline architectures. Evaluation results showed a mean opinion score (MOS) of 2.93 for naturalness and 3.35 for melodic similarity. These hybrid CNN-LSTM techniques allowed it to perform high-quality voice conversion with minimal training data, making it a promising solution for various applications.</p></abstract>

Active Defense Against Voice Conversion Through Generative Adversarial Network

Active Defense Against Voice Conversion Through Generative Adversarial Network

Enhancing Cross-Linguistic Image Caption Generation with Indian Multilingual Voice Interfaces using Deep Learning Techniques

The Multilingual Voice-Based Image Caption Generator (MVBICG) is a versatile tool with numerous applications spanning communications, culture preservation, business, and technology, making it indispensable in the interconnected world. The task of image caption generation combines computer vision and NLP (natural language processing) concepts, enabling the system to understand the details or complexities of the image context and describe them in natural language. Image descriptions serve as an invaluable solution for visually impaired individuals. The MVBICG system is designed to provide real-time image descriptions in the form of voice in multiple languages as per user requirements. With the use of an MVBICG, the descriptions can be obtained as a voice output in different languages. Converting a voice into multiple languages with the help of the Google Translate API is often referred to as “multilingual voice conversion” or “multilingual speech synthesis." It leverages the latest advancements in deep learning, particularly convolutional neural networks (CNNs) for image feature extraction and recurrent neural networks (RNNs) with attention mechanisms for natural language generation. In the future, image processing is expected to take center stage as a critical research domain primarily dedicated to the preservation and protection of human lives. The MVBICG demonstrates remarkable performance with BLEU scores of 0.483601 for BLEU-1 and 0.320112 for BLEU-2, indicating its proficiency in generating precise and contextually relevant image captions. These scores further underscore its value in bridging language barriers and enhancing accessibility, highlighting its potential for broader societal impact. Additionally, the system's training progress is illustrated by a loss plot, showing the convergence of the model over time. As image processing continues to advance, the MVBICG emerges as a pivotal research domain, focusing on the preservation and safeguarding of human lives through advanced technologies.

MetricCycleGAN-VC: Forcing CycleGAN-Based Voice Conversion Systems to Associate With Objective Quality Metrics

MetricCycleGAN-VC: Forcing CycleGAN-Based Voice Conversion Systems to Associate With Objective Quality Metrics

A Comparative Evaluation on Data Transformation Approach for Artificial Speech Detection

The rise of voice biometrics has transformed user authentication and offered enhanced security and convenience while phasing out less secure methods. Despite these advancements, Automatic Speaker Verification (ASV) systems remain vulnerable to spoofing, particularly with artificial speech generated swiftly using advanced speech synthesis and voice conversion algorithms. A recent data transformation technique achieved an impressive Equal Error Rate (EER) of 1.42% on the ASVspoof 2019 Logical Access Dataset. While this approach predominantly relies on Support Vector Machine (SVM) as the backend classifier for artificial speech detection, it is vital to explore a broader range of classifiers to enhance resilience. This paper addresses this research gap by systematically assessing classifier efficacy in artificial speech detection. The objectives are twofold: first, to evaluate various classifiers, not limited to SVM, and identify those best suited for artificial speech detection; second, to compare this approach's performance with existing methods. The evaluation demonstrated SVM-Polynomial as the top-performing classifier, surpassing the end-to-end learning approach. This work contributes to a deeper understanding of classifier efficacy and equips researchers and practitioners with a diversified toolkit for building robust ASV spoofing detection systems.

RefXVC: Cross-Lingual Voice Conversion with Enhanced Reference Leveraging

RefXVC: Cross-Lingual Voice Conversion with Enhanced Reference Leveraging

A Pitch-Controlled End-to-End Voice Conversion System for Brazilian Portuguese

A Pitch-Controlled End-to-End Voice Conversion System for Brazilian Portuguese

Fast end-to-end non-parallel voice conversion based on speaker-adaptive neural vocoder with cycle-consistent learning

Fast end-to-end non-parallel voice conversion based on speaker-adaptive neural vocoder with cycle-consistent learning

A novel communication system based on sign language recognition and voice conversion for differently abled person

Sign language is a form of communication that helps deaf and mute people interact with those who can hear. In the country, around 2.78% of people are unable to speak, i.e., are mute or deaf. Their primary means of communication is through hand motions and gestures. The paper proposes a new technique called the "artificial speaking mouth" for individuals with speech impairments. Humans connect with each other through thoughts and ideas, but some people lack the ability to speak. For them, sign language is the only means of communication. Nowadays, technology has reduced this communication gap with systems that translate sign language into speech. Sign Language Recognition (SLR) and gesture-based control are two major applications used for hand gesture recognition technologies. On the other hand, the controller converts sign language into text and speech, using text-to-speech conversion and analog-to-digital conversion.

StreamVoice+: Evolving Into End-to-End Streaming Zero-Shot Voice Conversion

StreamVoice+: Evolving Into End-to-End Streaming Zero-Shot Voice Conversion

Sign Recognition and Voice Conversion Device for Dumb

Sign Recognition and Voice Conversion Device for Dumb

Complementary regional energy features for spoofed speech detection

Automatic speaker verification systems are found to be vulnerable to spoof attacks such as voice conversion, text-to-speech, and replayed speech. As the security of biometric systems is vital, many countermeasures have been developed for spoofed speech detection. To satisfy the recent developments on speech synthesis, publicly available datasets became more and more challenging (e.g., ASVspoof 2019 and 2021 datasets). A variety of replay attack configurations were also considered in those datasets, as they do not require expertise, hence easily performed. This work utilizes regional energy features, which are experimentally proven to be more effective than the traditional frame-based energy features. The proposed energy features are independent from the utterance length and are extracted over nonoverlapping time-frequency regions of the magnitude spectrum. Different configurations are considered in the experiments to verify the regional energy features’ contribution to the performance. First, light convolutional neural network – long short-term memory (LCNN – LSTM) model with linear frequency cepstral coefficients is used to determine the optimal number of regional energy features. Then, SE-Res2Net model with log power spectrogram features is used, which achieved comparable results to the state-of-the-art for ASVspoof 2019 logical access condition. Physical access condition from ASVspoof 2019 dataset, logical access and deep fake conditions from ASVspoof 2021 dataset are also used in the experiments. The regional energy features achieved improvements for all conditions with almost no additional computational or memory loads (less than 1% increase in the model size for SE-Res2Net). The main advantages of the regional energy features can be summarized as i) capturing nonspeech segments, ii) extracting band-limited information. Both aspects are found to be discriminative for spoofed speech detection.

Method of interacting between humans and conversational voice agent systems

Voice agent systems are deployed widely in various devices such as mobile phones, smart speakers, televisions, and refrigerators. Although the market for voice agent systems has expanded significantly and the systems themselves are becoming intelligent, their usability remains limited. This study provides general and practical guidelines for voice interaction to help VUI (Voice User Interface) designers and developers in the process of designing voice agent systems. The current guidelines only offer generic information for VUI designers, as they only provide the static aspect of voice interaction rather than consider the entire conversation flow. Studies on designing voice interaction systems have shown that VUI designers and developers need more guidance. The new interaction guidelines were developed based on users’ cognitive strategy in the course of using voice agent systems. We proposed and evaluated 12 practical voice interaction design guidelines for conversational voice agent systems. To evaluate the guidelines, we recruited 60 experts with different levels of voice interaction. Participants evaluated the usefulness of the guidelines and provided insightful opinions on them.

Robust Spoofed Speech Detection with Denoised I-vectors

Spoofed speech detection is recently gaining attention of the researchers as speaker verification is shown to be vulnerable to spoofing attacks such as voice conversion, speech synthesis, replay, and impersonation. Although various different methods have been proposed to detect spoofed speech, their performances decrease dramatically under the mismatched conditions due to the additive or reverberant noises. Conventional speech enhancement methods fail to recover the performance gap, hence more advanced techniques seem to be necessary to solve the noisy spoofed speech detection problem. In this work, Denoising Autoencoder (DAE) is used to obtain clean estimates of i-vectors from their noisy versions. ASVspoof 2015 database is used in the experiments with five different noise types, added to the original utterances at 0, 10, and 20 dB signal-to-noise ratios (SNR). The experimental results verified that the DAE provides a more robust spoof detection, where the conventional methods fail.

Choosing only the best voice imitators: Top-K many-to-many voice conversion with StarGAN

Voice conversion systems have become increasingly important as the use of voice technology grows. Deep learning techniques, specifically generative adversarial networks (GANs), have enabled significant progress in the creation of synthetic media, including the field of speech synthesis. One of the most recent examples, StarGAN-VC, uses a single pair of generator and discriminator to convert voices between multiple speakers. However, the training stability of GANs can be an issue. The Top-K methodology, which trains the generator using only the best K generated samples that “fool” the discriminator, has been applied to image tasks and simple GAN architectures. In this work, we demonstrate that the Top-K methodology can improve the quality and stability of converted voices in a state-of-the-art voice conversion system like StarGAN-VC. We also explore the optimal time to implement the Top-K methodology and how to reduce the value of K during training. Through both quantitative and qualitative studies, it was found that the Top-K methodology leads to quicker convergence and better conversion quality compared to regular or vanilla training. In addition, human listeners perceived the samples generated using Top-K as more natural and were more likely to believe that they were produced by a human speaker. The results of this study demonstrate that the Top-K methodology can effectively improve the performance of deep learning-based voice conversion systems.

Huawei Smartphone Marketing Case Analysis

Smart phones have become a part of people's lives, and Huawei mobile phones are the mainstream brand in China. This article will introduce Huawei's sales strategies in the context of the epidemic, Huawei's response strategies when its products are subject to U.S. sanctions, and Huawei's marketing and profit making skills in sandbox countries. In recent years, the price of conversation voice resources of major telecom companies in China has continued to decline, making the use of conversation voice resources by customers more and more. There are three reasons: the purchase of low-price models is price-driven and does not require on-site confirmation. The on-site display of medium and high-end models can strengthen the brand's positioning in the high-end field. The cost of offline channels is high, and the premium of first-line models is needed to ensure profits. Under the influence of COVID-19 and US policies, Huawei's smart phone marketing abroad must be reduced and also needs to increase its online marketing efforts. The relationship between the profitability of high-end mobile phones and retail outlets is a particularly important sales strategy of Huawei. Huawei should focus on facing different consumer groups at different times.

Any-to-One Non-Parallel Voice Conversion System Using an Autoregressive Conversion Model and LPCNet Vocoder

We present an any-to-one voice conversion (VC) system, using an autoregressive model and LPCNet vocoder, aimed at enhancing the converted speech in terms of naturalness, intelligibility, and speaker similarity. As the name implies, non-parallel any-to-one voice conversion does not require paired source and target speeches and can be employed for arbitrary speech conversion tasks. Recent advancements in neural-based vocoders, such as WaveNet, have improved the efficiency of speech synthesis. However, in practice, we find that the trajectory of some generated waveforms is not consistently smooth, leading to occasional voice errors. To address this issue, we propose to use an autoregressive (AR) conversion model along with the high-fidelity LPCNet vocoder. This combination not only solves the problems of waveform fluidity but also produces more natural and clear speech, with the added capability of real-time speech generation. To precisely represent the linguistic content of a given utterance, we use speaker-independent PPG features (SI-PPG) computed from an automatic speech recognition (ASR) model trained on a multi-speaker corpus. Next, a conversion model maps the SI-PPG to the acoustic representations used as input features for the LPCNet. The proposed autoregressive structure enables our system to produce the following prediction step outputs from the acoustic features predicted in the previous step. We evaluate the effectiveness of our system by performing any-to-one conversion pairs between native English speakers. Experimental results show that the proposed method outperforms state-of-the-art systems, producing higher speech quality and greater speaker similarity.

W2VC: WavLM representation based one-shot voice conversion with gradient reversal distillation and CTC supervision

Non-parallel data voice conversion (VC) has achieved considerable breakthroughs due to self-supervised pre-trained representation (SSPR) being used in recent years. Features extracted by the pre-trained model are expected to contain more content information. However, in common VC with SSPR, there is no special implementation to remove speaker information in the content representation extraction by SSPR, which prevents further purification of the speaker information from SSPR representation. Moreover, in conventional VC, Mel-spectrogram is often selected as the reconstructed acoustic feature, which is not consistent with the input of the content encoder and results in some information lost. Motivated by the above, we proposed W2VC to settle the issues. W2VC consists of three parts: (1) We reconstruct feature from WavLM representation (WLMR) that is more consistent with the input of content encoder; (2) Connectionist temporal classification (CTC) is used to align content representation and text context from phoneme level, content encoder plus gradient reversal layer (GRL) based speaker classifier are used to remove speaker information in the content representation extraction; (3) WLMR-based HiFi-GAN is trained to convert WLMR to waveform speech. VC experimental results show that GRL can purify well the content information of the self-supervised model. The GRL purification and CTC supervision on the content encoder are complementary in improving the VC performance. Moreover, the synthesized speech using the WLMR retrained vocoder achieves better results in both subjective and objective evaluation. The proposed method is evaluated on the VCTK and CMU databases. It is shown the method achieves 8.901 in objective MCD, 4.45 in speech naturalness, and 3.62 in speaker similarity of subjective MOS score, which is superior to the baseline.

Data Augmentation based Cross-Lingual Multi-Speaker TTS using DL with Sentiment Analysis

Text to S peech (TTS) algorithms have made tremendous strides in recent years in terms of their ability to generate speech in a single language that sounds as natural as possible. However, because to a lack of available training data, the synthesis of speech from the same person in various languages continues to be difficult. It might be challenging to locate people who are proficient in numerous languages at a level equivalent to native speakers. Voice conversion is one method that may be used to create a polyglot corpus, which can then be used to solve this problem. This entails making use of a voice representation model that has been trained on 53 different languages through the application of hybrid deep learning in order to capture speaker-invariant qualities. In this study, we present a novel approach for the conversion of voices across different languages by employing Generated Adversarial Networks (GANs) to train a multilingual TTS system. The concept of individual likeness loss in order to address the special difficulty of maintaining one's individual speaking identity during the training process can be offered. This work is focused to provide the impression that voice data coming from a variety of languages and speakers was produced by the same individual, and one way to do so is by using this word. In order to determine the extent to which our model is useful, two experiments that compared it against benchmarks that made use of varying degrees of parameter sharing between languages are carried out. The purpose of these experiments was to evaluate the accuracy of pronunciation as well as the caliber of the synthetic voice during transitions between different languages.

Voice spoofing detection for multiclass attack classification using deep learning

Voice biometric authentication is increasingly gaining adoption in organisations with high-volume identity verifications and for providing access to physical and other virtual spaces. In this form of authentication, the user’s identity is verified with their voice. However, these systems are susceptible to voice spoofing attacks as malicious actors employ different types of attacks such as speech synthesis, voice conversion or imitations, and recorded replays to spoof the Automatic Speaker Verification (ASV) system or for spam communications. In this work, we provide a voice spoofing countermeasure as a binary classification problem, that classifies real and fake audio, and also as a multiclass classification problem to detect voice conversion, synthesis and replay attacks. We investigated numerous audio features and examined each feature capability alongside state-of-the-art deep learning algorithms including convolutional neural networks (CNN), WaveNet, and recurrent neural network variants — Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM) models. Using a large dataset of 419,426 audio files for experiments, we evaluated the deep learning models for their effectiveness against voice spoofing attacks. The binary class CNN achieved a false positive rate (FPR) of 0.0216, while the multiclass solutions using CNN, WaveNet, LSTMs and GRUs achieved an FPR of 0.003, 0.0260, 0.0302 and 0.0358 respectively. We extended the evaluation of the models by including the real-time classification using microphone voice audio and user-uploaded audio to demonstrate the practical implications and deployability.