Voice Conversion Research Articles

Anonymization of Voices in Spaces for Civic Dialogue: Measuring Impact on Empathy, Trust, and Feeling Heard

Anonymity is a powerful component of many participatory media platforms that can afford people greater freedom of expression and protection from external coercion and interference. However, it can be difficult to effectively implement on platforms that leverage spoken language due to distinct biomarkers present in the human voice. In this work, we explore the use of voice anonymization methods within the context of a technology-enhanced civic dialogue network based in the United States, whose purpose is to increase feelings of agency and being heard within civic processes. Specifically, we investigate the use of two different speech transformation and synthesis methods for anonymization: voice conversion (VC) and text-to-speech (TTS). Through a series of two studies, we examine the impact that each method has on 1) the empathy and trust that listeners feel towards a person sharing a personal story, and 2) a speaker's own perception of being heard, finding that voice conversion is an especially suitable method for our purposes. Our findings open up interesting potential research directions related to anonymous spoken discourse, as well as additional ways of engaging with voice-based civic technologies.

Improved Speech Authenticity Detection in Chinese–English Bilingual Contexts

The rapid evolution of voice technology has heightened the need for robust detection systems to distinguish between authentic and tampered speech. Recent competitions have significantly advanced the development of countermeasures against spoofing attacks. However, while advancements in detection technologies have been notable, existing methods often focus on a single type of tampering and language. Our contribution lies in developing an improved model that integrates an enhanced ResNet architecture with an LSTM to improve the detection of tampered audio, particularly in challenging multilingual scenarios. In the experiments, we built a hybrid dataset from self-recording Chinese speech and public VCTK2 English samples, enhanced the ResNet model generalization capabilities, and evaluated our approach using the bilingual dataset. Experiment results demonstrate that the proposed approach achieves a superior performance with an equal error rate of 11.62%, even in the face of bilingual conditions, and, more importantly, outperforms the leading models from ASVSpoof 2021 and ADD 2022 competitions. We also employed advanced tampering techniques, including CycleGAN voice conversion and auto splicing, to simulate real-world tampering scenarios and verify the effectiveness of the proposed approach.

CycleDiffusion: Voice Conversion Using Cycle-Consistent Diffusion Models

Voice conversion (VC) refers to the technique of modifying one speaker’s voice to mimic another’s while retaining the original linguistic content. This technology finds its applications in fields such as speech synthesis, accent modification, medicine, security, privacy, and entertainment. Among the various deep generative models used for voice conversion, including variational autoencoders (VAEs) and generative adversarial networks (GANs), diffusion models (DMs) have recently gained attention as promising methods due to their training stability and strong performance in data generation. Nevertheless, traditional DMs focus mainly on learning reconstruction paths like VAEs, rather than conversion paths as GANs do, thereby restricting the quality of the converted speech. To overcome this limitation and enhance voice conversion performance, we propose a cycle-consistent diffusion (CycleDiffusion) model, which comprises two DMs: one for converting the source speaker’s voice to the target speaker’s voice and the other for converting it back to the source speaker’s voice. By employing two DMs and enforcing a cycle consistency loss, the CycleDiffusion model effectively learns both reconstruction and conversion paths, producing high-quality converted speech. The effectiveness of the proposed model in voice conversion is validated through experiments using the VCTK (Voice Cloning Toolkit) dataset.

Dimensional Affective Speech Synthesis Based On Voice Conversion

Dimensional Affective Speech Synthesis Based On Voice Conversion

The Arduous and the Ordinary: Embodying the Conversational Voice in the Long Nineteenth Century

This article aims to chart out a cultural history of the voice in conversation in the nineteenth century. Often understood as a period of decline of the ‘art’ of conversation, as it was defined in the eighteenth century (and studied by historians of the period), the nineteenth century was witness to a shift away from conscious cultivation. Conversation, in the nineteenth century, became the counterpoint to more formalized modes of speech like oratory and debate, and was defined as ‘natural’. As such, it also became an object of study for experts specializing in vocal health, who saw this ‘natural’ form of vocal expression as a place to see the voice in action at its least artificial. Based on the work of these experts – scientific treatises, early forays in laryngoscopy, educational work and self-help manuals – I propose an analysis of their largely normative texts that focuses on sound and vocal practice, rather than the development of scientific discourse and its disciplinary character. I borrow central concepts from sound studies, like that of the audile technique and aurality in order to do so, and place the vocalizing body at the center of this analysis.

자막 제공 서비스 앱의 UI/UX 개선 연구 - 청각 장애인을 위한 자막 자동 생성 앱 ‘이어줌’의 사용성 향상을 위한 제안

This study aims to improve users' video content accessibility, usability, and aesthetic satisfaction by analyzing the UI/UX of <Eeojum>, an app that provides an automatic subtitle creation service for the hearing impaired, in terms of functionality and usability. To this end, we conducted preliminary research on automatic subtitle generation apps, analyzed the environment and actual conditions of mobile use by hearing impaired people, and analyzed domestic and international app accessibility policies and public institution UI design guidelines. Based on this, we looked at the UI/UX characteristics of <Eeojum>, derived UI/UX improvement items to improve app accessibility quality and usability, and proposed improvements to the voice conversion app UI/UX design draft. The biggest goal was to simplify the structure of IA and strengthen the visual formability of the display to improve usability, presenting design guidelines aimed at improving usability for the hearing impaired, the main users.

Zero-shot voice conversion based on feature disentanglement

Voice conversion (VC) aims to convert the voice from a source speaker to a target speaker without modifying the linguistic content. Zero-shot voice conversion has attracted significant attention in the task of VC because it can achieve conversion for speakers who did not appear during the training stage. Despite the significant progress made by previous methods in zero-shot VC, there is still room for improvement in separating speaker information and content information. In this paper, we propose a zero-shot VC method based on feature disentanglement. The proposed model uses a speaker encoder for extracting speaker embeddings, introduces mixed speaker layer normalization to eliminate residual speaker information in content encoding, and employs adaptive attention weight normalization for conversion. Furthermore, dynamic convolution is introduced to improve speech content modeling while requiring a small number of parameters. The experiments demonstrate that performance of the proposed model is superior to several state-of-the-art models, achieving both high similarity with the target speaker and intelligibility. In addition, the decoding speed of our model is much higher than the existing state-of-the-art models.

CLESSR-VC: Contrastive learning enhanced self-supervised representations for one-shot voice conversion

One-shot voice conversion (VC) has attracted more and more attention due to its broad prospects for practical application. In this task, the representation ability of speech features and the model’s generalization are the focus of attention. This paper proposes a model called CLESSR-VC, which enhances pre-trained self-supervised learning (SSL) representations through contrastive learning for one-shot VC. First, SSL features from the 23rd and 9th layers of the pre-trained WavLM are adopted to extract content embedding and SSL speaker embedding, respectively, to ensure the model’s generalization. Then, the conventional acoustic feature mel-spectrograms and contrastive learning are introduced to enhance the representation ability of speech features. Specifically, contrastive learning combined with the pitch-shift augmentation method is applied to disentangle content information from SSL features accurately. Mel-spectrograms are adopted to extract mel speaker embedding. The AM-Softmax and cross-architecture contrastive learning are applied between SSL and mel speaker embeddings to obtain the fused speaker embedding that helps improve speech quality and speaker similarity. Both objective and subjective evaluation results on the VCTK corpus confirm that the proposed VC model has outstanding performance and few trainable parameters.

SingVisio: Visual analytics of diffusion model for singing voice conversion

In this study, we present SingVisio, an interactive visual analysis system that aims to explain the diffusion model used in singing voice conversion. SingVisio provides a visual display of the generation process in diffusion models, showcasing the step-by-step denoising of the noisy spectrum and its transformation into a clean spectrum that captures the desired singer’s timbre. The system also facilitates side-by-side comparisons of different conditions, such as source content, melody, and target timbre, highlighting the impact of these conditions on the diffusion generation process and resulting conversions. Through comparative and comprehensive evaluations, SingVisio demonstrates its effectiveness in terms of system design, functionality, explainability, and user-friendliness. It offers users of various backgrounds valuable learning experiences and insights into the diffusion model for singing voice conversion.

Reimagining speech: a scoping review of deep learning-based methods for non-parallel voice conversion

Reimagining speech: a scoping review of deep learning-based methods for non-parallel voice conversion

DialogueNeRF: towards realistic avatar face-to-face conversation video generation

Conversation is an essential component of virtual avatar activities in the metaverse. With the development of natural language processing, significant breakthroughs have been made in text and voice conversation generation. However, face-to-face conversations account for the vast majority of daily conversations, while most existing methods focused on single-person talking head generation. In this work, we take a step further and consider generating realistic face-to-face conversation videos. Conversation generation is more challenging than single-person talking head generation, because it requires not only the generation of photo-realistic individual talking heads, but also the listener’s response to the speaker. In this paper, we propose a novel unified framework based on the neural radiance field (NeRF) to address these challenges. Specifically, we model both the speaker and the listener with a NeRF framework under different conditions to control individual expressions. The speaker is driven by the audio signal, while the response of the listener depends on both visual and acoustic information. In this way, face-to-face conversation videos are generated between human avatars, with all the interlocutors modeled within the same network. Moreover, to facilitate future research on this task, we also collected a new human conversation dataset containing 34 video clips. Quantitative and qualitative experiments evaluate our method in different aspects, e.g., image quality, pose sequence trend, and natural rendering of the scene in the generated videos. Experimental results demonstrate that the avatars in the resulting videos are able to carry on a realistic conversation, and maintain individual styles.

Target speaker filtration by mask estimation for source speaker traceability in voice conversion

Voice Conversion (VC) can manipulate the source speaker's identity of speech signal to make it sound like some specific target speaker, which makes it harder for a human being or a speaker verification/identification system to trace the real identity of the source speaker. However, extracting features of the source speaker from converted audio is challenging since the features of the target speaker are dominant in the converted audio, which hinders the extraction of the features of the source speaker. In this paper, to extract features of the source speaker from audios processed by VC methods, a speaker filtration block is designed, which uses mask estimation to identify source speakers from manipulated speech signals by filtering out the features of the target speaker in converted audio. Extensive experiments are conducted to evaluate the effectiveness of the proposed model in tracing source speakers of audios converted by ADAIN-VC, AGAIN-VC, VQMIVC, and FREEVC. Experimental results demonstrate the effectiveness of the proposed model by comparing to competitive baselines in speaker verification/identification scenarios. Notably, it has good performance even when being applied to unknown VC methods. Furthermore, the experiments also show that, training audios generated by multiple VC methods can improve the performance on the traceability of the source speaker.

Introduction to Special Issue: Centering Community Healing, Sustenance, and Resilience in STEM and Computing Education Through Art and Social Justice

Computing, technology, and engineering education are in particular need of examining how we can bring love, community, and joy to the center of learning experiences to support a diversity of youth to flourish. Computing and STEM disciplines are often taught in ways that detach knowledge from people and communities, making it difficult to explore a variety of personal, social, and societal connections that are not already at the center of the tech industry. These exclusionary practices limit the way work is accomplished—including who is involved in the design work and who technological innovations are designed for; as well as the way the work is situated within our society—including what problems technology is designed to solve and who benefits from particular innovations (Benjamin, 2019; Noble, 2018). Learning experiences and spaces for technology education should support youth as they develop new knowledge and skills in ways that recognize them as full humans, and enable them to ground it in their ideas, perspectives, interests, goals, identities, communities, relationships, and local environments. The goal of this special issue is to bring together educators’, researchers’, and youth voices in conversation about how we are centering self-empowerment in our educational work and how we can drive forward our practices.

Assessment of Self-Supervised Denoising Methods for Esophageal Speech Enhancement

Esophageal speech (ES) is a pathological voice that is often difficult to understand. Moreover, acquiring recordings of a patient’s voice before a laryngectomy proves challenging, thereby complicating enhancing this kind of voice. That is why most supervised methods used to enhance ES are based on voice conversion, which uses healthy speaker targets, things that may not preserve the speaker’s identity. Otherwise, unsupervised methods for ES are mostly based on traditional filters, which cannot alone beat this kind of noise, making the denoising process difficult. Also, these methods are known for producing musical artifacts. To address these issues, a self-supervised method based on the Only-Noisy-Training (ONT) model was applied, consisting of denoising a signal without needing a clean target. Four experiments were conducted using Deep Complex UNET (DCUNET) and Deep Complex UNET with Complex Two-Stage Transformer Module (DCUNET-cTSTM) for assessment. Both of these models are based on the ONT approach. Also, for comparison purposes and to calculate the evaluation metrics, the pre-trained VoiceFixer model was used to restore the clean wave files of esophageal speech. Even with the fact that ONT-based methods work better with noisy wave files, the results have proven that ES can be denoised without the need for clean targets, and hence, the speaker’s identity is retained.

Children’s conversational voice search as learning: a literature review

Purpose This paper aims to critically review the intersection of searching and learning among children in the context of voice-based conversational agents (VCAs). This study presents the opportunities and challenges around reconfiguring current VCAs for children to facilitate human learning, generate diverse data to empower VCAs, and assess children’s learning from voice search interactions. Design/methodology/approach The scope of this paper includes children’s use of VCAs for learning purposes with an emphasis on conceptualizing their VCA use from search as learning perspectives. This study selects representative works from three areas of literature: children’s perceptions of digital devices, children’s learning and searching, and children’s search as learning. This study also includes conceptual papers and empirical studies focusing on children from 3 to 11 because this age spectrum covers a vital transitional phase in children’s ability to understand and use VCAs. Findings This study proposes the concept of child-centered voice search systems and provides design recommendations for imbuing contextual information, providing communication breakdown repair strategies, scaffolding information interactions, integrating emotional intelligence, and providing explicit feedback. This study presents future research directions for longitudinal and observational studies with more culturally diverse child participants. Originality/value This paper makes important contributions to the field of information and learning sciences and children’s searching as learning by proposing a new perspective where current VCAs are reconfigured as conversational voice search systems to enhance children’s learning.

Speaking Practice using ChatGPT’s Voice Conversation: A Review on Potentials and Concerns

Investigating the pedagogical potentials and concerns of ChatGPT's voice conversation feature within the context of language learning, this study begins by exploring the historical context and limitations of traditional language learning methods, such as language exchange, conversation classes, and role-plays. The discussion then shifts to the rise of AI in education, highlighting ChatGPT's advanced capabilities in providing personalized, interactive, and efficient language instruction through natural language processing, machine learning, and speech recognition. While offering advantages such as accessibility, adaptability, and personalized feedback, the technology also presents limitations in accurately assessing pronunciation and contextual nuances. To effectively integrate ChatGPT into language instruction, educators must carefully consider its strengths and weaknesses. This research explores those things to optimize their use and ensure they complement, rather than replace, traditional language teaching methods.

Fake speech detection using VGGish with attention block

While deep learning technologies have made remarkable progress in generating deepfakes, their misuse has become a well-known concern. As a result, the ubiquitous usage of deepfakes for increasing false information poses significant risks to the security and privacy of individuals. The primary objective of audio spoofing detection is to identify audio generated through numerous AI-based techniques. Several techniques for fake audio detection already exist using machine learning algorithms. However, they lack generalization and may not identify all types of AI-synthesized audios such as replay attacks, voice conversion, and text-to-speech (TTS). In this paper, a deep layered model, i.e., VGGish, along with an attention block, namely Convolutional Block Attention Module (CBAM) for spoofing detection, is introduced. Our suggested model successfully classifies input audio into two classes: Fake and Real, converting them into mel-spectrograms, and extracting their most representative features due to the attention block. Our model is a significant technique to utilize for audio spoofing detection due to a simple layered architecture. It captures complex relationships in audio signals due to both spatial and channel features present in an attention module. To evaluate the effectiveness of our model, we have conducted in-depth testing using the ASVspoof 2019 dataset. The proposed technique achieved an EER of 0.52% for Physical Access (PA) attacks and 0.07 % for Logical Access (LA) attacks.

Scalability and diversity of StarGANv2-VC in Arabic emotional voice conversion: Overcoming data limitations and enhancing performance

Emotional Voice Conversion (EVC) for under-resourced languages like Arabic faces challenges due to limited emotional speech data. This study explored strategies to mitigate dataset scarcity and improve Arabic EVC performance. Fundamental experiments (Speaker-Dependent, Gender-Dependent, Gender-Independent) were conducted using the KSUEmotions dataset to analyze speaker, gender, and model impacts. Data augmentation techniques like time stretching and phase shuffling artificially increased data diversity. Attention mechanisms integrated into StarGANv2-VC aimed to better capture emotional cues. Transfer learning leveraged the larger English Emotional Speech Database (ESD) to enhance the Arabic system. A novel “Reordering Speaker-Emotion Data” approach treated each emotion as a separate speaker to expand the emotional variability. Our comprehensive approach, combining transfer learning, data augmentation, and architectural modifications, demonstrates the potential to overcome dataset limitations and enhance the performance of Arabic EVC systems.

EEASVI : A Voice-Driven Solution for the Visually Impaired

In the present scenario, everybody needs communication technology to connect with each other. Communication technologies are significant these days for the betterment of social and personal interaction. The combination of technologies with the internet makes communication easy. However, the person who is physically challenged suffered a lot to utilize this technology due to visual and physical difficulties. There are many technologies advancements have come though it is not possible to use like normal users. This paper aims at creating an email system that helps even new users or physically impaired people to use the system for communication without any previous practices. There is no use of keywords, only with the help of mouse actions and voice conversion the email system works. The person who is not literate can also send emails as it is based on speech recognition and text to speech. The system is completely based on responsive voice interaction to utilize the technology easy and hassle-free manner. The system is well designed to send the mails quickly. There are all the options available to send emails and perform all the functions for the email system. Key Words: Email, Internet, Voice, Speech recognition, physically challenged, Text to speech

Style Generative Adversarial Network Combined with Dynamic Fundamental Frequency Difference Compensation: A Practical and Efficient Method for Emotional Voice Conversion

This paper explores a novel emotional voice conversion (EVC) method based on the style generative adversarial network combined with dynamic fundamental frequency difference compensation. On the one hand, in terms of spectrum mapping, based on the method of EVC using star generative adversarial network, we propose to replace one-hot vectors with the emotion style features extracted from spectrum to represent emotional information. It enables the conversion not only to fully learn emotion style from the target speech, but also to transfer unseen emotion styles to a new utterance, that is, one-shot EVC. On the other hand, given that the traditional logarithm Gaussian normalization transformation and its some variants for prosody transfer are not enough to reflect the fine-grained fundamental frequency distribution difference between the different emotional speech, we propose the improved strategy of dynamic fundamental frequency difference compensation. The experimental results show that our proposed method can accomplish high quality one-shot EVC, significantly outperforming the baseline in terms of speech quality and emotional saturation in both objective and subjective evaluations.