Speaker Identity Information Research Articles

Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction

Speech signals are inherently complex as they encompass both global acoustic characteristics and local semantic information. However, in the task of target speech extraction, certain elements of global and local semantic information in the reference speech, which are irrelevant to speaker identity, can lead to speaker confusion within the speech extraction network. To overcome this challenge, we propose a self-supervised disentangled representation learning method. Our approach tackles this issue through a two-phase process, utilizing a reference speech encoding network and a global information disentanglement network to gradually disentangle the speaker identity information from other irrelevant factors. We exclusively employ the disentangled speaker identity information to guide the speech extraction network. Moreover, we introduce the adaptive modulation Transformer to ensure that the acoustic representation of the mixed signal remains undisturbed by the speaker embeddings. This component incorporates speaker embeddings as conditional information, facilitating natural and efficient guidance for the speech extraction network. Experimental results substantiate the effectiveness of our meticulously crafted approach, showcasing a substantial reduction in the likelihood of speaker confusion.

Noise-robust voice conversion using adversarial training with multi-feature decoupling

Most existing voice conversion methods focus primarily on separating speech content from speaker information while overlooking the decoupling of pitch information. Additionally, the quality of converted speech significantly degrades when the speech of the target speaker is contaminated by noises. To address these issues, this paper proposes a noise-robust voice conversion model with multi-feature decoupling based on adversarial training. The proposed framework utilizes three distinct encoders to encode speech content, speaker identity, and pitch information independently, which aims to enhance the performance of decoupling by minimizing their mutual information and reduce the correlations between feature vectors. Moreover, a gradient reversal layer and a noise decoupling discriminator are incorporated into the framework, which extracts noise-resistant speaker representations and content representations through adversarial training to facilitate the synthesis of high-quality speech. In order to optimize the learning process, a training strategy is developed which involves alternating between clean and noisy data during the training of the encoder. This strategy effectively guides and expedites the convergence of the model. Experimental results demonstrate that compared to the state-of-the-art baselines of noise-robust voice conversion, the proposed model achieves improvements around 0.31 and 0.39 in terms of speech naturalness and speaker similarity evaluation metrics, respectively.

Enhancing accuracy and privacy in speech-based depression detection through speaker disentanglement

Speech signals are valuable biomarkers for assessing an individual’s mental health, including identifying Major Depressive Disorder (MDD) automatically. A frequently used approach in this regard is to employ features related to speaker identity, such as speaker-embeddings. However, over-reliance on speaker identity features in mental health screening systems can compromise patient privacy. Moreover, some aspects of speaker identity may not be relevant for depression detection and could serve as a bias factor that hampers system performance. To overcome these limitations, we propose disentangling speaker-identity information from depression-related information. Specifically, we present four distinct disentanglement methods to achieve this — adversarial speaker identification (SID)-loss maximization (ADV), SID-loss equalization with variance (LEV), SID-loss equalization using Cross-Entropy (LECE) and SID-loss equalization using KL divergence (LEKLD). Our experiments, which incorporated diverse input features and model architectures, have yielded improved F1 scores for MDD detection and voice-privacy attributes, as quantified by Gain in Voice Distinctiveness (GVD) and De-Identification Scores (DeID). On the DAIC-WOZ dataset (English), LECE using ComparE16 features results in the best F1-Scores of 80% which represents the audio-only SOTA depression detection F1-Score along with a GVD of −1.1 dB and a DeID of 85%. On the EATD dataset (Mandarin), ADV using raw-audio signal achieves an F1-Score of 72.38% surpassing multi-modal SOTA along with a GVD of −0.89 dB dB and a DeID of 51.21%. By reducing the dependence on speaker-identity-related features, our method offers a promising direction for speech-based depression detection that preserves patient privacy.

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Persons

Automatically generating videos in which synthesized speech is synchronized with lip movements in a talking head has great potential in many human-computer interaction scenarios. In this paper, we present an automatic method to generate synchronized speech and talking-head videos on the basis of text and a single face image of an arbitrary person as input. In contrast to previous text-driven talking head generation methods, which can only synthesize the voice of a specific person, the proposed method is capable of synthesizing speech for any person. Specifically, the proposed method decomposes the generation of synchronized speech and talking head videos into two stages, i.e., a text-to-speech (TTS) stage and a speech-driven talking head generation stage. The proposed TTS module is a face-conditioned multi-speaker TTS model that gets the speaker identity information from face images instead of speech, which allows us to synthesize a personalized voice on the basis of the input face image. To generate the talking head videos from the face images, a facial landmark-based method that can predict both lip movements and head rotations is proposed. Extensive experiments demonstrate that the proposed method is able to generate synchronized speech and talking head videos for arbitrary persons, in which the timbre of the synthesized voice is in harmony with the input face, and the proposed landmark-based talking head method outperforms the state-of-the-art landmark-based method on generating natural talking head videos.

One-shot emotional voice conversion based on feature separation

The task of emotional voice conversion (EVC) aims to convert speech from one emotional state into another, while keeping linguistic content, speaker identity and other emotion-independent information unchanged. Because previous studies were limited to a specific set of emotions, it is challenging to realize the conversion of emotions never seen in training stage. In this paper, we propose a one-shot emotional voice conversion model based on feature separation. The proposed method could control emotional characteristics with Global Emotion Embeddings (GEEs), and introduce activation guidance (AG) and mutual information (MI) minimization to reduce the correlations between emotion embedding and emotion-independent representation. At run-time conversion, it could produce the desired emotional utterance from a single pairwise utterance without any emotion labels, whether the target emotion appears in the training set or not. The subjective and objective evaluations validate the effectiveness of our proposed model for both the degree of feature separation and emotion expression, even it could achieve unseen emotion conversion.

Speaker recognition based on short utterance compensation method of generative adversarial networks

On the basis of gaussian mixture model–universal background model (GMM–UBM) in the speaker recognition system, the paper proposes a short utterance sample compensation method based on the generative adversarial network (GAN) to solve the problem of the inadequate corpus data caused by short utterance, which has led to a serious reduction of recognition rate. The presented method compensates the short utterance samples into the speech samples with sufficient speaker identity information by completing the antagonistic training of generator network and discriminator network. In order to avoid the model crash and gradient instability in the process of GAN training, this paper adopts the condition information in the conditional GAN to guide the compensation process of the generator network, and proposes the generator compensation performance measurement training task and the feature tag training task of the discriminator to stabilize training process. Finally, the proposed short utterance compensation method is evaluated on the speaker recognition system based on GMM–UBM. The experimental results indicate that the presented method can effectively reduce the equal error rate of the speaker recognition system in short utterance environment.

Closed-set speaker conditioned acoustic-to-articulatory inversion using bi-directional long short term memory network.

Estimating articulatory movements from speech acoustic representations is known as acoustic-to-articulatory inversion (AAI). In this work, a speaker conditioned AAI (SC AAI) is proposed using a bi-directional LSTM neural network, where training is performed by pooling acoustic-articulatory data from multiple speakers along with their corresponding speaker identity information. For this work, 7.24 h of multi-speaker acoustic-articulatory data are collected from 20 speakers speaking 460 English sentences. Experiments with 20 speakers indicate that the SC AAI model performs better than SD AAI model with an improvement of correlation coefficient by 0.036 (absolute) between the original and estimated articulatory movements.

Addressing Text-Dependent Speaker Verification Using Singing Speech

The automatic speaker verification (ASV) has achieved significant progress in recent years. However, it is still very challenging to generalize the ASV technologies to new, unknown and spoofing conditions. Most previous studies focused on extracting the speaker information from natural speech. This paper attempts to address the speaker verification from another perspective. The speaker identity information was exploited from singing speech. We first designed and released a new corpus for speaker verification based on singing and normal reading speech. Then, the speaker discrimination was compared and analyzed between natural and singing speech in different feature spaces. Furthermore, the conventional Gaussian mixture model, the dynamic time warping and the state-of-the-art deep neural network were investigated. They were used to build text-dependent ASV systems with different training-test conditions. Experimental results show that the voiceprint information in the singing speech was more distinguishable than the one in the normal speech. More than relative 20% reduction of equal error rate was obtained on both the gender-dependent and independent 1 s-1 s evaluation tasks.

Emotion transplantation through adaptation in HMM-based speech synthesis

This paper proposes an emotion transplantation method capable of modifying a synthetic speech model through the use of CSMAPLR adaptation in order to incorporate emotional information learned from a different speaker model while maintaining the identity of the original speaker as much as possible. The proposed method relies on learning both emotional and speaker identity information by means of their adaptation function from an average voice model, and combining them into a single cascade transform capable of imbuing the desired emotion into the target speaker. This method is then applied to the task of transplanting four emotions (anger, happiness, sadness and surprise) into 3 male speakers and 3 female speakers and evaluated in a number of perceptual tests. The results of the evaluations show how the perceived naturalness for emotional text significantly favors the use of the proposed transplanted emotional speech synthesis when compared to traditional neutral speech synthesis, evidenced by a big increase in the perceived emotional strength of the synthesized utterances at a slight cost in speech quality. A final evaluation with a robotic laboratory assistant application shows how by using emotional speech we can significantly increase the students’ satisfaction with the dialog system, proving how the proposed emotion transplantation system provides benefits in real applications.

Cross-speaker generalisation in two phoneme-level perceptual adaptation processes

Speech perception is shaped by listeners' prior experience with speakers. Listeners retune their phonetic category boundaries after encountering ambiguous sounds in order to deal with variations between speakers. Repeated exposure to an unambiguous sound, on the other hand, leads to a decrease in sensitivity to the features of that particular sound. This study investigated whether these changes in the listeners' perceptual systems can generalise to the perception of speech from a novel speaker. Specifically, the experiments looked at whether visual information about the identity of the speaker could prevent generalisation from occurring. In Experiment 1, listeners retuned auditory category boundaries using audiovisual speech input. This shift in the category boundaries affected perception of speech from both the exposure speaker and a novel speaker. In Experiment 2, listeners were repeatedly exposed to unambiguous speech either auditorily or audiovisually, leading to a decrease in sensitivity to the features of the exposure sound. Here, too, the changes affected the perception of both the exposure speaker and the novel speaker. Together, these results indicate that changes in the perceptual system can affect the perception of speech from a novel speaker and that visual speaker identity information did not prevent this generalisation.

A Study of Bilinear Models in Voice Conversion

This paper presents a voice conversion technique based on bilinear models and introduces the concept of contextual modeling. The bilinear approach reformulates the spectral envelope representation from line spectral frequencies feature to a two-factor parameterization corresponding to speaker identity and phonetic information, the so-called style and content factors. This decomposition offers a flexible representation suitable for voice conversion and facilitates the use of efficient training algorithms based on singular value decomposition. In a contextual approach (bilinear) models are trained on subsets of the training data selected on the fly at conversion time depending on the characteristics of the feature vector to be converted. The performance of bilinear models and context modeling is evaluated in objective and perceptual tests by comparison with the popular GMM-based voice conversion method for several sizes and different types of training data.

Introduction to the Special Section on Rich Transcription

HE term Rich Transcription spans multiple areas in audio processing, and its study marks a broadening of the concerns of automatic speech recognition (ASR) to cover the affiliated areas necessary for maximally useful applications. Whereas classical speech recognition focuses purely on converting a sequence of audio words to a sequence of textual words—without regard for capitalization, punctuation, speaker identity, pragmatic intent, and other high-level information—rich transcription attempts to produce a more highly annotated and informative output. The study of rich transcription received a great impetus in 2002 when the Defense Advanced Research Projects Agency (DARPA) started the Effective Affordable Reusable Speech-to-Text (EARS) program. This program extended the previous HUB-4 and HUB-5 programs by adding an emphasis on metatdata extraction, in addition to traditional word recognition. The particular metadata tasks that were studied (http://nist.gov/speech/tests/rt/rt2004/fall/docs/rt04f-eval-planv14.doc) are as follows. • Speaker diarization: the problem of segmenting speech into regions where only one person is talking, and then linking together speech (possibly from disjoint regions of time) from the same speaker. • Identification of sentence-like units (SUs): the task of segmenting speech into units expressing separate thoughts or ideas, similar to sentences in written language, but taking into account that spoken language might not exhibit complete grammatical sentences. • Disfluency detection: the dual problems of detecting the speech locations where a fluent word stream is interrupted (interruption point detection), and identifying those words that need to be removed in order to obtain the fluent word sequence of the intended utterance. This involves the labeling of pause fillers (e.g., “uh”), edit words (e.g., “I mean”), and the words that the speaker meant to replace in a self-repair. Clearly, other forms and definitions of metadata are possible, and above tasks are offered only for illustrative purposes. While rich transcription adds a new emphasis on various forms of metadata annotation, it also maintains a strong focus on improving automatic speech recognition from a core word-error-rate point of view. This is reflected in the composition of the special issue, with about half the papers addressing ASR. Here, there is a great deal of current interest in topics such as discriminative training, the use of large amounts of training data, unsupervised and semisupervised training, and