Speaker Space Research Articles

The Art of Listening.

The Art of Listening.

Privacy and Utility of X-Vector Based Speaker Anonymization

We study the scenario where individuals (<i>speakers</i>) contribute to the publication of an anonymized speech corpus. Data <i>users</i> leverage this public corpus for downstream tasks, e.g., training an automatic speech recognition (ASR) system, while <i>attackers</i> may attempt to de-anonymize it using auxiliary knowledge. Motivated by this scenario, speaker anonymization aims to conceal speaker identity while preserving the quality and usefulness of speech data. In this article, we study x-vector based speaker anonymization, the leading approach in the VoicePrivacy Challenge, which converts the speaker&#x2019;s voice into that of a random pseudo-speaker. We show that the strength of anonymization varies significantly depending on how the pseudo-speaker is chosen. We explore four design choices for this step: the distance metric between speakers, the region of speaker space where the pseudo-speaker is picked, its gender, and whether to assign it to one or all utterances of the original speaker. We assess the quality of anonymization from the perspective of the three actors involved in our threat model, namely the speaker, the user and the attacker. To measure privacy and utility, we use respectively the linkability score achieved by the attackers and the decoding word error rate achieved by an ASR model trained on the anonymized data. Experiments on LibriSpeech show that the best combination of design choices yields state-of-the-art performance in terms of both privacy and utility. Experiments on Mozilla Common Voice further show that it guarantees the same anonymization level against re-identification attacks among 50 speakers as original speech among 20,000 speakers.

Deep Gaussian process based multi-speaker speech synthesis with latent speaker representation

This paper proposes deep Gaussian process (DGP)-based frameworks for multi-speaker speech synthesis and speaker representation learning. A DGP has a deep architecture of Bayesian kernel regression, and it has been reported that DGP-based single speaker speech synthesis outperforms deep neural network (DNN)-based ones in the framework of statistical parametric speech synthesis. By extending this method to multiple speakers, it is expected that higher speech quality can be achieved with a smaller number of training utterances from each speaker. To apply DGPs to multi-speaker speech synthesis, we propose two methods: one using DGP with one-hot speaker codes, and the other using a deep Gaussian process latent variable model (DGPLVM). The DGP with one-hot speaker codes uses additional GP layers to transform speaker codes into latent speaker representations. The DGPLVM directly models the distribution of latent speaker representations and learns it jointly with acoustic model parameters. In this method, acoustic speaker similarity is expressed in terms of the similarity of the speaker representations, and thus, the voices of similar speakers are efficiently modeled. We experimentally evaluated the performance of the proposed methods in comparison with those of conventional DNN and variational autoencoder (VAE)-based frameworks, in terms of acoustic feature distortion and subjective speech quality. The experimental results demonstrate that (1) the proposed DGP-based and DGPLVM-based methods improve subjective speech quality compared with a feed-forward DNN-based method, and (2) even when the amount of training data for target speakers is limited, the DGPLVM-based method outperforms other methods, including the VAE-based one. Additionally, (3) by using a speaker representation randomly sampled from the learned speaker space, the DGPLVM-based method can generate voices of non-existent speakers.

Rapid Speaker Adaptation Based on Combination of KPCA and Latent Variable Model

Speaker adaptation is implemented in order to shift the speaker-independent model closer to the new speaker speech characteristics to improve the speech recognition performance. The kernel eigenspace-based speaker adaptation methods provide satisfactory performance using only a small amount of adaptation data. In such adaptation methods, kernel principal component analysis (KPCA) is applied to the training speaker space in order to create kernel eigenspace. Then, the adapted acoustic model to the new user is calculated in that space. One limitation of KPCA is its inability to define a precise pre-image of the model adapted in the kernel eigenspace, back to the speaker space. Therefore, a huge amount of computations is required to perform adaptation. The previously developed solutions for calculation of an approximate pre-image of the adapted model do not necessarily lead to the optimal conditions. Therefore, in this paper, we propose an efficient solution for this problem to construct more reliable pre-image of the adapted model in the speaker space. For this purpose, we benefit from the latent variable model to define a probabilistic model for description of the applied mapping between the kernel eigenspace and the speaker space. The experiments were conducted on two speech databases: FARSDAT, a Persian, and TIMIT, an English speech database. Implementing a typical HMM-based automatic speech recognition system, it was verified that the proposed method, utilizing about three seconds of adaptation data, achieves up to 4.4% and 7.6% relative phoneme recognition accuracy rate over the speaker-independent model on FARSDAT and TIMIT, respectively. Moreover, the proposed approach demonstrated superior performance compared to the other kernel eigenspace-based adaptation methods.

Meeting the Enemy in World War I Poetry: Cognitive Dissonance as a Vehicle for Theme

Some World War I poems show an enemy soldier up close. This choice usually proves very effective for expressing the general irony of war, to be sure. However, I submit that showing interaction with the enemy also allows the speaker space to wrestle with internal conflict, guilt, or cognitive dissonance, and that it allows—or even forces—readers to participate in that struggle along with the speaker. While the poets’ writings no doubt had therapeutic effects for the poets themselves, I focus more on the literary effects, specifically arguing that the poems are powerful to us readers since they heighten the personal exposure of the poets’ psyches and since they make us share the dissonance as readers. I consider poems by Siegfried Sassoon, Wilfred Owen, Robert Graves, Ford Madox Ford, Herbert Read, and Robert Service.

Many-to-many and Completely Parallel-data-free Voice Conversion Based on Eigenspace DNN

Media conversion of image, text, speech, etc., generally requires a large amount of parallel data for training a conversion model. Recently, methods for training the model using no or a small amount of parallel data draw researchers’ attention. In many-to-many voice conversion, since it is often hard to collect parallel data from every pair of speakers, the conversion models requiring no parallel data are desired. Conventional many-to-many voice conversion models required a large amount of prestored parallel data to acquire prior knowledge of the entire speaker space. Then, a specific model from an arbitrary speaker to another can be realized by adapting a few model parameters. Although these conversion models certainly do not use parallel data in an adaptation step, they still use parallel data for prior training. In this study, we aim at realizing completely parallel-data-free and many-to-many voice conversion. The proposed method uses both Eigenvoice Gaussian mixture models EVGMM and Deep neural network DNN. EVGMM is a many-to-many conversion model that constructs the entire speaker space called eigenspace by analyzing mean vectors of Gaussian mixture models and it is used in our method to decompose training speakers’ features into their eigenspace components. By using the speaker features and the obtained components as pseudo parallel data, multiple DNNs are trained to realize conversion between them. With these DNNs, features of any target speaker can be represented by a weighted sum of the components. It should be noted that all the processes of our proposal do not require any parallel data. A key technique is to estimate covariance terms of EVGMM with no parallel data. Experiments indicate that individuality scores of the proposed method using no parallel data are comparable enough to those of a baseline system trained with parallel data.

Eigenvoice Speaker Adaptation with Minimal Data for Statistical Speech Synthesis Systems Using a MAP Approach and Nearest-Neighbors

Statistical speech synthesis (SSS) systems have the ability to adapt to a target speaker with a couple of minutes of adaptation data. Developing adaptation algorithms to further reduce the number of adaptation utterances to a few seconds of data can have substantial effect on the deployment of the technology in real-life applications such as consumer electronics devices. The traditional way to achieve such rapid adaptation is the eigenvoice technique which works well in speech recognition but known to generate perceptual artifacts in statistical speech synthesis. Here, we propose three methods to alleviate the quality problems of the baseline eigenvoice adaptation algorithm while allowing speaker adaptation with minimal data. Our first method is based on using a Bayesian eigenvoice approach for constraining the adaptation algorithm to move in realistic directions in the speaker space to reduce artifacts. Our second method is based on finding pre-trained reference speakers that are close to the target speaker and utilizing only those reference speaker models in a second eigenvoice adaptation iteration. Both techniques performed significantly better than the baseline eigenvoice method in the objective tests. Similarly, they both improved the speech quality in subjective tests compared to the baseline eigenvoice method. In the third method, tandem use of the proposed eigenvoice method with a state-of-the-art linear regression based adaptation technique is found to improve adaptation of excitation features.

Rapid speaker adaptation in latent speaker space with non-negative matrix factorization

A novel speaker adaptation algorithm based on Gaussian mixture weight adaptation is described. A small number of latent speaker vectors are estimated with non-negative matrix factorization (NMF). These latent vectors encode the distinctive systematic patterns of Gaussian usage observed when modeling the individual speakers that make up the training data. Expressing the speaker dependent Gaussian mixture weights as a linear combination of a small number of latent vectors reduces the number of parameters that must be estimated from the enrollment data. The resulting fast adaptation algorithm, using 3s of enrollment data only, achieves similar performance as fMLLR adapting on 100+s of data. In order to learn richer Gaussian usage patterns from the training data, the NMF-based weight adaptation is combined with vocal tract length normalization (VTLN) and speaker adaptive training (SAT), or with a simple Gaussian exponentiation scheme that lowers the dynamic range of the Gaussian likelihoods. Evaluation on the Wall Street Journal tasks shows a 5% relative word error rate (WER) reduction over the speaker independent recognition system which already incorporates VTLN. The WER can be lowered further by combining weight adaptation with Gaussian mean adaptation by means of eigenvoice speaker adaptation.

Kernel Eigenvoices (Revisited) for Large-Vocabulary Speech Recognition

Kernelized eigenvoice methods, which apply a nonlinear transform in speaker space, have previously been proposed for rapid adaptation. This paper examines, and addresses, a number of limitations and issues with the current schemes. First, the requirements for valid probability functions using kernel representations are discussed. Second, rapid speaker adaptation using these forms of representations is analyzed and the general update formulae for kernelized eigenvoice adaptation derived. The existing kernelized eigenvoice methods are then described within this formulation. This allows an EM-based, rather than gradient-descent-based, parameter estimation. To enable these approaches to be applied to large-vocabulary speech recognition tasks, eigenbases using transformations of an underlying canonical model are described and related to existing adaptation methods. Preliminary experiments on a large-vocabulary conversational telephone speech task are finally detailed.

Speaker Tracking Using Eigendecomposition and an Index Tree of Reference Models

This paper focuses on online speaker tracking for telephone conversations and broadcast news. Since the online applicability imposes some limitations on the tracking strategy, such as data insufficiency, a reliable approach should be applied to compensate for this shortage. In this framework, a set of reference speaker models are used as side information to facilitate online tracking. To improve the indexing accuracy, adaptation approaches in eigenvoice decomposition space are proposed in this paper. We believe that the eigenvoice adaptation techniques would help to embed the speaker space in the models and hence enrich the generality of the selected speaker models. Also, an index structure of the reference models is proposed to speed up the search in the model space. The proposed framework is evaluated on 2002 Rich Transcription Broadcast News and Conversational Telephone Speech corpus as well as a synthetic dataset. The indexing errors of the proposed framework on telephone conversations, broadcast news, and synthetic dataset are 8.77%, 9.36%, and 12.4%, respectively. Using the index tree structure approach, the run time of the proposed framework is improved by 22%.

Sequential organization of speech in computational auditory scene analysis

A human listener has the ability to follow a speaker’s voice over time in the presence of other talkers and non-speech interference. This paper proposes a general system for sequential organization of speech based on speaker models. By training a general background model, the proposed system is shown to function well with both interfering talkers and non-speech intrusions. To deal with situations where prior information about specific speakers is not available, a speaker quantization method is employed to extract representative models from a large speaker space and obtained generic models are used to perform sequential grouping. Our systematic evaluations show that grouping performance using generic models is only moderately lower than the performance level achieved with known speaker models.

Rapid Speaker Adaptation Using Clustered Maximum-Likelihood Linear Basis With Sparse Training Data

Speaker space-based adaptation methods for automatic speech recognition have been shown to provide significant performance improvements for tasks where only a few seconds of adaptation speech is available. However, these techniques are not widely used in practical applications because they require large amounts of speaker-dependent training data and large amounts of computer memory. The authors propose a robust, low-complexity technique within this general class that has been shown to reduce word error rate, reduce the large storage requirements associated with speaker space approaches, and eliminate the need for large numbers of utterances per speaker in training. The technique is based on representing speakers as a linear combination of clustered linear basis vectors and a procedure is presented for maximum-likelihood estimation of these vectors from training data. Significant word error rate reduction was obtained using these methods relative to speaker independent performance for the Resource Management and Wall Street Journal task domains.

Factorization for generating a library of mouth shapes

A library of mouth shapes is created by separating speaker-dependent and speaker independent variability. Preferably, speaker dependent variability is modeled by a speaker space while the speaker independent variability (i.e. context dependency), is modeled by a set of normalized mouth shapes that need be built only once. Given a small amount of data from a new speaker, it is possible to construct a corresponding mouth shape library by estimating a point in speaker space that maximizes the likelihood of adaptation data and by combining speaker dependent and speaker independent variability. Creation of talking heads is simplified because creation of a library of mouth shapes is enabled with only a few mouth shape instances. To build the speaker space, a context independent mouth shape parametric representation is obtained. Then a supervector containing the set of context-independent mouth shapes is formed for each speaker included in the speaker space. Dimensionality reduction is used to find the areas of the speaker space.

Speaker recognition by location in the space of reference speakers

Speaker representation by location in a reference space is a new technique of speaker recognition and adaptation. It consists in representing a speaker relatively rather than absolutely, by comparing him to a set of well-trained speakers. The main motivation is to obtain a compact modeling of every speaker, which gives similar performances to those of the state of the art GMM-UBM. Thus, instead of estimating numerous parameters of an absolute model of the speaker, only a few parameters of a model relatively to other speaker models called reference speakers are estimated. In this study, several points are addressed that are related to the concept of relative location in speaker recognition. Firstly, the reference speaker space is built. Then the appropriate metrics in this space are investigated in order to perform speaker recognition in a geometrical approach. Finally, a statistical approach for speaker location is used to eliminate the weaknesses of the geometrical approach. In-depth evaluations on a telephone database show that the concept of relative location is a promising technique for speaker verification. Therefore, it can be concluded that the most important motivation for using anchor models is their computational efficiency for indexing tasks.

Speech recognition using interphoneme dependencies based on a speaker space model

Speech recognition requires methods for calculating the acoustic likelihood of a speech sample using acoustic models that represent the spectrum diversity due to interspeaker variation. The acoustic features of a phoneme may vary over a large range between speakers; however, the relative relationships between phonemes are known to exhibit strong dependencies across speakers. In this paper, in consideration of such interphoneme dependencies, we propose a recognition method for speaker-independent speech recognition that uses the features of a given speaker without requiring adaptation data. By performing dimensionality reduction on the space of physical features of the five vowels of Japanese for multiple speakers, we can construct a model of the feature subspace (the speaker space) in which the five vowels of a given speaker will appear. We situate features consisting of the five vowels found in a candidate hypothesis created during speech recognition in the feature vector space and calculate its distance from the speaker space. We then use this distance as a measure of the validity of the hypothesis. Taking an N-best list of hypotheses and using the distance from each to the speaker space in addition to the acoustic model score and performing reranking, we obtained a 25p reduction in recognition errors. © 2005 Wiley Periodicals, Inc. Syst Comp Jpn, 36(8): 15–22, 2005; Published online in Wiley InterScience (). DOI 10.1002sscj.20337

Speech recognition using interphoneme dependencies based on a speaker space model

Speech recognition using interphoneme dependencies based on a speaker space model

Rapid online adaptation using speaker space model evolution

This paper presents a new approach to online adaptation of continuous density hidden Markov model (CDHMM) with a small amount of adaptation data based on speaker space model (SSM) evolution. The SSM which characterizes the a priori knowledge of the training speakers is effectively described in terms of the latent variable models such as the factor analysis or probabilistic principal component analysis. The SSM provides various sources of information such as the correlation information, the prior density, and the prior knowledge of the CDHMM parameters that are very useful for rapid online adaptation. We design the SSM evolution based on the quasi-Bayes estimation technique which incrementally updates the hyperparameters of the SSM and the CDHMM parameters simultaneously. In a series of speaker adaptation experiments on the continuous digit and large vocabulary recognition tasks, we demonstrate that the proposed approach not only achieves a good performance for a small amount of adaptation data but also maintains a good asymptotic convergence property as the data size increases.

Maximum a posteriori adaptation of HMM parameters based on speaker space projection

This paper presents a novel approach to rapid speaker adaptation based on the speaker space projection paradigm in which the adapted model is constrained to lie on a specific subspace spanned by a small number of basis vectors. In order to select the basis vectors that form the speaker space, we apply probabilistic principal component analysis (PPCA) technique to a set of training speaker models represented by a number of hidden Markov models (HMMs). The PPCA incorporates a probability model to the conventional principal component analysis (PCA) method, and finds the speaker space model by means of the expectation maximization (EM) algorithm which is computationally efficient. The PPCA model provides the information of correlation among different speech units as well as the prior probability density function (pdf) associated with each HMM parameter, which can be directly applied to the maximum a posteriori (MAP) adaptation framework. Through a series of supervised adaptation experiments on the tasks of connected digit and large vocabulary recognition, we show that the proposed approach not only achieves a good performance for a small amount of adaptation data but also guarantees a consistent estimate as the data size grows.

Context-dependent acoustic models for medium and large vocabulary speech recognition with eigenvoice training

A reduced dimensionality eigenvoice analytical technique is used during training to develop context-dependent acoustic models for allophones. The eigenvoice technique is also used during run time upon the speech of a new speaker. The technique removes individual speaker idiosyncrasies, to produce more universally applicable and robust allophone models. In one embodiment the eigenvoice technique is used to identify the centroid of each speaker, which may then be “subtracted out” of the recognition equation. In another embodiment maximum likelihood estimation techniques are used to develop common decision tree frameworks that may be shared across all speakers when constructing the eigenvoice representation of speaker space.

Vowel normalization by linear transformation of each speaker's acoustic space

Different speakers uttering phonetically equivalent vowels will not, in general, produce the same set of formant frequencies. Some of these variations are due to global differences in the shape of the acoustic space from one speaker to the next. We have investigated a family of linear transformation techniques for removing these global variations while preserving intervowel relationships. Patterns of interspeaker variation are first determined by PARAFAC three‐way factor analysis of the formant × vowel × speaker data set. The resulting factors suggest dimensions of consistent interspeaker variation which may eventually be identified with physiological or psychological factors affecting vowel production. For the purpose of normalization, each speaker's space is stretched in the directions of the (usually oblique) factor axes, in amounts inversely related to the person‐weights on the factors. This produces a restricted “least squares” optimal removal of individual differences due to these linear common factors of speaker variation. The effects of this normalization procedure will be demonstrated with data from several languages and compared with the results of Gerstman's range equalizing technique and Wakita's tract length normalizing technique. [Work supported by NSF.]