Rapid Speaker Adaptation Research Articles

Rapid Speaker Adaptation Based on Combination of KPCA and Latent Variable Model

Speaker adaptation is implemented in order to shift the speaker-independent model closer to the new speaker speech characteristics to improve the speech recognition performance. The kernel eigenspace-based speaker adaptation methods provide satisfactory performance using only a small amount of adaptation data. In such adaptation methods, kernel principal component analysis (KPCA) is applied to the training speaker space in order to create kernel eigenspace. Then, the adapted acoustic model to the new user is calculated in that space. One limitation of KPCA is its inability to define a precise pre-image of the model adapted in the kernel eigenspace, back to the speaker space. Therefore, a huge amount of computations is required to perform adaptation. The previously developed solutions for calculation of an approximate pre-image of the adapted model do not necessarily lead to the optimal conditions. Therefore, in this paper, we propose an efficient solution for this problem to construct more reliable pre-image of the adapted model in the speaker space. For this purpose, we benefit from the latent variable model to define a probabilistic model for description of the applied mapping between the kernel eigenspace and the speaker space. The experiments were conducted on two speech databases: FARSDAT, a Persian, and TIMIT, an English speech database. Implementing a typical HMM-based automatic speech recognition system, it was verified that the proposed method, utilizing about three seconds of adaptation data, achieves up to 4.4% and 7.6% relative phoneme recognition accuracy rate over the speaker-independent model on FARSDAT and TIMIT, respectively. Moreover, the proposed approach demonstrated superior performance compared to the other kernel eigenspace-based adaptation methods.

Unsupervised rapid speaker adaptation based on selective eigenvoice merging for user-specific voice interaction

Speaker adaptation transforms the standard speaker-independent acoustic models into an adapted model relevant to the user (called the target speaker) in order to provide reliable speech recognition performance. Although several conventional adaptation techniques, such as Maximum Likelihood Linear Regression (MLLR) and Maximum A Posteriori (MAP), have been successfully applied to speech recognition tasks, they demonstrate great dependency on the amount of adaptation data. However, the eigenvoice-based adaptation technique is known to provide reliable performance regardless of the amount of data, even for a very small amount. In this study, we propose an efficient eigenvoice adaptation approach to construct more reliable adapted models. The proposed approach merges eigenvoice sets for possible eigenvoice combinations, and then selects optimal eigenvoice sets that are most relevant to the target speaker. For this task, we propose an efficient unsupervised eigenvoice selection method as well as a rapid merging technique. On speech recognition experiments using the Defense Advanced Research Projects Agency׳s Resource Management corpus, the proposed approach exhibited superior performance, compared to conventional methods, in both recognition accuracy and time complexity.

Rapid speaker adaptation using compressive sensing

Speaker-space-based speaker adaptation methods can obtain good performance even if the amount of adaptation data is limited. However, it is difficult to determine the optimal dimension and basis vectors of the subspace for a particular unknown speaker. Conventional methods, such as eigenvoice (EV) and reference speaker weighting (RSW), can only obtain a sub-optimal speaker subspace. In this paper, we present a new speaker-space-based speaker adaptation framework using compressive sensing. The mean vectors of all mixture components of a conventional Gaussian-Mixture-Model-Hidden-Markov-Model (GMM-HMM)-based speech recognition system are concatenated to form a supervector. The speaker adaptation problem is viewed as recovering the speaker-dependent supervector from limited speech signal observations. A redundant speaker dictionary is constructed by a combination of all the training speaker supervectors and the supervectors derived from the EV method. Given the adaptation data, the best subspace for a particular speaker is constructed in a maximum a posterior manner by selecting a proper set of items from this dictionary. Two algorithms, i.e. matching pursuit and l1 regularized optimization, are adapted to solve this problem. With an efficient redundant basis vector removal mechanism and an iterative updating of the speaker coordinate, the matching pursuit based speaker adaptation method is fast and efficient. The matching pursuit algorithm is greedy and sub-optimal, while direct optimization of the likelihood of the adaptation data with an explicit l1 regularization term can obtain better approximation of the unknown speaker model. The projected gradient optimization algorithm is adopted and a few iterations of the matching pursuit algorithm can provide a good initial value. Experimental results show that matching pursuit algorithm outperforms the conventional testing methods under all testing conditions. Better performance is obtained when direct l1 regularized optimization is applied. Both methods can select a proper mixed set of the eigenvoice and reference speaker supervectors automatically for estimation of the unknown speaker models.

Rapid speaker adaptation in latent speaker space with non-negative matrix factorization

A novel speaker adaptation algorithm based on Gaussian mixture weight adaptation is described. A small number of latent speaker vectors are estimated with non-negative matrix factorization (NMF). These latent vectors encode the distinctive systematic patterns of Gaussian usage observed when modeling the individual speakers that make up the training data. Expressing the speaker dependent Gaussian mixture weights as a linear combination of a small number of latent vectors reduces the number of parameters that must be estimated from the enrollment data. The resulting fast adaptation algorithm, using 3s of enrollment data only, achieves similar performance as fMLLR adapting on 100+s of data. In order to learn richer Gaussian usage patterns from the training data, the NMF-based weight adaptation is combined with vocal tract length normalization (VTLN) and speaker adaptive training (SAT), or with a simple Gaussian exponentiation scheme that lowers the dynamic range of the Gaussian likelihoods. Evaluation on the Wall Street Journal tasks shows a 5% relative word error rate (WER) reduction over the speaker independent recognition system which already incorporates VTLN. The WER can be lowered further by combining weight adaptation with Gaussian mean adaptation by means of eigenvoice speaker adaptation.

Vocal Tract Length Normalization for Statistical Parametric Speech Synthesis

Vocal tract length normalization (VTLN) has been successfully used in automatic speech recognition for improved performance. The same technique can be implemented in statistical parametric speech synthesis for rapid speaker adaptation during synthesis. This paper presents an efficient implementation of VTLN using expectation maximization and addresses the key challenges faced in implementing VTLN for synthesis. Jacobian normalization, high-dimensionality features and truncation of the transformation matrix are a few challenges presented with the appropriate solutions. Detailed evaluations are performed to estimate the most suitable technique for using VTLN in speech synthesis. Evaluating VTLN in the framework of speech synthesis is also not an easy task since the technique does not work equally well for all speakers. Speakers have been selected based on different objective and subjective criteria to demonstrate the difference between systems. The best method for implementing VTLN is confirmed to be use of the lower order features for estimating warping factors.

Kernel Eigenvoices (Revisited) for Large-Vocabulary Speech Recognition

Kernelized eigenvoice methods, which apply a nonlinear transform in speaker space, have previously been proposed for rapid adaptation. This paper examines, and addresses, a number of limitations and issues with the current schemes. First, the requirements for valid probability functions using kernel representations are discussed. Second, rapid speaker adaptation using these forms of representations is analyzed and the general update formulae for kernelized eigenvoice adaptation derived. The existing kernelized eigenvoice methods are then described within this formulation. This allows an EM-based, rather than gradient-descent-based, parameter estimation. To enable these approaches to be applied to large-vocabulary speech recognition tasks, eigenbases using transformations of an underlying canonical model are described and related to existing adaptation methods. Preliminary experiments on a large-vocabulary conversational telephone speech task are finally detailed.

참조화자로부터 추정된 적응적 혼성 사전분포를 이용한 MAPLR 고속 화자적응

본 논문은 maximum a posteriori linear regression (MAPLR) 기반의 고속 화자적응 성능을 개선하기 위하여 사전분포를 추정하는 두 가지 방식을 제안한다. 일반적으로 MAPLR 방식에서 사용되는 변환행렬의 사전분포는 화자독립모델을 구성하는 훈련 화자들로부터 추정되어 모든 화자들에게 동등하게 적용된다. 본 논문에서는 새로운 화자에게 보다 더 적합한 사전분포를 적용하고자 적응 데이터를 이용하여 새로운 화자의 음향특성과 가까운 참조화자 집단을 선택한 후 참조화자 집단으로부터 사전분포를 추정하는 방법을 제안한다. 또한, 블록 대각 형태의 변환행렬의 사전분포를 추정하는 경우 사전분포의 평균행렬과 공분산행렬을 동일한 훈련 화자들로부터 얻어진 두 가지 형태의 변환행렬집단으로부터 각각 추정하는 방법을 제안한다. 제안된 방법의 성능 평가를 위하여 고립단어 인식실험을 통해 적응 단어의 개수에 따른 단어 인식률을 평가한다. 실험결과, 적응 단어 수가 매우 적을 때 기존의 MAPLR 방식에 비하여 통계적으로 유의미한 성능향상이 얻어짐을 보여준다. This paper proposes two methods of estimating prior distribution to improve the performance of rapid speaker adaptation based on maximum a posteriori linear regression (MAPLR). In general, prior distribution of the transformation matrix used in MAPLR adaptation is estimated from all of the training speakers who are employed to construct the speaker-independent model, and it is applied identically to all new speakers. In this paper, we propose a method in which prior distribution is estimated from a group of reference speakers, selected using adaptation data, so that the acoustic characteristics of the selected reference speakers may be similar to that of the new speaker. Additionally, in MAPLR adaptation with block-diagonal transformation matrix, we propose a method in which the mean matrix and covariance matrix of prior distribution are estimated from two groups of transformation matrices obtained from the same training speakers, respectively. To evaluate the performance of the proposed methods, we examine word accuracy according to the number of adaptation words in the isolated word recognition task. Experimental results show that, for very limited adaptation data, statistically significant performance improvement is obtained in comparison with the conventional MAPLR adaptation.

Evolutionary eigenvoice MLLR speaker adaptation

Abstract This paper considers the problem of rapid and robust speaker adaptation in automatic speech recognition (ASR) systems. We propose an approach using combination of eigenspace-based maximum likelihood linear regression (EMLLR) and evolutionary algorithms. To find the best solution for the coefficients estimation problem, we suggest using genetic algorithm (GA) for rapid speaker adaptation. This is due to the fact that genetic algorithms are not as sensitive as expectation maximization (EM) algorithm to the amount of adaptation data. Experimental results on TIMIT database illustrate that genetic algorithm, using random individuals in first population, leads to up to 1.03% improvement in phoneme recognition rate. Moreover, we show that if the first population contains coefficients initially estimated by maximum likelihood criterion, further improvement can be achieved as well. However, the amount of adaptation data does not have considerable effect on the proposed method.

A Closed-Form Solution of Linear Spectral Transformation for Robust Speech Recognition

The maximum likelihood linear spectral transformation (ML-LST) using a numerical iteration method has been previously proposed for robust speech recognition. The numerical iteration method is not appropriate for real-time applications due to its computational complexity. In order to reduce the computational cost, the objective function of the ML-LST is approximated and a closed-form solution is proposed in this paper. It is shown experimentally that the proposed closed-form solution for the ML-LST can provide rapid speaker and environment adaptation for robust speech recognition.

Speech separation using speaker-adapted eigenvoice speech models

We present a system for model-based source separation for use on single channel speech mixtures where the precise source characteristics are not known a priori. The sources are modeled using hidden Markov models (HMM) and separated using factorial HMM methods. Without prior speaker models for the sources in the mixture it is difficult to exactly resolve the individual sources because there is no way to determine which state corresponds to which source at any point in time. This is solved to a small extent by the temporal constraints provided by the Markov models, but permutations between sources remains a significant problem. We overcome this by adapting the models to match the sources in the mixture. We do this by representing the space of speaker variation with a parametric signal model-based on the eigenvoice technique for rapid speaker adaptation. We present an algorithm to infer the characteristics of the sources present in a mixture, allowing for significantly improved separation performance over that obtained using unadapted source models. The algorithm is evaluated on the task defined in the 2006 Speech Separation Challenge [Cooke, M.P., Lee, T.-W., 2008. The 2006 Speech Separation Challenge. Computer Speech and Language] and compared with separation using source-dependent models. Although performance is not as good as with speaker-dependent models, we show that the system based on model adaptation is able to generalize better to held out speakers.

Speaker Adaptation With Limited Data Using Regression-Tree-Based Spectral Peak Alignment

Spectral mismatch between training and testing utterances can cause significant degradation in the performance of automatic speech recognition (ASR) systems. Speaker adaptation and speaker normalization techniques are usually applied to address this issue. One way to reduce spectral mismatch is to reshape the spectrum by aligning corresponding formant peaks. There are various levels of mismatch in formant structures. In this paper, regression-tree-based phoneme- and state-level spectral peak alignment is proposed for rapid speaker adaptation using linearization of the vocal tract length normalization (VTLN) technique. This method is investigated in a maximum-likelihood linear regression (MLLR)-like framework, taking advantage of both the efficiency of frequency warping (VTLN) and the reliability of statistical estimations (MLLR). Two different regression classes are investigated: one based on phonetic classes (using combined knowledge and data-driven techniques) and the other based on Gaussian mixture classes. Compared to MLLR, VTLN, and global peak alignment, improved performance can be obtained for both supervised and unsupervised adaptations for both medium vocabulary (the RM1 database) and connected digits recognition (the TIDIGITS database) tasks. Performance improvements are largest with limited adaptation data which is often the case for ASR applications, and these improvements are shown to be statistically significant.

Discriminative cluster adaptive training

Multiple-cluster schemes, such as cluster adaptive training (CAT) or eigenvoice systems, are a popular approach for rapid speaker and environment adaptation. Interpolation weights are used to transform a multiple-cluster, canonical, model to a standard hidden Markov model (HMM) set representative of an individual speaker or acoustic environment. Maximum likelihood training for CAT has previously been investigated. However, in state-of-the-art large vocabulary continuous speech recognition systems, discriminative training is commonly employed. This paper investigates applying discriminative training to multiple-cluster systems. In particular, minimum phone error (MPE) update formulae for CAT systems are derived. In order to use MPE in this case, modifications to the standard MPE smoothing function and the prior distribution associated with MPE training are required. A more complex adaptive training scheme combining both interpolation weights and linear transforms, a structured transform (ST), is also discussed within the MPE training framework. Discriminatively trained CAT and ST systems were evaluated on a state-of-the-art conversational telephone speech task. These multiple-cluster systems were found to outperform both standard and adaptively trained systems

Eigenvoice-based MAP adaptation within correlation subspace

In recent years, the eigenvoice approach has proven to be an efficient method for rapid speaker adaptation, which directs the adaptation according to the analysis of full speaker vector space. In this article, we developed a new algorithm for eigenspace-based adaptation restricting eigenvoices in clustered subspaces, and maximum likelihood (ML) criterion was replaced with maximum aposteriori (MAP) criterion for better parameter estimation. Experiments show that even with one sentence adaptation data this algorithm would result in 6.45 % error ratio reduction relatively, which overcomes the instability of maximum likelihood linear regression (MLLR) with limited data and is much faster than traditional MAP method. This algorithm is not highly-dependent on subspace number of division, thus it proved to be a robust adaptation algorithm.

Segmental eigenvoice with delicate eigenspace for improved speaker adaptation

Eigenvoice techniques have been proposed to provide rapid speaker adaptation with very limited adaptation data, but the performance may be saturated when more adaptation data become available. This is because in these techniques an eigenspace with reduced dimensionality is established by properly utilizing the a priori knowledge from the large quantity of training data. The reduced dimensionality of the eigenspace requires less adaptation data to estimate the model parameters for the new speaker, but also makes it less easy to obtain more precise models with more adaptation data. In this paper, a new segmental eigenvoice approach is proposed, in which the eigenspace can be further segmented into N subeigenspaces by properly classifying the model parameters into N clusters. These N subeigenspaces can help to construct a more delicate eigenspace and more precise models when more adaptation data are available. It will be shown that there can be at least mixture-based, model-based and feature-based segmental eigenvoice approaches. Not only improved performance can be obtained, but these different approaches can be properly integrated to offer better performance. Two further approaches leading to improved segmental eigenvoice techniques with even better performance are also proposed. The experiments were performed with both a large vocabulary and a small vocabulary recognition tasks.

Systematic speaker variation and within-speaker center of gravity correlations in the TIMIT database

The systematicity of speaker variation in consonants was examined by measuring the noise component of consonants in the TIMIT database. Fricatives and stops were compared by measuring at the temporal middle of fricatives and at the release of stops. There is a high correlation between many of the center of gravity measures. For example, if a speaker has a particularly high center of gravity in the sound /t/, she or he will also have a high center of gravity in /d/, /s/, /p/, /k/, and other consonants. The full set of correlations will be described in the paper. The correlations appear to stem from individual differences and not from dialect variations. The implications of the results for rapid speaker adaptation in speech recognition will be explored. [Work supported by San Jose State University Faculty Grant.]

Maximum a posteriori adaptation of HMM parameters based on speaker space projection

This paper presents a novel approach to rapid speaker adaptation based on the speaker space projection paradigm in which the adapted model is constrained to lie on a specific subspace spanned by a small number of basis vectors. In order to select the basis vectors that form the speaker space, we apply probabilistic principal component analysis (PPCA) technique to a set of training speaker models represented by a number of hidden Markov models (HMMs). The PPCA incorporates a probability model to the conventional principal component analysis (PCA) method, and finds the speaker space model by means of the expectation maximization (EM) algorithm which is computationally efficient. The PPCA model provides the information of correlation among different speech units as well as the prior probability density function (pdf) associated with each HMM parameter, which can be directly applied to the maximum a posteriori (MAP) adaptation framework. Through a series of supervised adaptation experiments on the tasks of connected digit and large vocabulary recognition, we show that the proposed approach not only achieves a good performance for a small amount of adaptation data but also guarantees a consistent estimate as the data size grows.

Rapid speaker adaptation using probabilistic principal component analysis

In this letter, we propose a rapid speaker adaptation technique based on the probabilistic principal component analysis (PPCA). The PPCA is employed to obtain the canonical speaker models that provide the a priori knowledge of the training speakers. The proposed approach is conveniently incorporated into the Bayesian adaptation framework, where the parameters are adapted to the new speaker's speech according to the maximum a posteriori (MAP) criterion. Through a number of continuous digit recognition experiments, we can find the effectiveness of the PPCA-based approach compared to the other adaptation approaches with a small amount of adaptation data.

Discounted likelihood linear regression for rapid speaker adaptation

The widely used maximum likelihood linear regression speaker adaptation procedure suffers from overtraining when used for rapid adaptation tasks in which the amount of adaptation data is severely limited. This is a well known difficulty associated with the expectation maximization algorithm. We use an information geometric analysis of the expectation maximization algorithm as an alternating minimization of a Kullback–Leibler-type divergence to see the cause of this difficulty, and propose a more robust discounted likelihood estimation procedure. This gives rise to a discounted likelihood linear regression procedure, which is a variant of maximum likelihood linear regression suited for small adaptation sets. Our procedure is evaluated on an unsupervised rapid adaptation task defined on the Switchboard conversational telephone speech corpus, where our proposed procedure improves word error rate by 1.6% (absolute) with as little as 5 seconds of adaptation data, which is a situation in which maximum likelihood linear regression overtrains in the first iteration of adaptation. We compare several realizations of discounted likelihood linear regression with maximum likelihood linear regression and other simple maximum likelihood linear regression variants, and discuss issues that arise in implementing our discounted likelihood procedures.

A comparison of novel techniques for rapid speaker adaptation

This paper introduces two novel techniques for rapid speaker adaptation, reference speaker weighting and consistency modeling. Also presented is an adaptation technique called speaker cluster weighting (SCW) which provides a means for improving upon generic hierarchical speaker clustering techniques. Each of these adaptation methods attempts to utilize the underlying within-speaker correlations that are present between the acoustic realizations of different phones. By accounting for these correlations, a limited amount of adaptation data can be used to adapt the models of every phonetic acoustic model, including those for phones which have not been observed in the adaptation data. Results were obtained using the DARPA Resource Management corpus for a set of rapid adaptation experiments where single test utterances were used for adaptation and recognition simultaneously. Using the new adaptation techniques relative word error rate reductions ranging from 4.9% to 8.4% were obtained under various conditions. Using a combination of hierarchical speaker clustering techniques and the novel adaptation techniques, a word error rate reduction of 20% has been achieved from the baseline speaker independent (SI) recognition system.

Rapid speaker adaptation in eigenvoice space

This paper describes a new model-based speaker adaptation algorithm called the approach. The approach constrains the adapted model to be a linear combination of a small number of basis vectors obtained offline from a set of reference speakers, and thus greatly reduces the number of free parameters to be estimated from adaptation data. These eigenvoice basis vectors are orthogonal to each other and guaranteed to represent the most important components of variation between the reference speakers. Experimental results for a small-vocabulary task (letter recognition) given in the paper show that the approach yields major improvements in performance for tiny amounts of adaptation data. For instance, we obtained 16% relative improvement in error rate with one letter of supervised adaptation data, and 26% relative improvement with four letters of supervised adaptation data. After a comparison of the approach with other speaker adaptation algorithms, the paper concludes with a discussion of future work.