Small Amount Of Adaptation Data Research Articles

Rapid Speaker Adaptation Based on Combination of KPCA and Latent Variable Model

Speaker adaptation is implemented in order to shift the speaker-independent model closer to the new speaker speech characteristics to improve the speech recognition performance. The kernel eigenspace-based speaker adaptation methods provide satisfactory performance using only a small amount of adaptation data. In such adaptation methods, kernel principal component analysis (KPCA) is applied to the training speaker space in order to create kernel eigenspace. Then, the adapted acoustic model to the new user is calculated in that space. One limitation of KPCA is its inability to define a precise pre-image of the model adapted in the kernel eigenspace, back to the speaker space. Therefore, a huge amount of computations is required to perform adaptation. The previously developed solutions for calculation of an approximate pre-image of the adapted model do not necessarily lead to the optimal conditions. Therefore, in this paper, we propose an efficient solution for this problem to construct more reliable pre-image of the adapted model in the speaker space. For this purpose, we benefit from the latent variable model to define a probabilistic model for description of the applied mapping between the kernel eigenspace and the speaker space. The experiments were conducted on two speech databases: FARSDAT, a Persian, and TIMIT, an English speech database. Implementing a typical HMM-based automatic speech recognition system, it was verified that the proposed method, utilizing about three seconds of adaptation data, achieves up to 4.4% and 7.6% relative phoneme recognition accuracy rate over the speaker-independent model on FARSDAT and TIMIT, respectively. Moreover, the proposed approach demonstrated superior performance compared to the other kernel eigenspace-based adaptation methods.

An Adaptation Method in Noise Mismatch Conditions for DNN-based Speech Enhancement

The deep learning based speech enhancement has shown considerable success. However, it still suffers performance degradation under mismatch conditions. In this paper, an adaptation method is proposed to improve the performance under noise mismatch conditions. Firstly, we advise a noise aware training by supplying identity vectors (i-vectors) as parallel input features to adapt deep neural network (DNN) acoustic models with the target noise. Secondly, given a small amount of adaptation data, the noise-dependent DNN is obtained by using L2 regularization from a noise-independent DNN, and forcing the estimated masks to be close to the unadapted condition. Finally, experiments were carried out on different noise and SNR conditions, and the proposed method has achieved significantly 0.1%-9.6% benefits of STOI, and provided consistent improvement in PESQ and segSNR against the baseline systems.

A DNN-based text-to-speech synthesis system using speaker, gender, and age codes

Adaptation and manipulation techniques for creating various characteristics of synthetic speech are important research topics in the speech synthesis field. In this work, we investigate the performance of a DNN-based text-to-speech synthesis system that uses speaker, gender, and age codes as well as the text inputs (1) for modeling speaker-independent models called “average voice models,” (2) for performing speaker adaptation using a small amount of adaptation data, and also (3) for manipulating characteristics of synthetic speech based on the codes. For these purposes, we extracted a set of studio-quality speech data uttered by 68 males and 70 females, whose age vary between 10 and 80, from our large-scale Japanese corpus and carried out the three experiments: (1) We constructed a DNN-based speaker-independent model using one-hot vectors representing a set of the above speakers. (2) We performed speaker adaptation by estimating a code vector for a new speaker via the back-propagation. (3) We performed manual manipulation of the code vector to modify perceived characteristics, gender, and/or age of synthetic speech. Experimental results showed that high-performance speaker-independent models can be constructed using the proposed code vectors and additionally that adaptation and manipulation using the codes can also be performed effectively.

Speaker Adaptation of Hybrid NN/HMM Model for Speech Recognition Based on Singular Value Decomposition

Recently several speaker adaptation methods have been proposed for deep neural network (DNN) in many large vocabulary continuous speech recognition (LVCSR) tasks. However, only a few methods rely on tuning the connection weights in trained DNNs directly to optimize system performance since it is very prone to over-fitting especially when some class labels are missing in the adaptation data. In this paper, we propose a new speaker adaptation method for the hybrid NN/HMM speech recognition model based on singular value decomposition (SVD). We apply SVD on the weight matrices in trained DNNs and then tune rectangular diagonal matrices with the adaptation data. This alleviates the over-fitting problem via updating the weight matrices slightly by only modifying the singular values. We evaluate the proposed adaptation method in two standard speech recognition tasks, namely TIMIT phone recognition and large vocabulary speech recognition in the Switchboard task. Experimental results have shown that it is effective to adapt large DNN models using only a small amount of adaptation data. For example, recognition results in the Switchboard task have shown that the proposed SVD-based adaptation method may achieve up to 3-6 % relative error reduction using only a few dozens of adaptation utterances per speaker.

Fast Adaptation of Deep Neural Network Based on Discriminant Codes for Speech Recognition

Fast adaptation of deep neural networks (DNN) is an important research topic in deep learning. In this paper, we have proposed a general adaptation scheme for DNN based on discriminant condition codes, which are directly fed to various layers of a pre-trained DNN through a new set of connection weights. Moreover, we present several training methods to learn connection weights from training data as well as the corresponding adaptation methods to learn new condition code from adaptation data for each new test condition. In this work, the fast adaptation scheme is applied to supervised speaker adaptation in speech recognition based on either frame-level cross-entropy or sequence-level maximum mutual information training criterion. We have proposed three different ways to apply this adaptation scheme based on the so-called speaker codes: i) Nonlinear feature normalization in feature space; ii) Direct model adaptation of DNN based on speaker codes; iii) Joint speaker adaptive training with speaker codes. We have evaluated the proposed adaptation methods in two standard speech recognition tasks, namely TIMIT phone recognition and large vocabulary speech recognition in the Switchboard task. Experimental results have shown that all three methods are quite effective to adapt large DNN models using only a small amount of adaptation data. For example, the Switchboard results have shown that the proposed speaker-code-based adaptation methods may achieve up to 8-10% relative error reduction using only a few dozens of adaptation utterances per speaker. Finally, we have achieved very good performance in Switchboard (12.1% in WER) after speaker adaptation using sequence training criterion, which is very close to the best performance reported in this task (“Deep convolutional neural networks for LVCSR,” T. N. Sainath , Proc. IEEE Acoust., Speech, Signal Process., 2013).

Low Complexity On-Line Adaptation Techniques in Context of Assamese Spoken Query System

In this work, we present the development of an Assamese spoken query (SQ) system for accessing the price of agricultural commodities. The developed system intends to make the cultivators aware of the recent market trends. The SQ system enables the user to access the latest price of the commodity by calling the system using a landline/mobile phone. The spoken query input by the user is processed and then current price of the desired commodity in the given district is played back by the system. Features that make the system user friendly are incorporated into the design after taking feedbacks from local farmers. In other words, the system is tuned as per the needs of the users. Furthermore, the issues of adapting such query systems to the end user are also explored in this work. In case of the developed SQ system, the typical user responses are of extremely small duration (1---2 seconds due to isolated word response from the user). Moreover, the employed adaptation approach must keep the system latency low since these systems are meant for real-time applications. Consequently, adapting such systems to the end user becomes an extremely challenging task. In this regard, acoustic model interpolation based adaptation techniques are proposed that employ interpolation weights derived in an approximate fashion. The proposed approaches try to minimize the latency in the system response by avoiding the iterative weight estimation procedure used in the earlier reported works. Even with extremely small amount of adaptation data, the proposed approaches are found to result in a relative improvement of 12 % over the baseline ASR system.

Applying Multi- and Cross-Lingual Stochastic Phone Space Transformations to Non-Native Speech Recognition

In the context of hybrid HMM/MLP Automatic Speech Recognition (ASR), this paper describes an investigation into a new type of stochastic phone space transformation, which maps “source” phone (or phone HMM state) posterior probabilities (as obtained at the output of a Multilayer Perceptron/MLP) into “destination” phone (HMM phone state) posterior probabilities. The resulting stochastic matrix transformation can be used within the same language to automatically adapt to different phone formats (e.g., IPA) or across languages. Additionally, as shown here, it can also be applied successfully to non-native speech recognition. In the same spirit as MLLR adaptation, or MLP adaptation, the approach proposed here is directly mapping posterior distributions, and is trained by optimizing on a small amount of adaptation data a Kullback-Leibler based cost function, along a modified version of an iterative EM algorithm. On a non-native English database (HIWIRE), and comparing with multiple setups (monophone and triphone mapping, MLLR adaptation) we show that the resulting posterior mapping yields state-of-the-art results using very limited amounts of adaptation data in mono-, cross- and multi-lingual setups. We also show that “universal” phone posteriors, trained on a large amount of multilingual data, can be transformed to English phone posteriors, resulting in an ASR system that significantly outperforms a system trained on English data only. Finally, we demonstrate that the proposed approach outperforms alternative data-driven, as well as a knowledge-based, mapping techniques.

Speaker adaptation in the maximum a posteriori framework based on the probabilistic 2-mode analysis of training models

In this article, we describe a speaker adaptation method based on the probabilistic 2-mode analysis of training models. Probabilistic 2-mode analysis is a probabilistic extension of multilinear analysis. We apply probabilistic 2-mode analysis to speaker adaptation by representing each of the hidden Markov model mean vectors of training speakers as a matrix, and derive the speaker adaptation equation in the maximum a posteriori (MAP) framework. The adaptation equation becomes similar to the speaker adaptation equation using the MAP linear regression adaptation. In the experiments, the adapted models based on probabilistic 2-mode analysis showed performance improvement over the adapted models based on Tucker decomposition, which is a representative multilinear decomposition technique, for small amounts of adaptation data while maintaining good performance for large amounts of adaptation data.

Hermitian Polynomial for Speaker Adaptation of Connectionist Speech Recognition Systems

Model adaptation techniques are an efficient way to reduce the mismatch that typically occurs between the training and test condition of any automatic speech recognition (ASR) system. This work addresses the problem of increased degradation in performance when moving from speaker-dependent (SD) to speaker-independent (SI) conditions for connectionist (or hybrid) hidden Markov model/artificial neural network (HMM/ANN) systems in the context of large vocabulary continuous speech recognition (LVCSR). Adapting hybrid HMM/ANN systems on a small amount of adaptation data has been proven to be a difficult task, and has been a limiting factor in the widespread deployment of hybrid techniques in operational ASR systems. Addressing the crucial issue of speaker adaptation (SA) for hybrid HMM/ANN system can thereby have a great impact on the connectionist paradigm, which will play a major role in the design of next-generation LVCSR considering the great success reported by deep neural networks - ANNs with many hidden layers that adopts the pre-training technique - on many speech tasks. Current adaptation techniques for ANNs based on injecting an adaptable linear transformation network connected to either the input, or the output layer are not effective especially with a small amount of adaptation data, e.g., a single adaptation utterance. In this paper, a novel solution is proposed to overcome those limits and make it robust to scarce adaptation resources. The key idea is to adapt the hidden activation functions rather than the network weights. The adoption of Hermitian activation functions makes this possible. Experimental results on an LVCSR task demonstrate the effectiveness of the proposed approach.

Maximum likelihood subband polynomial regression for robust speech recognition

In this paper, we propose a model adaptation algorithm based on maximum likelihood subband polynomial regression (MLSPR) for robust speech recognition. In this algorithm, the cepstral mean vectors of prior trained hidden Markov models (HMMs) are converted to the log-spectral domain by the inverse discrete cosine transform (DCT) and each log-spectral mean vector is divided into several subband vectors. The relationship between the training and testing subband vectors is approximated by a polynomial function. The polynomial coefficients are estimated from adaptation data using the expectation–maximization (EM) algorithm under the maximum likelihood (ML) criterion. The experimental results show that the proposed MLSPR algorithm is superior to both the maximum likelihood linear regression (MLLR) adaptation and maximum likelihood subband weighting (MLSW) approach. In the MLSPR adaptation, only a very small amount of adaptation data is required and therefore it is more useful for fast model adaptation.

Adaptation of Hidden Markov Models Using Model-as-Matrix Representation

In this paper, we describe basis-based speaker adaptation techniques using the matrix representation of training models. Bases are obtained from training models by decomposition techniques for matrix-variate objects: two-dimensional principal component analysis (2DPCA) and generalized low rank approximations of matrices (GLRAM). The motivation for using matrix representation is that the sample covariance matrix of training models can be more accurately computed and the speaker weight becomes a matrix. Speaker adaptation equations are derived in the maximum-likelihood (ML) framework, and the adaptation equations can be solved using the maximum-likelihood linear regression technique. Additionally, novel applications of probabilistic 2DPCA and GLRAM to speaker adaptation are presented. From the probabilistic 2DPCA/GLRAM of training models, speaker adaptation equations are formulated in the maximum a posteriori (MAP) framework. The adaptation equations can be solved using the MAP linear regression technique. In the isolated-word experiments, the matrix representation-based methods in the ML and MAP frameworks outperformed maximum-likelihood linear regression adaptation, MAP adaptation, eigenvoice, and probabilistic PCA-based model for adaptation data longer than 20 s. Furthermore, the adaptation methods using probabilistic 2DPCA/GLRAM showed additional performance improvement over the adaptation methods using 2DPCA/GLRAM for small amounts of adaptation data.

Speech Recognition under Multiple Noise Environment Based on Multi-Mixture HMM and Weight Optimization by the Aspect Model

In this paper, we propose an acoustic model that is robust to multiple noise environments, as well as a method for adapting the acoustic model to an environment to improve the model. The model is called “the multi-mixture model, ” which is based on a mixture of different HMMs each of which is trained using speech under different noise conditions. Speech recognition experiments showed that the proposed model performs better than the conventional multi-condition model. The method for adaptation is based on the aspect model, which is a “mixture-of-mixture” model. To realize adaptation using extremely small amount of adaptation data (i.e., a few seconds), we train a small number of mixture models, which can be interpreted as models for “clusters” of noise environments. Then, the models are mixed using weights, which are determined according to the adaptation data. The experimental results showed that the adaptation based on the aspect model improved the word accuracy in a heavy noise environment and showed no performance deterioration for all noise conditions, while the conventional methods either did not improve the performance or showed both improvement and degradation of recognition performance according to noise conditions.

Acoustic model adaptation for ortolan bunting (Emberiza hortulana L.) song-type classification

Automatic systems for vocalization classification often require fairly large amounts of data on which to train models. However, animal vocalization data collection and transcription is a difficult and time-consuming task, so that it is expensive to create large data sets. One natural solution to this problem is the use of acoustic adaptation methods. Such methods, common in human speech recognition systems, create initial models trained on speaker independent data, then use small amounts of adaptation data to build individual-specific models. Since, as in human speech, individual vocal variability is a significant source of variation in bioacoustic data, acoustic model adaptation is naturally suited to classification in this domain as well. To demonstrate and evaluate the effectiveness of this approach, this paper presents the application of maximum likelihood linear regression adaptation to ortolan bunting (Emberiza hortulana L.) song-type classification. Classification accuracies for the adapted system are computed as a function of the amount of adaptation data and compared to caller-independent and caller-dependent systems. The experimental results indicate that given the same amount of data, supervised adaptation significantly outperforms both caller-independent and caller-dependent systems.

Rapid adaptation using linear spectral transformation for embedded speech recognisers

Embedded speech recognisers are typically used in unknown mobile environments where the acoustic conditions frequently change. Since a large amount of adaptation data is not usually available for such environments, the adaptation methods for the acoustic models of these recognisers must improve the recognition performance with only a small amount of adaptation data. In this Letter, we show that maximum likelihood linear spectral transformation provides the advantage of rapid adaptation using a very limited amount of adaptation data for the embedded acoustic models.

Kernel eigenvoice speaker adaptation

Eigenvoice-based methods have been shown to be effective for fast speaker adaptation when only a small amount of adaptation data, say, less than 10 s, is available. At the heart of the method is principal component analysis (PCA) employed to find the most important eigenvoices. In this paper, we postulate that nonlinear PCA using kernel methods may be even more effective. The eigenvoices thus derived will be called kernel eigenvoices (KEV), and we will call our new adaptation method kernel eigenvoice speaker adaptation. However, unlike the standard eigenvoice (EV) method, an adapted speaker model found by the kernel eigenvoice method resides in the high-dimensional kernel-induced feature space, which, in general, cannot be mapped back to an exact preimage in the input speaker supervector space. Consequently, it is not clear how to obtain the constituent Gaussians of the adapted model that are needed for the computation of state observation likelihoods during the estimation of eigenvoice weights and subsequent decoding. Our solution is the use of composite kernels in such a way that state observation likelihoods can be computed using only kernel functions without the need of a speaker-adapted model in the input supervector space. In this paper, we investigate two different composite kernels for KEV adaptation: direct sum kernel and tensor product kernel. In an evaluation on the TIDIGITS task, it is found that KEV speaker adaptation using both forms of composite Gaussian kernels are equally effective, and they outperform a speaker-independent model and adapted models found by EV, MAP, or MLLR adaptation using 2.1 and 4.1 s of speech. For example, with 2.1 s of adaptation data, KEV adaptation outperforms the speaker-independent model by 27.5%, whereas EV, MAP, or MLLR adaptation are not effective at all.

Unsupervised training of phoneme models using HMM sufficient statistics and a speaker distance function

AbstractWe propose an unsupervised method for the training of phoneme models using sufficient statistics from hidden Markov models and a speaker distance function. Our proposed method uses sufficient statistics calculated for acoustic models of speakers that are acoustically close to a test speaker to construct an accurately adapted model. The method involves (1) the selection of a group of speakers that are acoustically close to the test speaker and (2) the creation of phoneme models that are adapted to the test speaker using sufficient statistics from the models of the selected group of speakers. The calculation of sufficient statistics is performed prior to adaptation off‐line. The proposed method results in high recognition rates by performing adaptation based on the calculations of sufficient statistics of models of speakers that are acoustically close to the test speaker. In addition, only a small amount of adaptation data is required. The adaptation is performed rapidly since the sufficient statistics can be calculated off‐line. In comparison with speaker clustering methods, our proposed method allows for a more appropriate determination of the speaker cluster since this is selected dynamically on‐line using data from the test speaker. We show that in recognition experiments with small amounts of adaptation data, our proposed method results in a higher recognition rate than MLLR. © 2005 Wiley Periodicals, Inc. Electron Comm Jpn Pt 3, 88(9): 33–41, 2005; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecjc.20188

Rapid online adaptation based on transformation space model evolution

This paper presents a new approach to online linear regression adaptation of continuous density hidden Markov models based on transformation space model (TSM) evolution. The TSM which characterizes the a priori knowledge of the training speakers associated with maximum likelihood linear regression matrix parameters is effectively described in terms of the latent variable models such as the factor analysis or probabilistic principal component analysis. The TSM provides various sources of information such as the correlation information, the prior distribution, and the prior knowledge of the regression parameters that are very useful for rapid adaptation. The quasi-Bayes estimation algorithm is formulated to incrementally update the hyperparameters of the TSM and regression matrices simultaneously. The proposed TSM evolution is a general framework with batch TSM adaptation as a special case. Experiments on supervised speaker adaptation demonstrate that the proposed approach is more effective compared with the conventional quasi-Bayes linear regression technique when a small amount of adaptation data is available.

Rapid online adaptation using speaker space model evolution

This paper presents a new approach to online adaptation of continuous density hidden Markov model (CDHMM) with a small amount of adaptation data based on speaker space model (SSM) evolution. The SSM which characterizes the a priori knowledge of the training speakers is effectively described in terms of the latent variable models such as the factor analysis or probabilistic principal component analysis. The SSM provides various sources of information such as the correlation information, the prior density, and the prior knowledge of the CDHMM parameters that are very useful for rapid online adaptation. We design the SSM evolution based on the quasi-Bayes estimation technique which incrementally updates the hyperparameters of the SSM and the CDHMM parameters simultaneously. In a series of speaker adaptation experiments on the continuous digit and large vocabulary recognition tasks, we demonstrate that the proposed approach not only achieves a good performance for a small amount of adaptation data but also maintains a good asymptotic convergence property as the data size increases.

Maximum a posteriori adaptation of HMM parameters based on speaker space projection

This paper presents a novel approach to rapid speaker adaptation based on the speaker space projection paradigm in which the adapted model is constrained to lie on a specific subspace spanned by a small number of basis vectors. In order to select the basis vectors that form the speaker space, we apply probabilistic principal component analysis (PPCA) technique to a set of training speaker models represented by a number of hidden Markov models (HMMs). The PPCA incorporates a probability model to the conventional principal component analysis (PCA) method, and finds the speaker space model by means of the expectation maximization (EM) algorithm which is computationally efficient. The PPCA model provides the information of correlation among different speech units as well as the prior probability density function (pdf) associated with each HMM parameter, which can be directly applied to the maximum a posteriori (MAP) adaptation framework. Through a series of supervised adaptation experiments on the tasks of connected digit and large vocabulary recognition, we show that the proposed approach not only achieves a good performance for a small amount of adaptation data but also guarantees a consistent estimate as the data size grows.

Model‐based speaker normalization methods for speech recognition

AbstractA speaker normalization method using a speech generation model is proposed in order to achieve high‐performance speaker adaptation with a small amount of adaptation data. The speaker‐ and phoneme‐dependent vocal tract area function is approximated by the corresponding area function produced by the articulatory model of a standard speaker, combined with phoneme‐independent feature quantities of the vocal‐tract shape of the normalized target speaker as estimated from the formant frequencies of two vowels. The frequency warping functions are determined from the formant frequencies of speech calculated from the vocal‐tract area functions thus obtained, and normalization of the uttered speech is performed by stretching the speech spectrum in the frequency‐axis direction. Continuous phoneme recognition experiments using phoneme connection rules show that the recognition error using a gender‐dependent model is reduced by about 30% in the proposed method and that recognition performance superior to that of vocal‐tract length normalization is obtained. The recognition performance of the proposed method is also equivalent to that of speaker adaptation by moving vector field smoothing (VFS) using 10 phonetically balanced sentences, showing that high‐performance speaker adaptation using a small amount of adaptation data can be achieved by the proposed method. © 2003 Wiley Periodicals, Inc. Electron Comm Jpn Pt 2, 86(2): 45–56, 2003; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecjb.10119