Acoustic Model Adaptation Research Articles

Online adaptation of fourier series-based acoustic transfer function model and its application to sound source localization and separation

In this paper, we propose an online adaptation method for Fourier series-based acoustic transfer function (FS-ATF) models for robot audition systems using microphone array signal processing. The ATF represents the characteristics of signal propagation from a sound source to a microphone, which is essential for sound source localization (SSL) and sound source separation (SSS). For practical applications in dynamically changing real-world environments, ATF models must satisfy two criteria: (1) they must be adaptable to changes in the acoustic environment; and (2) they must be lightweight to be suitable for resource-constrained systems, such as robots with limited memory and computational capacity. The proposed method addresses these challenges using Fourier series expansions for interpolation, which reduces the memory footprint of the ATF model and facilitates online adaptation to acoustic environmental changes. The experimental results demonstrate that the proposed online adaptation method both improves the SSL and SSS performance while reducing the size of the ATF model, which represents a significant improvement over existing online ATF adaptation methods.

End-to-end emotional speech recognition using acoustic model adaptation based on knowledge distillation.

The end-to-end approach provides better performance in speech recognition compared to the traditional hidden Markov model-deep neural network (HMM-DNN)-based approach, but still shows poor performance in abnormal speech, especially emotional speech. The optimal solution is to build an acoustic model suitable for emotional speech recognition using only emotional speech data for each emotion, but it is impossible because it is difficult to collect sufficient amount of emotional speech data for each emotion. In this study, we propose a method to improve the emotional speech recognition performance by using the knowledge distillation technique that was originally introduced to decrease computational intensity of deep learning-based approaches by reducing the number of model parameters. In addition to its use as model compression, we employ this technique for model adaptation to emotional speech. The proposed method builds a basic model (referred to as a teacher model) with a number of model parameters using an amount of normal speech data, and then constructs a target model (referred to as a student model) with fewer model parameters using a small amount of emotional speech data (i.e., adaptation data). Since the student model is built with emotional speech data, it is expected to reflect the emotional characteristics of each emotion well. In the emotional speech recognition experiment, the student model maintained recognition performance regardless of the number of model parameters, whereas the teacher model degraded performance significantly as the number of parameters decreased, showing performance degradation of about 10% in word error rate. This result demonstrates that the student model serves as an acoustic model suitable for emotional speech recognition even though it does not require much emotional speech data.

Malaysian English Large Vocabulary Continuous Speech Recognizer: An Improvement Using Acoustic Model Adapation

This research project aims to develop Malaysian English Continuous Speech Recognition system by adapting US English acoustic model with Malaysian English speech corpus using Maximum a posteriori reasoning (MAP) and Maximum Likelihood Linear Regression (MLLR). During feature extraction stage, the Mel-Frequency Cepstral Coefficients (MFCC) technique was used. The Hidden Markov Model was used as the back end pattern comparison technique. For the purpose of implementation, the CMU Sphinx toolkit, which includes Pocketsphinx and Sphinxtrain as well as an acoustic model, was used to develop a speech recognition system for Malaysian English. Malaysian English speech sample will be recorded and transcribed to produce the training database required for acoustic model adaptation. The adaptation speech corpus were collected from a number of speakers. The outcome of this research could increase the application of Malaysian English speech recognition in Malaysia due to accent problem. As a result, speech recognition systems that have gone through the MAP adaptation had the best performance. Its average word error rate achieved was 32.84%. Average word recognition rate was 72.52% and average sentence error rate was 78.89%.

Prediction Method of Underwater Acoustic Transmission Loss Based on Deep Belief Net Neural Network

The prediction of underwater acoustic transmission loss in the sea plays a key role in generating situational awareness in complex naval battles and assisting underwater operations. However, the traditional classical underwater acoustic transmission loss models do not consider the regional hydrological elements, and the performance of underwater acoustic transmission loss prediction under complex environmental conditions in a wide range of sea areas is limited. In order to solve this problem, we propose a deep learning-based underwater acoustic transmission loss prediction method. First, we studied the application domains of typical underwater acoustic transmission loss models (ray model, normal model, fast field program model, parabolic equation model), analyzed the constraint rules of its characteristic parameters, and constructed a dataset according to the rules. Then, according to the characteristics of the dataset, we built a DBN (deep belief net) neural network model and used DBN to train and learn the dataset. Through the DBN method, the adaptation and calculation of the underwater acoustic transmission loss model under different regional hydrological elements were carried out in a simulation environment. Finally, the new method was verified with the measured transmission loss data of acoustic sea trials in a certain sea area. The results show that the RMSE error between the underwater acoustic transmission loss calculated by the new method and the measured data was less than 6.5 dB, the accuracy was higher than that of the traditional method, and the prediction speed was faster, the result was more accurate, and had a wide range of adaptability in complex seas.

Normal-to-Lombard adaptation of speech synthesis using long short-term memory recurrent neural networks

In this article, three adaptation methods are compared based on how well they change the speaking style of a neural network based text-to-speech (TTS) voice. The speaking style conversion adopted here is from normal to Lombard speech. The selected adaptation methods are: auxiliary features (AF), learning hidden unit contribution (LHUC), and fine-tuning (FT). Furthermore, four state-of-the-art TTS vocoders are compared in the same context. The evaluated vocoders are: GlottHMM, GlottDNN, STRAIGHT, and pulse model in log-domain (PML). Objective and subjective evaluations were conducted to study the performance of both the adaptation methods and the vocoders. In the subjective evaluations, speaking style similarity and speech intelligibility were assessed. In addition to acoustic model adaptation, phoneme durations were also adapted from normal to Lombard with the FT adaptation method. In objective evaluations and speaking style similarity tests, we found that the FT method outperformed the other two adaptation methods. In speech intelligibility tests, we found that there were no significant differences between vocoders although the PML vocoder showed slightly better performance compared to the three other vocoders.

Cross-lingual adaptation of a CTC-based multilingual acoustic model

Cross-lingual adaptation of a CTC-based multilingual acoustic model

Cross-lingual adaptation of a CTC-based multilingual acoustic model

Multilingual models for Automatic Speech Recognition (ASR) are attractive as they have been shown to benefit from more training data, and better lend themselves to adaptation to under-resourced languages. However, initialisation from monolingual context-dependent models leads to an explosion of context-dependent states. Connectionist Temporal Classification (CTC) is a potential solution to this as it performs well with monophone labels.We investigate multilingual CTC training in the context of adaptation and regularisation techniques that have been shown to be beneficial in more conventional contexts. The multilingual model is trained to model a universal International Phonetic Alphabet (IPA)-based phone set using the CTC loss function. Learning Hidden Unit Contribution (LHUC) is investigated to perform language adaptive training. During cross-lingual adaptation, the idea of extending the multilingual output layer to new phonemes is introduced and investigated. In addition, dropout during multilingual training and cross-lingual adaptation is also studied and tested in order to mitigate the overfitting problem.Experiments show that the performance of the universal phoneme-based CTC system can be improved by applying dropout and LHUC and it is extensible to new phonemes during cross-lingual adaptation. Updating all acoustic model parameters shows consistent improvement on limited data. Applying dropout during adaptation can further improve the system and achieve competitive performance with Deep Neural Network / Hidden Markov Model (DNN/HMM) systems on limited data.

TEDxSK and JumpSK: A New Slovak Speech Recognition Dedicated Corpus

Abstract This paper describes a new Slovak speech recognition dedicated corpus built from TEDx talks and Jump Slovakia lectures. The proposed speech database consists of 220 talks and lectures in total duration of about 58 hours. Annotated speech database was generated automatically in an unsupervised manner by using acoustic speech segmentation based on principal component analysis and automatic speech transcription using two complementary speech recognition systems. The evaluation data consisting of 50 manually annotated talks and lectures in total duration of about 12 hours, has been created for evaluation of the quality of Slovak speech recognition. By unsupervised automatic annotation of TEDx talks and Jump Slovakia lectures we have obtained 21.26% of new speech segments with approximately 9.44% word error rate, suitable for retraining or adaptation of acoustic models trained beforehand.

A Fast Speaker Adaptation Approach for Large Vocabulary Continuous Speech Recognition

Acoustic model adaptation is an efficient way to compensate the acoustic mismatch due to speaker variability. We propose a linear projection (LP) based on transformation matrix speaker adaptation approach to reduce the amount of the required adaptation data. The method adopts LP function to transform multiple sets of speaker adapted (SA) acoustic models into a new single set of models. To save storage space, the transformation matrix obtained by maximum likelihood linear regression (MLLR) is used to represent the specific speaker information. Maximum likelihood (ML) criterion is used to select the SA models with the most important information from the candidate models in order to reduce the amount of transformation parameters. Supervised and unsupervised speaker adaptation experiments are carried out on a large vocabulary continuous speech recognition task to evaluate its effectiveness. Experimental results show that the proposed approach can realize fast adaptation.

I-Vectors and Structured Neural Networks for Rapid Adaptation of Acoustic Models

A lot of interest has been risen in the last years on the adaptation of deep neural network (DNN) acoustic models, as the latter become the state-of-art in automatic speech recognition. This work focuses on approaches that allow for rapid and robust adaptation of such models. First, i-vectors are added to the DNN input as speaker-informed features. An informative prior is introduced to i-vector estimation to improve the robustness to limited adaptation data. I-vectors are then combined with a structured adaptive DNN, the multibasis adaptive neural network (MBANN), and the complementarity of these adaptation techniques is investigated. Moreover, i-vectors are used to predict the MBANN transforms, avoiding the initial decoding pass and alignment. These approaches are evaluated on a U.S. English Broadcast News (BN) transcription task with two distinct sets of test data. The first, from the BN task and BN-style Youtube videos, yields test data acoustically matched to the training data, while the second set is from acoustically mismatched Youtube videos of diverse context. The performance gains from these schemes are found to be sensitive to the level of mismatch between training and test sets. The MBANN system combined with i-vector input achieves best performance for BN test sets. The i-vector-based predictive MBANN scheme is proven to be more robust to acoustically mismatched conditions and outperforms the other adaptation schemes in such scenarios.

A combined evaluation of established and new approaches for speech recognition in varied reverberation conditions

Robustness to reverberation is a key concern for distant-microphone ASR. Various approaches have been proposed, including single-channel or multichannel dereverberation, robust feature extraction, alternative acoustic models, and acoustic model adaptation. However, to the best of our knowledge, a detailed study of these techniques in varied reverberation conditions is still missing in the literature. In this paper, we conduct a series of experiments to assess the impact of various dereverberation and acoustic model adaptation approaches on the ASR performance in the range of reverberation conditions found in real domestic environments. We consider both established approaches such as WPE and newer approaches such as learning hidden unit contribution (LHUC) adaptations, whose performance has not been reported before in this context, and we employ them in combination. Our results indicate that performing weighted prediction error (WPE) dereverberation on a reverberated test speech utterance and decoding using a deep neural network (DNN) acoustic model trained with multi-condition reverberated speech with feature-space maximum likelihood linear regression (fMLLR) transformed features, outperforms more recent approaches and helps significantly reduce the word error rate (WER).

Recent progresses in deep learning based acoustic models

In this paper, we summarize recent progresses made in deep learning based acoustic models and the motivation and insights behind the surveyed techniques. We first discuss models such as recurrent neural networks U+0028 RNNs U+0029 and convolutional neural networks U+0028 CNNs U+0029 that can effectively exploit variablelength contextual information, and their various combination with other models. We then describe models that are optimized end-to-end and emphasize on feature representations learned jointly with the rest of the system, the connectionist temporal classification U+0028 CTC U+0029 criterion, and the attention-based sequenceto-sequence translation model. We further illustrate robustness issues in speech recognition systems, and discuss acoustic model adaptation, speech enhancement and separation, and robust training strategies. We also cover modeling techniques that lead to more efficient decoding and discuss possible future directions in acoustic model research.

Gaussian mixture models for adaptation of deep neural network acoustic models in automatic speech recognition systems

Subject of Research. We study speaker adaptation of deep neural network (DNN) acoustic models in automatic speech recognition systems. The aim of speaker adaptation techniques is to improve the accuracy of the speech recognition system for a particular speaker. Method. A novel method for training and adaptation of deep neural network acoustic models has been developed. It is based on using an auxiliary GMM (Gaussian Mixture Models) model and GMMD (GMM-derived) features. The principle advantage of the proposed GMMD features is the possibility of performing the adaptation of a DNN through the adaptation of the auxiliary GMM. In the proposed approach any methods for the adaptation of the auxiliary GMM can be used, hence, it provides a universal method for transferring adaptation algorithms developed for GMMs to DNN adaptation.Main Results. The effectiveness of the proposed approach was shown by means of one of the most common adaptation algorithms for GMM models – MAP (Maximum A Posteriori) adaptation. Different ways of integration of the proposed approach into state-of-the-art DNN architecture have been proposed and explored. Analysis of choosing the type of the auxiliary GMM model is given. Experimental results on the TED-LIUM corpus demonstrate that, in an unsupervised adaptation mode, the proposed adaptation technique can provide, approximately, a 11–18% relative word error reduction (WER) on different adaptation sets, compared to the speaker-independent DNN system built on conventional features, and a 3–6% relative WER reduction compared to the SAT-DNN trained on fMLLR adapted features.

Differentiable Pooling for Unsupervised Acoustic Model Adaptation

We present a deep neural network (DNN) acoustic model that includes parametrised and differentiable pooling operators. Unsupervised acoustic model adaptation is cast as the problem of updating the decision boundaries implemented by each pooling operator. In particular, we experiment with two types of pooling parametrisations: learned $L_p$-norm pooling and weighted Gaussian pooling, in which the weights of both operators are treated as speaker-dependent. We perform investigations using three different large vocabulary speech recognition corpora: AMI meetings, TED talks and Switchboard conversational telephone speech. We demonstrate that differentiable pooling operators provide a robust and relatively low-dimensional way to adapt acoustic models, with relative word error rates reductions ranging from 5--20% with respect to unadapted systems, which themselves are better than the baseline fully-connected DNN-based acoustic models. We also investigate how the proposed techniques work under various adaptation conditions including the quality of adaptation data and complementarity to other feature- and model-space adaptation methods, as well as providing an analysis of the characteristics of each of the proposed approaches.

Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation

This work presents a broad study on the adaptation of neural network acoustic models by means of learning hidden unit contributions (LHUC) -- a method that linearly re-combines hidden units in a speaker- or environment-dependent manner using small amounts of unsupervised adaptation data. We also extend LHUC to a speaker adaptive training (SAT) framework that leads to a more adaptable DNN acoustic model, working both in a speaker-dependent and a speaker-independent manner, without the requirements to maintain auxiliary speaker-dependent feature extractors or to introduce significant speaker-dependent changes to the DNN structure. Through a series of experiments on four different speech recognition benchmarks (TED talks, Switchboard, AMI meetings, and Aurora4) comprising 270 test speakers, we show that LHUC in both its test-only and SAT variants results in consistent word error rate reductions ranging from 5% to 23% relative depending on the task and the degree of mismatch between training and test data. In addition, we have investigated the effect of the amount of adaptation data per speaker, the quality of unsupervised adaptation targets, the complementarity to other adaptation techniques, one-shot adaptation, and an extension to adapting DNNs trained in a sequence discriminative manner.

A Bayesian view on acoustic model-based techniques for robust speech recognition

This article provides a unifying Bayesian view on various approaches for acoustic model adaptation, missing feature, and uncertainty decoding that are well-known in the literature of robust automatic speech recognition. The representatives of these classes can often be deduced from a Bayesian network that extends the conventional hidden Markov models used in speech recognition. These extensions, in turn, can in many cases be motivated from an underlying observation model that relates clean and distorted feature vectors. By identifying and converting the observation models into a Bayesian network representation, we formulate the corresponding compensation rules. We thus summarize the various approaches as approximations or modifications of the same Bayesian decoding rule leading to a unified view on known derivations as well as to new formulations for certain approaches.

Differenced maximum mutual information criterion for robust unsupervised acoustic model adaptation

Discriminative criteria have been widely used for training acoustic models for automatic speech recognition (ASR). Many discriminative criteria have been proposed including maximum mutual information (MMI), minimum phone error (MPE), and boosted MMI (BMMI). Discriminative training is known to provide significant performance gains over conventional maximum-likelihood (ML) training. However, as discriminative criteria aim at direct minimization of the classification error, they strongly rely on having accurate reference labels. Errors in the reference labels directly affect the performance. Recently, the differenced MMI (dMMI) criterion has been proposed for generalizing conventional criteria such as BMMI and MPE. dMMI can approach BMMI or MPE if its hyper-parameters are properly set. Moreover, dMMI introduces intermediate criteria that can be interpreted as smoothed versions of BMMI or MPE. These smoothed criteria are robust to errors in the reference labels. In this paper, we demonstrate the effect of dMMI on unsupervised speaker adaptation where the reference labels are estimated from a first recognition pass and thus inevitably contain errors. In particular, we introduce dMMI-based linear regression (dMMI-LR) adaptation and demonstrate significant gains in performance compared with MLLR and BMMI-LR in two large vocabulary lecture recognition tasks.

Advanced acoustic modelling techniques in MP3 speech recognition

The automatic recognition of MP3 compressed speech presents a challenge to the current systems due to the lossy nature of compression which causes irreversible degradation of the speech wave. This article evaluates the performance of a recognition system optimized for MP3 compressed speech with current state-of-the-art acoustic modelling techniques and one specific front-end compensation method. The article concentrates on acoustic model adaptation, discriminative training, and additional dithering as prominent means of compensating for the described distortion in the task of phoneme and large vocabulary continuous speech recognition (LVCSR). The experiments presented on the phoneme task show a dramatic increase of the recognition error for unvoiced speech units as a direct result of compression. The application of acoustic model adaptation has proved to yield the highest relative contribution while the gain of discriminative training diminished with decreasing bit-rate. The application of additional dithering yielded a consistent improvement only for the MFCC features, but the overall results were still worse than those for the PLP features.

Bounded Conditional Mean Imputation with Observation Uncertainties and Acoustic Model Adaptation

Automatic speech recognition systems use noise compensation and acoustic model adaptation to increase robustness towards speaker and environmental variation. The current work focuses on noise compensation with bounded conditional mean imputation (BCMI). BCMI approaches are missing-data methods which operate on the assumption that noise-corrupted observations can be divided into reliable and unreliable components. BCMI methods substitute the unreliable components with a clean speech posterior distribution. The posterior means can be used as clean speech estimates and the posterior variances can be introduced in acoustic model likelihood calculation as observation uncertainties. In addition, we propose in the current work that similar uncertainties are introduced in acoustic model adaptation. Evaluation with speech data recorded in diverse public and car environments indicates that the proposed uncertainties improve adaptation performance. When uncertainties were used in acoustic model likelihood calculation and adaptation, the proposed imputation and adaptation system introduced 15%-84% relative error reductions to an uncompensated baseline system performance.

Reverberant speech recognition exploiting clarity index estimation

We present single-channel approaches to robust automatic speech recognition (ASR) in reverberant environments based on non-intrusive estimation of the clarity index (C 50). Our best performing method includes the estimated value of C 50 in the ASR feature vector and also uses C 50 to select the most suitable ASR acoustic model according to the reverberation level. We evaluate our method on the REVERB Challenge database employing two different C 50 estimators and show that our method outperforms the best baseline of the challenge achieved without unsupervised acoustic model adaptation, i.e. using multi-condition hidden Markov models (HMMs). Our approach achieves a 22.4 % relative word error rate reduction in comparison to the best baseline of the challenge.