Hidden Markov Model Adaptation Research Articles

Video-realistic expressive audio-visual speech synthesis for the Greek language

High quality expressive speech synthesis has been a long-standing goal towards natural human-computer interaction. Generating a talking head which is both realistic and expressive appears to be a considerable challenge, due to both the high complexity in the acoustic and visual streams and the large non-discrete number of emotional states we would like the talking head to be able to express. In order to cover all the desired emotions, a significant amount of data is required, which poses an additional time-consuming data collection challenge. In this paper we attempt to address the aforementioned problems in an audio-visual context. Towards this goal, we propose two deep neural network (DNN) architectures for Video-realistic Expressive Audio-Visual Text-To-Speech synthesis (EAVTTS) and evaluate them by comparing them directly both to traditional hidden Markov model (HMM) based EAVTTS, as well as a concatenative unit selection EAVTTS approach, both on the realism and the expressiveness of the generated talking head. Next, we investigate adaptation and interpolation techniques to address the problem of covering the large emotional space. We use HMM interpolation in order to generate different levels of intensity for an emotion, as well as investigate whether it is possible to generate speech with intermediate speaking styles between two emotions. In addition, we employ HMM adaptation to adapt an HMM-based system to another emotion using only a limited amount of adaptation data from the target emotion. We performed an extensive experimental evaluation on a medium sized audio-visual corpus covering three emotions, namely anger, sadness and happiness, as well as neutral reading style. Our results show that DNN-based models outperform HMMs and unit selection on both the realism and expressiveness of the generated talking heads, while in terms of adaptation we can successfully adapt an audio-visual HMM set trained on a neutral speaking style database to a target emotion. Finally, we show that HMM interpolation can indeed generate different levels of intensity for EAVTTS by interpolating an emotion with the neutral reading style, as well as in some cases, generate audio-visual speech with intermediate expressions between two emotions.

Text segmentation by language

There are two approaches for text segmentation by language: first, assuming that language changes happen in the “border” between sentences (never within a sentence); second, assuming that language changes can happen anyplace in the text. This work presents methods for both types of text’s segmentation by languages. On the first proposal, the text is initially segmented by sentence, then the language of each sentence is obtained; the second proposal is an adaptation of hidden Markov model to this task. Both cases, according to results obtained in experimental proofs, exceed the state of art.

HMM Adaptation for Improving a Human Activity Recognition System

When developing a fully automatic system for evaluating motor activities performed by a person, it is necessary to segment and recognize the different activities in order to focus the analysis. This process must be carried out by a Human Activity Recognition (HAR) system. This paper proposes a user adaptation technique for improving a HAR system based on Hidden Markov Models (HMMs). This system segments and recognizes six different physical activities (walking, walking upstairs, walking downstairs, sitting, standing and lying down) using inertial signals from a smartphone. The system is composed of a feature extractor for obtaining the most relevant characteristics from the inertial signals, a module for training the six HMMs (one per activity), and the last module for segmenting new activity sequences using these models. The user adaptation technique consists of a Maximum A Posteriori (MAP) approach that adapts the activity HMMs to the user, using some activity examples from this specific user. The main results on a public dataset have reported a significant relative error rate reduction of more than 30%. In conclusion, adapting a HAR system to the user who is performing the physical activities provides significant improvement in the system’s performance.

Vector Taylor series based model adaptation using noisy speech trained hidden Markov models

Conventionally, in vector Taylor series (VTS) based compensation for noise-robust speech recognition, hidden Markov models (HMMs) are usually trained with clean speech. However, it is known that better performance is generally obtained by training the HMM with noisy speech rather than clean speech. From this viewpoint, we propose a novel VTS-based HMM adaptation method for the noisy speech trained HMM. We derive a mathematical relation between the training and test noisy speech in the cepstrum-domain using VTS and the mean and covariance of the noisy speech trained HMM are adapted to the test noisy speech in an iterative expectation-maximization (EM) algorithm. In the experimental results on the Aurora 2 database, we could obtain about 10–25% relative improvements in word error rates (WERs) over multi-condition training (MTR) method depending on speech front-ends and the HMM complexity.

Open-vocabulary recognition of machine-printed Arabic text using hidden Markov models

In this paper, we present multi-font printed Arabic text recognition using hidden Markov models (HMMs). We propose a novel approach to the sliding window technique for feature extraction. The size and position of the cells of the sliding window adapt to the writing line of Arabic text and ink-pixel distributions. We employ a two-step approach for mixed-font text recognition, in which the input text line image is associated with the closest known font in the first step, using simple and effective features for font identification. The text line is subsequently recognized by the recognizer that was trained for the particular font in the next step. This approach proves to be more effective than text recognition using a recognizer trained on samples from multiple fonts. We also present a framework for the recognition of unseen fonts, which employs font association and HMM adaptation techniques. Experiments were conducted using two separate databases of printed Arabic text to demonstrate the effectiveness of the presented techniques. The presented techniques can be easily adapted to other scripts, such as Roman script.

Adaptation of Hidden Markov Models for Recognizing Speech of Reduced Frame Rate

The frame rate of the observation sequence in distributed speech recognition applications may be reduced to suit a resource-limited front-end device. In order to use models trained using full-frame-rate data in the recognition of reduced frame-rate (RFR) data, we propose a method for adapting the transition probabilities of hidden Markov models (HMMs) to match the frame rate of the observation. Experiments on the recognition of clean and noisy connected digits are conducted to evaluate the proposed method. Experimental results show that the proposed method can effectively compensate for the frame-rate mismatch between the training and the test data. Using our adapted model to recognize the RFR speech data, one can significantly reduce the computation time and achieve the same level of accuracy as that of a method, which restores the frame rate using data interpolation.

Synthesis of Child Speech With HMM Adaptation and Voice Conversion

The synthesis of child speech presents challenges both in the collection of data and in the building of a synthesizer from that data. We chose to build a statistical parametric synthesizer using the hidden Markov model (HMM)-based system HTS, as this technique has previously been shown to perform well for limited amounts of data, and for data collected under imperfect conditions. Six different configurations of the synthesizer were compared, using both speaker-dependent and speaker-adaptive modeling techniques, and using varying amounts of data. For comparison with HMM adaptation, techniques from voice conversion were used to transform existing synthesizers to the characteristics of the target speaker. Speaker-adaptive voices generally outperformed child speaker-dependent voices in the evaluation. HMM adaptation outperformed voice conversion style techniques when using the full target speaker corpus; with fewer adaptation data, however, no significant listener preference for either HMM adaptation or voice conversion methods was found.

Adaptation of hidden Markov models for half frame rate observations

In distributed speech recognition applications, the frame rate of observation may be reduced to match a resource limited front end device. A method to adapt the transition probabilities of hidden Markov models to match the frame rate of the observation is proposed. Experimental results on connected digit recognition show that the recognition rates of the proposed adapted system using half frame rate data were comparable to those of the original system using full frame rate data.

Environmental adaptation of hidden Markov models in an automatic sounds recognition system used for surveillance applications

Generally, the main goal of an automatic sound recognition system is to analyze in real time the sound environment of a habitat and to detect the abnormal sounds that could indicate a distress situation. In a surveillance application, the most encountered problem is the background noise that can be captured with the sounds to be identified. Thus, we propose to use a multistyle training system based on HMMs: one recognizer is trained on a database including different levels of background noises and is used as a universal recognizer for every environment. In order to enhance the system robustness, we will explore different adaptation algorithms that reduce the environmental variability. The algorithms closely examined are maximum likelihood linear regression (MLLR), maximum a posteriori (MAP), and the MAP/MLLR algorithm that combines MAP and MLLR. Experimental evaluation illustrates a recognition improvement over the baseline system (i.e., none adapted system) results. By using 156 adaptation scenes, the error rate of the adapted recognizer improves over the baseline system by more than 30% for rather all the considered algorithms, even under important noise degradation conditions.

Observation process adaptation for linear dynamic models

This work introduces two methods for adapting the observation process parameters of linear dynamic models (LDM) or other linear-Gaussian models. The first method uses the expectation-maximization (EM) algorithm to estimate transforms for location and covariance parameters, and the second uses a generalized EM (GEM) approach which reduces computation in making updates from O( p 6) to O( p 3), where p is the feature dimension. We present the results of speaker adaptation on TIMIT phone classification and recognition experiments with relative error reductions of up to 6%. Importantly, we find minimal differences in the results from EM and GEM. We therefore propose that the GEM approach be applied to adaptation of hidden Markov models which use non-diagonal covariances. We provide the necessary update equations.

Adaptation of Hidden Markov Models Using Maximum Model Distance Algorithm

This paper presents a new approach that uses the maximum model distance (MMD) method for the adaptation of hidden Markov models (HMMs). This method has the same framework as it is used for constructing speech recognizers with abundant data, and work effectively with any amount of adaptation data. All parameters of the HMMs with or without the adaptation data could be adapted. If the adaptation data is sufficient, then the adapted models will gradually become a speaker-dependent one. Both the dialect and the speaker adaptation experiments were conducted to investigate the effectiveness of the proposed algorithm. In the speaker adaptation experiments, up to 65.55% phoneme error reduction was achieved, and the MMD could reduce the phoneme error by 16.91% even only one adaptation utterance is available.

Quasi-Bayes linear regression for sequential learning of hidden Markov models

This paper presents an online/sequential linear regression adaptation framework for hidden Markov model (HMM) based speech recognition. Our attempt is to sequentially improve speaker-independent speech recognition system to handle the nonstationary environments via the linear regression adaptation of HMMs. A quasi-Bayes linear regression (QBLR) algorithm is developed to execute the sequential adaptation where the regression matrix is estimated using QB theory. In the estimation, we specify the prior density of regression matrix as a matrix variate normal distribution and derive the pooled posterior density belonging to the same distribution family. Accordingly, the optimal regression matrix can be easily calculated. Also, the reproducible prior/posterior pair provides a meaningful mechanism for sequential learning of prior statistics. At each sequential epoch, only the updated prior statistics and the current observed data are required for adaptation. The proposed QBLR is a general framework with maximum likelihood linear regression (MLLR) and maximum a posteriori linear regression (MAPLR) as special cases. Experiments on supervised and unsupervised speaker adaptation demonstrate that the sequential adaptation using QBLR is efficient and asymptotical to batch learning using MLLR and MAPLR in recognition performance.

Speaker adaptation of HMMs using evolutionary strategy-based linear regression

A new framework for speaker adaptation of continuous-density hidden Markov models (HMMs) is introduced. It aims to improve the robustness of speech recognizers by adapting HMM parameters to new conditions (e.g., from new speakers). It describes an optimization technique using an evolutionary strategy for linear regression-based spectral transformation. In classical iterative maximum likelihood linear regression (MLLR), a global transform matrix is estimated to make a general model better match particular target conditions. To permit adaptation on a small amount of data, a regression tree classification is performed. However, an important drawback of MLLR is that the number of regression classes is fixed. The new approach allows the degree of freedom of the global transform to be implicitly variable, as the evolutionary optimization permits the survival of only active classes. The fitness function is evaluated by the phoneme correctness through the evolution steps. The implementation requirements such as chromosome representation, selection function, genetic operators, and evaluation function have been chosen in order to lend more reliability to the global transformation matrix. Triphone experiments used the timit and arpa−rm1 databases. For new speakers, the new technique achieves 8 percent fewer word errors than the basic MLLR method.

Online adaptation of HMMs to real-life conditions: a unified framework

This paper introduces a unified framework for online adaptation of hidden Markov models (HMM) parameters to real-life conditions. Hence, it aims at improving the robustness of speech recognition systems. In addition, it describes some techniques developed to control the convergence of adaptation in unsupervised modes. Classically, two approaches have been used to adapt HMM parameters to new conditions, that is, Bayesian adaptation and spectral transformation-generally using linear regression. This paper lays out a unifying framework where both Bayesian adaptation and spectral transformation adaptation are seen as particular cases. In this sense, the framework attributes one transformation to each Gaussian distribution and partitions the latter automatically with respect to the adaptation data. Thus, the transformations of each class would share the same parameter vector. Consequently, the global transformation gets a data-driven freedom degree. The parameters of the global transformation are determined according to the maximum a posteriori (MAP) criterion using the original HMM a priori distributions. The general adaptation algorithm has been implemented within the CNET speech recognition system and the whole system evaluated on several field-telephone databases. The new adaptation method provides us with a systematic convergence in an online unsupervised mode of the speech recognition system toward a system enrolled with field data in a supervised mode.

Transformation-based Bayesian prediction for adaptation of HMMs

Due to inaccuracies in the modeling procedure, estimation errors, and poor data to parameter ratios, adaptation techniques can perform poorly when only a limited amount of data is available. Modeling inflexibility, on the other hand, limits their potential when large amounts of data are present. In this paper, we present a transformation-based Bayesian predictive approach to hidden Markov model (HMM) adaptation that addresses the above problems. The new technique, called Bayesian predictive adaptation (BPA), treats adaptation as model evolution arising from attempted transformation of the model parameters. The transformation is a structural representation of the assumed mismatch between the trained models and the adaptation data. Instead of estimating the transformation parameters directly, and blindly treating the estimates as if they are the true values, BPA averages over the variation of the parameters to generate a new model that can be used in the decoding process. By combining the power of Bayesian prediction to take into consideration the errors in estimation and modeling, with the power of transformation based techniques to use fewer parameters for adaptation, the proposed approach creates a new family of techniques that tend to be robust to estimation and modeling errors when only limited data are available, and to modeling inflexibility when large amounts of data are present. We present adaptation results under channel and speaker mismatches, and compare the performance of BPA to other adaptation techniques to demonstrate its effectiveness.

HMM adaptation for applications in telecommunication

The mismatch between the acoustic conditions during training and recognition often causes a performance deterioration in practical applications of speech recognition systems. Two important effects are the presence of a stationary background noise and the frequency response of the transmission channel from the speaker to the audio input of the recognizer. The original contribution of this work are two signal processing schemes for the estimation of the actual noise spectrum and the difference of the frequency responses between training and recognition. The estimated noise components are taken to adapt the cepstral parameters of the recognizer’s references which are described by hidden Markov models (HMMs). The adaptation process is based on the parallel model combination (PMC) approach (M.J.F. Gales, Model based techniques for noise robust speech recognition, Dissertation at the University of Cambridge, 1995). For speaker independent connected or isolated word recognition considerable improvements can be achieved in the presence of just one type of noise as well as in the presence of both types together. Furthermore this adaptation scheme is integrated as part of a complete dialogue and recognition system which is accessible via the public telephone network. The usability and the gain in recognition performance is shown for this application in a real telecommunication scenario under consideration of all real-time aspects.

A minimum cross-entropy approach to hidden Markov model adaptation

An adaptation algorithm using the theoretically optimal maximum a posteriori (MAP) formulation, and at the same time accounting for parameter correlation between different classes is desirable, especially when using sparse adaptation data. However, a direct implementation of such an approach may be prohibitive in many practical situations. We present an algorithm that approximates the above mentioned correlated MAP algorithm by iteratively maximizing the set of posterior marginals. With some simplifying assumptions, expressions for these marginals are then derived, using the principle of minimum cross-entropy. The resulting algorithm is simple, and includes conventional MAP estimation as a special case. The utility of the proposed method is tested in adaptation experiments for an alphabet recognition task.

Maximum-likelihood stochastic-transformation adaptation of hidden Markov models

The recognition accuracy in previous large vocabulary automatic speech recognition (ASR) systems is highly related to the existing mismatch between the training and testing sets. For example, dialect differences across the training and testing speakers result in a significant degradation in recognition performance. Some popular adaptation approaches improve the recognition performance of speech recognizers based on hidden Markov models with continuous mixture densities by using linear transformations to adapt the means, and possibly the covariances of the mixture Gaussians. The linear assumption, however, is too restrictive, and in this paper we propose a novel adaptation technique that adapts the means and, optionally, the covariances of the mixture Gaussians by using multiple stochastic transformations. We perform both speaker and dialect adaptation experiments, and we show that our method significantly improves the recognition accuracy and the robustness of our system. The experiments are carried out with SRI's DECIPHER speech recognition system.

Towards improving ASR robustness for PSN and GSM telephone applications

In real-life applications, errors in the speech recognition system are mainly due to inefficient detection of speech segments, unreliable rejection of Out-Of-Vocabulary (OOV) words, and insufficient account of noise and transmission channel effects. In this paper, we review a set of techniques developed at CNET in order to increase the robustness to mismatches between training and testing conditions. These techniques are divided in two classes: preprocessing techniques and Hidden Markov Models (HMM) parameters adaptation. The results of several experiments carried out on field databases, as well as on databases collected over PSN and GSM networks are presented. The main sources of errors are analyzed. We show that a blind equalization scheme significantly improves the recognition accuracy regarding both field and GSM data. Speech detection allows a system to delimit the boundaries of the words to be recognized. We also use preprocessing techniques to increase the robustness of such detectors to noisy GSM speech. We show that spectral subtraction improves speech detection under noisy GSM conditions. Bayesian adaptation of HMM parameters produces models which are robust to field and GSM conditions. Models robust to GSM conditions can also be generated by linear regression adaptation of HMM parameters. Our experiments show an equivalent performance obtained with both Bayesian and linear regression adaptation of HMM parameters. The results obtained also prove that HMM adaptation and preprocessing techniques can be advantageously combined to improve Automatic Speech Recognition (ASR) robustness.

Telephone speech recognition based on Bayesian adaptation of hidden Markov models

This paper presents an adaptation method of speech hidden Markov models (HMMs) for telephone speech recognition. Our goal is to automatically adapt the HMM parameters so that the adapted HMM parameters can match with the telephone environment. In this study, two kinds of transformation-based adaptations are investigated. One is the bias transformation and the other is the affine transformation. A Bayesian estimation technique which incorporates prior knowledge into the transformation is applied for estimating the transformation parameters. Experiments show that the proposed approach can be successfully employed for self adaptation as well as supervised adaptation. Besides, the performance of telephone speech recognition using Bayesian adaptation is shown to be superior to that using maximum-likelihood adaptation. The affine transformation is also demonstrated to be significantly better than the bias transformation.