DARPA Resource Management Research Articles

Independent vector analysis followed by HMM-based feature enhancement for robust speech recognition

This paper presents a feature-enhancement method that uses the outputs of independent vector analysis (IVA) for robust speech recognition. Although frequency-domain(FD) independent component analysis (ICA) can be successfully used in preprocessing of speech recognition because of its capability of blind source separation (BSS), the performance of the conventional ICA-based approaches is significantly degraded in underdetermined cases. Assuming the target speaker is located relatively close to microphones, the blind spatial subtraction array (BSSA) (Takahashi et al. [10]) tries to enhance target speech features by subtracting noise spectra estimated by FD ICA, even in the underdetermined cases. Unfortunately, the ICA may not be proficient at target speech estimation and then may cause inaccurate noise spectrum estimation. To improve robustness of speech recognition with the inaccurate noise spectra, we introduce Bayesian inference to estimate clean speech features. For a further improvement, FD ICA and delay-and-sum beamforming in the BSSA are replaced with IVA and its target speech output because IVA can improve separation performance without the permutation problem. Experimental results show that the proposed method can further reduce the relative word error rates by 60.11% and 20.07% on average compared to the BSSA for the AURORA2 and DARPA Resource Management databases, respectively.

New algorithms for improved speaker identification

This paper focuses on developing new algorithms for improving speaker identification accuracy. Forty nine male speakers from the DARPA resource management continuous speech database were used for training and testing. Mel-frequency cepstral coefficients (MFCC) components were used for training and testing. Vector quantisation (VQ) was used for classification. In addition to presenting identification results, this paper shows the error reduction rate relative to a baseline system.

Learning-Based Auditory Encoding for Robust Speech Recognition

This paper describes an approach to the optimization of the nonlinear component of a physiologically motivated feature extraction system for automatic speech recognition. Most computational models of the peripheral auditory system include a sigmoidal nonlinear function that relates the log of signal intensity to output level, which we represent by a set of frequency dependent logistic functions. The parameters of these rate-level functions are estimated to maximize the a posteriori probability of the correct class in training data. The performance of this approach was verified by the results of a series of experiments conducted with the CMU S phinx-III speech recognition system on the DARPA Resource Management, Wall Street Journal databases, and on the AURORA 2 database. In general, it was shown that feature extraction that incorporates the learned rate-nonlinearity, combined with a complementary loudness compensation function, results in better recognition accuracy in the presence of background noise than traditional MFCC feature extraction without the optimized nonlinearity when the system is trained on clean speech and tested in noise. We also describe the use of lattice structure that constraints the training process, enabling training with much more complicated acoustic models.

Frequency warping for VTLN and speaker adaptation by linear transformation of standard MFCC

Vocal tract length normalization (VTLN) for standard filterbank-based Mel frequency cepstral coefficient (MFCC) features is usually implemented by warping the center frequencies of the Mel filterbank, and the warping factor is estimated using the maximum likelihood score (MLS) criterion. A linear transform (LT) equivalent for frequency warping (FW) would enable more efficient MLS estimation. We recently proposed a novel LT to perform FW for VTLN and model adaptation with standard MFCC features. In this paper, we present the mathematical derivation of the LT and give a compact formula to calculate it for any FW function. We also show that our LT is closely related to different LTs previously proposed for FW with cepstral features, and these LTs for FW are all shown to be numerically almost identical for the sine-log all-pass transform (SLAPT) warping functions. Our formula for the transformation matrix is, however, computationally simpler and, unlike other previous LT approaches to VTLN with MFCC features, no modification of the standard MFCC feature extraction scheme is required. In VTLN and speaker adaptive modeling (SAM) experiments with the DARPA resource management (RM1) database, the performance of the new LT was comparable to that of regular VTLN implemented by warping the Mel filterbank, when the MLS criterion was used for FW estimation. This demonstrates that the approximations involved do not lead to any performance degradation. Performance comparable to front end VTLN was also obtained with LT adaptation of HMM means in the back end, combined with mean bias and variance adaptation according to the maximum likelihood linear regression (MLLR) framework. The FW methods performed significantly better than standard MLLR for very limited adaptation data (1 utterance), and were equally effective with unsupervised parameter estimation. We also performed speaker adaptive training (SAT) with feature space LT denoted CLTFW. Global CLTFW SAT gave results comparable to SAM and VTLN. By estimating multiple CLTFW transforms using a regression tree, and including an additive bias, we obtained significantly improved results compared to VTLN, with increasing adaptation data.

Tree-Based Covariance Modeling of Hidden Markov Models

In this paper, we present a tree-based, full covariance hidden Markov modeling technique for automatic speech recognition applications. A multilayered tree is built first to organize all covariance matrices into a hierarchical structure. Kullback-Leibler divergence is used in the tree-building to measure inter-Gaussian distortion and successive splitting is used to construct the multilayer covariance tree. To cope with the data sparseness problem in estimating a full covariance matrix, we interpolate the diagonal covariance matrix of a leaf-node at the bottom of the tree with the full covariance of its parent and ancestors along the path up to the root node. The interpolation coefficients are estimated in the maximum likelihood sense via the EM algorithm. The interpolation is performed in three different parametric forms: 1) inverse covariance matrix, 2) covariance matrix, and 3) off-diagonal terms of the full covariance matrix. The proposed algorithm is tested in three different databases: 1) the DARPA resource management (RM), 2) the switchboard, and 3) a Chinese dictation. In all three databases, we show that the proposed tree-based full covariance modeling consistently performs better than the baseline diagonal covariance modeling. The algorithm outperforms other covariance modeling techniques, including: 1) the semi-tied covariance modeling (STC), 2) heteroscedastic linear discriminant analysis (HLDA), 3) mixtures of inverse covariance (MIC), and 4) direct full covariance modeling

A comparison of novel techniques for rapid speaker adaptation

This paper introduces two novel techniques for rapid speaker adaptation, reference speaker weighting and consistency modeling. Also presented is an adaptation technique called speaker cluster weighting (SCW) which provides a means for improving upon generic hierarchical speaker clustering techniques. Each of these adaptation methods attempts to utilize the underlying within-speaker correlations that are present between the acoustic realizations of different phones. By accounting for these correlations, a limited amount of adaptation data can be used to adapt the models of every phonetic acoustic model, including those for phones which have not been observed in the adaptation data. Results were obtained using the DARPA Resource Management corpus for a set of rapid adaptation experiments where single test utterances were used for adaptation and recognition simultaneously. Using the new adaptation techniques relative word error rate reductions ranging from 4.9% to 8.4% were obtained under various conditions. Using a combination of hierarchical speaker clustering techniques and the novel adaptation techniques, a word error rate reduction of 20% has been achieved from the baseline speaker independent (SI) recognition system.

Maximum likelihood modelling of pronunciation variation

This paper addresses the problem of generating lexical word representations that properly represent natural pronunciation variations for the purpose of improved speech recognition accuracy. In order to create a consistent framework for optimisation of automatic speech recognition systems, we present a maximum likelihood based algorithm for fully automatic data-driven modelling of pronunciation, given a set of subword hidden Markov models (HMMs) and acoustic tokens of a word. We also propose an extension of this formulation in order to achieve optimal modelling of pronunciation variations. Since different words will not in general exhibit the same amount of pronunciation variation, the procedure allows words to be represented by a different number of baseforms. The methods improve the subword description of the vocabulary words and have been shown to improve recognition performance on the DARPA Resource Management task.

The importance of cepstral parameter correlations in speech recognition

In this work we demonstrate that explicit modeling of correlations between spectral parameters in speech recognition improves speech models both in terms of their descriptive power (higher likelihoods) and classification accuracy. Most large-vocabulary speech recognition systems are based on some form of hidden Markov models (HMMs) modeling sub-word speech segments. Most of the time speech segments are represented using short term spectra. In this work we employ three-state left-to-right phone models and LPC cepstral parameters including their first and second order time differentials. We investigate the importance of modeling correlations between cepstral parameters for high accuracy phone recognition. Several different types of distributions for each HMM state are compared. The simplest uses a single multivariate Gaussian distribution with a full covariance matrix. The next uses a weighted mixture of multivariate Gaussian distributions with diagonal covariances. It uses implicit rather than explicit modeling of parameter correlations. The most elaborate model employs a mixture of Gaussian distributions, just like the previous model, but in addition it uses a parameter space rotation which is specific to a given state in an HMM. It thus explicitly models parameter correlations in exactly the same way as the simplest model which uses a single distribution per state. The highest phone accuracy on the DARPA Resource Management task Feb 89 test set is obtained using the most elaborate model, with mixtures and space rotation - 82·4% phone accuracy. The next best result was achieved using single distributions, which also explicitly model parameter correlations, with 80·8% phone accuracy. The worst result was obtained using distributions which only implicitly model parameter correlations, achieving 78·7% phone accuracy. These results clearly demonstrate the importance of explicitly modeling parameter correlations for improving speech recognition performance.

Context-dependent connectionist probability estimation in a hybrid hidden Markov model-neural net speech recognition system

In this paper we present a training method and a network architecture for estimating context-dependent observation probabilities in the framework of a hybrid hidden Markov model (HMM)/multi layer perceptron (MLP) speaker-independent continuous speech recognition system. The context-dependent modeling approach we present here computes the HMM context-dependent observation probabilities using a Bayesian factorization in terms of context-conditioned posterior phone probabilities which are computed with a set of MLPs, one for every relevant context. The proposed network architecture shares the input-to-hidden layer among the set of context dependent MLPs in order to reduce the number of independent parameters. Multiple states for phone models with different context dependence for each state are used to model the different context effects at the beginning and end of phonetic segments. A new training procedure that "smooths" networks with different degrees of context dependence is proposed to obtain a robust estimate of the context-dependent probabilities. We have used this new architecture to model generalized biphone phonetic contexts. Tests with the speaker-independent DARPA Resource Management database have shown average reductions in word error rates of 28% using a word-pair grammar, compared to our earlier context-independent HMM/MLP hybrid.

High accuracy phone recognition using context clustering and quasi-triphonic models

A new phone recognizer has been implemented which extends the (phonotactic) decoding constraint to sequences of three phones. It is based on a structure similar to a second order ergodic hidden Markov model (HMM). This kind of a model assumes direct correspondence between the model states and phones, thus constraints on possible state sequences are equivalent to phonotactic constraints. Very high coverage by both left and right context-dependent phone models has been achieved using two methods. The first assumes that some contexts have the same or very similar effect on the phone in question. Thus they are merged into the same contextual class. The outcome is a set of 19 left context classes and 18 right context classes. The second assumes that left context mostly influences the beginning of a phone, whereas the right context influences the end of the phone. Each phone (a state in an ergodic HMM) is represented by a sequence of three probability density functions (pdfs), which is similar to a three state left-to-right HMM. We generate acoustic models such that the first pdf in the model is conditioned on the left context, the middle pdf is context independent (or it can also be context dependent), and the last pdf is conditioned on the right context. A large number of such quasi-triphonic acoustic models can be generated, thus providing a good triphone coverage for a given task, efficiently utilizing the available training data set. The current implementations of the recognizer described here have been applied to the DARPA Resource Management Task to demonstrate feasibility of performing phone (not phoneme) recognition using an untranscribed database, and the TIMIT database, for comparison to existing phone recognition systems. Since true phone sequences for the training utterances are not available for the RM database, they are estimated from text using a phone realization classification tree trained on the TIMIT database transcriptions. The estimates of the true phone sequences are used in training the models and generating reference phone sequences for scoring. The best phone recognition match between the most likely path through the classification tree and the phone recognizer output for the DARPA February 89 test set was 80·5% accurate and 84·0% correct. The best result obtained using the same recognizer structure on the TIMIT database is 69·4% accurate and 74·8% correct, which is a significant improvement over the best published result, when they are both reduced to the same phone set.

An application of recurrent nets to phone probability estimation

This paper presents an application of recurrent networks for phone probability estimation in large vocabulary speech recognition. The need for efficient exploitation of context information is discussed; a role for which the recurrent net appears suitable. An overview of early developments of recurrent nets for phone recognition is given along with the more recent improvements that include their integration with Markov models. Recognition results are presented for the DARPA TIMIT and Resource Management tasks, and it is concluded that recurrent nets are competitive with traditional means for performing phone probability estimation.

A hybrid segmental neural net/hidden Markov model system for continuous speech recognition

The current state-of-the-art in large-vocabulary, continuous speech recognition is based on the use of hidden Markov models (HMM). In an attempt to improve over HMM performance, the authors developed a hybrid system that combines the advantages of neural networks and HMM using a multiple hypothesis (or N-best) paradigm. The connectionist component of the system, the segmental neural net (SNN), models all the frames of a phonetic segment simultaneously, thus overcoming the well-known conditional-independence limitation of the HMM. They describe the hybrid system and discuss various aspects of SNN modeling, including network architectures, training algorithms and context modeling. Finally, they evaluate the hybrid system by performing several speaker-independent experiments with the DARPA Resource Management (RM) corpus, and demonstrate that the hybrid system shows a consistent improvement in performance over the baseline HMM system. >

Connectionist probability estimators in HMM speech recognition

The authors are concerned with integrating connectionist networks into a hidden Markov model (HMM) speech recognition system. This is achieved through a statistical interpretation of connectionist networks as probability estimators. They review the basis of HMM speech recognition and point out the possible benefits of incorporating connectionist networks. Issues necessary to the construction of a connectionist HMM recognition system are discussed, including choice of connectionist probability estimator. They describe the performance of such a system using a multilayer perceptron probability estimator evaluated on the speaker-independent DARPA Resource Management database. In conclusion, they show that a connectionist component improves a state-of-the-art HMM system.

Combining TDNN and HMM in a hybrid system for improved continuous-speech recognition

The paper presents a hybrid continuous-speech recognition system that leads to improved results on the speaker dependent DARPA Resource Management task. This hybrid system, called the combined system, is based on a combination of normalized neural network output scores with hidden Markov model (HMM) emission probabilities. The neural network is trained under mean square error and the HMM is trained under maximum likelihood estimation. In theory, whatever criterion may be used, the same word error rate should be reached if enough training data is available. As this is never the case, the idea of combining two different criteria, each of them extracting complementary characteristics of the feature is interesting. A state-of-the-art HMM system will be combined with a time delay neural network (TDNN) integrated in a Viterbi framework. A hierarchical TDNN structure is described that splits training into subtasks corresponding to subsets of phonemes. This structure makes training of TDNNs on large-vocabulary tasks manageable on workstations. It will be shown that the combined system, despite the low accuracy of the hierarchical TDNN, achieves a word error rate reduction of 15% with respect to our state-of-the-art HMM system. This reduction is obtained with a 10% increase only in the number of parameters.

A comparative study of discrete, semicontinuous, and continuous hidden Markov models

In this paper, we first extended the semicontinuous hidden Markov model to incorporate multiple code-books. The robustness of the semicontinuous output probability is enhanced by the combination of multiple codewords and multiple codebooks. In addition, we compared the semicontinuous model with the continuous mixture model and the discrete model in a large-vocabulary speaker-independent continuous speech recognition (DARPA resource management) task. The model assumption and parameter size issues are addressed in particular through these experiments. When the acoustic parameters are not well modelled by the continuous probability density, the model assumption problems may cause the recognition accuracy of the semicontinuous model or the continuous mixture model to be inferior to the discrete model. We also found that the SCHMM can have a large number of free parameters in comparison with the discrete HMM because of its smoothing ability. With explicit male and female clustered models and for conditional feature sets, we were able to reduce the error rate of discrete-model-based SPHINX by more than 20%.

On speaker-independent, speaker-dependent, and speaker-adaptive speech recognition

The DARPA Resource Management task is used as a domain for investigating the performance of speaker-independent, speaker-dependent, and speaker-adaptive speech recognition. The error rate of the speaker-independent recognition system, SPHINX, was reduced substantially by incorporating between-word triphone models additional dynamic features, and sex-dependent, semicontinuous hidden Markov models. The error rate for speaker-independent recognition was 4.3%. On speaker-dependent data, the error rate was further reduced to 2.6-1.4% with 600-2400 training sentences for each speaker. Using speaker-independent models, the authors studied speaker-adaptive recognition. Both codebooks and output distributions were considered for adaptation. It was found that speaker-adaptive systems outperform both speaker-independent and speaker-dependent systems, suggesting that the most effective system is one that begins with speaker-independent training and continues to adapt to users.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Shared-distribution hidden Markov models for speech recognition

A shared-distribution hidden Markov model (HMM) is presented for speaker-independent continuous speech recognition. The output distributions across different phonetic HMMs are shared with each other when they exhibit acoustic similarity. This sharing provides the freedom to use a larger number of Markov states for each phonetic model. Although an increase in the number of states will increase the total number of free parameters, with distribution sharing one can collapse redundant states while maintaining necessary ones. The shared-distribution model reduced the word error rate on the DARPA Resource Management task by 20% in comparison with the generalized-triphone model.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

On the use of inter-word context-dependent units for word juncture modeling

Word juncture coarticulation is one of the major sources of acoustic variability for initial and final word segments when spoken in fluent speech. One way to improve characterization of word pronuciations in continuous speech is to include inter-word contexts in lexical representations, similar to the way intra-word contexts are utilized. In this paper we investigate the issues related to the modeling of this set of inter-word, context-dependent units using continuous density hidden Markov models. Under such a modeling framework, it is usually required to have enough training tokens available for each unit in order reliably to estimate the parameters of the unit. Therefore, each context-dependent unit is included in the set of units to be modeled only when its frequency of occurrence in the training data exceeds a prescribed threshold. Testing such a unit selection and modeling strategy on the DARPA resource management task, it was found that the incorporation of inter-word units gave a 15–25% word error reduction compared to the base-line continuous speech recognition system using only intra-word units.

Improved acoustic modeling for large vocabulary continuous speech recognition

We report on some recent improvements to an HMM-based, continuous speech recognition system which is being developed at AT&T Bell Laboratories. These advances, which include the incorporation of inter-word, context-dependent units and an improved feature analysis, lead to a recognition system which gives a 95% word accuracy for speaker-independent recognition of the 1000-word DARPA resource management task using the standard word-pair grammar (with a perplexity of about 60). It will be shown that the incorporation of inter-word units into training results in better acoustic models of word juncture coarticulation and gives a 20% reduction in error rate. The effect of an improved set of spectral and log-energy, features is further to reduce word error rate by about 30%. Since we use a continuous density HMM to characterize each subword unit, it is simple and straightforward to add new features to the feature vector (initially a 24-element vector, consisting of 12 cepstral and 12 delta cepstral coefficients). We investigate augmenting the feature vector with 12 second difference (delta-delta) cepstral coefficients and with first (delta) and second difference (delta-delta) log energies, thereby giving a 38-element feature vector. Additional error rate reductions of 11% and 18% were achieved, respectively. With the improved acoustic modeling of subword units, the overall error rate reduction was over 42%. We also found that the spectral vectors, corresponding to the same speech unit, behave differently statistically, depending on whether they are at word boundaries or within a word. The results suggest that intra-word and inter-word units should be modeled independently, even when they appear in the same context. Using a set of subword units which included variants for intra-word and inter-word, context-dependent phones, an additional decrease of about 6–10% in word error rate resulted.

Voice across America: Toward robust speaker-independent speech recognition for telecommunications applications

The demand for telecommunications applications of automatic speech recognition has exploded in recent years. This area seems a natural candidate for speech recognition systems, since it embraces a tremendous variety of applications that rely entirely on audio signals and serial interfaces. However, the telecommunications environment strains the capabilities of current technology, given its broad range of uncontrollable variables, from speaker characteristics to telephone handsets and line quality. Current recognition systems have attained impressive performance levels on relatively controlled tasks, such as speaker-independent continuous digit recognition on laboratory databases comprising a few hundred speakers [l-3]. To comprehend the additional challenges of the telecommunications environment, we must study the effects on recognition of handset and channel characteristics, speaker accent, speaking style, and lexicon, as well as the interactions among these factors. No small amount of data will suffice to model these conditions. Simultaneous with the explosion of telecommunications applications has been the introduction of powerful statistical modeling techniques, known as hidden Markov models (HMMs), to speech recognition [4,5]. These computationally intensive algorithms introduce a large number of degrees of freedom into the speech recognition problem and hence exhibit slow convergence properties. As a consequence, they require orders of magnitude more training data than the previous generation of deterministic techniques. Many databases collected in the mid-l%Os, such as the DARPA Resource Management database [6] and the TIMIT Acoustic Phonetic database [7], while ambitious programs in their own right, have proven to consistently underrepresent important dimensions in HMM recognition systems due to their limited coverage. The Voice Across America (VAA) database being collected at Texas Instruments is designed to satisfy the data requirements of this next generation of speech recognition systems. Our goal is to collect data over standard long-distance telephone lines from 100,000 speakers representing a demographically and geographically balanced sample of the contiguous United States. This database will provide the foundation for a thorough investigation of factors affecting speaker-independent continuous speech recognition for American English. Similar projects are being planned for other countries, and will form the basis for research into recognition of Japanese, British English, and European languages. As of now, we have completed two phases of the VAAproject for a total of 50,000 utterances from nearly 3700 speakers. This paper describes the methods and motivation for VAA data collection and validation procedures, the current contents of the database, and the results of exploratory research on a 1088-speaker subset of the database. Our initial results underscore the need for an extensive database: even 1088 speakers-a large database by traditional standards-are insufficient to adequately represent the many dimensions of interest. One of our purposes here is to share the insights we have gained into telephone-based data collection, in the belief that the VAA model is likely to become the standard method of collecting data over the tele-