Automatic Speech Recognition Models Research Articles

Hidden Markov Acoustic Modeling With Bootstrap and Restructuring for Low-Resourced Languages

This paper proposes an acoustic modeling approach based on bootstrap and restructuring to dealing with data sparsity for low-resourced languages. The goal of the approach is to improve the statistical reliability of acoustic modeling for automatic speech recognition (ASR) in the context of speed, memory and response latency requirements for real-world applications. In this approach, randomized hidden Markov models (HMMs) estimated from the bootstrapped training data are aggregated for reliable sequence prediction. The aggregation leads to an HMM with superior prediction capability at cost of a substantially larger size. For practical usage the aggregated HMM is restructured by Gaussian clustering followed by model refinement. The restructuring aims at reducing the aggregated HMM to a desirable model size while maintaining its performance close to the original aggregated HMM. To that end, various Gaussian clustering criteria and model refinement algorithms have been investigated in the full covariance model space before the conversion to the diagonal covariance model space in the last stage of the restructuring. Large vocabulary continuous speech recognition (LVCSR) experiments on Pashto and Dari have shown that acoustic models obtained by the proposed approach can yield superior performance over the conventional training procedure with almost the same run-time memory consumption and decoding speed.

Semi-supervised learning for speech and audio processing

Semi-supervised learning requires one to make assumptions about the data. This talk will discuss two different assumptions, and algorithms that instantiate those assumptions, for the tasks of acoustic modeling and pronunciation modeling in automatic speech recognition. First, the acoustic spectra corresponding to different phonemes overlap, but there is a tendency for the instantiations of each phoneme to cluster within a well-defined region of the feature space—-a sort of “soft compactness” assumption. Softly compact distributions can be learned by an algorithm that encourages compactness without strictly requiring it, e.g., by maximizing likelihood of the unlabeled data, or even better, by minimizing its conditional class entropy. Second, the observed phone strings corresponding to coarticulated pronunciations of different words are also, often, indistinguishable, but can be transformed into a representation in which the degree of overlap is substantially reduced. The canonical phonetic pronunciations are transformed into an articulatory domain, possible mispronunciations are predicted based on a compactness criterion in the articulatory domain, and the result is transformed back into the phonetic domain, forming a finite state transducer that is able to effectively use hundreds of alternate pronunciations.

Collecting and evaluating speech recognition corpora for 11 South African languages

We describe the Lwazi corpus for automatic speech recognition (ASR), a new telephone speech corpus which contains data from the eleven official languages of South Africa. Because of practical constraints, the amount of speech per language is relatively small compared to major corpora in world languages, and we report on our investigation of the stability of the ASR models derived from the corpus. We also report on phoneme distance measures across languages, and describe initial phone recognisers that were developed using this data. We find that a surprisingly small number of speakers (fewer than 50) and around 10 to 20 h of speech per language are sufficient for the purposes of acceptable phone-based recognition.

An Adaptive Utterance Verification Framework Using Minimum Verification Error Training

This paper introduces an adaptive and integrated utterance verification (UV) framework using minimum verification error (MVE) training as a new set of solutions suitable for real applications. UV is traditionally considered an add-on procedure to automatic speech recognition (ASR) and thus treated separately from the ASR system model design. This traditional two-stage approach often fails to cope with a wide range of variations, such as a new speaker or a new environment which is not matched with the original speaker population or the original acoustic environment that the ASR system is trained on. In this paper, we propose an integrated solution to enhance the overall UV system performance in such real applications. The integration is accomplished by adapting and merging the target model for UV with the acoustic model for ASR based on the common MVE principle at each iteration in the recognition stage. The proposed iterative procedure for UV model adaptation also involves revision of the data segmentation and the decoded hypotheses. Under this new framework, remarkable enhancement in not only recognition performance, but also verification performance has been obtained.

Dual stream speech recognition using articulatory syllable models

Recent theoretical developments in neuroscience suggest that sublexical speech processing occurs via two parallel processing pathways. According to this Dual Stream Model of Speech Processing speech is processed both as sequences of speech sounds and articulations. We attempt to revise the "beads-on-a-string" paradigm of Hidden Markov Models in Automatic Speech Recognition (ASR) by implementing a system for dual stream speech recognition. A baseline recognition system is enhanced by modeling of articulations as sequences of syllables. An efficient and complementary model to HMMs is developed by formulating Dynamic Time Warping (DTW) as a probabilistic model. The DTW Model (DTWM) is improved by enriching syllable templates with constrained covariance matrices, data imputation, clustering and mixture modeling. The resulting dual stream system is evaluated on the N-Best Southern Dutch Broadcast News benchmark. Promising results are obtained for DTWM classification and ASR tests. We provide a discussion on the remaining problems in implementing dual stream speech recognition.

Characterisation and identification of non-native French accents

This paper focuses on foreign accent characterisation and identification in French. How many accents may a native French speaker recognise and which cues does (s)he use? Our interest concentrates on French productions stemming from speakers of six different mother tongues: Arabic, English, German, Italian, Portuguese and Spanish, also compared with native French speakers (from the Île-de-France region). Using automatic speech processing, our objective is to identify the most reliable acoustic cues distinguishing these accents, and to link these cues with human perception. We measured acoustic parameters such as duration and voicing for consonants, the first two formant values for vowels, word-final schwa-related prosodic features and the percentages of confusions obtained using automatic alignment including non-standard pronunciation variants. Machine learning techniques were used to select the most discriminant cues distinguishing different accents and to classify speakers according to their accents. The results obtained in automatic identification of the different linguistic origins under investigation compare favourably to perceptual data. Major identified accent-specific cues include the devoicing of voiced stop consonants, /b/ ∼/v/ and /s / ∼/z/ confusions, the “rolled r” and schwa fronting or raising. These cues can contribute to improve pronunciation modeling in automatic speech recognition of accented speech.

Dealing with noise in automatic speech recognition.

While automatic speech recognition (ASR) can work very well for clean speech, recognition accuracy often degrades significantly when the speech signal is subject to corruption, as occurs in many communication channels. This paper will survey recent methods for handling various distortions in practical ASR. The problem is often presented as an issue of mismatch between the models that are created during prior training phases and unforeseen environmental acoustic conditions that occur during the normal test phase. As one can never anticipate all possible future conditions, ASR analysis must be able to adapt to a wide variety of distortions. Human listeners furnish a useful standard of comparison for ASR in that humans are much more flexible in handling unexpected acoustic distortions than current ASR is. Methods that adapt ASR features and models will be compared against ASR methods that enhance the noisy input speech. Other topics to be discussed will include estimation of noise and channel parameters, RAS...

Phrase classes in two-level language models for ASR

In this work, we propose and compare two different approaches to a two-level language model. Both of them are based on phrase classes but they consider different ways of dealing with phrases into the classes. We provide a complete formulation consistent with the two approaches. The language models proposed were integrated into an Automatic Speech Recognition (ASR) system and evaluated in terms of Word Error Rate. Several series of experiments were carried out over a spontaneous human–machine dialogue corpus in Spanish, where users asked for information about long-distance trains by telephone. It can be extracted from the obtained results that the integration of phrases into classes when using the language models proposed leads to an improvement of the performance of an ASR system. Moreover, the obtained results seem to indicate that the history length with which the best performance is achieved is related to the features of the model itself. Thus, not all the models show the best results with the same value of history length.

Active learning process for spoken dialog systems

A large amount of human labor is required to transcribe and annotate a training corpus that is needed to create and update models for automatic speech recognition (ASR) and spoken language understanding (SLU). Active learning enables a reduction in the amount of transcribed and annotated data required to train ASR and SLU models. In one aspect of the present invention, an active learning ASR process and active learning SLU process are coupled, thereby enabling further efficiencies to be gained relative to a process that maintains an isolation of data in both the ASR and SLU domains.

Integration of Speech Recognition and Machine Translation in Computer-Assisted Translation

Parallel integration of automatic speech recognition (ASR) models and statistical machine translation (MT) models is an unexplored research area in comparison to the large amount of works done on integrating them in series, i.e., speech-to-speech translation. Parallel integration of these models is possible when we have access to the speech of a target language text and to its corresponding source language text, like a computer-assisted translation system. To our knowledge, only a few methods for integrating ASR models with MT models in parallel have been studied. In this paper, we systematically study a number of different translation models in the context of the N-best list rescoring. As an alternative to the N -best list rescoring, we use ASR word graphs in order to arrive at a tighter integration of ASR and MT models. The experiments are carried out on two tasks: English-to-German with an ASR vocabulary size of 17 K words, and Spanish-to-English with an ASR vocabulary of 58 K words. For the best method, the MT models reduce the ASR word error rate by a relative of 18% and 29% on the 17 K and the 58 K tasks, respectively.

A Portable robot audition software system for multiple simultaneous speech signals

Since a robot is deployed in various kinds of environments, the robot audition system should work with minimum prior information on environments to localize, separate and recognize utterances by multiple simultaneous talkers. For example, it should not assume either the number of speakers, the location of speakers for sound source separation (SSS), or specially tuned acoustic model for automatic speech recognition (ASR). We developed a portable robot audition that uses eight microphones installed on the surface of robot's body such as Honda ASIMO, and SIG‐2 and Robovie‐R2 at Kyoto University with different microphone configurations. The system integrates SSS and ASR by using the Missing‐Feature Theory. For SSS, we use Geometric Source Separation (GSS) [Parra 2002] and multichannel post‐filter [Cohen 2002] to separate each utterance. Since separated speech signals are distorted due to interfering talkers and sound source separation, multichannel post‐filter enhanced speech signals. At this process, we create a missing feature mask that specifies which acoustic features are reliable in time‐frequency domain. Multiband Julius [Nishimura 2004], a missing‐feature‐theory based ASR, uses this mask to avoid the influence of unreliable features in recognizing such distorted speech signals. The system demonstrated a waitress that accepts meal orders placed by three actual human talkers and a voice "Rock‐Scissors‐Paper" game referee that decides who wins the game.

Broader range of training voices improves performance of HMM model of phonemic identification

A model proposed by Lin (2005) learns phonetic categories from waveform input. Recorded speech from a set of male talkers is divided into training and test sets. The training set is separated into phonemes, subjected to cepstral analysis, and used as the input to a Hidden Markov Model, which clusters the phonemes into phonemic categories. After this unsupervised learning process, the model is then able to accurately identify speech segments in the test set, showing that relevant acoustic information is captured by the model. The current study explores the outcome when a model of this type is trained on a range of talkers differing in sex and vocal tract configuration. Preliminary results suggest that this approach can improve performance when testing is generalized to a wider range of new talkers. However, too wide a range of training voices reduces accurate categorization, while too narrow a range reduces generalizability. Continuing efforts seek to quantify the optimum range of training voices and to identify the variables that can predict the degree of improvement in performance on test voices. This work has implications for automatic speech recognition models as well as for issues of speaker normalization.

Environmental Independent ASR Model Adaptation/Compensation by Bayesian Parametric Representation

The mismatch between system training and operating conditions can seriously deteriorate the performance of automatic speech recognition (ASR) systems. Various techniques have been proposed to solve this problem in a specified speech environment. Employment of these techniques often involves modification on the ASR system structure. In this paper, we propose an environment-independent (EI) ASR model parameter adaptation approach based on Bayesian parametric representation (BPR), which is able to adapt ASR models to new environments without changing the structure of an ASR system. The parameter set of BPR is optimized by a maximum joint likelihood criterion which is consistent with that of the hidden Markov model (HMM)-based ASR model through an independent expectation-maximization (EM) procedure. Variations of the proposed approach are investigated in the experiments designed in two different speech environments: one is the noisy environment provided by the AURORA 2 database, and the other is the network environment provided by the NTIMIT database. Performances of the proposed EI ASR model compensation approach are compared to those of the cepstral mean normalization (CMN) approach, which is one of the standard techniques for additive noise compensation. The experimental results show that performances of ASR models in different speech environments are significantly improved after being adapted by the proposed BPR model compensation approach

Method and apparatus for rejection of speech recognition results in accordance with confidence level

An automatic speech recognition system for continuous speech recognition of vocabulary words for an autoattendent system proving hand-free telephone calling and utilizing a vocabulary comprising numbers or names of people to be called using known techniques for automatic speech recognition models of word sequencing resulting in high confidence levels of recognition.

Testing the performance of spoken dialogue systems by means of an artificially simulated user

This paper proposes a new technique to test the performance of spoken dialogue systems by artificially simulating the behaviour of three types of user (very cooperative, cooperative and not very cooperative) interacting with a system by means of spoken dialogues. Experiments using the technique were carried out to test the performance of a previously developed dialogue system designed for the fast-food domain and working with two kinds of language model for automatic speech recognition: one based on 17 prompt-dependent language models, and the other based on one prompt-independent language model. The use of the simulated user enables the identification of problems relating to the speech recognition, spoken language understanding, and dialogue management components of the system. In particular, in these experiments problems were encountered with the recognition and understanding of postal codes and addresses and with the lengthy sequences of repetitive confirmation turns required to correct these errors. By employing a simulated user in a range of different experimental conditions sufficient data can be generated to support a systematic analysis of potential problems and to enable fine-grained tuning of the system.

On the relevance of some spectral and temporal patterns for vowel classification

Many previous studies suggested that the information necessary for the identification of vowels from continuous speech is distributed both within and outside vowel boundaries. This information appears to be embedded in the speech signal in the form of various acoustic cues or patterns: spectral, energy, static, dynamic, and temporal. In a recent paper we identified seven types of acoustic patterns that might be exploited by listeners in the identification of coarticulated vowels. The current paper extends the previous study and quantizes the relevance for vowel classification of eight types of acoustic patterns, including static spectral patterns, dynamical spectral patterns, and temporal–durational patterns. Four of these eight patterns are not directly exploited by current automatic speech recognition techniques in computing the likelihood of each phonetic model. These four new patterns proved to contain significant vowel information. Two of these four new patterns represent static spectral patterns lying outside of the currently accepted boundaries of vowels, whereas one is a double-slope dynamical pattern and another one is a simple durational pattern. The findings of this paper may be important for both automatic speech recognition models and models of vowel/phoneme perception by humans.

A probabilistic framework for dialog simulation and optimal strategy learning

The design of Spoken Dialog Systems cannot be considered as the simple combination of speech processing technologies. Indeed, speech-based interface design has been an expert job for a long time. It necessitates good skills in speech technologies and low-level programming. Moreover, rapid development and reusability of previously designed systems remains uneasy. This makes optimality and objective evaluation of design very difficult. The design process is therefore a cyclic process composed of prototype releases, user satisfaction surveys, bug reports and refinements. It is well known that human intervention for testing is time-consuming and above all very expensive. This is one of the reasons for the recent interest in dialog simulation for evaluation as well as for design automation and optimization. In this paper we expose a probabilistic framework for a realistic simulation of spoken dialogs in which the major components of a dialog system are modeled and parameterized thanks to independent data or expert knowledge. Especially, an Automatic Speech Recognition (ASR) system model and a User Model (UM) have been developed. The ASR model, based on articulatory similarities in language models, provides task-adaptive performance prediction and Confidence Level (CL) distribution estimation. The user model relies on the Bayesian Networks (BN) paradigm and is used both for user behavior modeling and Natural Language Understanding (NLU) modeling. The complete simulation framework has been used to train a reinforcement-learning agent on two different tasks. These experiments helped to point out several potentially problematic dialog scenarios.

Can syllabification improve pronunciation by analogy of English?

In spite of difficulty in defining the syllable unequivocally, and controversy over its role in theories of spoken and written language processing, the syllable is a potentially useful unit in several practical tasks which arise in computational linguistics and speech technology. For instance, syllable structure might embody valuable information for building word models in automatic speech recognition, and concatenative speech synthesis might use syllables or demisyllables as basic units. In this paper, we first present an algorithm for determining syllable boundaries in the orthographic form of unknown words that works by analogical reasoning from a database or corpus of known syllabifications. We call this syllabification by analogy (SbA). It is similarly motivated to our existing pronunciation by analogy (PbA) which predicts pronunciations for unknown words (specified by their spellings) by inference from a dictionary of known word spellings and corresponding pronunciations. We show that including perfect (according to the corpus) syllable boundary information in the orthographic input can dramatically improve the performance of pronunciation by analogy of English words, but such information would not be available to a practical system. So we next investigate combining automatically-inferred syllabification and pronunciation in two different ways: the series model in which syllabification is followed sequentially by pronunciation generation; and the parallel model in which syllabification and pronunciation are simultaneously inferred. Unfortunately, neither improves performance over PbA without syllabification. Possible reasons for this failure are explored via an analysis of syllabification and pronunciation errors.

System and methods for acoustic and language modeling for automatic speech recognition with large vocabularies

A method for generating a language component vocabulary VC for a speech recognition system having a language vocabulary V of a plurality of word forms is disclosed. The method includes: partitioning the language vocabulary V into subsets of word forms based on frequencies of occurrence of the respective word forms; and in at least one of the subsets, splitting word forms having frequencies less than a threshold to thereby generate word form components. Also disclosed is a method for use in speech recognition including: splitting an acoustic vocabulary comprising baseforms into baseform components and storing the baseform components; and, performing sound to spelling mapping on the baseform components so as to generate a baseform components to word parts table for use in subsequent decoding of speech. A method for decoding a speech utterance using language model components and acoustic components, includes the steps of: generating from the utterance a stack of baseform component paths; concatenating baseform components in a path to generate concatenated baseforms, when the concatenated baseform components correspond to a baseform found in an acoustic vocabulary; mapping the concatenated baseforms into words; computing language model (LM) scores associated with the words using a language model, and performing further decoding of the utterance based thereupon.

On second-order statistics of log-periodogram with correlated components

We derive an explicit expression for the covariance of the log-periodogram power spectral density estimator for a zero mean Gaussian process. We do not make the assumption that the spectral components of the process are uncorrelated. Applications to spectral estimation and to cepstral modeling in automatic speech recognition are discussed.