Automatic Speech Recognition Performance Research Articles

Data-driven spectral basis functions for automatic speech recognition

Feature extraction plays a major role in any form of pattern recognition. Current feature extraction methods used for automatic speech recognition (ASR) and speaker verification rely mainly on properties of speech production (modeled by all-pole filters) and perception (critical-band integration simulated by Mel/Bark filter bank). We propose to use stochastic methods for designing feature extraction methods which are trained to alleviate the unwanted variability present in speech signals. In this paper we show that such data-driven methods provide significant advantages over the conventional methods both in terms of performance of ASR and in providing understanding about the nature of speech signal. The first part of the paper investigates the suitability of the cepstral features obtained by applying discrete cosine transform on logarithmic critical-band power spectra. An alternate set of basis functions were designed by linear discriminant analysis (LDA) of logarithmic critical-band power spectra. Discriminant features extracted by these alternate basis functions are shown to outperform the cepstral features in ASR experiments. The second part of the paper discusses the relevance of non-uniform frequency resolution used by current speech analysis methods like Mel frequency analysis and perceptual linear predictive analysis. It is shown that LDA of the short-time Fourier spectrum of speech yields spectral basis functions which provide comparatively lower resolution to the high-frequency region of spectrum. This is consistent with critical-band resolution and is shown to be caused by the spectral properties of vowel sounds.

An error-protected speech recognition system for wireless communications

Future wireless multimedia terminals will have a variety of applications that require speech recognition capabilities. We consider a robust distributed speech recognition system where representative parameters of the speech signal are extracted at the wireless terminal and transmitted to a centralized automatic speech recognition (ASR) server. We propose two unequal error protection schemes for the ASR bit stream and demonstrate the satisfactory performance of these schemes for typical wireless cellular channels. In addition, a "soft-feature" error concealment strategy is introduced at the ASR server that uses "soft-outputs" from the channel decoder to compute the marginal distribution of only the reliable features during likelihood computation at the speech recognizer. This soft-feature error concealment technique reduces the ASR error rate by more than a factor of 2.5 for certain channels. Also considered is a channel decoding technique with source information that improves ASR performance.

Analytic assessment of telephone transmission impact on ASR performance using a simulation model

This paper addresses the impact of telephone transmission channels on automatic speech recognition (ASR) performance. A real-time simulation model is described and implemented, which allows impairments that are encountered in traditional as well as modern (mobile, IP-based) networks to be flexibly and efficiently generated. The model is based on input parameters which are known to telephone network planners; thus, it can be applied without measuring specific network characteristics. It can be used for an analytic assessment of the impact of channel impairments on ASR performance, for producing training material with defined transmission characteristics, or for testing spoken dialogue systems in realistic network environments. In the present paper, we present an investigation of the first point. Two speech recognizers which are integrated into a spoken dialogue system for information retrieval are assessed in relation to controlled amounts of transmission degradations. The measured ASR performance degradation is compared to speech quality degradation in human-human communication. It turns out that ASR shows a different behavior than expected human quality judgments for some impairments. This fact has to be taken into account in both telephone network planning as well as in speech and language technology development.

Robust endpoint detection and energy normalization for real-time speech and speaker recognition

When automatic speech recognition (ASR) and speaker verification (SV) are applied in adverse acoustic environments, endpoint detection and energy normalization can be crucial to the functioning of both systems. In low signal-to-noise ratio (SNR) and nonstationary environments, conventional approaches to endpoint detection and energy normalization often fail and ASR performances usually degrade dramatically. The purpose of this paper is to address the endpoint problem. For ASR, we propose a real-time approach. It uses an optimal filter plus a three-state transition diagram for endpoint detection. The filter is designed utilizing several criteria to ensure accuracy and robustness. It has almost invariant response at various background noise levels. The detected endpoints are then applied to energy normalization sequentially. Evaluation results show that the proposed algorithm significantly reduces the string error rates in low SNR situations. The reduction rates even exceed 50% in several evaluated databases. For SV, we propose a batch-mode approach. It uses the optimal filter plus a two-mixture energy model for endpoint detection. The experiments show that the batch-mode algorithm can detect endpoints as accurately as using HMM forced alignment while the proposed one has much less computational complexity.

Creating conversational interfaces for children

Creating conversational interfaces for children is challenging in several respects. These include acoustic modeling for automatic speech recognition (ASR), language and dialog modeling, and multimodal-multimedia user interface design. First, issues in ASR of children's speech are introduced by an analysis of developmental changes in the spectral and temporal characteristics of the speech signal using data obtained from 456 children, ages five to 18 years. Acoustic modeling adaptation and vocal tract normalization algorithms that yielded state-of-the-art ASR performance on children's speech are described. Second, an experiment designed to better understand how children interact with machines using spoken language is described. Realistic conversational multimedia interaction data were obtained from 160 children who played a voice-activated computer game in a Wizard of Oz (WoZ) scenario. Results of using these data in developing novel language and dialog models as well as in a unified maximum likelihood framework for acoustic decoding in ASR and semantic classification for spoken language understanding are described. Leveraging the lessons learned from the WoZ study and a concurrent user experience evaluation, a multimedia personal agent prototype for children was designed. Details of the architecture and application details are described. Informal evaluation by children was found positive especially for the animated agent and the speech interface.

Issues Relating to Automatic Speech Recognition and Spatial Auditory Displays in High Noise, Stressful Tank Environments

The purpose of this study was to determine the extent to which automatic speech recognition (ASR) systems and human speech output to ASR systems are affected when users perform complex tasks in a noisy, stressful simulated tank environment. A second objective was to determine whether the type of auditory information input (monaural or spatial) affected user task performance, including the output of speech to ASR systems. Subjects performed simultaneous radio communication and tank driving tasks. Independent variables included auditory display configuration, driving speed, and subject type (soldier or civilian). Dependent variables included 1) the proportion of radio channel changes correctly recognized by the ASR; 2) the proportion and types of speech errors in radio channel changes; 3) the number of accurate responses to the radio messages; and 4) subject ratings of stress and workload. Subjects were six male U.S. Army personnel and six male civilians. Results indicated that ASR and human speech output were not significantly affected by any of the independent variables. A mean of 95.5% of all subject verbal commands was recognized by the ASR system. Spatialized radio messages significantly enhanced the ability of the user to understand speech communications from multiple channels. However, spatial audio information input had no significant effect on user speech output or on ASR performance. In general, ASR and spatial audio displays have potential as technologies of interest in high noise, stressful tank environments in which complex, concurrent tasks are performed.

On the Determinization of Weighted Finite Automata

We study the problem of constructing the deterministic equivalent of a nondeterministic weighted finite-state automaton (WFA). Determinization of WFAs has important applications in automatic speech recognition (ASR). We provide the first polynomial-time algorithm to test for the twins property, which determines if a WFA admits a deterministic equivalent. We also give upper bounds on the size of the deterministic equivalent; the bound is tight in the case of acyclic WFAs. Previously, Mohri presented a superpolynomial-time algorithm to test for the twins property, and he also gave an algorithm to determinize WFAs. He showed that the latter runs in time linear in the size of the output when a deterministic equivalent exists; otherwise, it does not terminate. Our bounds imply an upper bound on the running time of this algorithm. Given that WFAs can expand exponentially in size when determinized, we explore why those that occur in ASR tend to shrink when determinized. According to ASR folklore, this phenomenon is attributable solely to the fact that ASR WFAs have simple topology, in particular, that they are acyclic and layered. We introduce a very simple class of WFAs with this structure, but we show that the expansion under determinization depends on the transition weights: some weightings cause them to shrink, while others, including random weightings, cause them to expand exponentially. We provide experimental evidence that ASR WFAs exhibit this weight dependence. That they shrink when determinized, therefore, is a result of favorable weightings in addition to special topology. These analyses and observations have been used to design a new, approximate WFA determinization algorithm, reported in a separate paper along with experimental results showing that it achieves significant WFA size reduction with negligible impact on ASR performance.

Interactive Voice Response (IVR) Systems as Automatic Speech Recognition (ASR) Comes of Age

While interactive voice response (IVR) systems were rapidly making their way into the workplace, speech scientists were working hard to improve the performance of automatic speech recognition (ASR) systems to foster their acceptance among potential customers. In the last five years, great strides have been made in this regard, and the commercial use ASR is on the rise. The purpose of this panel is to explore the impact that ASR is (or is not) having on the design of IVR systems that were envisioned originally to operate solely via touch-tone input.

Robust speech recognition using the modulation spectrogram

The performance of present-day automatic speech recognition (ASR) systems is seriously compromised by levels of acoustic interference (such as additive noise and room reverberation) representative of real-world speaking conditions. Studies on the perception of speech by human listeners suggest that recognizer robustness might be improved by focusing on temporal structure in the speech signal that appears as low-frequency (below 16 Hz) amplitude modulations in subband channels following critical-band frequency analysis. A speech representation that emphasizes this temporal structure, the “modulation spectrogram”, has been developed. Visual displays of speech produced with the modulation spectrogram are relatively stable in the presence of high levels of background noise and reverberation. Using the modulation spectrogram as a front end for ASR provides a significant improvement in performance on highly reverberant speech. When the modulation spectrogram is used in combination with log-RASTA-PLP (log RelAtive SpecTrAl Perceptual Linear Predictive analysis) performance over a range of noisy and reverberant conditions is significantly improved, suggesting that the use of multiple representations is another promising method for improving the robustness of ASR systems.

Dialogue design in speech-mediated data-entry: the role of syntactic constraints and feedback

Human-computer interfaces which use speech as the medium for interaction present unique problems for human factors research, due to the fact that automatic speech recognition (ASR) technology is still error prone. The experiments described here address the design of ASR interfaces for data-entry tasks. Particular emphasis was placed on human factors, and users' data-entry performance was compared using not only quantitative measures of speed and accuracy but also more qualitative analyses of user-errors. Experiment 1 investigated the merits of using closed word-sets (syntax) to enhance recognition accuracy. Participants used a purely auditory interface (i.e. one with no visual component to it) programmed to exercise Full Syntactic constraints (FS), Partial Syntactic constraints (PS) or No Syntactic constraints (NS) on the set of words available for recognition at any given time in the data-entry dialogue. Comparisons of data-entry performance showed an advantage of syntax in terms of ASR performance, and when errors and their consequences were taken into account PS was shown to accommodate users' attempts at error-correction more readily than FS. Experiment 2 compared design options for visual prompts and feedback: a limited area of the visual display was dedicated to the provision of prompts and feedback supporting the spoken data-entry dialogue. Two styles of visual prompt were contrasted: Options Prompts (OP) which displayed the full set of current options for input, and Fieldname Prompts (FP) which displayed only the current Fieldname but could be expanded on command to include the relevant options. The results showed that overall OP led to more efficient performance than FP. The errors made by users in the absence of visual feedback were compared with those occurring when the visual component was included in the interface. Recommendations for design of ASR systems for data-entry tasks are made based on the experimental results.

Automatic Speech Recognition in Adverse Environments

Technological developments in automatic speech recognition (ASR) technology over the last couple of decades have resulted in ASR being considered for use in an ever-increasing number of applications. However, a number of physical and psychological environments are still deemed inappropriate for this technology. Several of the stressors creating adverse environments are considered here, in conjunction with their effects on human speech production, human performance in general, and the implications for ASR system design. We propose that a combination of continuing technological developments, together with the application of system engineering and human factors, will lead to ensuring the attainment of robust ASR performance in a number of environments previously considered as high risk.

Speech production models in automatic speech recognition—Forming a lasting marriage between speech science and speech technology

At present, the performance of automatic speech recognition (ASR) systems is still limited by variabilities within and between speakers, by acoustic differences between training and application environments, and by the sensitivity of ASR systems against changing communication channels. This talk considers the conjecture that the use of speech-production models in ASR systems can contribute to making ASR systems more robust with respect to these sources of variability. Although it is well known that production-oriented representations of speech may be used to exploit the continuity of articulatory movements, several obstacles stand in the way of incorporating speech production models in recognizers. These include the difficult problem of acoustic-to-articulatory mapping, the hugh complexity of searching an articulatory space, and the lack of sensitive diagnostic performance metrics for evaluating strengths and weaknesses of a particular production model. Several research laboratories are actively involved in efforts towards incorporating articulatory models in various forms in working ASR systems. In addition to summarizing this work, mechanisms will be suggested for stimulating closer interaction between researchers in production, perception, and processing.

How do humans process and recognize speech?

Until the performance of automatic speech recognition (ASR) hardware surpasses human performance in accuracy and robustness, we stand to gain by understanding the basic principles behind human speech recognition (HSR). This problem was studied exhaustively at Bell Labs between the years of 1918 and 1950 by Harvey Fletcher and his colleagues. The motivation for these studies was to quantify the quality of speech sounds in the telephone plant to both improve speech and preference. To do this he and his group studied the effects of filtering and noise on speech recognition accuracy for nonsense consonant-vowel-consonant (CVC) syllables, words, and sentences. Fletcher used the term as the probability of correct recognition for nonsense sounds, and intelligibility as the probability of correction recognition for words (sounds having meaning). In 1919, Fletcher found a way to transform data for filtered speech into an additive density function D(f) and found a formula that accurately predicts the average articulation. The area under D(S) is called the articulation index. Fletcher then went on to find relationships between the recognition errors for the nonsense speech sounds, words, and sentences. This work has recently been reviewed and partially replicated by Boothroyd and by Bronkhorst, et al. (1980). Taken as a whole, these studies tell us a great deal about how humans process and recognize speech sounds. >

A projection-based likelihood measure for speech recognition in noise

Investigates a projection-based likelihood measure that significantly improves automatic speech recognition performance in the presence of additive broadband noise. The measure was developed by modifying likelihood scores in continuous Gaussian density hidden Markov models (HMMs), resulting in the weighted projection measure (WPM). Experimental results using the proposed measure are reported for several performance factors: different cepstral-based parameters, normal and multistyle speech, and various noise signals, including white, jittering white, and broadband colored noise. In all cases, significant improvements in speaker-dependent, isolated word recognition were achieved using the WPM instead of the standard Gaussian likelihood measure (weighted Euclidean distance (WED)). As an example, at a SNR of 5 dB, the WPM resulted in improvement in recognition accuracy from 19.4 to 80.6% compared with the standard WED for the DFT mel-cepstral representation. >

Statistical treatment of the acoustic effects of vocal effort

Speech recognition performs best when trained and operated in the same environment. Such training is often impractical, especially in military applications. Apart from the difference in the noise environment, talkers change their speech habits when speaking over poor quality channels. The acoustic changes due to increased vocal effort reduce the performance of speech recognizers. While the acoustic effects of vocal effort are varied, it is possible to deal with a bulk of the differences using a statistical approach. A database of two talkers was collected under conditions which produced a variety of degrees of vocal effort. These conditions included both loud sidetone and noise which, when delivered over headphones, change stress and vocal effort. The talkers differed in their reaction to the extreme level in that one shouted and the other did not. Read materials consisted of lists of digits, randomized but the same across talkers. Acoustical parameters such as pitch, amplitude, and word duration were measured and compared across conditions. Using standard statistical procedures, transforms were computed to map the speech from several conditions into a common transformed space. Recognition experiments demonstrated that significant increases in automatic speech recognition performance can be achieved when templates from one condition are matched against unknown speech, when both have been transformed into the common space. An analysis of the recognition errors was made in light of the acoustic differences between vocal effort conditions.

Two useful techniques for automatic speech recognition

We describe two quite separate techniques that can enhance the performance of automatic speech recognition. One is a procedure that, in most task domains, will very significantly restrict the combinatoric explosion of possible interpretations. The other is a metric and a method for optimizing that metric for the discrimination of very similar words (diphones or other tokens). Two traditional paradigms that are similar to desirable properties of speech recognition are medical diagnosis and biological classification. In both fields, the procedures are to use rather simple methods to narrow the search to a few very similar choices, and then to perform specific tests that are optimum to discriminate these specific cases. We describe a technique that will automatically optimize a distance metric to give the best performance in differentiating a vocabulary item from its most confusable neighbors.