Automatic Speech Recognition Applications Research Articles

A Study of web-based oral activities enhanced by Automatic Speech Recognition for EFL college learning

Recently, a promising topic in computer-assisted language learning is the application of Automatic Speech Recognition (ASR) technology for assisting learners to engage in meaningful speech interactions. Simulated real-life conversation supported by the application of ASR has been suggested as helpful for speaking. In this study, a web-based conversation environment called CandleTalk, which allows learners to seemingly talk with the computer, was developed to help EFL learners receive explicit speech acts training that leads to better oral competence. CandleTalk is equipped with an ASR engine that judges whether learners provide appropriate input. Six speech acts are presented as the foci of the materials with local cultural information incorporated as the content of the dialogues to enhance student motivation. The materials were put to use on 29 English major and 20 non-English major students in order to investigate their learning outcome and perception in an EFL context. Oral proficiency assessment using the format of the Discourse Completion Test (DCT) given before and after the use of CandleTalk and an evaluation questionnaire were two instruments used for data collection. The results of the study showed that the application of ASR was helpful for the college freshmen in the teaching of speech acts, particularly for the non-English major students. Most learners perceived positively toward the instruction supported with speech recognition.

Acoustic variability and automatic recognition of children’s speech

This paper presents several acoustic analyses carried out on read speech collected from Italian children aged from 7 to 13 years and North American children aged from 5 to 17 years. These analyses aimed at achieving a better understanding of spectral and temporal changes in speech produced by children of various ages in view of the development of automatic speech recognition applications. The results of these analyses confirm and complement the results reported in the literature, showing that characteristics of children’s speech change with age and that spectral and temporal variability decrease as age increases. In fact, younger children show a substantially higher intra- and inter-speaker variability with respect to older children and adults. We investigated the use of several methods for speaker adaptive acoustic modeling to cope with inter-speaker spectral variability and to improve recognition performance for children. These methods proved to be effective in recognition of read speech with a vocabulary of about 11k words.

The use of speed-up techniques for a speech recognizer system

In speech recognition, not just the accuracy of an automatic speech recognition application is important, but also its speed. However, if we want to create a real-time speech recognizer, this requirement limits the time that is spent on searching for the best hypothesis, which can even affect the recognition accuracy. Thus the applied search method plays an important role in the speech recognition task, and so does its efficiency, i.e. how quickly it finds the uttered words. To speed up this search process, various ideas are available in the literature: we can use search heuristics, multi-pass search, or apply a family of aggregation operators. In this paper we test all these methods in turn, and combine them with a set of other novel speed-up ideas. The test results confirm that all of these techniques are valuable: using combinations of them helped make the speech recognition process over 12 times faster than the basic multi-stack decoding algorithm, and almost 11 times faster than the Viterbi beam search method.

Tree-Based Covariance Modeling of Hidden Markov Models

In this paper, we present a tree-based, full covariance hidden Markov modeling technique for automatic speech recognition applications. A multilayered tree is built first to organize all covariance matrices into a hierarchical structure. Kullback-Leibler divergence is used in the tree-building to measure inter-Gaussian distortion and successive splitting is used to construct the multilayer covariance tree. To cope with the data sparseness problem in estimating a full covariance matrix, we interpolate the diagonal covariance matrix of a leaf-node at the bottom of the tree with the full covariance of its parent and ancestors along the path up to the root node. The interpolation coefficients are estimated in the maximum likelihood sense via the EM algorithm. The interpolation is performed in three different parametric forms: 1) inverse covariance matrix, 2) covariance matrix, and 3) off-diagonal terms of the full covariance matrix. The proposed algorithm is tested in three different databases: 1) the DARPA resource management (RM), 2) the switchboard, and 3) a Chinese dictation. In all three databases, we show that the proposed tree-based full covariance modeling consistently performs better than the baseline diagonal covariance modeling. The algorithm outperforms other covariance modeling techniques, including: 1) the semi-tied covariance modeling (STC), 2) heteroscedastic linear discriminant analysis (HLDA), 3) mixtures of inverse covariance (MIC), and 4) direct full covariance modeling

Efficient client–server based implementations of mobile speech recognition services

The purpose of this paper is to demonstrate the efficiencies that can be achieved when automatic speech recognition (ASR) applications are provided to large user populations using client–server implementations of interactive voice services. It is shown that, through proper design of a client–server framework, excellent overall system performance can be obtained with minimal demands on the computing resources that are allocated to ASR. System performance is considered in the paper in terms of both ASR speed and accuracy in multi-user scenarios. An ASR resource allocation strategy is presented that maintains sub-second average speech recognition response latencies observed by users even as the number of concurrent users exceeds the available number of ASR servers by more than an order of magnitude. An architecture for unsupervised estimation of user-specific feature space adaptation and normalization algorithms is also described and evaluated. Significant reductions in ASR word error rate were obtained by applying these techniques to utterances collected from users of hand-held mobile devices. These results are important because, while there is a large body of work addressing the speed and accuracy of individual ASR decoders, there has been very little effort applied to dealing with the same issues when a large number of ASR decoders are used in multi-user scenarios.

Robust Beamforming of Microphone Array Using H Adaptive Filtering Technique

In ASR (Automatic Speech Recognition) applications, one of the most important issues in the real-time beamforming of microphone arrays is the inability to capture the whole acoustic dynamics via a finite-length of data and a finite number of array elements. For example, the reflected source signal impinging from the side-lobe direction presents a coherent interference, and the non-minimal phase channel dynamics may require an infinite amount of data in order to achieve perfect equalization (or inversion). All these factors appear as uncertainties or un-modeled dynamics in the receiving signals. Traditional adaptive algorithms such as NLMS that do not consider these errors will result in performance deterioration. In this paper, a time domain beamformer using H∞ filtering approach is proposed to adjust the beamforming parameters. Furthermore, this work also proposes a frequency domain approach called SPFDBB (Soft Penalty Frequency Domain Block Beamformer) using H∞ filtering approach that can reduce computational efforts and provide a purified data to the ASR application. Experimental results show that the adaptive H∞ filtering method is robust to the modeling errors and suppresses much more noise interference than that in the NLMS based method. Consequently, the correct rate of ASR is also enhanced.

MS/Ayad-Marwan Network: A New Architecture of an Artificial Neural Network

Due to its parallelism property in processing the data, the Artificial Neural Networks (ANN) has been gaining wide interest in recent years as a tool for processing data. Different ANN architectures have been defined for various applications. Yet, a number of difficulties existed when building an ANN like training time, over-training problem, re-training of the net for new types of data. This paper presents a new ANN architecture, which we called (MS stands for Multi-Stage), that compose of supernet and subnets (Stage architecture). The Computer-Assisted approach is used to develop this NN. Also, in this paper, a new learning algorithm has been proposed, which is required for adjusting weighting values for neural network. We called this algorithm a hybrid unsupervised/supervised learning paradigm This learning algorithm is used for training both supernet and subnets. Inheritance property of OOP plays an important role in the reasoning approach of Network, in that it deduces the final solution by gathering the property of the super network with the properties of its sub(s), to reach the ultimate goal. Thus, the results become more accurate. We called his reasoning approach Deductive auto-associative recalling. ANN has been applied to an Automatic Speech Recognition (ASR) application. This new network architecture is being able to recognize phonemes with >91% accuracy. We note through the results a promising advancement has been reached and hence a Multi-Stage strategy is useful in applications of high changeable data property, like ASR. MS/Ayad-Marwan Neural Network has been implemented and tested using MATLAB, as a part of the ASR system.

Frequency Domain Microphone Array Calibration and Beamforming for Automatic Speech Recognition

This investigation proposed two array beamformers SPFDBB (Soft Penalty Frequency Domain Block Beamformer) and FDABB (Frequency Domain Adjustable Block Beamformer). Compared with the conventional beamformers, these frequency-domain methods can significantly reduce the computation power requirement in ASR (Automatic Speech Recognition) based applications. Like other reference signal based techniques, SPFDBB and FDABB minimize microphone's mismatch, desired signal cancellation caused by reflection effects and resolution due to the array's position. Additionally, these proposed methods are suitable for both near-field and far-field environments. Generally, the convolution relation between channel and speech source in time domain cannot be modeled accurately as a multiplication in the frequency domain with a finite window size, especially in ASR applications. SPFDBB and FDABB can approximate this multiplication by treating several frames as a block to achieve a better beamforming result. Moreover, FDABB adjusts the number of frames on-line to cope with the variation of characteristics in both speech and interference signals. A better performance was found to be achievable by combining these methods with an ASR mechanism.

An elitist approach to automatic articulatory-acoustic feature classification for phonetic characterization of spoken language

A novel framework for automatic articulatory-acoustic feature extraction has been developed for enhancing the accuracy of place- and manner-of-articulation classification in spoken language. The “elitist” approach provides a principled means of selecting frames for which multi-layer perceptron, neural-network classifiers are highly confident. Using this method it is possible to achieve a frame-level accuracy of 93% on “elitist” frames for manner classification on a corpus of American English sentences passed through a telephone network (NTIMIT). Place-of-articulation information is extracted for each manner class independently, resulting in an appreciable gain in place-feature classification relative to performance for a manner-independent system. A comparable enhancement in classification performance for the elitist approach is evidenced when applied to a Dutch corpus of quasi-spontaneous telephone interactions (VIOS). The elitist framework provides a potential means of automatically annotating a corpus at the phonetic level without recourse to a word-level transcript and could thus be of utility for developing training materials for automatic speech recognition and speech synthesis applications, as well as aid the empirical study of spoken language.

Use of acoustic prior information for confidence measure in ASR (automatic speech recognition) applications

In this paper, a new acoustic confidence measure of automatic speech recognition hypothesis is proposed and it is compared to approaches proposed in the literature. This approach takes into account prior information on the acoustic model performance specific to each phoneme. The new method is tested on two types of recognition errors: the out-of-vocabulary words and the errors due to additive noise. An efficient way to interpret the raw confidence measure as a correctness prior probability is also proposed in the paper.

A new definition of Xhosa grapheme-to-phoneme rules for automatic transcription

A new definition of grapheme-to-phoneme rules was developed for automatically transcribing Xhosa text. An existing rule set was expanded and refined for the purposes of enhancing its accuracy and computational implementation. The new algorithm was implemented in various transcription and labelling processes involved in developing telephone speech databases for the AST project and proved to be of practical importance for linguistic research, Text-to-Speech and Automatic Speech Recognition applications.

Contemporaneous text as side-information in statistical language modeling

We propose new methods to exploit contemporaneous text, such as on-line news articles, to improve language models for automatic speech recognition and other natural language processing applications. In particular, we investigate the use of text from a resource-rich language to sharpen language models for processing a news story or article in a language with scarce linguistic resources. We demonstrate that even with fairly crude cross-language information retrieval and simple machine translation, one can construct story-specific Chinese language models which exploit cues from a side-corpus of English newswire to significantly improve the performance of language models estimated from a static Chinese corpus. Our investigations cover cases when the amount of available Chinese text is small, and a case when a large Chinese text corpus is available. We examine the effectiveness of our techniques both when the side-corpus contains English documents that are near-translations of the Chinese documents being processed, and when the English side-corpus is merely from contemporaneous and independent news sources. We present experimental results for automatic transcription of speech from the Mandarin Broadcast News corpus.

Graceful degradation of speech recognition performance over packet-erasure networks

This paper explores packet loss recovery for automatic speech recognition (ASR) in spoken dialog systems, assuming an architecture in which a lightweight client communicates with a remote ASR server. Speech is transmitted with source and channel codes optimized for the ASR application, i.e., to minimize word error rate. Unequal amounts of forward error correction, depending on the data's effect on ASR performance, are assigned to protect against packet loss. Experiments with simulated packet loss in a range of loss conditions are conducted on the DARPA Communicator (air travel information) task. Results show that the approach provides robust ASR performance which degrades gracefully as packet loss rates increase. Transmitting at 5.2 Kbps with up to 200 ms added delay, leads to only a 7% relative degradation in word error rate even under extremely adverse network conditions.

Phone inventory optimization for multilingual automatic speech recognition

This paper describes a phone inventory optimization procedure for application in multilingual automatic speech recognition (ASR). The optimization procedure is based on three knowledge sources that act collectively to guide phonological reduction and selection processes: (1) abstract (language-independent) phonological universals and tendencies that are used in the construction of a hierarchical structure that specifies phone class reduction paths; (2) language-dependent knowledge that includes information of the targeted languages’ phone inventories and individual phone frequencies in language data resources; (3) acoustic data that provides phone discriminability and similarity metrics. Using the optimization procedure, the phone inventories of six languages, American English, Mandarin Chinese, Egyptian Colloquial Arabic, Japanese, German, and Spanish, were merged to create an inventory consisting of 64 distinct cross-phonological units. This reduced phone set was used in all training and testing procedures and resources for the recognition of the six targeted languages. Preliminary recognition results are very encouraging: while purely data-driven approaches to multilingual ASR fail to reach word-recognition rates comparable to monolingual applications, the use of the optimized phone inventory in our multilingual ASR program yields recognition rates approximating that of monolingual ASR.

To ‘errrr’ is human: ecology and acoustics of speech disfluencies

Unlike read or laboratory speech, spontaneous speech contains high rates of disfluencies (e.g. repetitions, repairs, filled pauses, false starts). This paper aims to promote ‘disfluency awareness’ especially in the field of phonetics –which has much to offer in the way of increasing our understanding of these phenomena. Two broad claims are made, based on analyses of disfluencies in different corpora of spontaneous American English speech. First, an Ecology Claim suggests that disfluencies are related to aspects of the speaking environments in which they arise. The claim is supported by evidence from task effects, location analyses, speaker effects and sociolinguistic effects. Second, an Acoustics Claim argues that disfluency has consequences for phonetic and prosodic aspects of speech that are not represented in the speech patterns of laboratory speech. Such effects include modifications in segment durations, intonation, voice quality, vowel quality and coarticulation patterns. The ecological and acoustic evidence provide insights about human language production in real-world contexts. Such evidence can also guide methods for the processing of spontaneous speech in automatic speech recognition applications.

On the Determinization of Weighted Finite Automata

We study the problem of constructing the deterministic equivalent of a nondeterministic weighted finite-state automaton (WFA). Determinization of WFAs has important applications in automatic speech recognition (ASR). We provide the first polynomial-time algorithm to test for the twins property, which determines if a WFA admits a deterministic equivalent. We also give upper bounds on the size of the deterministic equivalent; the bound is tight in the case of acyclic WFAs. Previously, Mohri presented a superpolynomial-time algorithm to test for the twins property, and he also gave an algorithm to determinize WFAs. He showed that the latter runs in time linear in the size of the output when a deterministic equivalent exists; otherwise, it does not terminate. Our bounds imply an upper bound on the running time of this algorithm. Given that WFAs can expand exponentially in size when determinized, we explore why those that occur in ASR tend to shrink when determinized. According to ASR folklore, this phenomenon is attributable solely to the fact that ASR WFAs have simple topology, in particular, that they are acyclic and layered. We introduce a very simple class of WFAs with this structure, but we show that the expansion under determinization depends on the transition weights: some weightings cause them to shrink, while others, including random weightings, cause them to expand exponentially. We provide experimental evidence that ASR WFAs exhibit this weight dependence. That they shrink when determinized, therefore, is a result of favorable weightings in addition to special topology. These analyses and observations have been used to design a new, approximate WFA determinization algorithm, reported in a separate paper along with experimental results showing that it achieves significant WFA size reduction with negligible impact on ASR performance.

Application of current speech recognition technology to nonstandard domains

The typical input to automatic speech recognition (ASR) algorithms is a word-length or longer acoustic waveform. The typical output consists of names of items identified from stored vocabularies. Most algorithms also use scoring procedures to find the top matches between the input and the stored information. These evaluation scores, probabilities or distance metrics, may also be output. There are many nonstandard applications of ASR technology that combine identification and evaluation scores. Examples to be discussed include speech training to improve intelligibility, spoken language proficiency of non-native speakers, and language training for adults with developmental disabilities. Most current ASR algorithms are fine tuned for species-specific properties of human speech and language. In addition, many incorporate psychophysical properties of the human auditory system in the initial signal processing. However, the underlying pattern recognition algorithms are also applicable to a wide range of animal vocalizations. Some commercial and laboratory systems will be discussed in relation to nonstandard applications of ASR. [Work supported by NIH, RO1 DC-02229, R44 DC-02213, and R43 HD-35425.]

Application of automatic speech recognition to evaluation of speech transmission quality in analog communication systems

The preliminary results of application of automatic recognition of isolated words to objective evaluation of speech transmission quality in analog telephone channels are presented. A memoryless, finite state recognition system with LPC, FFT, and BF-FFT (where the speech signal was filtered in Bark bands) parametrization was applied. In classification stage a dynamic time warping and nearest-neighbor algorithm were utilized. Nonsense word lists consisting of 100 logotoms were recorded in a studio by a professional male speaker and utilized next as a test material. Speech transmission quality was examined in laboratory models of telephone channels with frequency bands of 300–3400, 400–2500, and 100–6000 Hz for speech-to-white-noise ratios in the range of +15 to −15 dB. The results of objective measurements expressed in percent of logotoms correctly recognized by the recognition system were compared under the same transmission conditions with subjectively measured logotom intelligibility. The best agreement between subjective and objective evaluation of speech transmission quality was obtained for automatic speech recognition utilizing BF–FFT parametrization. The results of objective evaluation of speech transmission quality by means of the presented method are encouraging and the experiments will be continued for other communication channels (e.g., digital) and different distortions and disturbances.

Research and development of robust speech recognition

This paper describes recent research and development activities on robust ASR (automatic speech recognition) in NTT Human Interface Laboratories. ASR system design has been changing from the experimental to the commercial level. A relevant issue in achieving practical ASR is the robustness against environmental noise, and speaker and circuit differences. Adaptation technique has been widely investigated for improving the robustness. It should be noted that customers or users generally prefer simple procedures and quicker response. Thus real time speaker/noise adaptation is getting more and more popular. Confidence measure is also effective in enhancing speech recognition performance by rejecting out-of-vocabulary words, lip, and breath noises. Another relevant issue is how to quickly reach the goal of the human-machine dialog via the telephone without visual information. The voice quality through a telephone is not good enough to provide perfect syllable intelligibility even for a human. This suggests that an appropriate human-machine dialog is needed to support an ASR system in acquiring the information or request intended by the user. Recent research topics including spectral estimation, dialog control, and other new approaches related to above discussion will be shown in the presentation. ASR application examples and related problems will also be shown.

Ergonomics and automatic speech recognition applications for deaf and hard-of-hearing users

The potential of automatic speech recognition (ASR) for deaf and hard-of-hearing people has been recognized. This paper reviews the use environment for the deaf and hard-of-hearing person and an ASR device, using ergonomic analysis techniques. Within the deaf and hard-of-hearing population, there are many interpersonal differences, stemming from audiological and rehabilitation characteristics and social and occupational settings. Some members of the population may find ASR devices inadequate or prefer other communication aids. Environmental characteristics, particularly noise and light, affect ASR device use. The technology of automatic speech recognition itself introduces device limitations and challenges. In ergonomic analysis, these three aspects - user, environment and device - are connected by the task, in this case the communication task. Deaf and hard-of-hearing users need to communicate with many different people, ranging from intimate associates to perfect strangers and disembodied voices. They need to communicate in one-to-one situations and in groups both small and large. In many of these communication settings, relationships are more important than information and use of ASR would interfere with valuable eye contact as much as aid communication. Imposing order on many communication settings is problematic, but multiple conversations and interruptions create interference with selecting the correct signal to transcribe. © 1997 Published by Elsevier Science Ireland Ltd.