Recognizer Output Voting Error Reduction Research Articles

Consensus Automatic Speech Recognition (CASR) in Cognitive Testing

AbstractBackgroundScoring verbal cognitive tests with automatic speech recognition (ASR) engines increases the efficiency of scoring and provides word timestamps that enable detailed temporal analyses of spoken responses. Here, we describe consensus ASR (CASR) procedures that incorporate multiple ASR engines to increase transcription and timing accuracy and generate CASR transcript confidence scores.MethodSeven ASR engines produced automatic transcriptions of both speech database samples (GMU Speech Accent Archive and NUS Auditory English Lexicon Project) and verbal test responses of 41 subjects from the California Cognitive Assessment Battery (CCAB). A novel Recognizer Output Voting Error Reduction (ROVER) algorithm was used to mutually align the transcripts, and a Bayesian weighted voting algorithm produced the best CASR transcript, mean word timestamps, and consensus scores. Word error rates (WER) gauged CASR accuracy against either predetermined or manually corrected transcripts.ResultDatabase sentence WERs from 1767 subjects ranged from a mean of 22% (Windows10 UWP) to 6% (Rev.ai) with CASR producing 5%, with no significant gender or age effects but better performance for native english speakers (Figure 1). In CCAB test responses, for limited word response tests CASR WERs ranged from 3% to less than 1% (Figure 2); for expansive word response tests CASR WERs ranged from 8% to 2% (Figure 3); and for discursive speech, CASR WERs ranged from 6% to 5%. Word start time ASR estimates for 594 database words in lists ranged in mean deviations from true times from 250ms std.dev. (Google) to 17ms std.dev. (Amazon) with CASR obtaining 14ms errors (Figure 4). Finally, consensus confidence scores from CCAB test responses, ranging from 0 to 1 (1 = complete agreement across ASR engines), show that CASR words with consensus scores above 0.8 and 0.9 are correct >99% and >99.8% of the time, respectively (Figure 5).ConclusionCASR produces transcripts for verbal test responses accurate enough for estimating scores in most limited word response tests. In large vocabulary response tests, CASR transcripts facilitate quick manual correction, and confidence values can identify transcript words needing manual correction. Patterns in CASR errors also indicate future substantial reductions to CASR WER on a per test basis.

A lexicon-free approach for 3D handwriting recognition using classifier combination

Recent developments in depth sensing technology such as Leap Motion have opened novel directions in Human-Computer-Interaction (HCI) research domain. The sensor extends the way of writing from traditional method to a gesture based writing in the 3D space. The online text written in 3D space over the sensorâs viewing field is different from traditional 2D handwriting in several ways. The 3D handwriting does not consist any stroke information, since all characters are connected by a single stroke. Moreover, non-uniform text styles and jitters during writing in 3D space create additional challenge for the recognition task. Because of these challenges in 3D handwriting, recognition of cursive words is not satisfied using a single classifier. In this paper, we present a lexicon free approach for the recognition of 3D handwritten words in Latin and Devanagari scripts by combining multiple classifiers. The individual recognition systems are computed using Bidirectional Long-Short Term Memory Neural Network (BLSTM-NN) classifier with the help of different features. The combination of multiple classifier is performed by aligning the output word sequence of each classifier using the Recognizer Output Voting Error Reduction (ROVER) framework. Accuracies of 72.25% and 71.86% are recorded using the proposed methodology for Latin and Devanagari scripts, respectively.

Automatic quality estimation for ASR system combination

Recognizer Output Voting Error Reduction (ROVER) has been widely used for system combination in automatic speech recognition (ASR). In order to select the most appropriate words to insert at each position in the output transcriptions, some ROVER extensions rely on critical information such as confidence scores and other ASR decoder features. This information, which is not always available, highly depends on the decoding process and sometimes tends to overestimate the real quality of the recognized words. In this paper we propose a novel variant of ROVER that takes advantage of ASR quality estimation (QE) for ranking the transcriptions at “segment level” instead of: i) relying on confidence scores, or ii) feeding ROVER with randomly ordered hypotheses. We first introduce an effective set of features to compensate for the absence of ASR decoder information. Then, we apply QE techniques to perform accurate hypothesis ranking at segment-level before starting the fusion process. The evaluation is carried out on two different tasks, in which we respectively combine hypotheses coming from independent ASR systems and multi-microphone recordings. In both tasks, it is assumed that the ASR decoder information is not available. The proposed approach significantly outperforms standard ROVER and it is competitive with two strong oracles that exploit prior knowledge about the real quality of the hypotheses to be combined. Compared to standard ROVER, the absolute WER improvements in the two evaluation scenarios range from 0.5% to 7.3%.

Microphone Array Processing Strategies for Distant-Based Automatic Speech Recognition

Robust distant speech recognition (DSR) is necessary in many speech technology applications using multiple microphones but has received only limited treatment in the literature. In this paper, we work on communicating with vehicle voice-controlled system which is one of the applications of DSR. Two approaches for DSR are i) signal-level combination using beamforming followed by automatic speech recognition (ACR), and ii) word hypothesis-level combination using several speech recognition engines followed by confusion network combination or followed by recognizer output voting error reduction (ROVER). In addition to these approaches, it is possible to examine training-level combination by training the recognizer on audio signals from multiple channels (microphones). In this paper, the authors investigate how these methods can be leveraged for in-vehicle ACR using the CU-Move corpus. The authors propose various combinations of these three methods to find an optimum structure for in-vehicle ACR. The authors also investigate the effect of speaker adaptation (SA). The author's experience shows that applying SA on individual channels and merging the results with ROVER reduces the negative effects of SA reported by others in the field, and illustrates the overall improvement obtained with front-end enhancement techniques in DSR.

Combination of multiple acoustic models with unsupervised adaptation for lecture speech transcription

Automatic speech recognition systems (ASR) have achieved considerable progress in real applications because of skilled design of the architecture with advanced techniques and algorithms. However, how to design a system efficiently integrating these various techniques to obtain advanced performance is still a challenging task. In this paper, we introduced an ensemble model combination and adaptation based ASR system with two characteristics: (1) large-scale combination of multiple ASR systems based on a Recognizer Output Voting Error Reduction (ROVER) system, and (2) multi-pass unsupervised speaker adaptation for deep neural network acoustic models and topic adaptation on language model. The multiple acoustic models were trained with different acoustic features and model architectures which helped to provide complementary and discriminative information in the ROVER process. With these multiple acoustic models, a better estimation of word confidence could be obtained from ROVER process which helped in selecting data for unsupervised adaptation on the previously trained acoustic models. The final recognition result was obtained using multi-pass decoding, ROVER, and adaptation processes. We tested the system on lecture speeches with topics related to Technology, Entertainment and Design (TED) that were used in the international workshop on spoken language translation (IWSLT) evaluation campaign, and obtained 6.5%, 7.0%, 10.6%, and 8.4% word error rates for test sets in 2011, 2012, 2013, and 2014, which to our knowledge are the best results for these evaluation sets.

A SYSTEM COMBINATION FOR MALAY BROADCAST NEWS TRANSCRIPTION

In this paper, we propose a post decoding system combination approach for automatic transcribing Malay broadcast news. This approach combines the hypotheses produced by parallel automatic speech recognition (ASR) systems. Each ASR system uses different language models, one which is generic domain model and another is domain specific model. The main idea is to take advantage of different ASR knowledge to improve ASR decoding result. It uses the language score and time information to produce a 1-best lattice, and then rescore the 1-best lattice to get the most likely word sequence as the final output. The proposed approach was compared with conventional combination approach, the recognizer output voting error reduction (ROVER). Our proposed approach improved the word error rate (WER) from 33.9% to 30.6% with an average relative WER improvement of 9.74%, and it is better than the conventional ROVER approach.

Speech recognition in reverberant and noisy environments employing multiple feature extractors and i-vector speaker adaptation

The REVERB challenge provides a common framework for the evaluation of feature extraction techniques in the presence of both reverberation and additive background noise. State-of-the-art speech recognition systems perform well in controlled environments, but their performance degrades in realistic acoustical conditions, especially in real as well as simulated reverberant environments. In this contribution, we utilize multiple feature extractors including the conventional mel-filterbank, multi-taper spectrum estimation-based mel-filterbank, robust mel and compressive gammachirp filterbank, iterative deconvolution-based dereverberated mel-filterbank, and maximum likelihood inverse filtering-based dereverberated mel-frequency cepstral coefficient features for speech recognition with multi-condition training data. In order to improve speech recognition performance, we combine their results using ROVER (Recognizer Output Voting Error Reduction). For two- and eight-channel tasks, to get benefited from the multi-channel data, we also use ROVER, instead of the multi-microphone signal processing method, to reduce word error rate by selecting the best scoring word at each channel. As in a previous work, we also apply i-vector-based speaker adaptation which was found effective. In speech recognition task, speaker adaptation tries to reduce mismatch between the training and test speakers. Speech recognition experiments are conducted on the REVERB challenge 2014 corpora using the Kaldi recognizer. In our experiments, we use both utterance-based batch processing and full batch processing. In the single-channel task, full batch processing reduced word error rate (WER) from 10.0 to 9.3 % on SimData as compared to utterance-based batch processing. Using full batch processing, we obtained an average WER of 9.0 and 23.4 % on the SimData and RealData, respectively, for the two-channel task, whereas for the eight-channel task on the SimData and RealData, the average WERs found were 8.9 and 21.7 %, respectively.

Improvement Comparison of Different Lattice-based Discriminative Training Methods in Chinese-monolingual and Chinese-English-bilingual Speech Recognition

AbstractDiscriminative training approaches such as minimum phone error (MPE), feature minimum phone error (fMPE) and boosted maximum mutual information (BMMI) have brought remarkable improvement to the speech community in recent years, however, much work still remains to be done. This paper investigates the performances of three lattice-based discriminative training methods in detail, and does a comparison of different I-smoothing methods to obtain more robust models in the Chinese-monolingual situation. The complementary properties of the different discriminative training methods are explored to perform a system combination by recognizer output voting error reduction (ROVER). Although discriminative training is normally used in monolingual systems, this paper systematically investigates its use for bilingual speech recognition, including MPE, fMPE, and BMMI. A new method is proposed to generate significantly better lattices for training the bilingual model, and complementary discriminative training models are also explored to get the best ROVER performance in the bilingual situation. Experimental results show that all forms of discriminative training can reduce the word error rate in both monolingual and bilingual systems, and that combining complementary discriminative training methods can improve the performance significantly.

Combining diverse on-line and off-line systems for handwritten text line recognition

In this paper we present a multiple classifier system (MCS) for on-line handwriting recognition. The MCS combines several individual recognition systems based on hidden Markov models (HMMs) and bidirectional long short-term memory networks (BLSTM). Beside using two different recognition architectures (HMM and BLSTM), we use various feature sets based on on-line and off-line features to obtain diverse recognizers. Furthermore, we generate a number of different neural network recognizers by changing the initialization parameters. To combine the word sequences output by the recognizers, we incrementally align these sequences using the recognizer output voting error reduction framework (ROVER). For deriving the final decision, different voting strategies are applied. The best combination ensemble has a recognition rate of 84.13%, which is significantly higher than the 83.64% achieved if only one recognition architecture (HMM or BLSTM) is used for the combination, and even remarkably higher than the 81.26% achieved by the best individual classifier. To demonstrate the high performance of the classification system, the results are compared with two widely used commercial recognizers from Microsoft and Vision Objects.