Continuous Speech Recognizer Research Articles

Innovative Teaching Strategies for Civics Courses Facing Virtual Reality

Abstract This paper innovatively applies the teaching system integrating virtual reality and intelligent voice interaction to Civics classroom teaching. Using 3ds Max modeling, I can build a multimedia blackboard, code display table, characters, and other three-dimensional teaching models. Rendering optimization of the model and the scene is carried out to construct a real and rich teaching scene. Viterbi technology realizes the search of keywords in the speech through the process of continuous speech recognizer, keyword searcher, confidence confirmation, and keyword confirmation in order to discover intelligent voice interaction. The results of the Civics test, questionnaire, and structural modeling show that there is a significant difference between the experimental group’s performance and the control group in terms of the total performance (T=2.367, P=0.032), the vividness and intuition of the teaching content (X²=5.743, P=0.022) and the cognitive load (T=0.78, P=0.000<0.05). For all dimensions, the immersive VR environment scored higher than the interactive VR environment. The p-values of perceived usefulness and want-to-use attitude are 0.022 and 0.024, respectively. Based on this, the Civics teaching system of virtual reality and intelligent voice interaction can effectively improve students’ performance, classroom satisfaction, and acceptance.

Malaysian English Large Vocabulary Continuous Speech Recognizer: An Improvement Using Acoustic Model Adapation

This research project aims to develop Malaysian English Continuous Speech Recognition system by adapting US English acoustic model with Malaysian English speech corpus using Maximum a posteriori reasoning (MAP) and Maximum Likelihood Linear Regression (MLLR). During feature extraction stage, the Mel-Frequency Cepstral Coefficients (MFCC) technique was used. The Hidden Markov Model was used as the back end pattern comparison technique. For the purpose of implementation, the CMU Sphinx toolkit, which includes Pocketsphinx and Sphinxtrain as well as an acoustic model, was used to develop a speech recognition system for Malaysian English. Malaysian English speech sample will be recorded and transcribed to produce the training database required for acoustic model adaptation. The adaptation speech corpus were collected from a number of speakers. The outcome of this research could increase the application of Malaysian English speech recognition in Malaysia due to accent problem. As a result, speech recognition systems that have gone through the MAP adaptation had the best performance. Its average word error rate achieved was 32.84%. Average word recognition rate was 72.52% and average sentence error rate was 78.89%.

TAMEEM V1.0: speakers and text independent Arabic automatic continuous speech recognizer

This research work aims to disseminate the efforts towards developing and evaluating TAMEEM V1.0, which is a state-of-the-art pure Modern Standard Arabic (MSA), automatic, continuous, speaker independent, and text independent speech recognizer using high proportion of the spoken data of the phonetically rich and balanced MSA speech corpus. The speech corpus contains speech recordings of Arabic native speakers from 11 Arab countries representing Levant, Gulf, and Africa regions of the Arabic World, which make about 45.30 h of speech data. The recordings contain about 39.28 h of 367 sentences that are considered phonetically rich and balanced, which are used for training TAMEEM V1.0 speech recognizer, and another 6.02 h of another 48 sentences that are used for testing purposes, which are mostly text independent and foreign to the training sentences. TAMEEM V1.0 speech recognizer is developed using the Carnegie Mellon University (CMU) Sphinx 3 tools in order to evaluate the speech corpus, whereby the speech engine uses three-emitting state Continuous Density Hidden Markov Model for tri-phone based acoustic models, and the language model contains uni-grams, bi-grams, and tri-grams. Using three different testing data sets, this work obtained 7.64% average Word Error Rate (WER) for speakers dependent with text independent data set. For speakers independent with text dependent data set, this work obtained 2.22% average WER, whereas 7.82% average WER is achieved for speakers independent with text independent data set.

Automatic continuous speech recogniser for Dravidian languages using the auto associative neural network

In recent times with the extensive improvement of computers, numerous methods of data interchange between man and computer are revealed. It aims to provide an efficient way for human to communicate with computers exclusively for people with disabilities who face diversity of obstacles while using computers. This paper predominantly focuses on developing an efficient speech recognition system for Dravidian languages such as Tamil, Malayalam, Telugu and Kannada. The proposed CSR system comprises of four steps namely pre-processing, feature extraction, automatic continuous speech segmentation and classification. The most powerful and widely used short term energy and zero crossing rate is used for continuous speech segmentation and Mel frequency cepstral coefficients MFCC, linear predictive cepstral coefficients LPCC and shifted delta cepstrum SDC feature extractions are used for recognition system. Experiments are carried out with real time Dravidian languages speech signal. It is observed from the results that the proposed system gives significant results in AANN classifier with MFCC feature when compared with LPCC and SDC features.

Topic segmentation on spoken documents using self-validated acoustic cuts

Topic segmentation serves as a necessary prerequisite for multimedia content analysis and management. The normalized cuts (NCuts) approach has shown superior performance in topic segmentation of spoken document. However, in this method, the number of topics in a document has to be known prior to segmentation. This is impractical for real-world applications with exponential growth of multimedia data. On the other hand, previous lexical-based spoken document segmentation approaches, including NCuts, work on text transcripts generated by a large vocabulary continuous speech recognizer (LVCSR). As we know, training such a recognizer requires a large amount of transcribed speech data and language-specific knowledges. Moreover, inevitable speech recognition errors and the out-of-vocabulary (OOV) problem apparently affect the segmentation performance. This paper addresses these problems by a self-validated acoustic normalized cuts approach, namely SACuts. First, as compared with NCuts, our approach can determine the topic number in a spoken document automatically without extra computation load. Second, as compared with lexical approaches that rely on a high-resource speech recognizer, our approach can achieve comparable and even better segmentation performance using only acoustic-level information. Evaluation on a broadcast news topic segmentation task shows the superiority of the proposed approach.

Tackling speaking mode varieties in EMG-based speech recognition.

An electromyographic (EMG) silent speech recognizer is a system that recognizes speech by capturing the electric potentials of the human articulatory muscles, thus enabling the user to communicate silently. After having established a baseline EMG-based continuous speech recognizer, in this paper, we investigate speaking mode variations, i.e., discrepancies between audible and silent speech that deteriorate recognition accuracy. We introduce multimode systems that allow seamless switching between audible and silent speech, investigate different measures which quantify speaking mode differences, and present the spectral mapping algorithm, which improves the word error rate (WER) on silent speech by up to 14.3% relative. Our best average silent speech WER is 34.7%, and our best WER on audibly spoken speech is 16.8%.

Acoustic Modeling With Hierarchical Reservoirs

Accurate acoustic modeling is an essential requirement of a state-of-the-art continuous speech recognizer. The Acoustic Model (AM) describes the relation between the observed speech signal and the non-observable sequence of phonetic units uttered by the speaker. Nowadays, most recognizers use Hidden Markov Models (HMMs) in combination with Gaussian Mixture Models (GMMs) to model the acoustics, but neural-based architectures are on the rise again. In this work, the recently introduced Reservoir Computing (RC) paradigm is used for acoustic modeling. A reservoir is a fixed - and thus non-trained - Recurrent Neural Network (RNN) that is combined with a trained linear model. This approach combines the ability of an RNN to model the recent past of the input sequence with a simple and reliable training procedure. It is shown here that simple reservoir-based AMs achieve reasonable phone recognition and that deep hierarchical and bi-directional reservoir architectures lead to a very competitive Phone Error Rate (PER) of 23.1% on the well-known TIMIT task.

HMM/ANN hybrid model for continuous Malayalam speech recognition

This paper describes the development of a context independent, small vocabulary, connectionist-statistical continuous Malayalam speech recognition system which combines the time normalization property of Hidden Markov Models (HMMs) with the superior discriminative ability of Artificial Neural Networks (ANNs). In this work, the HMM based phoneme models use the emission probabilities estimated from the posterior probabilities obtained through Multi Layer Perceptrons. We evaluated the performance of our proposed system on a small vocabulary, speaker independent continuous Malayalam speech corpus and our system has produced a promising result of 86.67% word and 66.67% sentence recognition rates. This is the first reported result for a Malayalam speaker independent continuous speech recognizer based on an HMM/ANN hybrid framework.

Fast Language Model Look-ahead Algorithm Using Extended N -gram Model

摘要: 针对基于动态解码网络的大词汇量连续语音识别器,本文提出了一种采用扩展N元文法模 型进行快速语言模型(Language model, LM)预测的方法.扩展N元文法模型统一了语言模型和语言模型预测树的 表示与分数计算方法,从而大大简化了解码器的实现,极大地提升了语言模型预测的速度,使得高阶语言模型预测成为可能.扩展N元文法模型在解码之前离线生成,生成过程利 用了N元文法的稀疏性加速计算过程,并采用了词尾节点前推和分数量化的方法压缩模 型存储空间大小.实验表明,相比于采用动态规划在解码过程中实时计算语言模型预测分 数的传统方法,本文提出的方法在相同的字错误率下使得整个识别系统识别速率提升了5～ 9 倍,并且采用高阶语言模型预测可获得比低阶预测更优的解码速度与精度. 关键词: 语音识别 / 语言模型预测 / N元文法模型 / 解码

A large vocabulary continuous speech recognition system for Persian language

The first large vocabulary speech recognition system for the Persian language is introduced in this paper. This continuous speech recognition system uses most standard and state-of-the-art speech and language modeling techniques. The development of the system, called Nevisa, has been started in 2003 with a dominant academic theme. This engine incorporates customized established components of traditional continuous speech recognizers and its parameters have been optimized for real applications of the Persian language. For this purpose, we had to identify the computational challenges of the Persian language, especially for text processing and extract statistical and grammatical language models for the Persian language. To achieve this, we had to either generate the necessary speech and text corpora or modify the available primitive corpora available for the Persian language. In the proposed system, acoustic modeling is based on hidden Markov models, and optimized decoding, pruning and language modeling techniques were used in the system. Both statistical and grammatical language models were incorporated in the system. MFCC representation with some modifications was used as the speech signal feature. In addition, a VAD was designed and implemented based on signal energy and zero-crossing rate. Nevisa is equipped with out-of-vocabulary capability for applications with medium or small vocabulary sizes. Powerful robustness techniques were also utilized in the system. Model-based approaches like PMC, MLLR and MAP, along with feature robustness methods such as CMS, PCA, RCC and VTLN, and speech enhancement methods like spectral subtraction and Wiener filtering, along with their modified versions, were diligently implemented and evaluated in the system. A new robustness method called PC-PMC was also proposed and incorporated in the system. To evaluate the performance and optimize the parameters of the system in noisy-environment tasks, four real noisy speech data sets were generated. The final performance of Nevisa in noisy environments is similar to the clean conditions, thanks to the various robustness methods implemented in the system. Overall recognition performance of the system in clean and noisy conditions assures us that the system is a real-world product as well as a competitive ASR engine.

MLN-based Bangla ASR using context sensitive triphone HMM

Building a continuous speech recognizer for the Bangla (widely used as Bengali) language is a challenging task due to the unique inherent features of the language like long and short vowels and many instances of allophones. Stress and accent vary in spoken Bangla language from region to region. But in formal read Bangla speech, stress and accents are ignored. There are three approaches to continuous speech recognition (CSR) based on the sub-word unit viz. word, phoneme and syllable. Pronunciation of words and sentences are strictly governed by set of linguistic rules. Many attempts have been made to build continuous speech recognizers for Bangla for small and restricted tasks. However, medium and large vocabulary CSR for Bangla is relatively new and not explored. In this paper, the authors have attempted for building automatic speech recognition (ASR) method based on context sensitive triphone acoustic models. The method comprises three stages, where the first stage extracts phoneme probabilities from acoustic features using a multilayer neural network (MLN), the second stage designs triphone models to catch context of both sides and the final stage generates word strings based on triphone hidden Markov models (HMMs). The objective of this research is to build a medium vocabulary triphone based continuous speech recognizer for Bangla language. In this experimentation using Bangla speech corpus prepared by us, the recognizer provides higher word accuracy as well as word correct rate for trained and tested sentences with fewer mixture components in HMMs.

Malay Language Speech Recogniser with Hybrid Hidden Markov Model and Artificial Neural Network(HMM/ANN)

There are many artificial intelligence approaches used in the development of Automatic Speech Recognition (ASR), and hybrid approach is one of them. The common hybrid method in speech recognition is the combination of Hidden Markov Model (HMM) and Artificial Neural Network (ANN). The hybrid HMM/ANN is able to combine the strength of HMM in sequential modeling structure and ANN in pattern classification. Thus, this paper proposed a speaker independent and continuous Malay language speech recogniser by using the hybrid HMM/ANN method. In addition to that, this paper presents a study on Standard Malay's phonetic and phonology to help in the recognition of Malay words. The CSLU toolkit is utilized for building the recogniser, and the experimental results showed that the proposed HMM/ANN model outperformed the conventional HMM model. The performances of the recognisers are measured in term of word accuracy and sentence accuracy.

Word and triphone based approaches in continuous speech recognition for Tamil language

Building a continuous speech recognizer for the Indian language like Tamil is a challenging task due to the unique inherent features of the language like long and short vowels, lack of aspirated st...

Hybrid statistical pronunciation models designed to be trained by a medium-size corpus

Generating pronunciation variants of words is an important subject in speech research and is used extensively in automatic speech recognition and segmentation systems. Decision trees are well known tools in modeling pronunciation over words or sub-word units. In the case of word units and very large vocabulary, in order to train necessary decision trees, a huge amount of speech utterances are required. This training data must contain all of the needed words in the vocabulary with a sufficient number of repetitions for each one. Additionally, an extra corpus is needed for every word which is not included in the original training corpus and may be added to the vocabulary in the future. To overcome these drawbacks, we have designed generalized decision trees, which can be trained using a medium-size corpus over groups of similar words to share information on pronunciation, instead of training a separate tree for every single word. Generalized decision trees predict places in the word where substitution, deletion and insertion of phonemes may occur. After this step, appropriate statistical contextual rules are applied to the permitted places, in order to specifically determine word variants. The hybrids of generalized decision trees and contextual rules are designed in static and dynamic versions. The hybrid static pronunciation models take into account word phonological structures, unigram probabilities, stress and phone context information simultaneously, while the hybrid dynamic models consider an extra feature, speaking rate, to generate pronunciation variants of words. Using the word variants, generated by static and dynamic models, in the lexicon of the SHENAVA Persian continuous speech recognizer, relative word error rate reductions as high as 8.1% and 11.6% are obtained, respectively.

Online Speech Detection and Dual-Gender Speech Recognition for Captioning Broadcast News

This paper describes a new method to detect speech segments online with identifying gender attributes for efficient dual gender-dependent speech recognition and broadcast news captioning. The proposed online speech detection performs dual-gender phoneme recognition and detects a start-point and an end-point based on the ratio between the cumulative phoneme likelihood and the cumulative non-speech likelihood with a very small delay from the audio input. Obtaining the speech segments, the phoneme recognizer also identifies gender attributes with high discrimination in order to guide the subsequent dual-gender continuous speech recognizer efficiently. As soon as the start-point is detected, the continuous speech recognizer with paralleled gender-dependent acoustic models starts a search and allows search transitions between male and female in a speech segment based on the gender attributes. Speech recognition experiments on conversational commentaries and field reporting from Japanese broadcast news showed that the proposed speech detection method was effective in reducing the false rejection rate from 4.6% to 0.53% and also recognition errors in comparison with a conventional method using adaptive energy thresholds. It was also effective in identifying the gender attributes, whose correct rate was 99.7% of words. With the new speech detection and the gender identification, the proposed dual-gender speech recognition significantly reduced the word error rate by 11.2% relative to a conventional gender-independent system, while keeping the computational cost feasible for real-time operation.

Comparison between different feature extraction techniques for audio-visual speech recognition

The current audio-only speech recognition still lacks the expected robustness when the Signal to Noise Ratio (SNR) decreases. The video information is not affected by noise which makes it an ideal candidate for data fusion for speech recognition benefit. In the paper [1] the authors have shown that most of the techniques used for extraction of static visual features result in equivalent features or at least the most informative features exhibit this property. We argue that one of the main problems of existing methods is that the resulting features contain no information about the motion of the speaker’s lips. Therefore, in this paper we will analyze the importance of motion detection for speech recognition. For this we will first present the Lip Geometry Estimation(LGE) method for static feature extraction. This method combines an appearance based approach with a statistical based approach for extracting the shape of the mouth. The method was introduced in [2] and explored in detail in [3]. Further more, we introduce a second method based on a novel approach that captures the relevant motion information with respect to speech recognition by performing optical flow analysis on the contour of the speaker’s mouth. For completion, a middle way approach is also analyzed. This third method considers recovering the motion information by computing the first derivatives of the static visual features. All methods were tested and compared on a continuous speech recognizer for Dutch. The evaluation of these methods is done under different noise conditions. We show that the audio-video recognition based on the true motion features, namely obtained by performing optical flow analysis, outperforms the other settings in low SNR conditions.

Objective quality evaluation in blind source separation for speech recognition in a real room

The determination of quality of the signals obtained by blind source separation is a very important subject for development and evaluation of such algorithms. When this approach is used as a pre-processing stage for automatic speech recognition, the quality measure of separation applied for assessment should be related to the recognition rates of the system. Many measures have been used for quality evaluation, but in general these have been applied without prior research of their capabilities as quality measures in the context of blind source separation, and often they require experimentation in unrealistic conditions. Moreover, these measures just try to evaluate the amount of separation, and this value could not be directly related to recognition rates. Presented in this work is a study of several objective quality measures evaluated as predictors of recognition rate of a continuous speech recognizer. Correlation between quality measures and recognition rates is analyzed for a separation algorithm applied to signals recorded in a real room with different reverberation times and different kinds and levels of noise. A very good correlation between weighted spectral slope measure and the recognition rate has been verified from the results of this analysis. Furthermore, a good performance of total relative distortion and cepstral measures for rooms with relatively long reverberation time has been observed.

Continuous space language models

This paper describes the use of a neural network language model for large vocabulary continuous speech recognition. The underlying idea of this approach is to attack the data sparseness problem by performing the language model probability estimation in a continuous space. Highly efficient learning algorithms are described that enable the use of training corpora of several hundred million words. It is also shown that this approach can be incorporated into a large vocabulary continuous speech recognizer using a lattice rescoring framework at a very low additional processing time. The neural network language model was thoroughly evaluated in a state-of-the-art large vocabulary continuous speech recognizer for several international benchmark tasks, in particular the N ist evaluations on broadcast news and conversational speech recognition. The new approach is compared to four-gram back-off language models trained with modified Kneser–Ney smoothing which has often been reported to be the best known smoothing method. Usually the neural network language model is interpolated with the back-off language model. In that way, consistent word error rate reductions for all considered tasks and languages were achieved, ranging from 0.4% to almost 1% absolute.

Contextual effects on the continuancy of /■/

This research examined the segmental-contextual effects on the continuancy of the voiced dental /■/ in American-English and whether certain acoustic attributes were preserved despite possible modification. Word-initial /■/ cases, extracted from continuous speech of 146 speakers from the TIMIT database, were frequently (averaging 65%) stoplike when it was in utterance-initial position or when its preceding phoneme was voiceless and/or [-continuant]. This stoplike modification occurred less frequently (averaging 39%) when /■/ was preceded by a voiced fricative and rarely (averaging 13%) when preceded by a vowel or liquid consonant. A comparison of stoplike /■/ and /d/ cases under similar contexts showed that the burst peak location, burst spectrum shape, and F2 at vowel onset averages were all statistically different between the two groups. In addition, the acoustic data suggested that the dental place of articulation was preserved for the modified /■/. Preliminary automatic classification experiments involving salient acoustic attributes seemed to indicate that F2 at vowel onset may be a reliable cue for the dental place of articulation in /■/. [Work supported by NIH DC02978 and T32DC00038.]

Introduction to the Special Issue on Data Mining of Speech, Audio, and Dialog

ATA mining is concerned with the science, technology, and engineering of discovering patterns and extracting potentially useful or interesting information automatically or semi-automatically from data. Data mining was introduced in the 1990s and has deep roots in the fields of statistics, artificial intelligence, and machine learning. With the advent of inexpensive storage space and faster processing over the past decade or so, data mining research has started to penetrate new grounds in areas of speech and audio processing as well as spoken language dialog. It has been fueled by the influx of audio data that are becoming more widely available from a variety of multimedia sources including webcasts, conversations, music, meetings, voice messages, lectures, television, and radio. Algorithmic advances in automatic speech recognition have also been a major, enabling technology behind the growth in data mining. Current state-of-the-art, large-vocabulary, continuous speech recognizers are now trained on a record amount of data—several hundreds of millions of words and thousands of hours of speech. Pioneering research in robust speech processing, large-scale discriminative training, finite state automata, and statistical hidden Markov modeling have resulted in real-time recognizers that are able to transcribe spontaneous speech with a word accuracy exceeding 85%. With this level of accuracy, the technology is now highly attractive for a variety of speech mining applications. Speech mining research includes many ways of applying machine learning, speech processing, and language processing algorithms to benefit and serve commercial applications. It also raises and addresses several new and interesting fundamental research challenges in the areas of prediction, search, explanation, learning, and language understanding. These basic challenges are becoming increasingly important in revolutionizing business processes by providing essential sales and marketing information about services, customers, and product offerings. They are also enabling a new class of learning systems to be created that can infer knowledge and trends automatically from data, analyze and report application performance, and adapt and improve over time with minimal or zero human involvement. Effective techniques for mining speech, audio, and dialog data can impact numerous business and government applications. The technology for monitoring conversational speech to discover patterns, capture useful trends, and generate alarms is essential for intelligence and law enforcement organizations as well as for enhancing call center operation. It is useful for an