Connectionist Temporal Classification Research Articles

Deep Neural Networks for Acoustic Modeling in the Presence of Noise

Systems using deep neural network (DNN) have shown promising results in automatic speech recognition (ASR), where one of the biggest challenges is the recognition in noisy speech signals. We have combined two famous architectures of deep learning, the convolutional neural networks (CNN) for acoustic approach and a recurrent architecture with connectionist temporal classification (CTC) for sequential modeling, in order to decode the frames in a sequence forming a word. Experimental results show that the proposed architecture achieves improved performance over classical models, such as hidden model Markov (HMM) for labeling in variable time sequences in BioChaves database.

Unconstrained handwriting recoganization basedon neural network using connectionist temporal classification token passing algorithm

Recognition of human handwriting which offers the new way to improve the computer interface with the human and this process is very much useful for documents.Keyword spotting refers the spontaneous recognition of handwritten text, letter, and scripts from historical hand written books and the procedure of recovering all instance of a known keyword from an article. With a specific end goal to choose new components this paper, propose "a repetitive neural system manually written acknowledgment framework" for watchword spotting.The watchword seeing is finished utilizing an adjustment of the connectionist temporat classification Token Passing calculation in coincidence with a repetitive neural system. The proposed watchword spotting technique for written by hand message utilizing neural system, with another adaptation of connectionist temporat classification Token Pass calculation with quick and reliable catchphrase spotting can be executed without utilizing any content line or portioning separate words.

Baidu Meizu Deep Learning Competition: Arithmetic Operation Recognition Using End-to-End Learning OCR Technologies

The end-to-end learning approaches were proposed for an arithmetic expression recognition task in the Baidu Meizu Deep Learning Competition by a deep convolutional neural network (DCNN) with parallel dense layers and component-connection-based detection pipeline with the convolutional recurrent neural network (CRNN) model. Two effective pipelines for DCNN and CRNN to identify long and complex expressions are presented and compared. In the first task, a DCNN connected to parallel dense layers for digital arithmetic operations was developed, which achieves 99.985% accuracy. In the second task, the CRNN with connectionist temporal classification was adopted, combined with the text region detection technique to recognize more complex pictures with both assignment operations and calculation formulas, which achieves 98.087% accuracy.

Hybrid CTC/Attention Architecture for End-to-End Speech Recognition

Conventional automatic speech recognition (ASR) based on a hidden Markov model (HMM)/deep neural network (DNN) is a very complicated system consisting of various modules such as acoustic, lexicon, and language models. It also requires linguistic resources, such as a pronunciation dictionary, tokenization, and phonetic context-dependency trees. On the other hand, end-to-end ASR has become a popular alternative to greatly simplify the model-building process of conventional ASR systems by representing complicated modules with a single deep network architecture, and by replacing the use of linguistic resources with a data-driven learning method. There are two major types of end-to-end architectures for ASR; attention-based methods use an attention mechanism to perform alignment between acoustic frames and recognized symbols, and connectionist temporal classification (CTC) uses Markov assumptions to efficiently solve sequential problems by dynamic programming. This paper proposes hybrid CTC/attention end-to-end ASR, which effectively utilizes the advantages of both architectures in training and decoding. During training, we employ the multiobjective learning framework to improve robustness and achieve fast convergence. During decoding, we perform joint decoding by combining both attention-based and CTC scores in a one-pass beam search algorithm to further eliminate irregular alignments. Experiments with English (WSJ and CHiME-4) tasks demonstrate the effectiveness of the proposed multiobjective learning over both the CTC and attention-based encoder–decoder baselines. Moreover, the proposed method is applied to two large-scale ASR benchmarks (spontaneous Japanese and Mandarin Chinese), and exhibits performance that is comparable to conventional DNN/HMM ASR systems based on the advantages of both multiobjective learning and joint decoding without linguistic resources.

In-air handwritten English word recognition using attention recurrent translator

As a new human–computer interaction way, in-air handwriting allows users to write in the air in a natural, unconstrained way. Compared with conventional online handwriting based on touch devices, in-air handwriting is much more challenging due to its unique characteristics. The in-air handwriting is always finished in a single stroke and thus lacks pen-down and pen-up information. Moreover, the in-air handwriting suffers less friction and space restriction so that the users write more casually. In this paper, we present an in-air handwriting system for effectively recognizing handwritten English words. An attention-based model, called attention recurrent translator, is proposed for the in-air handwritten English word recognition, which is considerably different from connectionist temporal classification (CTC). We evaluate the proposed approach on a newly collected dataset containing a total of 150,480 recordings that cover 2280 English words. The proposed approach achieves a word recognition accuracy of 97.74%. The experimental results show that the proposed recognizer is comparable with CTC and is extremely effective for in-air handwritten English word recognition.

Lattice Based Transcription Loss for End-to-End Speech Recognition

End-to-end speech recognition systems have been successfully implemented and have become competitive replacements for hybrid systems. A common loss function to train end-to-end systems is connectionist temporal classification (CTC). This method maximizes the log likelihood between the feature sequence and the associated transcription sequence. However there are some weaknesses with CTC training. The main weakness is that the training criterion is different from the test criterion, since the training criterion is log likelihood, while the test criterion is word error rate. In this work, we introduce a new lattice based transcription loss function to address this deficiency of CTC training. Compared to the CTC function, our new method optimizes the model directly using the transcription loss. We evaluate this new algorithm in both a small speech recognition task, the Wall Street Journal (WSJ) dataset, a large vocabulary speech recognition task, the Switchboard dataset and a low resource speech recognition task, OpenKWS16. Results demonstrate that our algorithm outperforms a traditional CTC criterion, and achieves 7% WER relative reduction. In addition, we compare our new algorithm to some discriminative training algorithms, such as state-level minimum Bayes risk (SMBR) and minimum word error (MWE), with results supporting the benefits of the new algorithm.

CTC Regularized Model Adaptation for Improving LSTM RNN Based Multi-Accent Mandarin Speech Recognition

This paper proposes a novel regularized adaptation method for long short term memory (LSTM) recurrent neural network (RNN) based acoustic model trained with connectionist temporal classification (CTC) loss function (LSTM-RNN-CTC) to improve the performance of multi-accent Mandarin speech recognition task. In general, directly adjusting the network parameters with a small adaptation set may lead to over-fitting. In order to avoid this problem, we add a regularization term to the original training criterion. It forces the conditional probability distribution over initial and final (I/F) sequences estimated from the adapted model to be close to the accent independent (AI) model. Meanwhile, hidden layers of LSTM RNN should not be adjusted, but only the accent-specific output layer needs to be fine-tuned using this adaptation method. Experiments on RASC863 and CASIA regional accent speech corpus show that the proposed method obtains obvious improvement when compared with LSTM-RNN-CTC baseline model.

Long short-term memory recurrent neural network-based acoustic model using connectionist temporal classification on a large-scale training corpus

A Long Short-Term Memory (LSTM) Recurrent Neural Network (RNN) has driven tremendous improvements on an acoustic model based on Gaussian Mixture Model (GMM). However, these models based on a hybrid method require a forced aligned Hidden Markov Model (HMM) state sequence obtained from the GMM-based acoustic model. Therefore, it requires a long computation time for training both the GMM-based acoustic model and a deep learning-based acoustic model. In order to solve this problem, an acoustic model using CTC algorithm is proposed. CTC algorithm does not require the GMM-based acoustic model because it does not use the forced aligned HMM state sequence. However, previous works on a LSTM RNN-based acoustic model using CTC used a small-scale training corpus. In this paper, the LSTM RNN-based acoustic model using CTC is trained on a large-scale training corpus and its performance is evaluated. The implemented acoustic model has a performance of 6.18% and 15.01% in terms of Word Error Rate (WER) for clean speech and noisy speech, respectively. This is similar to a performance of the acoustic model based on the hybrid method.

Recent progress in deep end-to-end models for spoken language processing

End-to-end models (or sequence-to-sequence models) based on deep neural networks have recently become popular within the machine learning community. These techniques are also increasingly used in automatic speech recognition as an alternative to the state-of-the-art, hybrid HMM-DNN (hidden Markov model, deep neural network) system. The end-to-end systems contain a purely neural architecture that eliminates the need of any time alignment between the input acoustic feature vector sequence and output phone sequence. In this paper, we present progress within the IBM Watson Multimodal Group on end-to-end models for spoken language processing. We present our work on two types of end-to-end models applied to speech-to-text and keyword search tasks, namely, 1) recurrent neural networks (RNNs) based on connectionist temporal classification loss, and 2) attention-based encoder–decoder RNNs. We present results on several languages (such as Pashto, Mongolian, Javanese, Amharic, Guarani, Dholuo, Igbo, and Georgian) from the Intelligence Advanced Research Projects Activity funded Babel Program. We also present a detailed analysis of some salient characteristics of these models compared with the state-of-the-art HMM-DNN hybrid systems, and also discuss future challenges in using such models for spoken language processing.

LSTM $$^{2}$$ 2 : Multi-Label Ranking for Document Classification

Multi-label document classification is a typical challenge in many real-world applications. Multi-label ranking is a common approach, while existing studies usually disregard the effects of context and the relationships among labels during the scoring process. In this paper, we propose an Long Short Term Memory (LSTM)-based multi-label ranking model for document classification, namely LSTM $$^2$$ consisting of repLSTM—an adaptive data representation process and rankLSTM—a unified learning-ranking process. In repLSTM, the supervised LSTM is used to learn document representation by incorporating the document labels. In rankLSTM, the order of the documents labels is rearranged in accordance with a semantic tree, in which the semantics are compatible with and appropriate to the sequential learning of LSTM. The model can be wholly trained by sequentially predicting labels. Connectionist Temporal Classification is performed in rankLSTM to address the error propagation for a variable number of labels in each document. Moreover, a variety of experiments with document classification conducted on three typical datasets reveal the impressive performance of our proposed approach.

Maximum-<italic>a-Posteriori</italic>-Based Decoding for End-to-End Acoustic Models

This paper presents a novel decoding framework for acoustic models (AMs) based on end-to-end neural networks (e.g., connectionist temporal classification). The end-to-end training of AMs has recently demonstrated high accuracy and efficiency in automatic speech recognition (ASR). When using the trained AM in decoding, although a language model (LM) is implicitly involved in such an end-to-end AM, it is still essential to integrate an external LM trained with a large text corpus to achieve the best results. While there is no theoretical justification, most of the studies suggest using a naive interpolation of the end-to-end AM score and the external LM score, empirically. In this paper, we propose a more theoretically sound decoding framework derived from a maximization of the posterior probability of a word sequence given an observation. As a consequence of the theory, the subword LM is newly introduced to seamlessly integrate the external LM score with the end-to-end AM score. Our proposed method can be achieved by a small modification of the conventional weighted finite-state transducer-based implementation, without having to heavily increase the graph size. We tested the proposed decoding framework on ASR experiments with the Corpus of the Wall Street Journal and the Corpus of Spontaneous Japanese. The results showed that the proposed framework achieved significant and consistent improvements over the conventional interpolation-based decoding framework.

Recent progresses in deep learning based acoustic models

In this paper, we summarize recent progresses made in deep learning based acoustic models and the motivation and insights behind the surveyed techniques. We first discuss models such as recurrent neural networks U+0028 RNNs U+0029 and convolutional neural networks U+0028 CNNs U+0029 that can effectively exploit variablelength contextual information, and their various combination with other models. We then describe models that are optimized end-to-end and emphasize on feature representations learned jointly with the rest of the system, the connectionist temporal classification U+0028 CTC U+0029 criterion, and the attention-based sequenceto-sequence translation model. We further illustrate robustness issues in speech recognition systems, and discuss acoustic model adaptation, speech enhancement and separation, and robust training strategies. We also cover modeling techniques that lead to more efficient decoding and discuss possible future directions in acoustic model research.

Long Short-Term Memory Projection Recurrent Neural Network Architectures for Piano’s Continuous Note Recognition

Long Short-Term Memory (LSTM) is a kind of Recurrent Neural Networks (RNN) relating to time series, which has achieved good performance in speech recogniton and image recognition. Long Short-Term Memory Projection (LSTMP) is a variant of LSTM to further optimize speed and performance of LSTM by adding a projection layer. As LSTM and LSTMP have performed well in pattern recognition, in this paper, we combine them with Connectionist Temporal Classification (CTC) to study piano’s continuous note recognition for robotics. Based on the Beijing Forestry University music library, we conduct experiments to show recognition rates and numbers of iterations of LSTM with a single layer, LSTMP with a single layer, and Deep LSTM (DLSTM, LSTM with multilayers). As a result, the single layer LSTMP proves performing much better than the single layer LSTM in both time and the recognition rate; that is, LSTMP has fewer parameters and therefore reduces the training time, and, moreover, benefiting from the projection layer, LSTMP has better performance, too. The best recognition rate of LSTMP is 99.8%. As for DLSTM, the recognition rate can reach 100% because of the effectiveness of the deep structure, but compared with the single layer LSTMP, DLSTM needs more training time.

Phone Synchronous Speech Recognition With CTC Lattices

Connectionist temporal classification CTC has recently shown improved performance and efficiency in automatic speech recognition. One popular decoding implementation is to use a CTC model to predict the phone posteriors at each frame and then perform Viterbi beam search on a modified WFST network. This is still within the traditional frame synchronous decoding framework. In this paper, the peaky posterior property of CTC is carefully investigated and it is found that ignoring blank frames will not introduce additional search errors. Based on this phenomenon, a novel phone synchronous decoding framework is proposed by removing tremendous search redundancy due to blank frames, which results in significant search speed up. The framework naturally leads to an extremely compact phone-level acoustic space representation: CTC lattice. With CTC lattice, efficient and effective modular speech recognition approaches, second pass rescoring for large vocabulary continuous speech recognition LVCSR, and phone-based keyword spotting KWS, are also proposed in this paper. Experiments showed that phone synchronous decoding can achieve 3-4 times search speed up without performance degradation compared to frame synchronous decoding. Modular LVCSR with CTC lattice can achieve further WER improvement. KWS with CTC lattice not only achieved significant equal error rate improvement, but also greatly reduced the KWS model size and increased the search speed.

End-to-end neural network modeling for Japanese speech recognition

This study proposes end-to-end neural network modeling to adapt direct speech-to-text decoding to Japanese. End-to-end speech recognition systems using deep neural networks (DNNs) are currently being investigated. These systems do not need intermediate phonetic representation. Instead, many of them utilize Recurrent Neural Networks (RNNs) trained by using much more data than ever before. The end-to-end approach makes acoustic models simpler to train. Typically, previous works have dealt with phonogram labels such alphabetic characters. Ideograms such as Kanji, however, make end-to-end speech recognition more complex. A single Kanji can have multiple readings, such as On-yomi (Chinese reading) and Kun-yomi (Japanese reading). In addition, whereas alphabets have at most 100 labels, Japanese has over 2000 labels to predict, such as Kanji, Hiragana, Katakana, the Roman alphabet, digits, and punctuation marks. To resolve this problem, we attempt to make end-to-end neural network modeling allows speech recognition of Japanese without phonetic representation. This method trains RNN and adopts the connectionist temporal classification (CTC) objective function. The proposed method was able to deal with a large amount of character labels. We also analyzed the decoding results and examined ideas for improving the accuracy of word error rate (WER).

Regular expressions for decoding of neural network outputs

This article proposes a convenient tool for decoding the output of neural networks trained by Connectionist Temporal Classification (CTC) for handwritten text recognition. We use regular expressions to describe the complex structures expected in the writing. The corresponding finite automata are employed to build a decoder. We analyze theoretically which calculations are relevant and which can be avoided. A great speed-up results from an approximation. We conclude that the approximation most likely fails if the regular expression does not match the ground truth which is not harmful for many applications since the low probability will be even underestimated. The proposed decoder is very efficient compared to other decoding methods. The variety of applications reaches from information retrieval to full text recognition. We refer to applications where we integrated the proposed decoder successfully.

Improved End-to-End Speech Recognition Using Adaptive Per-Dimensional Learning Rate Methods

The introduction of deep neural networks (DNNs) leads to a significant improvement of the automatic speech recognition (ASR) performance. However, the whole ASR system remains sophisticated due to the dependent on the hidden Markov model (HMM). Recently, a new end-to-end ASR framework, which utilizes recurrent neural networks (RNNs) to directly model context-independent targets with connectionist temporal classification (CTC) objective function, is proposed and achieves comparable results with the hybrid HMM/DNN system. In this paper, we investigate per-dimensional learning rate methods, ADAGRAD and ADADELTA included, to improve the recognition of the end-to-end system, based on the fact that the blank symbol used in CTC technique dominates the output and these methods give frequent features small learning rates. Experiment results show that more than 4% relative reduction of word error rate (WER) as well as 5% absolute improvement of label accuracy on the training set are achieved when using ADADELTA, and fewer epochs of training are needed.

Offline cursive Urdu-Nastaliq script recognition using multidimensional recurrent neural networks

Optical Character Recognition of cursive scripts remains a challenging task due to a large number of character shapes, inter- and intra-word overlaps, context sensitivity and diagonality of text. This paper presents an implicit segmentation based recognition system for Urdu text lines in Nastaliq script. The proposed technique relies on sliding overlapped windows on lines of text and extracting a set of statistical features. The extracted features are fed to a multi-dimensional long short term memory recurrent neural network (MDLSTM RNN) with a connectionist temporal classification (CTC) output layer that labels the character sequences. Experimental study of the proposed technique is carried out on the standard Urdu Printed Text-line Images (UPTI) database which comprises 10,000 text lines in Nastaliq font. Evaluations under different experimental settings realize promising recognition rates with a highest character recognition rate of 96.40%.

Keyword spotting exploiting Long Short-Term Memory

We investigate various techniques for keyword spotting which are exclusively based on acoustic modeling and do not presume the existence of an in-domain language model. Since adequate context modeling is nevertheless necessary for word spotting, we show how the principle of Long Short-Term Memory (LSTM) can be incorporated into the decoding process. We propose a novel technique that exploits LSTM in combination with Connectionist Temporal Classification in order to improve performance by using a self-learned amount of contextual information. All considered approaches are evaluated on read speech as contained in the TIMIT corpus as well as on the SEMAINE database which consists of spontaneous and emotionally colored speech. As further evidence for the effectiveness of LSTM modeling for keyword spotting, results on the CHiME task are shown.

Noise robust ASR in reverberated multisource environments applying convolutive NMF and Long Short-Term Memory

This article proposes and evaluates various methods to integrate the concept of bidirectional Long Short-Term Memory (BLSTM) temporal context modeling into a system for automatic speech recognition (ASR) in noisy and reverberated environments. Building on recent advances in Long Short-Term Memory architectures for ASR, we design a novel front-end for context-sensitive Tandem feature extraction and show how the Connectionist Temporal Classification approach can be used as a BLSTM-based back-end, alternatively to Hidden Markov Models (HMM). We combine context-sensitive BLSTM-based feature generation and speech decoding techniques with source separation by convolutive non-negative matrix factorization. Applying our speaker adapted multi-stream HMM framework that processes MFCC features from NMF-enhanced speech as well as word predictions obtained via BLSTM networks and non-negative sparse classification (NSC), we obtain an average accuracy of 91.86% on the PASCAL CHiME Challenge task at signal-to-noise ratios ranging from −6 to 9dB. To our knowledge, this is the best result ever reported for the CHiME Challenge task.