Connectionist Temporal Classification Loss Research Articles

LipNet: End-to-End Lipreading

Lipreading is the task of decoding text from the movement of a speaker’s mouth. This research presents the development of an advanced end-to-end lipreading system. Leveraging deep learning architectures and multimodal fusion techniques, the proposed system interprets spoken language solely from visual cues, such as lip movements. Through meticulous data collection, annotation, preprocessing, model development, and evaluation, diverse datasets encompassing various speakers, accents, languages, and environmental conditions are curated to ensure robustness and generalization. Conventional methods divided the task into two phases: prediction and designing or learning visual characteristics. Most deep lipreading methods are trainable from end to end. In the past, lipreading has been tackled using tedious and sometimes unsatisfactory techniques that break down speech into smaller units like phonemes or visemes. But these methods often fail when faced with real-world problems, such contextual factors, accents, and differences in speech patterns. Nevertheless, current research on end-to-end trained models only carries out word classification; sentence-level sequence prediction is not included. LipNet is an end-to-end trained model that uses spatiotemporal convolutions, a recurrent network, and the connectionist temporal classification loss to translate a variable-length sequence of video frames to text. LipNet breaks from this traditional paradigm by using an all-encompassing, end-to-end approach supported by deep learning algorithms, Convolutional neural networks (CNNs) and recurrent neural networks (RNNs), which are skilled at processing sequential data and extracting high-level representations, are fundamental to LipNet's architecture.LipNet achieves 95.2% accuracy in sentence-level on the GRID corpus, overlapped speaker split task, outperforming experienced human lipreaders and the previous 86.4% word-level state-of-the-art accuracy.The results underscore the transformative potential of the lipreading system in real-world applications, particularly in domains such as assistive technology and human-computer interaction, where it can significantly improve communication accessibility and inclusivity for individuals with hearing impairments.

SilentInterpreter: Analysis of Lip Movement and Extracting Speech Using Deep Learning

Lip reading presents a captivating avenue for advancing speech recognition algorithms, leveraging visual cues from lip movements to recognise spoken words. This paper introduces a novel method employing deep neural networks to convert lip motions into textual representations. The methodology integrates convolutional neural networks for visual feature extraction, recurrent neural networks to capture temporal context, and the Connectionist Temporal Classification loss function for aligning lip features with phonemes. Additionally, dynamic learning rate scheduling and a unique callback mechanism for training visualization are incorporated into the process. Post-training on a sizeable dataset, the model demonstrates notable convergence, showcasing its ability to discern intricate temporal correlations. Comprehensive evaluations, combining quantitative metrics and qualitative assessments, validate the model's effectiveness. Visual inspections of lip reading capabilities and standard speech recognition criteria evaluation highlight its performance. The study delves into the impact of various model topologies and hyperparameters on performance, providing valuable insights for future research directions. This research contributes a deep learning framework for accurate and efficient speech recognition, expanding the landscape of lip reading technologies. The findings open paths for further refinement and deployment across diverse domains, including assistive technologies, audio-visual communication systems, and human-computer interaction. Keywords: lip reading, deep learning, convolutional neural networks, recurrent neural networks, CTC loss, speech recognition

Survey on Silentinterpreter : Analysis of Lip Movement and Extracting Speech using Deep Learning

Lip reading is a complex but interesting path for the growth of speech recognition algorithms. It is the ability of deciphering spoken words by evaluating visual cues from lip movements. In this study, we suggest a unique method for lip reading that converts lip motions into textual representations by using deep neural networks. Convolutional neural networks are used in the methodology to extract visual features, recurrent neural networks are used to simulate temporal context, and the Connectionist Temporal Classification loss function is used to align lip features with corresponding phonemes. The study starts with a thorough investigation of data loading methods, which include alignment extraction and video preparation. A well selected dataset with video clips and matching phonetic alignments is presented. We select relevant face regions, convert frames to grayscale, then standardize the resulting data so that it can be fed into a neural network. The neural network architecture is presented in depth, displaying a series of bidirectional LSTM layers for temporal context understanding after 3D convolutional layers for spatial feature extraction. Careful consideration of input shapes, layer combinations, and parameter selections forms the foundation of the model's design. To train the model, we align predicted phoneme sequences with ground truth alignments using the CTC loss. Dynamic learning rate scheduling and a unique callback mechanism for training visualization of predictions are integrated into the training process. After training on a sizable dataset, the model exhibits remarkable convergence and proves its capacity to understand intricate temporal correlations. Through the use of both quantitative and qualitative evaluations, the results are thoroughly assessed. We visually check the model's lip reading abilities and assess its performance using common speech recognition criteria. It is explored how different model topologies and hyperparameters affect performance, offering guidance for future research. The trained model is tested on external video samples to show off its practical application. Its accuracy and resilience in lip-reading spoken phrases are demonstrated. By providing a deep learning framework for precise and effective speech recognition, this research adds to the rapidly changing field of lip reading devices. The results offer opportunities for additional development and implementation in various fields, such as assistive technologies, audio-visual communication systems, and human-computer interaction.

Self-Distillation Regularized Connectionist Temporal Classification Loss for Text Recognition: A Simple Yet Effective Approach

Text recognition methods are gaining rapid development. Some advanced techniques, e.g., powerful modules, language models, and un- and semi-supervised learning schemes, consecutively push the performance on public benchmarks forward. However, the problem of how to better optimize a text recognition model from the perspective of loss functions is largely overlooked. CTC-based methods, widely used in practice due to their good balance between performance and inference speed, still grapple with accuracy degradation. This is because CTC loss emphasizes the optimization of the entire sequence target while neglecting to learn individual characters. We propose a self-distillation scheme for CTC-based model to address this issue. It incorporates a framewise regularization term in CTC loss to emphasize individual supervision, and leverages the maximizing-a-posteriori of latent alignment to solve the inconsistency problem that arises in distillation between CTC-based models. We refer to the regularized CTC loss as Distillation Connectionist Temporal Classification (DCTC) loss. DCTC loss is module-free, requiring no extra parameters, longer inference lag, or additional training data or phases. Extensive experiments on public benchmarks demonstrate that DCTC can boost text recognition model accuracy by up to 2.6%, without any of these drawbacks.

Improving Continuous Sign Language Recognition with Consistency Constraints and Signer Removal

Deep-learning-based continuous sign language recognition (CSLR) models typically consist of a visual module, a sequential module, and an alignment module. However, the effectiveness of training such CSLR backbones is hindered by limited training samples, rendering the use of a single connectionist temporal classification loss insufficient. To address this limitation, we propose three auxiliary tasks to enhance CSLR backbones. First, we enhance the visual module, which is particularly sensitive to the challenges posed by limited training samples, from the perspective of consistency. Specifically, since sign languages primarily rely on signers’ facial expressions and hand movements to convey information, we develop a keypoint-guided spatial attention module that directs the visual module to focus on informative regions, thereby ensuring spatial attention consistency. Furthermore, recognizing that the output features of both the visual and sequential modules represent the same sentence, we leverage this prior knowledge to better exploit the power of the backbone. We impose a sentence embedding consistency constraint between the visual and sequential modules, enhancing the representation power of both features. The resulting CSLR model, referred to as consistency-enhanced CSLR, demonstrates superior performance on signer-dependent datasets, where all signers appear during both training and testing. To enhance its robustness for the signer-independent setting, we propose a signer removal module based on feature disentanglement, effectively eliminating signer-specific information from the backbone. To validate the effectiveness of the proposed auxiliary tasks, we conduct extensive ablation studies. Notably, utilizing a transformer-based backbone, our model achieves state-of-the-art or competitive performance on five benchmarks, including PHOENIX-2014, PHOENIX-2014-T, PHOENIX-2014-SI, CSL, and CSL-Daily. Code and models are available at https://github.com/2000ZRL/LCSA_C2SLR_SRM.

Efficient CRNN: Towards end-to-end low resource Urdu text recognition using depthwise separable convolutions and gated recurrent units

In this study, a novel technique is proposed to recognize printed text in images for Urdu, a low-resource language with a scarcity of benchmark datasets. The proposed technique is called Efficient CRNN which uses depthwise separable convolutional and bidirectional gated recurrent unit layers, followed by connectionist temporal classification loss. The proposed technique is computationally more efficient than the existing text recognition techniques, requiring fewer parameters and computations. A multi-font printed Urdu text lines corpus is also presented, consisting of 245,000 text line images rendered using 7 different fonts. The corpus is called the MMU-Extension-22 and is used to train and evaluate existing state-of-the-art end-to-end text recognition techniques. Efficient CRNN is also evaluated using the proposed corpus. The proposed technique is first trained using a total of 196,000 text line images and then tested using 49,000 images. The Efficient CRNN technique achieved the minimum character and word error rates of 0.91% and 1.49% respectively for Urdu text line recognition under different settings, outperforming the existing computationally more complex techniques. The simple nature of the proposed technique not only makes it more efficient but also more robust for Urdu text line recognition, achieving a 2.23% reduced character error rate and a 71%11Percentage Decrease = 100 * (Baseline Value - Changed Value)/Baseline Value. decrease in character error rate as compared to the best performing existing Recurrent Neural Networks based technique. Also, the proposed technique outperforms Vision Transformer-based network achieving a 0.79% reduced character error rate accounting for a 41% decrease in error. Also, the Efficient CRNN has 49.16% reduced parameters compared to the baseline Vision Transformer technique.

SoftCorrect: Error Correction with Soft Detection for Automatic Speech Recognition

Error correction in automatic speech recognition (ASR) aims to correct those incorrect words in sentences generated by ASR models. Since recent ASR models usually have low word error rate (WER), to avoid affecting originally correct tokens, error correction models should only modify incorrect words, and therefore detecting incorrect words is important for error correction. Previous works on error correction either implicitly detect error words through target-source attention or CTC (connectionist temporal classification) loss, or explicitly locate specific deletion/substitution/insertion errors. However, implicit error detection does not provide clear signal about which tokens are incorrect and explicit error detection suffers from low detection accuracy. In this paper, we propose SoftCorrect with a soft error detection mechanism to avoid the limitations of both explicit and implicit error detection. Specifically, we first detect whether a token is correct or not through a probability produced by a dedicatedly designed language model, and then design a constrained CTC loss that only duplicates the detected incorrect tokens to let the decoder focus on the correction of error tokens. Compared with implicit error detection with CTC loss, SoftCorrect provides explicit signal about which words are incorrect and thus does not need to duplicate every token but only incorrect tokens; compared with explicit error detection, SoftCorrect does not detect specific deletion/substitution/insertion errors but just leaves it to CTC loss. Experiments on AISHELL-1 and Aidatatang datasets show that SoftCorrect achieves 26.1% and 9.4% CER reduction respectively, outperforming previous works by a large margin, while still enjoying fast speed of parallel generation.

TLWSR: Weakly supervised real‐world scene text image super‐resolution using text label

AbstractScene text image super‐resolution (STISR) has recently received considerable attention. Existing STISR methods are applicable to the situation that all the LR‐HR pairs are available. However, in real‐world scenarios, it is difficult and expensive to collect ground‐truth HR labels and align them with LR images, and thus it is essential to find a way to implement weakly supervised learning. We investigate the STISR problem in the situation that only a subset of HR labels is available and design a weak supervision framework using coarse‐grained text labels named TLWSR, which combines incomplete supervision and inexact supervision. Specifically, a lightweight text recognition network and connectionist temporal classification loss are used to guide the super‐resolution of text images during training. Extensive experiments on the benchmark TextZoom demonstrate that TLWSR generates distinguishable text images and exceeds the fully supervised baseline TSRN in boosting text recognition accuracywith only 50% HR labels available. Meanwhile, TLWSR can be applied to different super‐resolution backbones and significantly improves their performance. Furthermore, TLWSR shows good generalization capability to low‐quality images on scene text recognition benchmarks, which verifies the effectiveness of this framework. To the authors' knowledge, this is the first work exploring the problem of STISR in weakly supervised scenarios.

Speech Recognition for Air Traffic Control via Feature Learning and End-to-End Training

In this work, we propose a new automatic speech recognition (ASR) system based on feature learning and an end-to-end training procedure for air traffic control (ATC) systems. The proposed model integrates the feature learning block, recurrent neural network (RNN), and connectionist temporal classification loss to build an end-to-end ASR model. Facing the complex environments of ATC speech, instead of the handcrafted features, a learning block is designed to extract informative features from raw waveforms for acoustic modeling. Both the SincNet and 1D convolution blocks are applied to process the raw waveforms, whose outputs are concatenated to the RNN layers for the temporal modeling. Thanks to the ability to learn representations from raw waveforms, the proposed model can be optimized in a complete end-to-end manner, i.e., from waveform to text. Finally, the multilingual issue in the ATC domain is also considered to achieve the ASR task by constructing a combined vocabulary of Chinese characters and English letters. The proposed approach is validated on a multilingual real-world corpus (ATCSpeech), and the experimental results demonstrate that the proposed approach outperforms other baselines, achieving a 6.9% character error rate.

Training a Singing Transcription Model Using Connectionist Temporal Classification Loss and Cross-Entropy Loss

In this paper, we propose a method that uses a combination of the Connectionist Temporal Classification (CTC) loss and the cross-entropy loss to train a note-level singing transcription model. By considering the task as predicting a note sequence of the input audio, we can compute the CTC loss between the prediction and the groundtruth note sequence, and further use it with the traditional cross-entropy loss to optimize the transcription model. By comparing the proposed method with a baseline that only utilizes the cross-entropy loss, the results show improved model performance on all the evaluation metrics. Furthermore, using the CTC loss allows the transcription model to learn from weakly labeled data, which is easier to annotate than traditional strongly labeled data. Moreover, we point out the issue of the intrinsic global time shift on the onset labels between datasets. By automatically estimating and calibrating the global time shift of the training dataset, the performance of the singing transcription model is then not affected by the global time shift in the cross-dataset evaluation scenario.

Seamless equal accuracy ratio for inclusive CTC speech recognition

Concerns have been raised regarding performance disparity in automatic speech recognition (ASR) systems as they provide unequal transcription accuracy for different user groups defined by different attributes that include gender, dialect, and race. In this paper, we propose “equal accuracy ratio”, a novel inclusiveness measure for ASR systems that can be seamlessly integrated into the standard connectionist temporal classification (CTC) training pipeline of an end-to-end neural speech recognizer to increase the recognizer’s inclusiveness. We also create a novel multi-dialect benchmark dataset to study the inclusiveness of ASR, by combining data from existing corpora in seven dialects of English (African American, General American, Latino English, British English, Indian English, Afrikaaner English, and Xhosa English). Experiments on this multi-dialect corpus show that using the equal accuracy ratio as a regularization term along with CTC loss, succeeds in lowering the accuracy gap between user groups and reduces the recognition error rate compared with a non-regularized baseline. Experiments on additional speech corpora that have different user groups also confirm our findings.

Towards multilingual end‐to‐end speech recognition for air traffic control

In this work, an end-to-end framework is proposed to achieve multilingual automatic speech recognition (ASR) in air traffic control (ATC) systems. Considering the standard ATC procedure, a recurrent neural network (RNN) based framework is selected to mine the temporal dependencies among speech frames. Facing the distributed feature space caused by the radio transmission, a hybrid feature embedding block is designed to extract high-level representations, in which multiple convolutional neural networks are designed to accommodate different frequency and temporal resolutions. The residual mechanism is performed on the RNN layers to improve the trainability and the convergence. To integrate the multilingual ASR into a single model and relieve the class imbalance, a special vocabulary is designed to unify the pronunciation of the vocabulary in Chinese and English, i.e., pronunciation-oriented vocabulary. The proposed model is optimized by the connectionist temporal classification loss and is validated on a real-world speech corpus (ATCSpeech). A character error rate of 4.4% and 5.9% is achieved for Chinese and English speech, respectively, which outperforms other popular approaches. Most importantly, the proposed approach achieves the multilingual ASR task in an end-to-end manner with considerable high performance.

Affect-salient event sequence modelling for continuous speech emotion recognition

Continuous speech emotion recognition, which faces the problems of delay caused by annotators’ reaction time and noise caused by non-emotional segments, is a challenging subject in the field of affective computing. To solve these problems, we propose a new affect-salient event sequence modelling (ASESM) method based on connectionist temporal classification (CTC). This method treats a sentence’s label as a chain of affect-salient event (ASE) and non-affect-salient event Null states rather than continuous emotional value. With this representation, a CTC-based convolutional neural network (CNN) is built to automatically label the sentence’s emotional segments with ASE and non-emotional segments with Null, so as to reduce the impact of noise caused by non-emotional segments. Furthermore, we propose an event probability vector decoding (EPVD) algorithm to search the optimal ASE sequence from the CTC loss matrix and mark the occurrence time of each event within this sequence. Then, the arousal and valence ground-truth annotations of each ASE are used to represent the continuous emotional value of a segment which is predicted as the ASE. Since the ground-truth annotations of each ASE have contained different time-delays, taking events as the target can avoid the additional reaction delay compensation. We test our method on the RECOLA and AVEC 2014 benchmark databases. The experimental results demonstrate that the proposed event-based method can improve the performance of continuous emotion recognition and the improvement is more obvious when the selected ASE has high annotation consistency.

Connectionist temporal classification loss for vector quantized variational autoencoder in zero-shot voice conversion

Vector quantized variational autoencoder (VQ-VAE) has recently become an increasingly popular method in non-parallel zero-shot voice conversion (VC). The reason behind is that VQ-VAE is capable of disentangling the content and the speaker representations from the speech by using a content encoder and a speaker encoder, which is suitable for the VC task that makes the speech of a source speaker sound like the speech of the target speaker without changing the linguistic content. However, the converted speech is not satisfying because it is difficult to disentangle the pure content representations from the acoustic features due to the lack of linguistic supervision for the content encoder. To address this issue, under the framework of VQ-VAE, connectionist temporal classification (CTC) loss is proposed to guide the content encoder to learn pure content representations by using an auxiliary network. Based on the fact that the CTC loss is not affected by the sequence length of the output of the content encoder, adding the linguistic supervision to the content encoder can be much easier. This non-parallel many-to-many voice conversion model is named as CTC-VQ-VAE. VC experiments on the CMU ARCTIC and VCTK corpus are carried out to evaluate the proposed method. Both the objective and the subjective results show that the proposed approach significantly improves the speech quality and speaker similarity of the converted speech, compared with the traditional VQ-VAE method.

Single-Stage Intake Gesture Detection Using CTC Loss and Extended Prefix Beam Search.

Accurate detection of individual intake gestures is a key step towards automatic dietary monitoring. Both inertial sensor data of wrist movements and video data depicting the upper body have been used for this purpose. The most advanced approaches to date use a two-stage approach, in which (i) frame-level intake probabilities are learned from the sensor data using a deep neural network, and then (ii) sparse intake events are detected by finding the maxima of the frame-level probabilities. In this study, we propose a single-stage approach which directly decodes the probabilities learned from sensor data into sparse intake detections. This is achieved by weakly supervised training using Connectionist Temporal Classification (CTC) loss, and decoding using a novel extended prefix beam search decoding algorithm. Benefits of this approach include (i) end-to-end training for detections, (ii) simplified timing requirements for intake gesture labels, and (iii) improved detection performance compared to existing approaches. Across two separate datasets, we achieve relative F1 score improvements between 1.9% and 6.2% over the two-stage approach for intake detection and eating/drinking detection tasks, for both video and inertial sensors.

Brain2Char: a deep architecture for decoding text from brain recordings

Objective. Decoding language representations directly from the brain can enable new brain–computer interfaces (BCIs) for high bandwidth human–human and human–machine communication. Clinically, such technologies can restore communication in people with neurological conditions affecting their ability to speak. Approach. In this study, we propose a novel deep network architecture Brain2Char, for directly decoding text (specifically character sequences) from direct brain recordings (called electrocorticography, ECoG). Brain2Char framework combines state-of-the-art deep learning modules—3D Inception layers for multiband spatiotemporal feature extraction from neural data and bidirectional recurrent layers, dilated convolution layers followed by language model weighted beam search to decode character sequences, and optimizing a connectionist temporal classification loss. Additionally, given the highly non-linear transformations that underlie the conversion of cortical function to character sequences, we perform regularizations on the network’s latent representations motivated by insights into cortical encoding of speech production and artifactual aspects specific to ECoG data acquisition. To do this, we impose auxiliary losses on latent representations for articulatory movements, speech acoustics and session specific non-linearities. Main results. In three (out of four) participants reported here, Brain2Char achieves 10.6%, 8.5%, and 7.0% word error rates respectively on vocabulary sizes ranging from 1200 to 1900 words. Significance. These results establish a new end-to-end approach on decoding text from brain signals and demonstrate the potential of Brain2Char as a high-performance communication BCI.

A Unified Framework for Multilingual Speech Recognition in Air Traffic Control Systems

This work focuses on robust speech recognition in air traffic control (ATC) by designing a novel processing paradigm to integrate multilingual speech recognition into a single framework using three cascaded modules: an acoustic model (AM), a pronunciation model (PM), and a language model (LM). The AM converts ATC speech into phoneme-based text sequences that the PM then translates into a word-based sequence, which is the ultimate goal of this research. The LM corrects both phoneme- and word-based errors in the decoding results. The AM, including the convolutional neural network (CNN) and recurrent neural network (RNN), considers the spatial and temporal dependences of the speech features and is trained by the connectionist temporal classification loss. To cope with radio transmission noise and diversity among speakers, a multiscale CNN architecture is proposed to fit the diverse data distributions and improve the performance. Phoneme-to-word translation is addressed via a proposed machine translation PM with an encoder-decoder architecture. RNN-based LMs are trained to consider the code-switching specificity of the ATC speech by building dependences with common words. We validate the proposed approach using large amounts of real Chinese and English ATC recordings and achieve a 3.95% label error rate on Chinese characters and English words, outperforming other popular approaches. The decoding efficiency is also comparable to that of the end-to-end model, and its generalizability is validated on several open corpora, making it suitable for real-time approaches to further support ATC applications, such as ATC prediction and safety checking.

Connectionist Temporal Classification Model for Dynamic Hand Gesture Recognition using RGB and Optical flow Data

Automatic classification of dynamic hand gesture is challenging due to the large diversity in a different class of gesture, Low resolution, and it is performed by finger. Due to a number of challenges many researchers focus on this area. Recently deep neural network can be used for implicit feature extraction and Soft Max layer is used for classification. In this paper, we propose a method based on a two-dimensional convolutional neural network that performs detection and classification of hand gesture simultaneously from multimodal Red, Green, Blue, Depth (RGBD) and Optical flow Data and passes this feature to Long-Short Term Memory (LSTM) recurrent network for frame-to-frame probability generation with Connectionist Temporal Classification (CTC) network for loss calculation. We have calculated an optical flow from Red, Green, Blue (RGB) data for getting proper motion information present in the video. CTC model is used to efficiently evaluate all possible alignment of hand gesture via dynamic programming and check consistency via frame-to-frame for the visual similarity of hand gesture in the unsegmented input stream. CTC network finds the most probable sequence of a frame for a class of gesture. The frame with the highest probability value is selected from the CTC network by max decoding. This entire CTC network is trained end-to-end with calculating CTC loss for recognition of the gesture. We have used challenging Vision for Intelligent Vehicles and Applications (VIVA) dataset for dynamic hand gesture recognition captured with RGB and Depth data. On this VIVA dataset, our proposed hand gesture recognition technique outperforms competing state-of-the-art algorithms and gets an accuracy of 86%

Keyword retrieving in continuous speech using connectionist temporal classification

The issue of Speech Keyword Retrieval (SKR) has received considerable critical attention. SKR aims to retrieve data from a speech repository given by a spoken query. The accuracy of retrieval often depends on the performance of acoustic model. In this paper, we proposed a new speech keyword retrieval framework called DCNN-CTC using Deep Convolutional Neural Network (DCNN) based on Connectionist Temporal Classification (CTC). The proposed method provides new insights into multimedia information retrieval. The pre-trained models are fine-tuned with a CTC loss to predict target keywords, and the features are extracted by DCNN, which is a complete end-to-end acoustic model training. It does not need to align and label the data one by one in advance, and CTC directly outputs the probability of sequence prediction, which greatly improves the processing performance of the speech retrieval system. Our experimental results on benchmark datasets show that our approach leads to stable and robust retrieval performance, and the precision rate and recall rate of DCNN-CTC are much higher than the baseline system.

A new joint CTC-attention-based speech recognition model with multi-level multi-head attention

A method called joint connectionist temporal classification (CTC)-attention-based speech recognition has recently received increasing focus and has achieved impressive performance. A hybrid end-to-end architecture that adds an extra CTC loss to the attention-based model could force extra restrictions on alignments. To explore better the end-to-end models, we propose improvements to the feature extraction and attention mechanism. First, we introduce a joint model trained with nonnegative matrix factorization (NMF)-based high-level features. Then, we put forward a hybrid attention mechanism by incorporating multi-head attentions and calculating attention scores over multi-level outputs. Experiments on TIMIT indicate that the new method achieves state-of-the-art performance with our best model. Experiments on WSJ show that our method exhibits a word error rate (WER) that is only 0.2% worse in absolute value than the best referenced method, which is trained on a much larger dataset, and it beats all present end-to-end methods. Further experiments on LibriSpeech show that our method is also comparable to the state-of-the-art end-to-end system in WER.