Attention-based Encoder-decoder Research Articles

Combining hybrid DNN-HMM ASR systems with attention-based models using lattice rescoring

The traditional hybrid deep neural network (DNN)–hidden Markov model (HMM) system and attention-based encoder–decoder (AED) model are both commonly used automatic speech recognition (ASR) approaches with distinct characteristics and advantages. While hybrid systems are per-frame-based and highly modularised to leverage external phonetic and linguistic knowledge, AED models operate on a per-label basis and jointly learn the acoustic and language information using a single model in an end-to-end trainable fashion. In this paper, we propose combining these two approaches in a two-pass rescoring framework. The first-pass uses hybrid ASR systems to facilitate streaming and controllable ASR, and the second-pass re-scores the N-best hypotheses or lattices produced by the first-pass hybrid DNN-HMM system with AED models. We also propose an improved algorithm for lattice rescoring with AED models. Experiments show the combined two-pass systems achieve competitive performance without using extra speech or text data on two standard ASR tasks. For the 80-hour AMI IHM dataset, the combined system has a 13.7% word error rate (WER) on the evaluation set and is up to a 29% relative WER reduction over the individual systems. For the 300-hour Switchboard dataset, the WERs of the combined system are 5.7% and 12.1% on Switchboard and CallHome subsets of Hub5’00, and 13.2% and 7.6% on Switchboard Cellular and Fisher subsets of RT03, and are up to a 33% relative reduction in WER over the individual systems.

Attention-Based Multiscale Spatiotemporal Network for Traffic Forecast with Fusion of External Factors

Periodic traffic prediction and analysis is essential for urbanisation and intelligent transportation systems (ITS). However, traffic prediction is challenging due to the nonlinear flow of traffic and its interdependencies on spatiotemporal features. Traffic flow has a long-term dependence on temporal features and a short-term dependence on local and global spatial features. It is strongly influenced by external factors such as weather and points of interest. Existing models consider long-term and short-term predictions in Euclidean space. In this paper, we design an attention-based encoder–decoder with stacked layers of LSTM to analyse multiscale spatiotemporal dependencies in non-Euclidean space to forecast traffic. The attention weights are obtained adaptively and external factors are fused with the output of the decoder to evaluate region-wide traffic predictions. Extensive experiments are conducted to evaluate the performance of the proposed attention-based non-Euclidean spatiotemporal network (ANST) on real-world datasets. The proposed model has improved prediction accuracy over previous methods. The insights obtained from traffic prediction would be beneficial for daily commutation and logistics.

Multi-step forecasting of multivariate time series using multi-attention collaborative network

Multi-step forecasting of multivariate time series plays a critical role in many fields, such as disaster warning and financial analysis. While attention-based recurrent neural networks (RNNs) achieved encouraging performance, two limitations exist in current models: i) Existing approaches merely focus on variables’ interactions, and ignore the negative noise of non-predictive variables, ii) These methods cannot model the difference in the temporal importance of the target and the non-predictive series to prediction. To tackle these challenges, we propose a triangle structured Multi-Attention Collaborative Network (MACN), which includes a backbone network T-net with attention-based encoder–decoder framework, and an auxiliary hierarchical network NP-net. NP-net focuses on non-predictive variables, capturing the most relevant variables and temporal dependencies through the proposed variables-distillation attention network (VDN) and long short-term memory network (LSTM). T-net executes on target variable, and its encoder and decoder are both connected to NP-net, thereby using the output of NP-net to assist learning and decision-making. Specifically we design a knowledge-enhanced LSTM (KeLSTM) as the encoder and decoder of T-net. In the coding stage, KeLSTM refines the output of NP-net to strengthen the latent semantics of the target variable. In the decoding stage, KeLSTM captures subtle differences between the target and the non-predictive variables’ contribution to prediction, and improves model’s predictive ability by alleviating such conflicts. Experiments on three real-world datasets demonstrate that MACN outperforms different types of state-of-the-art methods.

Multiple attention-based encoder–decoder networks for gas meter character recognition

Factories swiftly and precisely grasp the real-time data of the production instrumentation, which is the foundation for the development and progress of industrial intelligence in industrial production. Weather, light, angle, and other unknown circumstances, on the other hand, impair the image quality of meter dials in natural environments, resulting in poor dial image quality. The remote meter reading system has trouble recognizing dial pictures in extreme settings, challenging it to meet industrial production demands. This paper provides multiple attention and encoder–decoder-based gas meter recognition networks (MAEDR) for this problem. First, from the acquired dial photos, the dial images with extreme conditions such as overexposure, artifacts, blurring, incomplete display of characters, and occlusion are chosen to generate the gas meter dataset. Then, a new character recognition network is proposed utilizing multiple attention and an encoder–decoder structure. Convolutional neural networks (CNN) extract visual features from dial images, encode visual features employing multi-head self-attention and position information, and facilitate feature alignment using the connectionist temporal classification (CTC) method. A novel two-step attention decoder is presented to improve the accuracy of recognition results. convolutional block attention module (CBAM) reweights the visual features from the CNN and the semantic features computed by the encoder to improve model performance; long short-term memory attention (LSTM attention) focuses on the relationship between feature sequences. According to experimental data, our system can effectively and efficiently identify industrial gas meter picture digits with 91.1% identification accuracy, faster inference speed, and higher accuracy than standard algorithms. The accuracy and practicality of the recognition can fulfill the needs of instrument data detection and recognition in industrial production, and it has a wide range of applications.

Monotonic Gaussian regularization of attention for robust automatic speech recognition

The Attention-based Encoder–Decoder (AED) models are one of the most popular models for Automatic Speech Recognition (ASR). However, instability can occur in AED with problems such as incorrect insertions or word repetitions due to the violation of the inherent monotonic alignment property. To address these problems, we propose a monotonic Gaussian regularization method to guide the attention training, where the guiding map is depicted as a sequence of Gaussian distributions with monotonically moving centers. Experiments show our method reduces the insertion error rate by a relative 7% on the HKUST dataset, relative 20% and 16% on two large industrial datasets, and a relative 21% on an out-of-domain test set. The overall Character Error Rates (CERs) are all reduced at the same time, indicating that the model’s recognition ability is well maintained. Therefore, our proposed method improves model performance by enhancing monotonic alignment, and provides better robustness.

Dynamic Co-Attention Networks for multi-horizon forecasting in multivariate time series

Although attention-based encoder–decoder models achieve encouraging performance in multivariate time series multi-horizon forecasting, two key limitations exist in current methods: (i) These methods merely focus on the spatial or temporal correlations among series, and ignore harmful interactions between non-predictive and target variables. (ii) Time series forecasting benefits significantly from knowledge of the recent past, but existing approaches only consider the encoder and neglect the previous decoder units. To tackle these challenges, we propose a novel prediction framework, termed Dynamic Co-Attention Networks (DCAN). For the first issue, we propose a two-stage variables embedding network as DCAN’s underlying structure to better capture the potential semantics that integrates target and non-predictive variables. The first stage selects the most important non-predictive variables and generates a spatial vector, and the second stage trades off how much information the potential semantics consider from the target variable and spatial vector. To alleviate the second issue, we design a dynamic attention synergetic network (DASN) as the superstructure of the proposed model, including a contextual attention and a filtration gate. A contextual attention expands the observation horizon for current prediction by continuously retrieving previous decoder states, and a filtration gate module selectively incorporates previous decoder information into the current decoder unit, thus improving the credibility and accuracy of the model decision-making. Experiments on three real-life datasets demonstrate that DCAN outperforms different types of state-of-the-art baselines.

CloudRaednet: residual attention-based encoder–decoder network for ground-based cloud images segmentation in nychthemeron

ABSTRACT Obtaining accurate cloudage information through ground-based cloud observation is of great significance to astronomical telescope observatory site selection. This paper proposes a residual attention-based encoder–decoder network (CloudRAEDNet) for ground-based cloud image segmentation in nychthemeron. CloudRAEDNet uses ImageNet pre-trained ResNet50 as the encoder backbone network, which reduces the number of network training. The network decoder introduces residual modules to solve the problem of network degradation caused by the increase in the number of network layers. CloudRAEDNet connects encoder and decoder through attention gates to suppress the features of irrelevant areas and automatically focus on areas with prominent features. In addition, the segmentation performance of the network is further improved by the ranger optimizer. The comparative experimental results show that CloudRAEDNet can segment the local details of the ground-based cloud images more finely without increasing the time complexity. Compared with CloudSegNet, EFCN, CloudU-Net and CloudU-Netv2, CloudRAEDNet has the best segmentation performance. The results of ablation experiments show that the attention module contributes the most to CloudRAEDNet and the residual module contributes the least to CloudRAEDNet. In addition, the pre-training and Ranger optimizer also contribute to improving the segmentation performance of CloudRAEDNet.

Self-Supervised Pre-Training for Attention-Based Encoder-Decoder ASR Model

End-to-end (E2E) models, including the attention-based encoder-decoder (AED) models, have achieved promising performance on the automatic speech recognition (ASR) task. However, the supervised training process of the E2E model needs a large amount of speech-text paired data. In contrast, self-supervised pre-training can pre-train the model on the unlabeled data and then fine-tune it on the limited labeled data to realize better performance. Most of the previous self-supervised pre-training methods focus on learning hidden representations from speech but ignore how to utilize the unpaired text. As a result, previous works often pre-train an acoustic encoder and then fine-tune it as a classification based ASR model, such as Connectionist Temporal Classification (CTC) based model, rather than an AED model. In this paper, we propose a self-supervised pre-training method for the AED model (SP-AED). The SP-AED method contains acoustic pre-training for the encoder, linguistic pre-training for the decoder, and an adaptive combination fine-tuning for the whole system. We first design a linguistic pre-training method for decoder by utilizing the text-only data. The decoder will be pre-trained as a noise-condition language model to learn the prior distribution of the text. Then, we pre-train the AED encoder with the wav2vec2.0 method with some modifications. Finally, we combine the pre-trained encoder and decoder and fine-tune them on the limited labeled data. We design an adaptive combination method during fine-tuning by modifying the decoder's input and output to prevent catastrophic forgetting. Experiments prove that compared with the random initialized models, the SP-AED pre-trained models can realize up to 17% relative improvement. And with similar model size or computational cost, we can get comparable results to other classification-based models on both English and Chinese corpus.

Improving Mandarin End-to-End Speech Recognition With Word N-Gram Language Model

Despite the rapid progress of end-to-end (E2E) automatic speech recognition (ASR), it has been shown that incorporating external language models (LMs) into the decoding can further improve the recognition performance of E2E ASR systems. To align with the modeling units adopted in E2E ASR systems, subword-level (e.g., characters, BPE) LMs are usually used to cooperate with current E2E ASR systems. However, the use of subword-level LMs will ignore the word-level information, which may limit the strength of the external LMs in E2E ASR. Although several methods have been proposed to incorporate word-level external LMs in E2E ASR, these methods are mainly designed for languages with clear word boundaries such as English and cannot be directly applied to languages like Mandarin, in which each character sequence can have multiple corresponding word sequences. To this end, we propose a novel decoding algorithm where a word-level lattice is constructed on-the-fly to consider all possible word sequences for each partial hypothesis. Then, the LM score of the hypothesis is obtained by intersecting the generated lattice with an external word N-gram LM. The proposed method is examined on both Attention-based Encoder-Decoder (AED) and Neural Transducer (NT) frameworks. Experiments suggest that our method consistently outperforms subword-level LMs, including N-gram LM and neural network LM. We achieve state-of-the-art results on both Aishell-1 (CER 4.18%) and Aishell-2 (CER 5.06%) datasets and reduce CER by 14.8% relatively on a 21K-hour Mandarin dataset.

CSA-UNet: Channel-Spatial Attention-Based Encoder–Decoder Network for Rural Blue-Roofed Building Extraction From UAV Imagery

Building extraction is a critical part of remote sensing (RS) image interpretation, and it is a popular research topic in the RS community. However, building extraction from RS images is a difficult task due to its various shape, size, and complex scene. The extracted feature of existing deep learning methods is lack of discrimination, resulting in incomplete buildings and irregular boundaries. Most studies are mainly concentrated on urban areas, ignoring illegal blue-roofed building extraction in rural areas. To address the above-mentioned problems, a channel-spatial attention-based encoder-decoder network (CSA-UNet) is proposed for rural blue-roofed building extraction tasks from RS images. To extract the key areas of buildings, the CSA-UNet employed channel-spatial attention to the fused features of encoder and decoder for achieving discriminative and attentive features. At the same time, considering the problem of false negative predictions, a joint loss function is designed by giving weight to positive samples to alleviate this problem and optimize the CSA-UNet model. Furthermore, blue-roofed buildings are a special type of illegal building, so we take blue-roofed buildings as an example to carry out related research. And a blue roof dataset termed UAVBlue is built through unmanned aerial vehicle (UAV). Experimental results exhibit that the CSA-UNet is better than some state-of-the-art (SOTA) methods.

Bornon: Bengali Image Captioning with Transformer-Based Deep Learning Approach

Image captioning using encoder–decoder-based approach where CNN is used as the Encoder and sequence generator like RNN as Decoder has proven to be very effective. However, this method has a drawback, that is, sequence needs to be processed in order. To overcome this drawback, some researchers have utilized the transformer model to generate captions from images using English datasets. However, none of them generated captions in Bengali using the transformer model. As a result, we utilized three different Bengali datasets to generate Bengali captions from images using the transformer model. Additionally, we compared the performance of the transformer-based model with a visual attention-based encoder–decoder approach. Finally, we compared the result of the transformer-based model with other models that employed different Bengali image captioning datasets.

DWNet: Dual-Window Deep Neural Network for Time Series Prediction

Multivariate time series prediction is a very important task, which plays a huge role in climate, economy, and other fields. We usually use an Attention-based Encoder-Decoder network to deal with multivariate time series prediction because the attention mechanism makes it easier for the model to focus on the really important attributes. However, the Encoder-Decoder network has the problem that the longer the length of the sequence is, the worse the prediction accuracy is, which means that the Encoder-Decoder network cannot process long series and therefore cannot obtain detailed historical information. In this paper, we propose a dual-window deep neural network (DWNet) to predict time series. The dual-window mechanism allows the model to mine multigranularity dependencies of time series, such as local information obtained from a short sequence and global information obtained from a long sequence. Our model outperforms nine baseline methods in four different datasets.

Pooling Attention-based Encoder–Decoder Network for semantic segmentation

Aiming to the challenge of poor pixel-consistency in inter-category and pixel-similarity in inter-category, in this paper, we propose an Encoder–Decoder network for image semantic segmentation using pooling SE-ResNet attention module, called PAEDN. It is an effective of attention mechanism to get aggregated information. According to the principle of SE-ResNet, a collection of Average, Maximum and Stochastic global pooling, which concentrate on contoured, detailed, and generalized information in a certain semantic segmentation, form attention modules. Channel Pooling Attention Module (CPAM) and Position Pooling Attention Module (PPAM) are designed and integrated into the Encoder to extract discriminative features from input images, and the Decoder is developed through SE-ResNet attention module to fuse the feature map in high-resolution with that in low-resolution. Experimental evaluations performed on the data sets PASCAL and Cityscapes, show the proposed Encoder–Decoder with pooling attention module produces good pixel-consistency semantic label, achieves 15.1% improvement to FCN.

Code generation from a graphical user interface via attention-based encoder–decoder model

Code generation from a graphical user interface via attention-based encoder–decoder model

Deep-Learning Forecasting Method for Electric Power Load via Attention-Based Encoder-Decoder with Bayesian Optimization

Short-term electrical load forecasting plays an important role in the safety, stability, and sustainability of the power production and scheduling process. An accurate prediction of power load can provide a reliable decision for power system management. To solve the limitation of the existing load forecasting methods in dealing with time-series data, causing the poor stability and non-ideal forecasting accuracy, this paper proposed an attention-based encoder-decoder network with Bayesian optimization to do the accurate short-term power load forecasting. Proposed model is based on an encoder-decoder architecture with a gated recurrent units (GRU) recurrent neural network with high robustness on time-series data modeling. The temporal attention layer focuses on the key features of input data that play a vital role in promoting the prediction accuracy for load forecasting. Finally, the Bayesian optimization method is used to confirm the model’s hyperparameters to achieve optimal predictions. The verification experiments of 24 h load forecasting with real power load data from American Electric Power (AEP) show that the proposed model outperforms other models in terms of prediction accuracy and algorithm stability, providing an effective approach for migrating time-serial power load prediction by deep-learning technology.

Integrating Knowledge Into End-to-End Speech Recognition From External Text-Only Data

Attention-based encoder-decoder (AED) models have achieved promising performance in speech recognition. However, because of the end-to-end training, an AED model is usually trained with speech-text paired data. It is challenging to incorporate external text-only data into AED models. Another issue of the AED model is that it does not use the right context of a text token while predicting the token. To alleviate the above two issues, we propose a unified method called LST (Learn Spelling from Teachers) to integrate knowledge into an AED model from the external text-only data and leverage the whole context in a sentence. The method is divided into two stages. First, in the representation stage, a language model is trained on the text. It can be seen as that the knowledge in the text is compressed into the LM. Then, at the transferring stage, the knowledge is transferred to the AED model via teacher-student learning. To further use the whole context of the text sentence, we propose an LM called causal cloze completer (COR), which estimates the probability of a token, given both the left context and the right context of it. Therefore, with LST training, the AED model can leverage the whole context in the sentence. Different from fusion based methods, which use LM during decoding, the proposed method does not increase any extra complexity at the inference stage. We conduct experiments on two scales of public Chinese datasets AISHELL-1 and AISHELL-2. The experimental results demonstrate the effectiveness of leveraging external text-only data and the whole context in a sentence with our proposed method, compared with baseline hybrid systems and AED model based systems.

Fast End-to-End Speech Recognition Via Non-Autoregressive Models and Cross-Modal Knowledge Transferring From BERT

Attention-based encoder-decoder (AED) models have achieved promising performance in speech recognition. However, because the decoder predicts text tokens (such as characters or words) in an autoregressive manner, it is difficult for an AED model to predict all tokens in parallel. This makes the inference speed relatively slow. We believe that because the encoder already captures the whole speech utterance, which has the token-level relationship implicitly, we can predict a token without explicitly autoregressive language modeling. When the prediction of a token does not rely on other tokens, the parallel prediction of all tokens in the sequence is realizable. Based on this idea, we propose a non-autoregressive speech recognition model called LASO (Listen Attentively, and Spell Once). The model consists of an encoder, a decoder, and a position dependent summarizer (PDS). The three modules are based on basic attention blocks. The encoder extracts high-level representations from the speech. The PDS uses positional encodings corresponding to tokens to convert the acoustic representations into token-level representations. The decoder further captures token-level relationships with the self-attention mechanism. At last, the probability distribution on the vocabulary is computed for each token position. Therefore, speech recognition is re-formulated as a position-wise classification problem. Further, we propose a cross-modal transfer learning method to refine semantics from a large-scale pre-trained language model BERT for improving the performance.

Gated Recurrent Context: Softmax-Free Attention for Online Encoder-Decoder Speech Recognition

Recently, attention-based encoder-decoder (AED) models have shown state-of-the-art performance in automatic speech recognition (ASR). As the original AED models with global attentions are not capable of online inference, various online attention schemes have been developed to reduce ASR latency for better user experience. However, a common limitation of the conventional softmax-based online attention approaches is that they introduce an additional hyperparameter related to the length of the attention window, requiring multiple trials of model training for tuning the hyperparameter. In order to deal with this problem, we propose a novel softmax-free attention method and its modified formulation for online attention, which does not need any additional hyperparameter at the training phase. Through a number of ASR experiments, we demonstrate the tradeoff between the latency and performance of the proposed online attention technique can be controlled by merely adjusting a threshold at the test phase. Furthermore, the proposed methods showed competitive performance to the conventional global and online attentions in terms of word-error-rates (WERs).

Convolutional recurrent neural network with attention for Vietnamese speech to text problem in the operating room

We introduce automatic Vietnamese speech recognition (ASR) system for converting Vietnamese speech to text on a real operating room ambient noise recorded during liver surgery. First, we propose applying a combination between convolutional neural network (CNN) and bidirectional long short-term memory (BLSTM) for investigating local speech feature learning, sequence modelling, and transcription for speech recognition. We also extend the CNN-LSTM framework with an attention mechanism to decode the frames into a sequence of words. The CNN, LSTM and attention models are combining into a unified architecture. In addition, we combine connectionist temporal classification (CTC) and attention's loss functions in training phase. The length of the output label sequence from CTC is applied to the attention-based decoder predictions to make the final label sequence. This process helps to decrease irregular alignments and make speedup of the label sequence estimation during training and inference, instead of only relying on the data-driven attention-based encoder-decoder for estimating the label sequence in long sentences. The proposed system is evaluated using a real operating room database. The results show that our method significantly enhances the performance of the ASR system. We find that our approach provides a 13.05% in WER and outperforms standard methods.

Attention-Based Fully Gated CNN-BGRU for Russian Handwritten Text

This article considers the task of handwritten text recognition using attention-based encoder–decoder networks trained in the Kazakh and Russian languages. We have developed a novel deep neural network model based on a fully gated CNN, supported by multiple bidirectional gated recurrent unit (BGRU) and attention mechanisms to manipulate sophisticated features that achieve 0.045 Character Error Rate (CER), 0.192 Word Error Rate (WER), and 0.253 Sequence Error Rate (SER) for the first test dataset and 0.064 CER, 0.24 WER and 0.361 SER for the second test dataset. Our proposed model is the first work to handle handwriting recognition models in Kazakh and Russian languages. Our results confirm the importance of our proposed Attention-Gated-CNN-BGRU approach for training handwriting text recognition and indicate that it can lead to statistically significant improvements (p-value < 0.05) in the sensitivity (recall) over the tests dataset. The proposed method’s performance was evaluated using handwritten text databases of three languages: English, Russian, and Kazakh. It demonstrates better results on the Handwritten Kazakh and Russian (HKR) dataset than the other well-known models.