Attention-based Encoder-decoder Research Articles

HINDIA: a deep-learning-based model for spell-checking of Hindi language

The spelling error is a mistake occurred while typing the text document. The applications like search engines, information retrieval, emails, etc., require user typing. In such applications, good spell-checker is essential to rectify the misspelling. Spell-checkers for western languages like English are very powerful and can handle any type of spelling errors, whereas in the case of Indian languages like Hindi, Urdu, Bengali, Kannada, Assamese, etc., the available spell-checkers are very basic ones. These spell-checkers are developed using traditional methods like statistical methods and rule-based methods. This article presents a novel model HINDIA to handle the spelling errors of the Hindi language, one of the most spoken languages in India. It utilizes a deep-learning method for spelling error detection and correction. The proposed spell-checking model works in two phases. In the first phase model identifies the erroneous words in the input sample and in the second phase it replaces the wrong words with the most probable correct words. Model HINDIA is developed using the attention-based encoder–decoder bidirectional recurrent neural network (BiRNN) which uses long short-term memory cells. Several modifications in the BiRNN have been made and network is fine-tuned to process the spelling errors of Hindi language. It uses publicly available dataset ‘monolingual corpus’ developed by IIT Mumbai for training and testing. The performance of the proposed model is evaluated in two scenarios. In the first scenario where the testing dataset is generated using split function. HINDIA performs significantly well with precision 0.86, recall 0.72, f-measure 0.78 and accuracy 0.80. Further, in the second scenario, where a dataset is manually generated its performance is fairly good with precision 0.81, recall 0.72, f-measure 0.76 and accuracy 0.74. Model HINDIA gives better performance than the deep-learning-based Malayalam spell-checker and some other deep-learning-based correction models present in the literature.

Dual Attention-Based Encoder-Decoder: A Customized Sequence-to-Sequence Learning for Soft Sensor Development.

Soft sensor techniques have been applied to predict the hard-to-measure quality variables based on the easy-to-measure process variables in industry scenarios. Since the products are usually produced with prearranged processing orders, the sequential dependence among different variables can be important for the process modeling. To use this property, a dual attention-based encoder-decoder is developed in this article, which presents a customized sequence-to-sequence learning for soft sensor. We reveal that different quality variables in the same process are sequentially dependent on each other and the process variables are natural time sequences. Hence, the encoder-decoder is constructed to explicitly exploit the sequential information of both the input, that is, the process variables, and the output, that is, the quality variables. The encoder and decoder modules are specified as the long short-term memory network. In addition, since different process variables and time points impose different effects on the quality variables, a dual attention mechanism is embedded into the encoder-decoder to concurrently search the quality-related process variables and time points for a fine-grained quality prediction. Comprehensive experiments are performed based on a real cigarette production process and a benchmark multiphase flow process, which illustrate the effectiveness of the proposed encoder-decoder and its sequence to sequence learning for soft sensor.

Breathing Sound Segmentation and Detection Using Transfer Learning Techniques on an Attention-Based Encoder-Decoder Architecture.

This paper focuses on the use of an attention-based encoder-decoder model for the task of breathing sound segmentation and detection. This study aims to accurately segment the inspiration and expiration of patients with pulmonary diseases using the proposed model. Spectrograms of the lung sound signals and labels for every time segment were used to train the model. The model would first encode the spectrogram and then detect inspiratory or expiratory sounds using the encoded image on an attention-based decoder. Physicians would be able to make a more precise diagnosis based on the more interpretable outputs with the assistance of the attention mechanism.The respiratory sounds used for training and testing were recorded from 22 participants using digital stethoscopes or anti-noising microphone sets. Experimental results showed a high 92.006% accuracy when applied 0.5 second time segments and ResNet101 as encoder. Consistent performance of the proposed method can be observed from ten-fold cross-validation experiments.

Text Normalization Using Encoder–Decoder Networks Based on the Causal Feature Extractor

The encoder–decoder architecture is a well-established, effective and widely used approach in many tasks of natural language processing (NLP), among other domains. It consists of two closely-collaborating components: An encoder that transforms the input into an intermediate form, and a decoder producing the output. This paper proposes a new method for the encoder, named Causal Feature Extractor (CFE), based on three main ideas: Causal convolutions, dilatations and bidirectionality. We apply this method to text normalization, which is a ubiquitous problem that appears as the first step of many text-to-speech (TTS) systems. Given a text with symbols, the problem consists in writing the text exactly as it should be read by the TTS system. We make use of an attention-based encoder–decoder architecture using a fine-grained character-level approach rather than the usual word-level one. The proposed CFE is compared to other common encoders, such as convolutional neural networks (CNN) and long-short term memories (LSTM). Experimental results show the feasibility of CFE, achieving better results in terms of accuracy, number of parameters, convergence time, and use of an attention mechanism based on attention matrices. The obtained accuracy ranges from 83.5% to 96.8% correctly normalized sentences, depending on the dataset. Moreover, the proposed method is generic and can be applied to different types of input such as text, audio and images.

Spatial Relational Attention Using Fully Convolutional Networks for Image Caption Generation

Attention-based encoder–decoder framework has greatly improved image caption generation tasks. The attention mechanism plays a transitional role by transforming static image features into sequential captions. To generate reasonable captions, it is of great significance to detect spatial characteristics of images. In this paper, we propose a spatial relational attention approach to consider spatial positions and attributes. Image features are firstly weighted by the attention mechanism. Then they are concatenated with contextual features to form a spatial–visual tensor. The tensor is feature extracted by a fully convolutional network to produce visual concepts for the decoder network. The fully convolutional layers maintain spatial topology of images. Experiments conducted on the three benchmark datasets, namely Flickr8k, Flickr30k and MSCOCO, demonstrate the effectiveness of our proposed approach. Captions generated by the spatial relational attention method precisely capture spatial relations of objects.

Attention-Based Encoder-Decoder Model for Photovoltaic Power Generation Prediction

The weather factors that affect the output of photovoltaic power generation systems have great volatility and discontinuities. Thus how to accurately predict the output of photovoltaic power generation has become a crucial issue. In this paper, we propose an attention-based Encoder-Decoder model for photovoltaic power generation. Filtered data based on maximum information coefficient is used as one of the features to reconstruct the experiment data. Then the attention mechanism is introduced to the Encoder-Decoder model, which constructed by Long Short-Term Memory (LSTM) neurons. We implement this experiment based on actual photovoltaic power plant examples and experimental results confirm the accuracy and applicability of the proposed model in predicting photovoltaic power generation

Multivariate time series forecasting via attention-based encoder–decoder framework

Time series forecasting is an important technique to study the behavior of temporal data and forecast future values, which is widely applied in many fields, e.g. air quality forecasting, power load forecasting, medical monitoring, and intrusion detection. In this paper, we firstly propose a novel temporal attention encoder–decoder model to deal with the multivariate time series forecasting problem. It is an end-to-end deep learning structure that integrates the traditional encode context vector and temporal attention vector for jointly temporal representation learning, which is based on bi-directional long short-term memory networks (Bi-LSTM) layers with temporal attention mechanism as the encoder network to adaptively learning long-term dependency and hidden correlation features of multivariate temporal data. Extensive experimental results on five typical multivariate time series datasets showed that our model has the best forecasting performance compared with baseline methods.

A Hybrid Short-Term Load Forecasting Framework with an Attention-Based Encoder–Decoder Network Based on Seasonal and Trend Adjustment

Accurate electrical load forecasting plays an important role in power system operation. An effective load forecasting approach can improve the operation efficiency of a power system. This paper proposes the seasonal and trend adjustment attention encoder–decoder (STA–AED), a hybrid short-term load forecasting approach based on a multi-head attention encoder–decoder module with seasonal and trend adjustment. A seasonal and trend decomposing technique is used to preprocess the original electrical load data. Each decomposed datum is regressed to predict the future electric load value by utilizing the encoder–decoder network with the multi-head attention mechanism. With the multi-head attention mechanism, STA–AED can interpret the prediction results more effectively. A large number of experiments and extensive comparisons have been carried out with a load forecasting dataset from the United States. The proposed hybrid STA–AED model is superior to the other five counterpart models such as random forest, gradient boosting decision tree (GBDT), gated recurrent units (GRUs), Encoder–Decoder, and Encoder–Decoder with multi-head attention. The proposed hybrid model shows the best prediction accuracy in 14 out of 15 zones in terms of both root mean square error (RMSE) and mean absolute percentage error (MAPE).

Pattern generation strategies for improving recognition of Handwritten Mathematical Expressions

Recognition of Handwritten Mathematical Expressions (HMEs) is a challenging problem because of the ambiguity and complexity of two-dimensional handwriting. Moreover, the lack of large training data is a serious issue, especially for academic recognition systems. In this paper, we propose pattern generation strategies that generate shape and structural variations to improve the performance of recognition systems based on a small training set. For data generation, we employ the public databases: CROHME 2014 and 2016 of online HMEs. The first strategy employs local and global distortions to generate shape variations. The second strategy decomposes an online HME into sub-online HMEs to get more structural variations. The hybrid strategy combines both these strategies to maximize shape and structural variations. The generated online HMEs are converted to images for offline HME recognition. We tested our strategies in an end-to-end recognition system constructed from a recent deep learning model: Convolutional Neural Network and attention-based encoder-decoder. The results of experiments on the CROHME 2014 and 2016 databases demonstrate the superiority and effectiveness of our strategies: our hybrid strategy achieved classification rates of 48.78% and 45.60%, respectively, on these databases. These results are competitive compared to others reported in recent literature. Our generated datasets are openly available for research community and constitute a useful resource for the HME recognition research in future.

RNN-LSTM-GRU based language transformation

In past, rule-based and statistical machine translation techniques were employed to solve Urdu transliteration techniques. As mentioned in the literature, Urdu is considered as low-resource language. An impressive effort has been made for Arabic, French and Chinese language transliteration as compared to the Urdu language. Machine translation of Urdu language is a challenging problem. A very minute research work has been conducted toward Urdu transliteration. Factors behind the ignorance of Urdu language in research may be for its morphological complexity, diversity and most importantly due to the lack of reasonable bilingual parallel dataset. Getting a corpus for a language transliteration is the main resource to work on. This paper demonstrates the application of neural machine translation (NMT) for Urdu language transliteration, with the emphasis on contextual coverage of a language, which helps to improve transliteration accuracy. Build a robust NMT model which delivers efficient performance when trained over bilingual parallel corpora. Neural machine translation is an emerging technique depicting impressive performance, better than traditional MT methods in multiple aspects. In this research, we build the NMT model for the Urdu language to improve transliteration quality. An attention-based encoder–decoder system is proposed, and our experiment proves the efficiency of the proposed approach. To the best of our knowledge, this is the first effort for Urdu language bidirectional transliteration toward neural machine translation.

Video Summarization With Attention-Based Encoder–Decoder Networks

This paper addresses the problem of supervised video summarization by formulating it as a sequence-to-sequence learning problem, where the input is a sequence of original video frames, and the output is a keyshot sequence. Our key idea is to learn a deep summarization network with attention mechanism to mimic the way of selecting the keyshots of human. To this end, we propose a novel video summarization framework named attentive encoder-decoder networks for video summarization (AVS), in which the encoder uses a bidirectional long short-term memory (BiLSTM) to encode the contextual information among the input video frames. As for the decoder, two attention-based LSTM networks are explored by using additive and multiplicative objective functions, respectively. Extensive experiments are conducted on two video summarization benchmark datasets, i.e., SumMe and TVSum. The results demonstrate the superiority of the proposed AVS-based approaches against the state-of-the-art approaches, with remarkable improvements on both datasets.

An End-to-End Recognition System for Unconstrained Vietnamese Handwriting

Inspired by recent successes in neural machine translation and image caption generation, we present an attention-based encoder–decoder model (AED) to recognize handwritten Vietnamese text. The model consists of two parts: a Convolutional Neural Network-based (CNN) encoder and a Long Short-Term Memory-based (LSTM) decoder. The encoder is based on DenseNet for extracting invariant features. The LSTM-based decoder with an attention model incorporated generates output text. The input of the encoder is a handwritten text image and the target of the decoder is the corresponding text of the input image. Our model is trained end-to-end to predict the text from a given input image since all the parts are differential components. In the experiment section, we evaluate our proposed AED model on the VNOnDB-Word and VNOnDB-Line datasets to verify its efficiency. The experiential results show that our model achieves 4.10% of character error eate (CER) and 10.24% of word error rate (WER) on the testing set of VNOnDB-Word and 4.67% of CER and 13.33% of WER on the testing set of VNOnDB-Line without using any language model. This result is higher than our previous system, Nguyen et al.’s system, and the system provided by Google in the Vietnamese Online Handwritten Text Recognition competition.

Attention-Based Personalized Encoder-Decoder Model for Local Citation Recommendation.

With a tremendous growth in the number of scientific papers, researchers have to spend too much time and struggle to find the appropriate papers they are looking for. Local citation recommendation that provides a list of references based on a text segment could alleviate the problem. Most existing local citation recommendation approaches concentrate on how to narrow the semantic difference between the scientific papers' and citation context's text content, completely neglecting other information. Inspired by the successful use of the encoder-decoder framework in machine translation, we develop an attention-based encoder-decoder (AED) model for local citation recommendation. The proposed AED model integrates venue information and author information in attention mechanism and learns relations between variable-length texts of the two text objects, i.e., citation contexts and scientific papers. Specifically, we first construct an encoder to represent a citation context as a vector in a low-dimensional space; after that, we construct an attention mechanism integrating venue information and author information and use RNN to construct a decoder, then we map the decoder's output into a softmax layer, and score the scientific papers. Finally, we select papers which have high scores and generate a recommended reference paper list. We conduct experiments on the DBLP and ACL Anthology Network (AAN) datasets, and the results illustrate that the performance of the proposed approach is better than the other three state-of-the-art approaches.

Long short-term memory network with external memories for image caption generation

In long short-term memory (LSTM) neural networks, the input gates and output gates control information flowing into and out of memory cells. For sequence-to-sequence learning problems, each element is input into the network only once. If the input gates are closed at a certain step, the information is lost and is not input again. The same problem exists for the output gates. Therefore, the input and output gates do not fully support the roles of gating. An LSTM network with external memories, in which separate memories are installed for the input and output gates, is proposed. Information that is blocked by the input gates is preserved in the input memories, enabling the cells to read these memories when necessary. Similarly, information blocked by the output gates is preserved in the output memories and flows out to hidden units of the network at an appropriate time. In addition, a dynamic attention model is proposed to take into account the attention history. It provides guidance when predicting the attention weights at each step. The proposed model exploits attention-based encoder–decoder architecture to generate image captions. Experiments were conducted on three benchmark datasets, namely Flickr8k, Flickr30k, and MSCOCO, to demonstrate the effectiveness of the proposed approach. Captions generated by the proposed method are longer and more informative than those obtained with the original LSTM network.

A Human-Inspired Recognition System for Pre-Modern Japanese Historical Documents

Recognition of historical documents is a challenging problem due to the noised, damaged characters, and background. However, in Japanese historical documents, not only contains the mentioned problems, pre-modern Japanese characters were written in cursive and are connected. Therefore, character segmentation-based methods do not work well. This leads to the idea of creating a new recognition system. In this paper, we propose a human-inspired document reading system to recognize multiple lines of pre-modern Japanese historical documents. During the reading, people employ eyes movement to determine the start of a text line. Then, they move the eyes from the current character/word to the next character/word. They can also determine the end of a line or skip a figure to move to the next line. The eyes movement integrates with visual processing to operate the reading process in the brain. We employ attention-based encoder–decoder to implement this recognition system. First, the recognition system detects were to start a text line. Second, the system scans and recognize character by character until the text line is completed. Then, the system continues to detect the start of the next text line. This process is repeated until reading the whole document. As results, the system is successful to recognize multiple lines, connected and cursive characters without performing character/line segmentation. Besides, we also employ a coverage model which stores the history of eyes movement to predict the next movement more precisely. We tested our human-inspired recognition system on the pre-modern Japanese historical document provided by the PRMU Kuzushiji competition. The results of the experiments demonstrate the superiority and effectiveness of our proposed system by achieving Sequence Error Rate of 9.87% and 53.81% on level 2 and level 3 of the dataset, respectively. These results outperform to any other systems participated in the PRMU Kuzushiji competition.

MindID

Person identification technology recognizes individuals by exploiting their unique, measurable physiological and behavioral characteristics. However, the state-of-the-art person identification systems have been shown to be vulnerable, e.g., anti-surveillance prosthetic masks can thwart face recognition, contact lenses can trick iris recognition, vocoder can compromise voice identification and fingerprint films can deceive fingerprint sensors. EEG (Electroencephalography)-based identification, which utilizes the user's brainwave signals for identification and offers a more resilient solution, has recently drawn a lot of attention. However, the state-of-the-art systems cannot achieve similar accuracy as the aforementioned methods. We propose MindID, an EEG-based biometric identification approach, with the aim of achieving high accuracy and robust performance. At first, the EEG data patterns are analyzed and the results show that the Delta pattern contains the most distinctive information for user identification. Next, the decomposed Delta signals are fed into an attention-based Encoder-Decoder RNNs (Recurrent Neural Networks) structure which assigns varying attention weights to different EEG channels based on their importance. The discriminative representations learned from the attention-based RNN are used to identify the user through a boosting classifier. The proposed approach is evaluated over 3 datasets (two local and one public). One local dataset (EID-M) is used for performance assessment and the results illustrate that our model achieves an accuracy of 0.982 and significantly outperforms the state-of-the-art and relevant baselines. The second local dataset (EID-S) and a public dataset (EEG-S) are utilized to demonstrate the robustness and adaptability, respectively. The results indicate that the proposed approach has the potential to be widely deployed in practical settings.

Modeling coverage with semantic embedding for image caption generation

This paper presents a coverage-based image caption generation model. The attention-based encoder–decoder framework has enhanced state-of-the-art image caption generation by learning where to attend of the visual field. However, there exists a problem that in some cases it ignores past attention information, which tends to lead to over-recognition and under-recognition. To solve this problem, a coverage mechanism is incorporated into attention-based image caption generation. A sequential updated coverage vector is applied to preserve the attention historical information. At each time step, the attention model takes the coverage vector as auxiliary input to focus more on unattended features. Besides, to maintain the semantics of an image, we propose semantic embedding as global guidance to coverage and attention model. With semantic embedding, the attention and coverage mechanisms consider more about features relevant to the semantics of an image. Experiments conducted on the three benchmark datasets, namely Flickr8k, Flickr30k and MSCOCO, demonstrate the effectiveness of our proposed approach. In addition to solve the over-recognition and under-recognition problems, it behaves better on long descriptions.

Multi-Channel Encoder for Neural Machine Translation

Attention-based Encoder-Decoder has the effective architecture for neural machine translation (NMT), which typically relies on recurrent neural networks (RNN) to build the blocks that will be lately called by attentive reader during the decoding process. This design of encoder yields relatively uniform composition on source sentence, despite the gating mechanism employed in encoding RNN. On the other hand, we often hope the decoder to take pieces of source sentence at varying levels suiting its own linguistic structure: for example, we may want to take the entity name in its raw form while taking an idiom as a perfectly composed unit. Motivated by this demand, we propose Multi-channel Encoder (MCE), which enhances encoding components with different levels of composition. More specifically, in addition to the hidden state of encoding RNN, MCE takes 1) the original word embedding for raw encoding with no composition, and 2) a particular design of external memory in Neural Turing Machine NTM) for more complex composition, while all three encoding strategies are properly blended during decoding. Empirical study on Chinese-English translation shows that our model can improve by 6.52 BLEU points upon a strong open source NMT system: DL4MT1. On the WMT14 English-French task, our single shallow system achieves BLEU=38.8, comparable with the state-of-the-art deep models.

Hybrid CTC/Attention Architecture for End-to-End Speech Recognition

Conventional automatic speech recognition (ASR) based on a hidden Markov model (HMM)/deep neural network (DNN) is a very complicated system consisting of various modules such as acoustic, lexicon, and language models. It also requires linguistic resources, such as a pronunciation dictionary, tokenization, and phonetic context-dependency trees. On the other hand, end-to-end ASR has become a popular alternative to greatly simplify the model-building process of conventional ASR systems by representing complicated modules with a single deep network architecture, and by replacing the use of linguistic resources with a data-driven learning method. There are two major types of end-to-end architectures for ASR; attention-based methods use an attention mechanism to perform alignment between acoustic frames and recognized symbols, and connectionist temporal classification (CTC) uses Markov assumptions to efficiently solve sequential problems by dynamic programming. This paper proposes hybrid CTC/attention end-to-end ASR, which effectively utilizes the advantages of both architectures in training and decoding. During training, we employ the multiobjective learning framework to improve robustness and achieve fast convergence. During decoding, we perform joint decoding by combining both attention-based and CTC scores in a one-pass beam search algorithm to further eliminate irregular alignments. Experiments with English (WSJ and CHiME-4) tasks demonstrate the effectiveness of the proposed multiobjective learning over both the CTC and attention-based encoder–decoder baselines. Moreover, the proposed method is applied to two large-scale ASR benchmarks (spontaneous Japanese and Mandarin Chinese), and exhibits performance that is comparable to conventional DNN/HMM ASR systems based on the advantages of both multiobjective learning and joint decoding without linguistic resources.

Recent progress in deep end-to-end models for spoken language processing

End-to-end models (or sequence-to-sequence models) based on deep neural networks have recently become popular within the machine learning community. These techniques are also increasingly used in automatic speech recognition as an alternative to the state-of-the-art, hybrid HMM-DNN (hidden Markov model, deep neural network) system. The end-to-end systems contain a purely neural architecture that eliminates the need of any time alignment between the input acoustic feature vector sequence and output phone sequence. In this paper, we present progress within the IBM Watson Multimodal Group on end-to-end models for spoken language processing. We present our work on two types of end-to-end models applied to speech-to-text and keyword search tasks, namely, 1) recurrent neural networks (RNNs) based on connectionist temporal classification loss, and 2) attention-based encoder–decoder RNNs. We present results on several languages (such as Pashto, Mongolian, Javanese, Amharic, Guarani, Dholuo, Igbo, and Georgian) from the Intelligence Advanced Research Projects Activity funded Babel Program. We also present a detailed analysis of some salient characteristics of these models compared with the state-of-the-art HMM-DNN hybrid systems, and also discuss future challenges in using such models for spoken language processing.