Neural Encoder-decoder Model Research Articles

The metamorphosis of machine translation: The rise of neural machine translation and its challenges

Machine translation refers to the process of using computers to translate source language into target language, which has undergone significant transformations since its inception, with the current mainstream neural machine translation achieving satisfactory translation performance. This paper overviews the three developmental stages of machine translation: rule-based machine translation, statistical machine translation, and neural machine translation, with a focus on neural machine translation. It introduces the key models that emerged in the development process of neural machine translation, namely the recurrent neural network encoder-decoder model, recurrent neural network search model, and Transformer, and compares their strengths and limitations. Other relevant technologies and models developed alongside neural machine translation are also discussed. Addressing the current challenges of neural machine translation, the paper delves into issues of overfitting, low-resource translation, structural optimization of Transformer models, and enhancement of neural machine translation interpretability. Finally, the paper explores the prospects of applying neural machine translation to multimodal translation.

Convolutional neural networks applied to differential dynamic microscopy reduces noise when quantifying heterogeneous dynamics.

Differential dynamic microscopy (DDM) typically relies on movies containing hundreds or thousands of frames to accurately quantify motion in soft matter systems. Using movies much shorter in duration produces noisier and less accurate results. This limits the applicability of DDM to situations where the dynamics are stationary over extended times. Here, we investigate a method to denoise the DDM process, particularly suited to when a limited number of imaging frames are available or when dynamics are quickly evolving in time. We use a convolutional neural network encoder-decoder (CNN-ED) model to reduce the noise in the intermediate scattering function that is computed via DDM. We demonstrate this approach of combining machine learning and DDM on samples containing diffusing micron-sized colloidal particles. We quantify how the particles' diffusivities change over time as the fluid they are suspended in gels. We also quantify how the diffusivity of particles varies with position in a sample containing a viscosity gradient. These test cases demonstrate how studies of non-equilibrium dynamics and high-throughput screens could benefit from a method to denoise the outputs of DDM.

A Systematic Review of Deep Learning Approaches for Vessel Segmentation in Retinal Fundus Images

Retinal vessel segmentation (RVS) is important to accurately differentiate retinal vasculature for diagnosing and monitoring various ocular and systemic diseases. The traditional methods for RVS have mostly involved supervised learning, although semi-supervised and unsupervised techniques are on the rise. This paper reviews the increase in complexity of developments in RVS primarily after 2020. The methods were chosen to cover both the gradual transition over time and a variety of unorthodox or combinatorial approaches. This includes convolutional neural networks, encoder-decoder models, generative models, and other multi-modal or hybrid techniques. CNN approaches discussed employ Zero Phase Component Analysis, Global Contrast Normalization, and reinforcement learning. Encoder-decoder models include approaches such as the use of skip and residual connections, spatial attention, and atrous enhancement U-Net. Generative models propose short link connections, recurrent residual blocks, and multi-scale features to refine convolutional blocks. Hybrid methods involve the use of connectivity features, the MISODATA Algorithm, cross-domain adaptation, and multiple filters (such as morphological, match, and Gabor). All the frameworks are compared based on their performance on the benchmark dataset DRIVE to provide a comprehensive understanding of the current state of RVS.

A Survey of Knowledge-enhanced Text Generation

The goal of text-to-text generation is to make machines express like a human in many applications such as conversation, summarization, and translation. It is one of the most important yet challenging tasks in natural language processing (NLP). Various neural encoder-decoder models have been proposed to achieve the goal by learning to map input text to output text. However, the input text alone often provides limited knowledge to generate the desired output, so the performance of text generation is still far from satisfaction in many real-world scenarios. To address this issue, researchers have considered incorporating (i) internal knowledge embedded in the input text and (ii) external knowledge from outside sources such as knowledge base and knowledge graph into the text generation system. This research topic is known as knowledge-enhanced text generation . In this survey, we present a comprehensive review of the research on this topic over the past five years. The main content includes two parts: (i) general methods and architectures for integrating knowledge into text generation; (ii) specific techniques and applications according to different forms of knowledge data. This survey can have broad audiences, researchers and practitioners, in academia and industry.

Using Neural Encoder-Decoder Models With Continuous Outputs for Remote Sensing Image Captioning

Remote sensing image captioning involves generating a concise textual description for an input aerial image. The task has received significant attention, and several recent proposals are based on neural encoder-decoder models. Most previous methods are trained to generate discrete outputs corresponding to word tokens that match the reference sentences word-by-word, thereby optimizing the generation locally at token-level instead of globally at sentence-level. This paper explores an alternative generation method based on continuous outputs, which generates sequences of embedding vectors instead of directly predicting word tokens. We argue that continuous output models have the potential to better capture the global semantic similarity between captions and images, e.g. by facilitating the use of loss functions matching different views of the data. This includes comparing representations for individual tokens and for the entire captions, and also comparing captions against intermediate image representations. We experimentally compare discrete versus continuous output captioning methods over the UCM and RSICD datasets, which are extensively used in the area despite some issues which we also discuss. Results show that the alternative encoder-decoder framework with continuous outputs can indeed lead to better results on the two datasets, compared to the standard approach based on discrete outputs. The proposed approach is also competitive against the state-of-the-art model in the area.

Classification of Arrhythmia Based on Convolutional Neural Networks and Encoder-Decoder Model

As a common and high-risk type of disease, heart disease seriously threatens people’s health. At the same time, in the era of the Internet of Thing (IoT), smart medical device has strong practical significance for medical workers and patients because of its ability to assist in the diagnosis of diseases. Therefore, the research of real-time diagnosis and classification algorithms for arrhythmia can help to improve the diagnostic efficiency of diseases. In this paper, we design an automatic arrhythmia classification algorithm model based on Convolutional Neural Network (CNN) and Encoder-Decoder model. The model uses Long Short-Term Memory (LSTM) to consider the influence of time series features on classification results. Simultaneously, it is trained and tested by the MIT-BIH arrhythmia database. Besides, Generative Adversarial Networks (GAN) is adopted as a method of data equalization for solving data imbalance problem. The simulation results show that for the inter-patient arrhythmia classification, the hybrid model combining CNN and Encoder-Decoder model has the best classification accuracy, of which the accuracy can reach 94.05%. Especially, it has a better advantage for the classification effect of supraventricular ectopic beats (class S) and fusion beats (class F).

A Position-Aware Transformer for Image Captioning

Image captioning aims to generate a corresponding description of an image. In recent years, neural encoder-decoder models have been the dominant approaches, in which the Convolutional Neural Network (CNN) and Long Short Term Memory (LSTM) are used to translate an image into a natural language description. Among these approaches, the visual attention mechanisms are widely used to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. However, most conventional visual attention mechanisms are based on high-level image features, ignoring the effects of other image features, and giving insufficient consideration to the relative positions between image features. In this work, we propose a Position-Aware Transformer model with image-feature attention and position-aware attention mechanisms for the above problems. The image-feature attention firstly extracts multi-level features by using Feature Pyramid Network (FPN), then utilizes the scaled-dot-product to fuse these features, which enables our model to detect objects of different scales in the image more effectively without increasing parameters. In the position-aware attention mechanism, the relative positions between image features are obtained at first, afterwards the relative positions are incorporated into the original image features to generate captions more accurately. Experiments are carried out on the MSCOCO dataset and our approach achieves competitive BLEU-4, METEOR, ROUGE-L, CIDEr scores compared with some state-of-the-art approaches, demonstrating the effectiveness of our approach.

Decoding Methods in Neural Language Generation: A Survey

Neural encoder-decoder models for language generation can be trained to predict words directly from linguistic or non-linguistic inputs. When generating with these so-called end-to-end models, however, the NLG system needs an additional decoding procedure that determines the output sequence, given the infinite search space over potential sequences that could be generated with the given vocabulary. This survey paper provides an overview of the different ways of implementing decoding on top of neural network-based generation models. Research into decoding has become a real trend in the area of neural language generation, and numerous recent papers have shown that the choice of decoding method has a considerable impact on the quality and various linguistic properties of the generation output of a neural NLG system. This survey aims to contribute to a more systematic understanding of decoding methods across different areas of neural NLG. We group the reviewed methods with respect to the broad type of objective that they optimize in the generation of the sequence—likelihood, diversity, and task-specific linguistic constraints or goals—and discuss their respective strengths and weaknesses.

A Survey on Segmentation Techniques for Image Processing

The segmentation methods for image processing are studied in the presented work. Image segmentation can be defined as a vital step in digital image processing. Also, it is used in various applications including object co-segmentation, recognition tasks, medical imaging, content based image retrieval, object detection, machine vision and video surveillance. A lot of approaches were created for image segmentation. In addition, the main goal of segmentation is to facilitate and alter the image representation into something which is more important and simply to be analyzed. The approaches of image segmentation are splitting the images into a few parts on the basis of image’s features including texture, color, pixel intensity value and so on. With regard to the presented study, many approaches of image segmentation are reviewed and discussed. The techniques of segmentation might be categorized into six classes: First, thresholding segmentation techniques such as global thresholding (iterative thresholding, minimum error thresholding, otsu's, optimal thresholding, histogram concave analysis and entropy based thresholding), local thresholding (Sauvola’s approach, T.R Singh’s approach, Niblack’s approaches, Bernsen’s approach Bruckstein’s and Yanowitz method and Local Adaptive Automatic Binarization) and dynamic thresholding. Second, edge-based segmentation techniques such as gray-histogram technique, gradient based approach (laplacian of gaussian, differential coefficient approach, canny approach, prewitt approach, Roberts approach and sobel approach). Thirdly, region based segmentation approaches including Region growing techniques (seeded region growing (SRG), statistical region growing, unseeded region growing (UsRG)), also merging and region splitting approaches. Fourthly, clustering approaches, including soft clustering (fuzzy C-means clustering (FCM)) and hard clustering (K-means clustering). Fifth, deep neural network techniques such as convolution neural network, recurrent neural networks (RNNs), encoder-decoder and Auto encoder models and support vector machine. Finally, hybrid techniques such as evolutionary approaches, fuzzy logic and swarm intelligent (PSO and ABC techniques) and discusses the pros and cons of each method.

Generating Instructive Questions from Multiple Articles to Guide Reading in E-Bibliotherapy.

E-Bibliotherapy deals with adolescent psychological stress by manually or automatically recommending multiple reading articles around their stressful events, using electronic devices as a medium. To make E-Bibliotherapy really useful, generating instructive questions before their reading is an important step. Such a question shall (a) attract teens’ attention; (b) convey the essential message of the reading materials so as to improve teens’ active comprehension; and most importantly (c) highlight teens’ stress to enable them to generate emotional resonance and thus willingness to pursue the reading. Therefore in this paper, we propose to generate instructive questions from the multiple recommended articles to guide teens to read. Four solutions based on the neural encoder-decoder model are presented to tackle the task. For model training and testing, we construct a novel large-scale QA dataset named TeenQA, which is specific to adolescent stress. Due to the extensibility of question expressions, we incorporate three groups of automatic evaluation metrics as well as one group of human evaluation metrics to examine the quality of the generated questions. The experimental results show that the proposed Encoder-Decoder with Summary on Contexts with Feature-rich embeddings (ED-SoCF) solution can generate good questions for guiding reading, achieving comparable performance on some semantic similarity metrics with that of humans.

Deep Learning-Based Short Story Generation for an Image Using the Encoder-Decoder Structure

Research that applies artificial intelligence (AI) to generate the captions for an image has been extensively studied in recent years. However, the length of these captions was short, and the number of generated captions was limited. In addition, it is unknown whether a short story can be generated based on the image, because many sentences have to be connected to create a fluent short story. As a result, this study introduces an encoder-decoder framework structure to generate a short story captioning (SSCap) using a common image caption dataset and a manually collected story corpus. This manuscript has three main contributions, which include 1) an unsupervised deep learning-based framework that combines a recurrent neural network (RNN) structure and encoder-decoder model for composing a short story for an image, 2) a huge story corpus, which includes two different genres (horror and romantic), manually collected and validated. Extensive experiments demonstrated that short stories created by the proposed model show creative content compared to existing systems that can only make concise sentences. Therefore, the demonstrated framework has the potential to motivate the development of a more robust AI story writer and motivates the integration of the suggested model into practical applications to help the story writers find a new idea.

Global Encoding for Long Chinese Text Summarization

Text summarization is one of the significant tasks of natural language processing, which automatically converts text into a summary. Some summarization systems, for short/long English, and short Chinese text, benefit from advances in the neural encoder-decoder model because of the availability of large datasets. However, the long Chinese text summarization research has been limited to datasets of a couple of hundred instances. This article aims to explore the long Chinese text summarization task. To begin with, we construct a first large-scale, long Chinese text summarization corpus, the Long Chinese Summarization of Police Inquiry Record Text (LCSPIRT). Based on this corpus, we propose a sequence-to-sequence (Seq2Seq) model that incorporates a global encoding process with an attention mechanism. Our model achieves a competitive result on the LCSPIRT corpus compared with several benchmark methods.

Deep in the Bowel: Highly Interpretable Neural Encoder-Decoder Networks Predict Gut Metabolites from Gut Microbiome

BackgroundTechnological advances in next-generation sequencing (NGS) and chromatographic assays [e.g., liquid chromatography mass spectrometry (LC-MS)] have made it possible to identify thousands of microbe and metabolite species, and to measure their relative abundance. In this paper, we propose a sparse neural encoder-decoder network to predict metabolite abundances from microbe abundances.ResultsUsing paired data from a cohort of inflammatory bowel disease (IBD) patients, we show that our neural encoder-decoder model outperforms linear univariate and multivariate methods in terms of accuracy, sparsity, and stability. Importantly, we show that our neural encoder-decoder model is not simply a black box designed to maximize predictive accuracy. Rather, the network’s hidden layer (i.e., the latent space, comprised only of sparsely weighted microbe counts) actually captures key microbe-metabolite relationships that are themselves clinically meaningful. Although this hidden layer is learned without any knowledge of the patient’s diagnosis, we show that the learned latent features are structured in a way that predicts IBD and treatment status with high accuracy.ConclusionsBy imposing a non-negative weights constraint, the network becomes a directed graph where each downstream node is interpretable as the additive combination of the upstream nodes. Here, the middle layer comprises distinct microbe-metabolite axes that relate key microbial biomarkers with metabolite biomarkers. By pre-processing the microbiome and metabolome data using compositional data analysis methods, we ensure that our proposed multi-omics workflow will generalize to any pair of -omics data. To the best of our knowledge, this work is the first application of neural encoder-decoders for the interpretable integration of multi-omics biological data.

Automatic Question Generation u sing Sequence to Sequence RNN Model

Automatic Question Generation (AQG) has recently received growing focus in the processing of natural language (NLP). This attempts to create questions from a text paragraph, where certain sub-spans of the passage in question will answer the questions produced.. Traditional methods predominantly use rigid heuristic rules to turn a sentence into related questions. In this research, we suggest using the neural encoder-decoder model to produce substantive and complex questions from the sentences of natural language. We apply a attention-based sequence to sequence learning paradigm for the task and analyze the impact of encoding sentence vs. knowledge at paragraph level. Information retrieval and NLP are the core components of AQG. It incorporates the application of production rules; recurrent neural network (RNN) based encoder-decoder sequence to sequence (seq2seq) models, and other intelligent techniques. RNN is used because of its long short term memory power (LSTM).The proposed system focus on generating factual WH type questions.

Exploring a Long Short-Term Memory based Encoder-Decoder framework for multi-step-ahead flood forecasting

Operational flood control systems depend on reliable and accurate forecasts with a suitable lead time to take necessary actions against flooding. This study proposed a Long Short-Term Memory based Encoder-Decoder (LSTM-ED) model for multi-step-ahead flood forecasting for the first time. The Shihmen Reservoir catchment in Taiwan constituted the case study. A total of 12,216 hourly hydrological data collected from 23 typhoon events were allocated into three datasets for model training, validation, and testing. The input sequence of the model contained hourly reservoir inflows and rainfall data (traced back to the previous 8 h) of ten gauge stations, and the output sequence stepped into 1- up to 6-hour-ahead reservoir inflow forecasts. A feed forward neural network-based Encoder-Decoder (FFNN-ED) model was established for comparison purposes. This study conducted model training a number of times with various initial weights to evaluate the accuracy, stability, and reliability of the constructed FFNN-ED and LSTM-ED models. The results demonstrated that both models, in general, could provide suitable multi-step ahead forecasts, and the proposed LSTM-ED model not only could effectively mimic the long-term dependence between rainfall and runoff sequences but also could make more reliable and accurate flood forecasts than the FFNN-ED model. Concerning the time delay between the time horizons of model inputs (rainfall) and model outputs (runoff), the impact assessment of this time-delay on model performance indicated that the LSTM-ED model achieved similar forecast performance when fed with antecedent rainfall either at a shorter horizon of 4 h in the past (T − 4) or at horizons longer than 7 h in the past (>T − 7). We conclude that the proposed LSTM-ED that translates and links the rainfall sequence with the runoff sequence can improve the reliability of flood forecasting and increase the interpretability of model internals.

Exploiting Background Knowledge in Compact Answer Generation for Why-Questions

This paper proposes a novel method for generating compact answers to open-domain why-questions, such as the following answer, “Because deep learning technologies were introduced,” to the question, “Why did Google’s machine translation service improve so drastically?” Although many works have dealt with why-question answering, most have focused on retrieving as answers relatively long text passages that consist of several sentences. Because of their length, such passages are not appropriate to be read aloud by spoken dialog systems and smart speakers; hence, we need to create a method that generates compact answers. We developed a novel neural summarizer for this compact answer generation task. It combines a recurrent neural network-based encoderdecoder model with stacked convolutional neural networks and was designed to effectively exploit background knowledge, in this case a set of causal relations (e.g., “[Microsoft’s machine translation has made great progress over the last few years]effect since [it started to use deep learning.]cause”) that was extracted from a large web data archive (4 billion web pages). Our experimental results show that our method achieved significantly better ROUGE F-scores than existing encoder-decoder models and their variations that were augmented with query-attention and memory networks, which are used to exploit the background knowledge.

Visual Dialog.

We introduce the task of Visual Dialog, which requires an AI agent to hold a meaningful dialog with humans in natural, conversational language about visual content. Specifically, given an image, a dialog history, and a question about the image, the agent has to ground the question in image, infer context from history, and answer the question accurately. Visual Dialog is disentangled enough from a specific downstream task so as to serve as a general test of machine intelligence, while being sufficiently grounded in vision to allow objective evaluation of individual responses and benchmark progress. We develop a novel two-person real-time chat data-collection protocol to curate a large-scale Visual Dialog dataset (VisDial). VisDial v0.9 has been released and consists of dialog question-answer pairs from 10-round, human-human dialogs grounded in images from the COCO dataset.

Improving Low-Resource Morphological Learning with Intermediate Forms from Finite State Transducers

Neural encoder-decoder models are usually applied to morphology learning as an end-to-end process without considering the underlying phonological representations that linguists posit as abstract forms before morphophonological rules are applied. Finite State Transducers for morphology, on the other hand, are developed to contain these underlying forms as an intermediate representation. This paper shows that training a bidirectional two-step encoder-decoder model of Arapaho verbs to learn two separate mappings between tags and abstract morphemes and morphemes and surface allomorphs improves results when training data is limited to 10,000 to 30,000 examples of inflected word forms.

A Neural Machine Translation Model for Arabic Dialects That Utilises Multitask Learning (MTL).

In this research article, we study the problem of employing a neural machine translation model to translate Arabic dialects to modern standard Arabic. The proposed solution of the neural machine translation model is prompted by the recurrent neural network-based encoder-decoder neural machine translation model that has been proposed recently, which generalizes machine translation as sequence learning problems. We propose the development of a multiytask learning (MTL) model which shares one decoder among language pairs, and every source language has a separate encoder. The proposed model can be applied to limited volumes of data as well as extensive amounts of data. Experiments carried out have shown that the proposed MTL model can ensure a higher quality of translation when compared to the individually learned model.

Leveraging Orthographic Similarity for Multilingual Neural Transliteration

We address the task of joint training of transliteration models for multiple language pairs ( multilingual transliteration). This is an instance of multitask learning, where individual tasks (language pairs) benefit from sharing knowledge with related tasks. We focus on transliteration involving related tasks i.e., languages sharing writing systems and phonetic properties ( orthographically similar languages). We propose a modified neural encoder-decoder model that maximizes parameter sharing across language pairs in order to effectively leverage orthographic similarity. We show that multilingual transliteration significantly outperforms bilingual transliteration in different scenarios (average increase of 58% across a variety of languages we experimented with). We also show that multilingual transliteration models can generalize well to languages/language pairs not encountered during training and hence perform well on the zeroshot transliteration task. We show that further improvements can be achieved by using phonetic feature input.