Long Short-term Memory Decoder Research Articles

LSTM-based graph attention network for vehicle trajectory prediction

Vehicle Trajectory Prediction (VTP) is one of the key technologies for autonomous driving, which can improve the safety and collaboration of the autonomous driving system. The interaction behavior among vehicles in reality has an impact on VTP. However, many methods ignore the interaction among vehicles, which results in limited accuracy of prediction results. Therefore, we propose a Long Short-Term Memory (LSTM)-based Graph Attention Network (GAT) method for VTP, which encodes vehicle trajectory information with LSTM networks and represents vehicle interactions with GAT. Firstly, in order to capture the temporal relationship between positions and consider their influence, we use LSTM model to encode the position data. Meanwhile, to comprehensively model vehicle motion and use multidimensional feature representation, we employ another LSTM model to encode the motion data, including position, velocity and acceleration. Secondly, to learn distinct feature representation, we use one GAT module to process the LSTM position encoding features for capturing spatial relationships of position information. Another GAT module is employed to process the LSTM motion encoding features for fully considering multidimensional motion dynamics and spatial–temporal dependencies. Finally, the LSTM decoder receives all features and predicts the vehicle trajectory. The experimental results show that the proposed method demonstrates superior predictive performance by using the Next Generation Simulation (NGSIM) dataset.

An intelligent chatbot design and implementation model using long short-term memory with recurrent neural networks and attention mechanism

Assistant Conversational Agents (chatbots) are becoming increasingly complex and sophisticated since they are enabled to understand and respond to human language. Chatbots have varying organizational functions, from customer support to personal assistance, and have dominated the digital market as they are efficient and effective. Chatbots can collect valuable information from customers through interaction which can result in a better customer experience, but an inappropriate or inadequate response can also lead to the loss of customers. Therefore, the chatbots must be intelligent enough to provide the most efficient and effective response to positively impact the customers and add value to the business and organization. Over the years, several streams of research have been conducted, and multiple chatbots have been created, but they have their limitation, which has been discussed in this article. We suggest a method to upgrade the performance of the chatbots so that they can respond to the user query with better accuracy. The proposed model shows the implementation of a chatbot using Long Short-Term Memory (LSTM), attention mechanism, Bag of Words (BOW), and beam search decoding. The sequence-to-sequence (Seq2Seq) architecture with an LSTM encoder and decoder has been used. The dialog dataset is preferred to train and test the model, and the BleuN algorithm is used to determine the chatbot’s accuracy.

Prediction of wave-induced ship motions based on integrated neural network system and spatiotemporal wave-field data

This study introduces an artificial neural network system for ship motion prediction in seaways. To consider the physical characteristics of wave-induced ship motions, neural networks based on a Long Short-Term Memory (LSTM) encoder and decoder, and a convolutional neural network (CNN) are integrated. The LSTM encoder computes the state vector representing the memory effects of motion-induced radiated waves based on past motion records, that is, a sequence-to-one model. In the LSTM decoder, the motion time series is predicted using the encoded initial state vector and foreseen information on the ocean wave field around a vessel, that is, a sequence-to-sequence model. In addition, a CNN is adopted to compress the wave data into a vector sequence. Particularly, the present CNN uses spatiotemporal wave-field data, not a wave signal at single location. To validate the proposed system, a database for training the integrated system was constructed using a physics-based seakeeping program for various sea states. By applying the trained model, deterministic predictions were performed for a new ocean environment, and the accuracy and reliability of the testing results are investigated according to the input data and neural network structures. From the simulation results, it was confirmed that the present encoder–decoder system can conduct ship motion forecasting by effectively considering the motion memory effects and wave excitations as in the ship hydrodynamic model. In addition, excitations and resulting motion responses by short-crested waves can be considered through CNN-based wave-field data processing. Finally, the present machine-learning model also showed the capability of extracting ship operation information (maneuvering quantities) from the given wave-field data.

Diverse Image Captioning via Conditional Variational Autoencoder and Dual Contrastive Learning

Diverse image captioning has achieved substantial progress in recent years. However, the discriminability of generative models and the limitation of cross entropy loss are generally overlooked in the traditional diverse image captioning models, which seriously hurts both the diversity and accuracy of image captioning. In this paper, aiming to improve diversity and accuracy simultaneously, we propose a novel Conditional Variational Autoencoder (DCL-CVAE) framework for diverse image captioning by seamlessly integrating sequential variational autoencoder with contrastive learning. In the encoding stage, we first build conditional variational autoencoders to separately learn the sequential latent spaces for a pair of captions. Then, we introduce contrastive learning in the sequential latent spaces to enhance the discriminability of latent representations for both image-caption pairs and mismatched pairs. In the decoding stage, we leverage the captions sampled from the pre-trained Long Short-Term Memory (LSTM), LSTM decoder as the negative examples and perform contrastive learning with the greedily sampled positive examples, which can restrain the generation of common words and phrases induced by the cross entropy loss. By virtue of dual constrastive learning, DCL-CVAE is capable of encouraging the discriminability and facilitating the diversity, while promoting the accuracy of the generated captions. Extensive experiments are conducted on the challenging MSCOCO dataset, showing that our proposed methods can achieve a better balance between accuracy and diversity compared to the state-of-the-art diverse image captioning models.

Classification of arrhythmias using an LSTM- and GAN-based approach to ECG signal augmentation

Abstract Funding Acknowledgements Type of funding sources: Private grant(s) and/or Sponsorship. Main funding source(s): British Heart Foundation Introduction Automated classification of arrhythmias in ECGs is becoming increasingly important. Publicly available ECG datasets have been widely used by the research community to create novel artificial intelligence models that improve these detection rates. The development of these models requires access to large volume of labelled data. However, access to such databases is becoming increasingly limited. In addition, the datasets are often unbalanced because abnormal rhythms are far outweighed by normal samples. The unbalanced nature of the datasets can lead to less accurate models. Therefore, generating realistic synthetic signals can augment the real signals found in such databases and provide data that allows sophisticated model development. Purpose In this study, we propose a deep learning-based approach for synthetic ECG signal generation that uses long short-term memory (LSTM) autoencoder and generative adversarial networks (GAN) to generate signals that mimic the distribution of arrhythmia signals (Figure 1). Methods The LSTM autoencoder is composed of two parts: an encoder and a decoder (Figure 1b). The encoder takes original ECG signal as its input and uses LSTM layers to compress the signal into a set of features. The decoder is formed by reversing the encoding process, which uses the encoded features as its input and converts them back into the original signal. To generate synthetic signals, we inserted GANs between the LSTM encoder and the decoder. GANs are composed of a generator and a discriminator (Figure 1c). The generator produces synthetic ECG features based on noise, whereas the discriminator tries to distinguish between real features and results received from the generator. The pathological beats studies were: left bundle branch block (LBBB), right bundle branch block (RBBB), aberrated atrial premature (AA), and normal beats (N) from the MIT-BIH arrhythmia database, using lead II only. To evaluate the quality of our synthetic signals, we trained an LSTM classifier on a combination of our real and synthetic data and compared the testing results with a model trained on real data alone. Results The LSTM encoder, decoder and GAN were trained individually for each beat type, and examples of generated signals are illustrated in Figure 2. The average accuracy of the classification for the original dataset was 90%, with a recall of 98% for N, 36% for AA, 39% for LBBB and 97% for RBBB. Once synthetic signals were added to the training set, the average testing classification accuracy increased to 98%, with a recall of 99% for N, 83% for AA, 99% for LBBB and 99% for RBBB. For fair comparison, the testing set contained only real data and remained unchanged for both groups. Conclusion In this work, we proved the capability of GANs to generate realistic synthetic signals that helped to improve the detection rates of arrhythmias as measured by both increased overall accuracy and recall.

A Parallel Turbo Decoder Based on Recurrent Neural Network

A neural network-based decoder, based on a long short-term memory (LSTM) network, is proposed to solve the problem that large decoding delay and performance degradation under non-Gaussian noise due to poor parallelism of existing turbo decoding algorithms. The proposed decoder refers to a unique component coding concept of turbo codes. First, each component decoder is designed based on an LSTM network. Next, each layer of the component decoder is trained, and the trained weights are loaded into the turbo code decoding neural network as initialization parameters. Then, the turbo code decoding network is trained end-to-end. Finally, a complete turbo decoder is realized. The structural advantage of turbo code component coding is fully considered in the design process, and the problem of decoding delay caused by the existence of interleaver is cleverly avoided. The introduction of deep learning technology provides a new idea to solve the traditional communication problems. Simulation results show that the performance of the proposed decoder is improved by 0.5–1.5 dB compared with the traditional serial decoding algorithm in Gaussian white noise and t-distribution noise. When BER performance is close, the LSTM decoder requires half or even less than that of BCJR. Moreover, the results demonstrate that the proposed decoder is adaptive and can be applied to communication systems with various turbo codes. The LSTM decoder shows lower bit error rate, computational complexity and higher decoding efficiency under the same conditions. Therefore, it is necessary to study the turbo code decoding technology based on deep learning combined with the actual channel environment.

Modeling Fabric-Type Actuator Using Point Clouds by Deep Learning

Flexible actuators are popular in the consumer and medical fields because of their flexibility and compliance. However, they are typically difficult to model because of their viscoelasticity and nonlinearity. This letter proposes a method for correcting the deformation of the simulated flexible robots to make it similar to the deformation of real robots using point clouds by deep learning. Long short-term memory (LSTM) can simulate the next frame of actuator deformation from the previous frames of deformations. In this study, we presented the robots with four different muscle structures. We found that using an encoder–LSTM–decoder network can improve the similarity between the deformation of a learned muscle structure and the real deformation and is also effective in correcting the deformation of the unlearned structures. Our correction method reduced the average Chamfer distance of the simulated point clouds of the basic-type structure actuator from 15.89 to 7.81. This research can provide a new concept for future flexible robot modeling using point clouds.

Improve Representation for Cross-Language Clone Detection by Pretrain Using Tree Autoencoder

With the rise of deep learning in recent years, many code clone detection (CCD) methods use deep learning techniques and achieve promising results, so is cross-language CCD. However, deep learning techniques require a dataset to train the models. The dataset is typically small and has a gap between real-world clones due to the difficulty of collecting datasets for cross-language CCD. This creates a data bottleneck problem: data scale and quality issues will cause that model with a better design can still not reach its full potential. To mitigate this, we propose a tree autoencoder (TAE) architecture. It uses unsupervised learning to pretrain with abstract syntax trees (ASTs) of a large-scale dataset, then fine-tunes the trained encoder in the downstream CCD task. Our proposed TAE contains a tree Long Short-Term Memory (LSTM) encoder and a tree LSTM decoder. We design a novel embedding method for AST nodes, including type embedding and value embedding. In the training of TAE, we present an “encode and decode by layers” strategy and a node-level batch size design. For the CCD dataset, we propose a negative sampling method based on probability distribution. The experimental results on two datasets verify the effeteness of our embedding method, as well as that TAE and its pretrain enhance the performance of the CCD model. The node context information is well captured, and the reconstruction accuracy of the node-value reaches 95.45%. TAE pretrain improves the performance of CCD with a 4% increase in F1 score, which alleviates the data bottleneck problem.

Estimation of Lower Limb Kinematics during Squat Task in Different Loading Using sEMG Activity and Deep Recurrent Neural Networks.

The aim of the present study was to predict the kinematics of the knee and the ankle joints during a squat training task of different intensities. Lower limb surface electromyographic (sEMG) signals and the 3-D kinematics of lower extremity joints were recorded from 19 body builders during squat training at four loading conditions. A long-short term memory (LSTM) was used to estimate the kinematics of the knee and the ankle joints. The accuracy, in terms root-mean-square error (RMSE) metric, of the LSTM network for the knee and ankle joints were 6.774 ± 1.197 and 6.961 ± 1.200, respectively. The LSTM network with inputs processed by cross-correlation (CC) method showed 3.8% and 4.7% better performance in the knee and ankle joints, respectively, compared to when the CC method was not used. Our results showed that in the prediction, regardless of the intensity of movement and inter-subject variability, an off-the-shelf LSTM decoder outperforms conventional fully connected neural networks.

LSTM auto‐encoder based representative scenario generation method for hybrid hydro‐PV power system

The increasing penetration of renewable energy sources causes complex uncertainties of the power system. To capture such uncertainties in power system planning, an important step is to generate representative scenarios. In this work, a long short term memory (LSTM) auto-encoder based approach is proposed to generate representative scenarios in an integrated hydro-photovoltaic (PV) power generation system, which consists of feature extraction by LSTM Encoder, scenario clustering in feature domain by combining gap statistics method and K-means++, and representative scenario reconstruction by using LSTM Decoder. Compared with traditional scenario selection and generation methods, the proposed method can better capture the patterns of multivariate time-series data in both temporal and spatial dimensions. A case study in southwest China is used to demonstrate the effectiveness of the proposed method, which outperforms other existing methods by achieving the lowest SSE and DBI indices of 0.89 and 0.12, respectively, and obtaining the best SIL and CHI scores of 0.93 and 2.30, respectively, In addition, the case study shows the proposed model setup works more stable for scenario generation.

Using An Attention-Based LSTM Encoder–Decoder Network for Near Real-Time Disturbance Detection

Accurate prediction of future observations based on past data is the key to near real-time disturbance detection using satellite image time series (SITS). To overcome the limitations of existing methods, we present an attention-based long-short-term memory (LSTM) encoder-decoder model in which the historical time series of a pixel is encoded with a bidirectional LSTM encoder while the future time series is produced by another LSTM decoder. An attention mechanism is integrated into the encoder-decoder model to align the input time series with the output time series and to dynamically choose the most relevant contextual information while forecasting. Based on the proposed model, we develop a framework for near real-time disturbance detection and verify its effectiveness in the case of burned area mapping. The prediction accuracy of the proposed model is evaluated using moderate resolution imaging spectroradiometer (MODIS) time series and compared with state-of-the-art models. Experimental results show that our model achieves the best results in terms of lower prediction error and higher model fitness. We also evaluate the disturbance detection ability of the proposed framework. The proposed approach improves the detection rate of disturbances while suppressing false alarms, and increases the temporal accuracy. We suggest that the proposed methods provide new tools for enhancing current early warning systems in real time.

Performance Evaluation of Channel Decoder based on Recurrent Neural Network

Recently, recurrent neural network (RNN) has demonstrated superior performance for channel decoder, which motivates us to explore which RNN decoder can be more efficient. In this paper, we propose three kinds of RNN decoders, which are built upon long short term memory (LSTM), gated recurrent unit (GRU) and bidirectional gated recurrent units (Bi-GRU), respectively. The performance of these three RNN decoders are evaluated through lots of simulations, which indicate that the GRU decoder with simplest structure and least computational time, has similar bit error rate (BER) performance as that of the LSTM decoder. The Bi-GRU decoder has the best BER performance at the expenses of more computational time. However, it is prone to overfitting. Furthermore, we find that the BER performance of RNN decoders without dropout is better than that of the decoders with dropout when decoding models are underfitting, while the RNN decoders are better to dropout when decoding models are overfitting.

Constrained representation learning for recurrent policy optimisation under uncertainty

Learning to make decisions in partially observable environments is a notorious problem that requires a complex representation of controllers. In most work, the controllers are designed as a non-linear mapping from a sequence of temporal observations to actions. These problems can, in principle, be formulated as a partially observable Markov decision process whose policy can be parameterised through the use of recurrent neural networks. In this paper, we will propose an alternative framework that (a) uses the Long-Short-Term-Memory (LSTM) Encoder-Decoder framework to learn an internal state representation for historical observations and then (b) integrates it into existing recurrent policy models to improve the task performance. The LSTM Encoder encodes a history of observations as input into a representation of internal states. The LSTM Decoder can perform two alternative decoding tasks: predicting the same input observation sequence or predicting future observation sequences. The first proposed decoder acts like an auto-encoder that will guide and constrain the learning of a useful internal state for the policy optimisation task. The second proposed decoder decodes the learnt internal state by the encoder to predict future observation sequences. This idea makes the network act like a non-linear predictive state representation model. Both these decoding parts, which introduce constraints to policy representation, will help guide both the policy optimisation problem and latent state representation learning. The integration of representation learning and policy optimisation aims to help learn more complex policies and improve the performance of policy learning tasks.

High-Quality Image Captioning With Fine-Grained and Semantic-Guided Visual Attention

The soft-attention mechanism is regarded as one of the representative methods for image captioning. Based on the end-to-end convolutional neural network (CNN)-long short term memory (LSTM) framework, the soft-attention mechanism attempts to link the semantic representation in text (i.e., captioning) with relevant visual information in the image for the first time. Motivated by this approach, several state-of-the-art attention methods are proposed. However, due to the constraints of CNN architecture, the given image is only segmented to the fixed-resolution grid at a coarse level. The visual feature extracted from each grid indiscriminately fuses all inside objects and/or their portions. There is no semantic link between grid cells. In addition, the large area “stuff” (e.g., the sky or a beach) cannot be represented using the current methods. To address these problems, this paper proposes a new model based on the fully convolutional network (FCN)-LSTM framework, which can generate an attention map at a fine-grained grid-wise resolution. Moreover, the visual feature of each grid cell is contributed only by the principal object. By adopting the grid-wise labels (i.e., semantic segmentation), the visual representations of different grid cells are correlated to each other. With the ability to attend to large area “stuff,” our method can further summarize an additional semantic context from semantic labels. This method can provide comprehensive context information to the language LSTM decoder. In this way, a mechanism of fine-grained and semantic-guided visual attention is created, which can accurately link the relevant visual information with each semantic meaning inside the text. Demonstrated by three experiments including both qualitative and quantitative analyses, our model can generate captions of high quality, specifically high levels of accuracy, completeness, and diversity. Moreover, our model significantly outperforms all other methods that use VGG-based CNN encoders without fine-tuning.

Decoding hindlimb kinematics from primate motor cortex using long short-term memory recurrent neural networks.

Recent machine learning techniques have become a powerful tool in a variety of tasks, including neural decoding. Deep neural networks, particularly recurrent models, leverage the temporal evolution of neural ensemble activity to decode complex movement and sensory signals. Using single-unit recordings from microelectrode arrays implanted in the leg area of primary motor cortex in non-human primates, we decode the positions and angles of hindlimb joints during a locomotion task using a long short-term memory (LSTM) network. The LSTM decoder improved decoding over traditional filtering methods, such as Wiener and Kalman filters. However, dramatic improvements over other machine learning (e.g. XGBoost) and latent state-space methods were not observed.

Video Captioning With Attention-Based LSTM and Semantic Consistency

Recent progress in using long short-term memory (LSTM) for image captioning has motivated the exploration of their applications for video captioning. By taking a video as a sequence of features, an LSTM model is trained on video-sentence pairs and learns to associate a video to a sentence. However, most existing methods compress an entire video shot or frame into a static representation, without considering attention mechanism which allows for selecting salient features. Furthermore, existing approaches usually model the translating error, but ignore the correlations between sentence semantics and visual content. To tackle these issues, we propose a novel end-to-end framework named aLSTMs, an attention-based LSTM model with semantic consistency, to transfer videos to natural sentences. This framework integrates attention mechanism with LSTM to capture salient structures of video, and explores the correlation between multimodal representations (i.e., words and visual content) for generating sentences with rich semantic content. Specifically, we first propose an attention mechanism that uses the dynamic weighted sum of local two-dimensional convolutional neural network representations. Then, an LSTM decoder takes these visual features at time $t$ and the word-embedding feature at time $t$ $-$ 1 to generate important words. Finally, we use multimodal embedding to map the visual and sentence features into a joint space to guarantee the semantic consistence of the sentence description and the video visual content. Experiments on the benchmark datasets demonstrate that our method using single feature can achieve competitive or even better results than the state-of-the-art baselines for video captioning in both BLEU and METEOR.

First-Feed LSTM model for video description

Video description (VD) aims to automatically generate descriptive natural language for videos. With its successful implementations and a broad range of applications, lots of work based on deep neural network (DNN) models have been put forward by researchers. This paper takes inspiration from an image caption model and develops an end-to-end VD model using long short-term memory (LSTM). Single video feature is fed to the first unit of LSTM decoder, and subsequent words of sentence are generated on previous predicted words. Experimental results on two publicly available datasets demonstrate that the performance of the proposed model outperforms that of baseline.