Long-term Dependency Problem Research Articles

Memory-Enhanced Knowledge Reasoning with Reinforcement Learning

In recent years, the emergence of large-scale language models, such as ChatGPT, has presented significant challenges to research on knowledge graphs and knowledge-based reasoning. As a result, the direction of research on knowledge reasoning has shifted. Two critical issues in knowledge reasoning research are the algorithm of the model itself and the selection of paths. Most studies utilize LSTM as the path encoder and memory module. However, when processing long sequence data, LSTM models may encounter the problem of long-term dependencies, where memory units of the model may decay gradually with an increase in time steps, leading to forgetting earlier input information. This can result in a decline in the performance of the LSTM model in long sequence data. Additionally, as the data volume and network depth increase, there is a risk of gradient disappearance. This study improved and optimized the LSTM model to effectively address the problems of gradient explosion and gradient disappearance. An attention layer was employed to alleviate the issue of long-term dependencies, and ConvR embedding was used to guide path selection and action pruning in the reinforcement learning inference model. The overall model achieved excellent reasoning results.

Using Machine Learning and Finite Element Analysis to Extract Traction-Separation Relations at Bonding Wire Interfaces of Insulated Gate Bipolar Transistor Modules.

For insulated gate bipolar transistor (IGBT) modules using wire bonding as the interconnection method, the main failure mechanism is cracking of the bonded interface. Studying the mechanical properties of the bonded interface is crucial for assessing the reliability of IGBT modules. In this paper, first, shear tests are conducted on the bonded interface to test the bonded interface's strength. Then, finite element-cohesive zone modeling (FE-CZM) is established to describe the mechanical behavior of the bonded interface. A novel machine learning (ML) architecture integrating a convolutional neural network (CNN) and a long short-term memory (LSTM) network is used to identify the shape and parameters of the traction separation law (TSL) of the FE-CZM model accurately and efficiently. The CNN-LSTM architecture not only has excellent feature extraction and sequence-data-processing abilities but can also effectively address the long-term dependency problem. A total of 1800 sets of datasets are obtained based on numerical computations, and the CNN-LSTM architecture is trained with load-displacement (F-δ) curves as input parameters and TSL shapes and parameters as output parameters. The results show that the error rate of the model for TSL shape prediction is only 0.186%. The performance metric's mean absolute percentage error (MAPE) is less than 3.5044% for all the predictions of the TSL parameters. Compared with separate CNN and LSTM architectures, the proposed CNN-LSTM-architecture approach exhibits obvious advantages in recognizing TSL shapes and parameters. A combination of the FE-CZM and ML methods in this paper provides a promising and effective solution for identifying the mechanical parameters of the bonded interfaces of IGBT modules.

An ensemble model using temporal convolution and dual attention gated recurrent unit to analyze risk of civil aircraft

As the most important issue in civil aviation, safety needs to be maintained on an acceptable risk level in the air transport system at all times. Due to the high safety requirements in civil aviation, effective emergency response to high-risk events is of great significance for safety insurance. Therefore, it is very necessary to judge the risk level immediately after receiving the unsafe event reports. Unsafe event reports collected in the Aviation Safety Report System (ASRS) contain highly unstructured short texts, which pose a challenge for rapid risk analysis. In this paper, a risk analysis model that integrates a temporal convolutional network (TCN) and a dual attention gated recurrent unit (DAGRU) neural network is proposed for predicting the risk of civil aircraft based on the ASRS report. On the one hand, the causal convolution and dilated convolution structures of TCN are utilized to obtain higher-level text sequence features. On the other hand, the gated recurrent unit (GRU) neural network with dual attention gates is used to learn the contextual information of ASRS report. The introduced attention mechanism of GRU pays much attention to the significant information during the prediction procedure. The ability of GRU to solve long-term dependency problems is greatly enhanced. Finally, in order to verify the validity and reliability of the proposed ensemble model, the proposed TCN-DAGRU is compared with seven benchmark models. The comparison results show that the proposed TCN-DAGRU ensemble model can achieve the highest prediction accuracy and provide reliable support for rapid prediction of aviation safety risk.

Lithium Battery SOC Estimation Based on Multi-Head Attention Mechanism and GRU Algorithm

Pure electric vehicles have been widely used due to their non-pollution, low noise, high energy conversion efficiency and other advantages. SOC (State of Charge) is a crucial indicator for lithium batteries and pure electric vehicles. SOC cannot be directly measured. This article designs a new network structure. It is the GRU-Attention network structure. The stacked GRU algorithm in GRU-Attention network extracts the temporal characteristics of lithium battery test data, and the stacked multi-head self-attention network extracts the global information. The GRU-Attention network can avoid long-term dependency and gradient disappearance problems. The proposed network utilizes Stacked FFN as the dense layer. This article will test the network designed in the public data set at the University of Maryland. Simultaneously, this article compares the effects of different BatchSize on the performance of the algorithm. The network training process converges more effectively with a smaller BatchSize. Both too large and too small BatchSize have a negative impact on the generalization performance of the network. The extraction of the time-order character, however, may be hampered if the timestamp is too small. At the same time, the paper also compares the GRU-Attention network horizontally with the GRU and Attention networks. Eventually, the GRU-Attention network proposed in this article could better meet the estimate of the lithium battery SOC.

Informer-Based Temperature Prediction Using Observed and Numerical Weather Prediction Data.

This paper proposes an Informer-based temperature prediction model to leverage data from an automatic weather station (AWS) and a local data assimilation and prediction system (LDAPS), where the Informer as a variant of a Transformer was developed to better deal with time series data. Recently, deep-learning-based temperature prediction models have been proposed, demonstrating successful performances, such as conventional neural network (CNN)-based models, bi-directional long short-term memory (BLSTM)-based models, and a combination of both neural networks, CNN-BLSTM. However, these models have encountered issues due to the lack of time data integration during the training phase, which also lead to the persistence of a long-term dependency problem in the LSTM models. These limitations have culminated in a performance deterioration when the prediction time length was extended. To overcome these issues, the proposed model first incorporates time-periodic information into the learning process by generating time-periodic information and inputting it into the model. Second, the proposed model replaces the LSTM with an Informer as an alternative to mitigating the long-term dependency problem. Third, a series of fusion operations between AWS and LDAPS data are executed to examine the effect of each dataset on the temperature prediction performance. The performance of the proposed temperature prediction model is evaluated via objective measures, including the root-mean-square error (RMSE) and mean absolute error (MAE) over different timeframes, ranging from 6 to 336 h. The experiments showed that the proposed model relatively reduced the average RMSE and MAE by 0.25 °C and 0.203 °C, respectively, compared with the results of the CNN-BLSTM-based model.

Long Short-Term Memory and Attention Models for Simulating Urban Densification

This paper introduces a novel cellular automata model that combines Long Short-Term Memory, Attention, and Neural Network models to capture spatio-temporal Land Use Change (LUC) behaviors while addressing the challenge of imbalanced datasets. The proposed method is developed and validated using data from Belgium, defined as three (100x100) m raster-based built-up maps for 2000, 2010, and 2020. The model is trained and validated using data from 2000 to 2010, and its effectiveness is tested using data from 2010 to 2020. The key contribution of our approach lies in its ability to tackle long-term temporal dependency and class imbalance problems in LUC science. Our proposed method significantly enhances the performance of spatio-temporal LUC simulation. Additionally, we adopt a data splitting strategy that takes into account the different transitions between classes, improving the accuracy of the model predictions of minority class. The obtained results demonstrate the efficiency of the proposed model in capturing complex spatio-temporal dynamics and reducing the impact of imbalanced datasets surpassing existing methods. The implications of our study extend beyond LUC modeling, as the proposed approach can be applied to a wide range of applications where machine learning is used to model complex environmental and geographical phenomena.

Bridging the Gap: Enhancing Storm Surge Prediction and Decision Support with Bidirectional Attention-Based LSTM

Accurate storm surge forecasting is vital for saving lives and avoiding economic and infrastructural damage. Failure to accurately predict storm surge can have catastrophic repercussions. Advances in machine learning models show the ability to improve accuracy of storm surge prediction by leveraging vast amounts of historical and realtime data such as weather and tide patterns. This paper proposes a bidirectional attention-based LSTM storm surge architecture (BALSSA) to improve prediction accuracy. Training and evaluation utilized extensive meteorological and tide level data from 77 typhoon incidents in Hong Kong and Macao between 2017 and 2022. The proposed methodology is able to model complex non-linearities between large amounts of data from different sources and identify complex relationships between variables that are typically not captured by traditional physical methods. BALSSA effectively resolves the problem of long-term dependencies in storm surge prediction by the incorporation of an attention mechanism. It enables selective emphasis on significant features and boosts the prediction accuracy. Evaluation has been conducted using real-world datasets from Macao to validate our storm surge prediction model. Results show that accuracy and robustness of predictions were significantly improved by the incorporation of attention mechanisms in our models. BALSSA captures temporal dynamics effectively, providing highly accurate storm surge forecasts (MAE: 0.0126, RMSE: 0.0003) up to 72 h in advance. These findings have practical significance for disaster risk reduction strategies, saving lives through timely evacuation and early warnings. Experiments comparing BALSSA variations with other machine learning algorithms consistently validate BALSSA’s superior predictive performance. It offers an additional risk management tool for civil-protection agencies and governments, as well as an ideal solution for enhancing storm surge prediction accuracy, benefiting coastal communities.

GCN-ONLSTM: Process Next Event Prediction Method based on Spatio-Temporal Feature Fusion

The task of predicting the next event of a process is the focus of research in the field of predictive process monitoring. Most of the existing methods to achieve this task only process the event log trace as a one-dimensional sequence or regular two-dimensional image data, without considering the simultaneous internal synchronization of the event log. It contains temporal and spatial feature information. In addition, existing studiesignore that trace data is a non-Euclidean structure with topological relationships. In order to solve the above problems and further improve the accuracy of model prediction, this paper constructs a GCN-ONLSTM network that fuses temporal and spatial dimension features, constructs the spatial relationship between events through a two-layer graph convolution network, and improves the feature expression ability of data. And combined with ONLSTM (Ordered Neurons LSTM) network to process the hierarchical structure of trace sequence, to further solve the long-term dependency problem. The ablation experiments and comparison experiments are carried out through 6 BPI public event log data. The results show that the proposed method has significantly improved prediction accuracy in each event log compared with other existing deep learning methods, and the highest is higher than the traditional LSTM (Long Short-term Memory) increased by 8.63%, it can be considered that this method has better performance for the next event prediction task of the process.

Approximation and Analysis of Natural Data Based on NARX Neural Networks Involving Wavelet Filtering

Recurrent neural network (RNN) models continue the theory of the autoregression integrated moving average (ARIMA) model class. In this paper, we consider the architecture of the RNN with embedded memory—«Process of Nonlinear Autoregressive Exogenous Model» (NARX). Though it is known that NN is a universal approximator, certain difficulties and restrictions in different NN applications are still topical and call for new approaches and methods. In particular, it is difficult for an NN to model noisy and significantly nonstationary time series. The paper suggests optimizing the modeling process for a complicated-structure time series by NARX networks involving wavelet filtering. The developed procedure of wavelet filtering includes the application of the construction of wavelet packets and stochastic thresholds. A method to estimate the thresholds to obtain a solution with a defined confidence level is also developed. We introduce the algorithm of wavelet filtering. It is shown that the proposed wavelet filtering makes it possible to obtain a more accurate NARX model and improves the efficiency of the forecasting process for a natural time series of a complicated structure. Compared to ARIMA, the suggested method allows us to obtain a more adequate model of a nonstationary time series of complex nonlinear structure. The advantage of the method, compared to RNN, is the higher quality of data approximation for smaller computation efforts at the stages of network training and functioning that provides the solution to the problem of long-term dependencies. Moreover, we develop a scheme of approach realization for the task of data modeling based on NARX and anomaly detection. The necessity of anomaly detection arises in different application areas. Anomaly detection is of particular relevance in the problems of geophysical monitoring and requires method accuracy and efficiency. The effectiveness of the suggested method is illustrated in the example of processing of ionospheric parameter time series. We also present the results for the problem of ionospheric anomaly detection. The approach can be applied in space weather forecasting to predict ionospheric parameters and to detect ionospheric anomalies.

Next-Day Medical Activities Recommendation Model with Double Attention Mechanism Using Generative Adversarial Network.

Medical activities recommendation is a key aspect of an intelligent healthcare system, which can assist doctors with little clinical experience in clinical decision making. Medical activities recommendation can be seen as a kind of temporal set prediction. Previous studies about them are based on Recurrent Neural Network (RNN), which does not incorporate personalized medical history or differentiate between the impact of medical activities. To address the above-given issues, this paper proposes a Next-Day Medical Activities Recommendation (NDMARec) model. Specifically, our model firstly proposes an inpatient day embedding method based on soft-attention which balances the impact of different medical activities to get a joint representation of medical activities that occurred within the same day. Then, a fusion module is designed to combine features of inpatient day and medical history to achieve personalization. These features are learned by the self-attention mechanism that solves the long-term dependency problem of RNNs. Last, adversarial training is introduced to improve the generalization ability of our model. Extensive experiments on a real dataset from a hospital are conducted to show that NDMARec outperformed both classical and state-of-the-art methods.

소스 코드 취약점 탐지를 위한 서브워드 토큰화 기반의 딥러닝 모델

The study of vulnerability detection in source code has been attracting attention in practice and academia because web applications can be vulnerable to attacks from the outside due to the open access characteristics. This study aims to build deep learning models and evaluate their performances for the field of source code vulnerability detection. The proposed deep learning models tackle class imbalance problem, long-term dependency problem, and out-of-vocabulary problem which are challenging problems in detecting source code vulnerabilities. As an experiment result, the precision of the subword tokenization-based one-dimensional convolution model showed 39%, which is about 20 times higher than the expected precision of 1.92% of the model predicted by chance. Although Conv1d+BT model using the BERT tokenizer showed the highest AUC value of 0.9116, the precision and recall of this model were 0.39 and 0.35, so it is judged that further improvement is needed for practical application.

Asymmetric CNN for Image Superresolution

Deep convolutional neural networks (CNNs) have been widely applied for low-level vision over the past five years. According to the nature of different applications, designing appropriate CNN architectures is developed. However, customized architectures gather different features via treating all pixel points as equal to improve the performance of given application, which ignores the effects of local power pixel points and results in low training efficiency. In this article, we propose an asymmetric CNN (ACNet) comprising an asymmetric block (AB), a memory enhancement block (MEB), and a high-frequency feature enhancement block (HFFEB) for image superresolution (SR). The AB utilizes one-dimensional (1-D) asymmetric convolutions to intensify the square convolution kernels in horizontal and vertical directions for promoting the influences of local salient features for single image SR (SISR). The MEB fuses all hierarchical low-frequency features from AB via a residual learning technique to resolve the long-term dependency problem and transforms obtained low-frequency features into high-frequency features. The HFFEB exploits low- and high-frequency features to obtain more robust SR features and address the excessive feature enhancement problem. Additionally, it also takes charge of reconstructing a high-resolution image. Extensive experiments show that our ACNet can effectively address SISR, blind SISR, and blind SISR of blind noise problems. The code of the ACNet is shown at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/hellloxiaotian/ACNet</uri> .

A Time Series Transformer based method for the rotating machinery fault diagnosis

Fault diagnosis of rotating machinery is a significant engineering problem. In recent years, fault diagnosis methods have matured based on the Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN). However, these traditional models have the problem of Long-Term Dependencies, leading to their feature extraction ability defect. To address these issues, we proposed a new method based on the Time Series Transformer (TST) to recognize the fault modes of the various rotating machinery. In this paper, firstly, we design a new tokens sequences generation method that can handle data in 1D format, namely time series tokenizer. Then the TST combining time series tokenizer and Transformer is presented. The test results on the given datasets show that the proposed method has better fault identification capability than traditional CNN and RNN models. Secondly, the effect of structural hyperparameters on fault diagnosis performance, computational complexity, and parameters number of the TST is analyzed in detail through experiments. The influence laws of some hyperparameters are obtained as well. Finally, the feature vectors in the embedding space are visualized via the t-Distributed Stochastic Neighbor Embedding (t-SNE) dimensionality reduction method. On this basis, the working pattern of TST is explained to a certain extent. Moreover, we find that the feature vectors extracted by the proposed method show the best intra-class compactness and inter-class separability compared with CNN and RNN models by analyzing their distribution form, which further demonstrates the effectiveness of the proposed method.

Physical Model versus Artificial Neural Network (ANN) Model: A Comparative Study on Modeling Car-Following Behavior at Signalized Intersections

Many studies have simulated traffic behavior at signalized intersections using various Car-Following (CF) models. However, the performance of which CF Model is superior at signalized intersections has not been thoroughly analyzed and evaluated. In this study, two novel Artificial Neural Network (ANN) CF models, the Convolutional Neural Network—Long Short-term Memory (CNN-LSTM) and the Convolution-LSTM (Conv-LSTM)—are first applied to predict CF behaviors at signalized intersections. Both models can extract spatial and temporal information to address the long-term dependency problem more effectively. Based on the filtered NGSIM dataset, we conduct a comparative empirical study of three conventional CF models and five ANN CF models. The dataset is divided into two categories based on the characteristics of CF behavior at signalized intersections: continuous and discontinuous. The experiments demonstrated that ANN CF models outperformed conventional CF models when the output was the velocity in two categories of traffic flow but only failed to do so when the output was acceleration in discontinuous traffic flow. The proposed models were capable of accurately predicting acceleration, but the traffic fluctuations also existed as time passed. Additionally, it was discovered that while the ANN CF model is preferable for traffic flow simulation, the conventional CF model still cannot be ignored for discontinuous traffic flow simulation, particularly when acceleration is required.

Sentiment analysis by using Naïve‐Bayes classifier with stacked CARU

A long sequence always contains long-term dependency problems, which leads to paragraph-based sentiment analysis being a very challenging task and difficult to evaluate by using a simple RNN network. It is proposed in this letter to use a stacked CARU network to extract the main information in a paragraph. The resulting network also points out how to use a CNN-based extractor to explore complete passages and capture useful features in their hidden state. In particular, instead of using the Softmax function, the Naïve-Bayes classifier is connected to the end of the CNN-based extractor. The proposed models also take into account the conditional independence of the observed results under the hidden variables, which aims to project features into a probability distribution appreciated for its simplicity and interpretability. The advantages of these models in sentiment analysis are empirically investigated by combining the usual classifiers with the results of GloVe embedding on the SST-5 and IMDB datasets.

A Dual CNN for Image Super-Resolution

High-quality images have an important effect on high-level tasks. However, due to human factors and camera hardware, digital devices collect low-resolution images. Deep networks can effectively restore these damaged images via their strong learning abilities. However, most of these networks depended on deeper architectures to enhance clarities of predicted images, where single features cannot deal well with complex screens. In this paper, we propose a dual super-resolution CNN (DSRCNN) to obtain high-quality images. DSRCNN relies on two sub-networks to extract complementary low-frequency features to enhance the learning ability of the SR network. To prevent a long-term dependency problem, a combination of convolutions and residual learning operation is embedded into dual sub-networks. To prevent information loss of an original image, an enhanced block is used to gather original information and obtained high-frequency information of a deeper layer via sub-pixel convolutions. To obtain more high-frequency features, a feature learning block is used to learn more details of high-frequency information. The proposed method is very suitable for complex scenes for image resolution. Experimental results show that the proposed DSRCNN is superior to other popular in SR networks. For instance, our DSRCNN has obtained improvement of 0.08 dB than that of MemNet on Set5 for ×3.

Temporal attention augmented transformer Hawkes process

In recent years, mining the knowledge from asynchronous sequences by Hawkes process is a subject worthy of continued attention, and Hawkes processes based on the neural network have gradually become the most hotly researched fields, especially based on the recurrence neural network (RNN). However, these models still contain some inherent shortcomings of RNN, such as vanishing and exploding gradient and long-term dependency problems. Meanwhile, transformer-based on self-attention has achieved great success in sequential modeling like text processing and speech recognition. Although the Transformer Hawkes process (THP) has gained huge performance improvement, THPs do not effectively utilize the temporal information in the asynchronous events, for these asynchronous sequences, the event occurrence instants are as important as the types of events, while conventional THPs simply convert temporal information into position encoding and add them as the input of transformer. With this in mind, we come up with a new kind of Transformer-based Hawkes process model, temporal attention augmented transformer Hawkes Process (TAA-THP), we modify the traditional dot-product attention structure and introduce the temporal encoding into attention structure. We conduct numerous experiments on a wide range of synthetic and real-life datasets to validate the performance of our proposed TAA-THP model, a significant improvement compared with existing baseline models on the different measurements is achieved, including log-likelihood on the test dataset, and prediction accuracies of event types and occurrence times. In addition, through the ablation studies, we vividly demonstrate the merit of introducing additional temporal attention by comparing the performance of the model with and without temporal attention.

Improving de novo Molecule Generation by Embedding LSTM and Attention Mechanism in CycleGAN.

The application of deep learning in the field of drug discovery brings the development and expansion of molecular generative models along with new challenges in this field. One of challenges in de novo molecular generation is how to produce new reasonable molecules with desired pharmacological, physical, and chemical properties. To improve the similarity between the generated molecule and the starting molecule, we propose a new molecule generation model by embedding Long Short-Term Memory (LSTM) and Attention mechanism in CycleGAN architecture, LA-CycleGAN. The network layer of the generator in CycleGAN is fused head and tail to improve the similarity of the generated structure. The embedded LSTM and Attention mechanism can overcome long-term dependency problems in treating the normally used SMILES input. From our quantitative evaluation, we present that LA-CycleGAN expands the chemical space of the molecules and improves the ability of structure conversion. The generated molecules are highly similar to the starting compound structures while obtaining expected molecular properties during cycle generative adversarial network learning, which comprehensively improves the performance of the generative model.

Text Embedding Bank for Detailed Image Paragraph Captioning

Existing deep learning-based models for image captioning typically consist of an image encoder to extract visual features and a language model decoder, an architecture that has shown promising results in single high-level sentence generation. However, only the word-level guiding signal is available when the image encoder is optimized to extract visual features. The inconsistency between the parallel extraction of visual features and sequential text supervision limits its success when the length of the generated text is long (more than 50 words). We propose a new module, called the Text Embedding Bank (TEB), to address this problem for image paragraph captioning. This module uses the paragraph vector model to learn fixed-length feature representations from a variable-length paragraph. We refer to the fixed-length feature as the TEB. This TEB module plays two roles to benefit paragraph captioning performance. First, it acts as a form of global and coherent deep supervision to regularize visual feature extraction in the image encoder. Second, it acts as a distributed memory to provide features of the whole paragraph to the language model, which alleviates the long-term dependency problem. Adding this module to two existing state-of-the-art methods achieves a new state-of-the-art result on the paragraph captioning Stanford Visual Genome dataset.

Medical image super-resolution via deep residual neural network in the shearlet domain

This paper proposes aconvolutional neural network (CNN)-based efficient medical image super-resolution (SR) method in the shearlet domain. Because of differences between imaging mechanisms optimized for natural images and medical images, the design begins with building a medical image dataset for medical image SR and extracting effective areas to remarkably enhance the training effects of the CNN-based method. Then, a new medical image SR network structure—deep medical super-resolution network (DMSRN)—has been designed in which local residual learning is implemented through a recursive network and combined with global residual learning to heighten the depth of the network on the ground with no parameter increase. This effectively fixes the long-term dependency problem, which causes the prior state layers to barely have any effect on the following state layers. Last, the design addresses the problem of too-smooth reconstruction effects in the CNN-based method in the image space domain; shearlet transform is introduced to DMSRN to restore global topology through low-frequency sub-bands and restore local edge detail information through high-frequency sub-bands. Experimental results show that the proposed method is better than other state-of-the-art methods for medical image SR, which significantly promotes the restoration ability of texture structure and edge details.