Long Short-term Memory Structure Research Articles

Exploring the development and application of LSTM variants

Long Short-Term Memory (LSTM) is receiving increasing attention as the development of deep learning technology. The gate structure of LSTM enhances long-term memory, forming its superior capacity to complete tasks that challenge traditional RNN. However, considering the wide variety of applications, a comprehensive understanding of the development and application of the model, which is vital for future research, is comparatively lacking. Therefore, this paper is produced with the hope of offering an overview of the development of LSTM. It shows the process of development from RNN to LSTM and explains the aim and necessity of LSTMs birth. After that it introduces the structure of LSTM, analyses its advantages over RNN, and discusses the application of some popular LSTM variants, such as peephole LSTM, bidirectional LSTM, and GRU. Hopefully, this work can provide a more profound knowledge of LSTM's benefits and potential, identifying worthwhile avenues or fields of future research.

A Novel Variant of LSTM Stock Prediction Method Incorporating Attention Mechanism

Long Short-Term Memory (LSTM) is an effective method for stock price prediction. However, due to the nonlinear and highly random nature of stock price fluctuations over time, LSTM exhibits poor stability and is prone to overfitting, resulting in low prediction accuracy. To address this issue, this paper proposes a novel variant of LSTM that couples the forget gate and input gate in the LSTM structure, and adds a “simple” forget gate to the long-term cell state. In order to enhance the generalization ability and robustness of the variant LSTM, the paper introduces an attention mechanism and combines it with the variant LSTM, presenting the Attention Mechanism Variant LSTM (AMV-LSTM) model along with the corresponding backpropagation algorithm. The parameters in AMV-LSTM are updated using the Adam gradient descent method. Experimental results demonstrate that the variant LSTM alleviates the instability and overfitting issues of LSTM, effectively improving prediction accuracy. AMV-LSTM further enhances accuracy compared to the variant LSTM, and compared to AM-LSTM, it exhibits superior generalization ability, accuracy, and convergence capability.

State of charge estimation for Li-ion battery during dynamic driving process based on dual-channel deep learning methods and conditional judgement

The accurate estimation of the state of charge (SOC) is crucial for the safe and reliable operation of Li-ion batteries. In this paper, novel dual-channel deep learning methods were proposed for SOC estimation during dynamic driving cycles. For dual-channel deep learning methods, each channel of the deep learning methods could be a single convolutional neural network (CNN) layer, single long short-term memory (LSTM) layer or sequentially combined structure of CNN and LSTM. After evaluation, it could be found that dual-channel deep learning methods could achieve higher accuracy than conventional single-channel method, however, the model size and floating-point operations (FLOPs) could be either reduced or increased. Among all the dual-channel deep learning methods, CNN + LSTM-CNN + LSTM method could achieve the highest accuracy and the shortest training time with sacrifices in computational complexity. CNN–CNN + LSTM method could achieve higher accuracy than the single-channel deep learning method with reduced computational complexity and model size. As the error of SOC estimation is highly correlated with voltage signals, conditional judgement was then integrated with dual-channel deep learning method to obtain higher accuracy. After integrated with conditional judgement, the averaged error of SOC estimation could be reduced by more than 45% without a substantial sacrifice in computational time.

A Comparative Study of LSTM and Temporal Convolutional Network Models for Semisubmersible Platform Wave Runup Prediction

Abstract Wave runup prediction is necessary for offshore structure designs and early warnings. Data-driven methods based on machine learning have inspired reduced-order solutions for wave–structure interaction problems. This study provides the quantification of deep learning algorithms’ potential for wave runup prediction. Two prominent deep learning models were utilized to predict the wave runups along the fore column of semisubmersible under head seas. The long short-term memory (LSTM) and the temporal convolutional networks (TCNs) were comprehensively compared based on the datasets from a model test carried out in the deep ocean basin. The LSTM and TCN model structures were optimized to compare prediction accuracy and computational complexity reasonably. The results reveal that (1) both developed TCN and LSTM models had a satisfied prediction accuracy of over 90%. Their predictions were extended to 10 s into the future with accuracies over 80% and 45%, respectively. (2) With the noise-extended datasets, the LSTM model was robust to noises, while the TCN model showed better prediction performance on the extreme wave runup events. (3) The incident wave and dominant rotation provided the major information for wave runup prediction. TCN and LSTM models’ prediction accuracies were 91.5% and 89.3% based on the simplified input tensors composed of incident wave and pitch. The comparison showed the great potential of the TCN model to predict the nonlinear wave runup with less time and memory costs. The input tensors’ design and optimization based on physical understanding also play a significant role in the prediction performance.

TLN-LSTM: an automatic modulation recognition classifier based on a two-layer nested structure of LSTM network for extremely long signal sequences

Purpose Automatic modulation recognition (AMR) is a challenging problem in intelligent communication systems and has wide application prospects. At present, although many AMR methods based on deep learning have been proposed, the methods proposed by these works cannot be directly applied to the actual wireless communication scenario, because there are usually two kinds of dilemmas when recognizing the real modulated signal, namely, long sequence and noise. This paper aims to effectively process in-phase quadrature (IQ) sequences of very long signals interfered by noise. Design/methodology/approach This paper proposes a general model for a modulation classifier based on a two-layer nested structure of long short-term memory (LSTM) networks, called a two-layer nested structure (TLN)-LSTM, which exploits the time sensitivity of LSTM and the ability of the nested network structure to extract more features, and can achieve effective processing of ultra-long signal IQ sequences collected from real wireless communication scenarios that are interfered by noise. Findings Experimental results show that our proposed model has higher recognition accuracy for five types of modulation signals, including amplitude modulation, frequency modulation, gaussian minimum shift keying, quadrature phase shift keying and differential quadrature phase shift keying, collected from real wireless communication scenarios. The overall classification accuracy of the proposed model for these signals can reach 73.11%, compared with 40.84% for the baseline model. Moreover, this model can also achieve high classification performance for analog signals with the same modulation method in the public data set HKDD_AMC36. Originality/value At present, although many AMR methods based on deep learning have been proposed, these works are based on the model’s classification results of various modulated signals in the AMR public data set to evaluate the signal recognition performance of the proposed method rather than collecting real modulated signals for identification in actual wireless communication scenarios. The methods proposed in these works cannot be directly applied to actual wireless communication scenarios. Therefore, this paper proposes a new AMR method, dedicated to the effective processing of the collected ultra-long signal IQ sequences that are interfered by noise.

Data-Driven Detection of Video Forgery for Secured Microcontrollers: Dataset and Method

The microcontrollers in a camera often capture videos at a low frame rate due to limited processing capability. To satisfy the requirement of high quality of service, low-frame-rate videos are often forged as the high-frame-rate ones by the Frame Rate Up-Conversion (FRUC) operation. Therefore, detecting the existence of FRUC has become a necessary job for secured microcontrollers. In this paper, we propose a data-driven detection to identify whether a video is forged by FRUC. The core of detection is the creation of a large-scale video dataset VifFRUC (Videos forged by FRUC). Various types of forged videos can continue to be added into VifFRUC, making the detection more universal and robust. To match with VifFRUC, we have also designed a neural network, which trains a number of Long Short Term Memory (LSTM) units in parallel to learn the data-driven detection. The parallel LSTM structure of network can continually adapt to the newly added FRUC methods in VifFRUC. Extensive experiments on VifFRUC demonstrate the effectiveness of data-driven detection for FRUC, resulting in the security improvement of microcontrollers.

A novel deep convolutional neural network and its application to fault diagnosis of the squirrel-cage asynchronous motor under noisy environment

The intelligent classification achieved through the utilization of deep learning networks, which possess the capability to automatically extract essential features from data, has garnered significant attention within the domain of fault diagnosis research. Nevertheless, within the industrial production process, the data collected inevitably suffers from noise contamination, thereby adversely affecting the network’s diagnostic results. To enhance the denoising prowess and mitigate the risks associated with overfitting in deep learning networks, this paper introduces the input gate structure of long short-term memory and an attention module into the convolutional neural network to propose a novel architecture known as the gate convolutional attention neural network (gate-CANN), which subsequently finds application in the domain of squirrel-cage asynchronous motor fault diagnosis. Firstly, the sensor-acquired time domain vibration undergo conversion into two-dimensional time–frequency images through the employment of continuous wavelet transform (CWT). Subsequently, the CWT images in two directions are put into gate-CANN for feature extraction, respectively. Finally, feature fusion and fault diagnosis are achieved in the end of network. To validate the effectiveness of the proposed method, it undergoes verification using the fault diagnosis testbed specific to squirrel cage asynchronous motors. The obtained results demonstrate that, in comparison to alternative diagnostic methods, the proposed approach exhibits superior capabilities in terms of noise resistance and generalization.

A Deep-Learning-Based Graph Neural Network-Long-Short-Term Memory Model for Reservoir Simulation and Optimization With Varying Well Controls

Summary A new deep-learning-based surrogate model is developed and applied for predicting dynamic oil rate and water rate with different well controls. The surrogate model is based on the graph neural networks (GNNs) and long-short-term memory (LSTM) techniques. The GNN models are used to characterize the connections of injector-producer pairs and producer-producer pairs, while an LSTM structure is developed to simulate the evolution of the constructed GNN models over time. In this way, we use geological attributes at wells and well controls with different times as input data. The oil rates and water rates at different times are generated. In this study, the GNN-LSTM surrogate model is applied to a high dimensional oil-gas-water field case with flow driven by 189 wells (i.e., 96 producers and 93 injectors) operating under time-varying control specifications. A total of 500 high-fidelity training simulations are performed in the offline stage, out of which 450 simulations are used for training the GNN-LSTM surrogate model, which takes about 150 minutes on an RTX2060 GPU. The trained model is then used to provide production forecasts under various well control scenarios, which are shown to be consistent with those obtained from the high-fidelity simulations (e.g., around 4.8% and 4.3% average relative errors for water production rates and oil production rates, respectively). The online computations from our GNN-LSTM model take about 0.3 seconds per run, achieving a speedup of over a factor of 1,000 relative to the high-fidelity simulations, which takes about 363 seconds per run. Overall, this model is shown to provide reliable and fast predictions of oil rates and water rates with a large level of perturbations in the well controls. Finally, the proposed GNN-LSTM model, in conjunction with the particle swarm optimization (PSO) technique, is applied to optimize the field oil production by varying the well control schedule of all injectors. Due to the significant speedup and high accuracy of the proposed surrogate model, the improved well-control strategies can be efficiently obtained.

Improved Equilibrium Optimizer for Short-Term Traffic Flow Prediction

Meta-heuristic algorithms have been widely used in deep learning. A hybrid algorithm EO-GWO is proposed to train the parameters of long short-term memory (LSTM), which greatly balances the abilities of exploration and exploitation. It utilizes the grey wolf optimizer (GWO) to further search the optimal solutions acquired by equilibrium optimizer (EO) and does not add extra evaluation of objective function. The short-term prediction of traffic flow has the characteristics of high non-linearity and uncertainty and has a strong correlation with time. This paper adopts the structure of LSTM and EO-GWO to implement the prediction, and the hyper parameters of the LSTM are optimized by EO-GWO to transcend the problems of backpropagation. Experiments show that the algorithm has achieved wonderful results in the accuracy and computation time of the three prediction models in the highway intersection.

Hyperparameter-Optimization-Inspired Long Short-Term Memory Network for Air Quality Grade Prediction

In the world, with the continuous development of modern society and the acceleration of urbanization, the problem of air pollution is becoming increasingly salient. Methods for predicting the air quality grade and determining the necessary governance are at present most urgent problems waiting to be solved by human beings. In recent years, more and more machine-learning-based methods have been used to solve the air quality prediction problem. However, the uncertainty of environmental changes and the difficulty of precisely predicting quantitative values seriously influence prediction results. In this paper, the proposed air pollutant quality grade prediction method based on a hyperparameter-optimization-inspired long short-term memory (LSTM) network provides two advantages. Firstly, the definition of air quality grade is introduced in the air quality prediction task, which turns a fitting problem into a classification problem and makes the complex problem simple; secondly, the hunter–prey optimization algorithm is used to optimize the hyperparameters of the LSTM structure to obtain the optimal network structure adaptively determined through the use of input data, which can include more generalization abilities. The experimental results from three real Xi’an air quality datasets display the effectiveness of the proposed method.

Predicting remaining useful life of a machine based on embedded attention parallel networks

Remaining useful life (RUL) prediction of mechanical equipment is the core task of the mechanical system of prediction and health management (PHM). To find data that is highly correlated with equipment degradation in enormous data, the mutual use of long short-term memory (LSTM) network and attention mechanism achieves good performance. However, the different combination of the two will greatly affect the efficiency of data feature extraction, and some disadvantages of the traditional sequential LSTM structure and the basic attention mechanism also limit the performance of the network. In this paper, we propose an embedded attention-based parallel network (EAPN) framework for RUL prediction on machine equipment. A new LSTM-based structure, ResLSTMa, is designed for feature extraction of data, and the Query branch of the attention mechanism is improved. Moreover, the ResLSTMa structure is embedded in the Query branch to improve the attention of the relevant features of the attention network, which is combined with the gate recurrent unit (GRU) basic feature extraction network of the parallel branch to achieve multi-level feature extraction and fusion to improve the performance of RUL prediction. The proposed EAPN is evaluated on the aircraft engine dataset and compared with some state-of-the-art prediction methods, and the experimental results demonstrate that our algorithm outperforms these state-of-the-art methods.

Research on Key Method of Cyber Security Situation Awareness Based on ResMLP and LSTM Network

Cyber security situation awareness,has become a hotpot of research. However, the existing cyber security situation awareness methods are difficult to extract high-order features from network traffic data. In this work, we present an improved cyber security situation awareness method based on ResMLP and LSTM network from a new perspective. Our work focus on cyber attack behavior analysis, that is a key research content of cyber security situation awareness. It introduces the Residual Multi-Layer Perceptrons in deep learning into the network structure of long-short term memory. It can effectively extract the spatial and temporal characteristics of network traffic data, reduce the computational complexity, and improve the accuracy of cyber security situation awareness. Firstly, we extract the spatial features using the ResMLP network. Secondly, we extract the temporal characteristics using the LSTM network. The architecture of the ResMLP network replaces the self-noticing layer with a linear interaction layer, and this design architecture allows the model to guarantee accurate cyber attack behavior analysis performance while balancing the computational cost of the model, which can improve the detection efficiency of the model. Considering that the network data are fed into the model in the form of time series after processing, the model incorporates LSTM networks to avoid the gradient problem while better bringing up the temporal characteristics in the data.The experimental results show that the proposed method can model the future cyber security situation in a network environment more accurately than other similar methods.

Predicting Output Responses of Nonlinear Dynamical Systems With Parametrized Inputs Using LSTM

Long Short-Term Memory (LSTM) has been more and more used to predict time evolution of dynamics for many problems, especially the fluid dynamics. Usually, it is applied to the latent space after dimension reduction of the full dynamical system by proper orthogonal decomposition (POD), autoencoder (AE) or convolutional autoencoder (CAE). In this work, we propose to directly apply LSTM to the data of the output without dimension reduction for output response prediction. The dimension of the output is usually small, and no dimension reduction is necessary, thus no accuracy loss is caused by dimension reduction. Based on the standard LSTM structure, we propose an LSTM network with modified activation functions which is shown to be much more robust for predicting periodic waveforms. We are especially interested in showing the efficiency of LSTM for predicting the output responses corresponding to time-vary input signals, which is rarely considered in the literature. However, such systems are of great interests in electrical engineering, mechanical engineering, and control engineering, etc. Numerical results for models from circuit simulation, neuron science and a electrochemical reaction have shown the efficiency of LSTM in predicting the dynamics of output responses.

Egocentric Action Recognition by Automatic Relation Modeling.

Egocentric videos, which record the daily activities of individuals from a first-person point of view, have attracted increasing attention during recent years because of their growing use in many popular applications, including life logging, health monitoring and virtual reality. As a fundamental problem in egocentric vision, one of the tasks of egocentric action recognition aims to recognize the actions of the camera wearers from egocentric videos. In egocentric action recognition, relation modeling is important, because the interactions between the camera wearer and the recorded persons or objects form complex relations in egocentric videos. However, only a few of existing methods model the relations between the camera wearer and the interacting persons for egocentric action recognition, and moreover they require prior knowledge or auxiliary data to localize the interacting persons. In this work, we consider modeling the relations in a weakly supervised manner, i.e., without using annotations or prior knowledge about the interacting persons or objects, for egocentric action recognition. We form a weakly supervised framework by unifying automatic interactor localization and explicit relation modeling for the purpose of automatic relation modeling. First, we learn to automatically localize the interactors, i.e., the body parts of the camera wearer and the persons or objects that the camera wearer interacts with, by learning a series of keypoints directly from video data to localize the action-relevant regions with only action labels and some constraints on these keypoints. Second, more importantly, to explicitly model the relations between the interactors, we develop an ego-relational LSTM (long short-term memory) network with several candidate connections to model the complex relations in egocentric videos, such as the temporal, interactive, and contextual relations. In particular, to reduce human efforts and manual interventions needed to construct an optimal ego-relational LSTM structure, we search for the optimal connections by employing a differentiable network architecture search mechanism, which automatically constructs the ego-relational LSTM network to explicitly model different relations for egocentric action recognition. We conduct extensive experiments on egocentric video datasets to illustrate the effectiveness of our method.

An Improved Self-Organizing Migration Algorithm for Short-Term Load Forecasting with LSTM Structure Optimization

Establishing an accurate and robust short-term load forecasting (STLF) model for a power system in safe operation and rational dispatching is both required and beneficial. Although deep long short-term memory (LSTM) networks have been widely used in load forecasting applications, it still has some problems to optimize, such as unstable network performance and long optimization time. This study proposes an adaptive step size self-organizing migration algorithm (AS-SOMA) to improve the predictive performance of LSTM. First, an optimization model for LSTM prediction is developed, which divides the LSTM structure seeking into two stages. One is the optimization of the number of hidden layer layers, and the other optimizes the number of neurons, time step, learning rate, epochs, and batch size. Then, a logistic chaotic mapping and an adaptive step size method were proposed to overcome slow convergence problems and stacking into local optimum of SOMA. Comparison experiments with SOMA, PSO, CPSO, LSOMA, and OSMA on test function sets show the advantages of the improved algorithm. Finally, the AS-SOMA-LSTM network prediction model is used to solve the STLF problem to verify the effectiveness of the proposed algorithm. Simulation experiments show that the AS-SOMA exhibits higher accuracy and convergence speed on the standard test function set and has strong prediction ability in STLF application with LSTM.

Predicting the Frequency of Marine Accidents by Navigators’ Watch Duty Time in South Korea Using LSTM

Despite the development of advanced technology, marine accidents have not decreased. To prevent marine accidents, it is necessary to predict accidents in advance. With the recent development of artificial intelligence (AI), AI technologies such as deep learning have been applied to create and analyze predictive models in various fields. The purpose of this study is to develop a model for predicting the frequency of marine accidents using a long-short term memory (LSTM) network. In this study, a prediction model was developed using marine accidents from 1981 to 2019, and the proposed model was evaluated by predicting the accidents in 2020. As a result, we found that marine accidents mainly occurred during the third officer’s duty time, representing that the accidents are highly related to the navigator’s experience. In addition, the proposed LSTM model performed reliably to predict the frequency of marine accidents with a small mean absolute percentage error (best MAPE: 0.059) that outperformed a traditional statistical method (i.e, ARIMA). This study could help us build LSTM structures for marine accident prediction and could be used as primary data to prevent the accidents by predicting the number of marine accidents by the navigator’s watch duty time.

A computationally efficient learning model to classify audio signal attributes

<p>The era of machine learning has opened up groundbreaking realities and opportunities in the field of medical diagnosis. However, it is also observed that faster and proper diagnosis of any diseases/medical conditions require proper analysis and classification of digital signal data. It indicates the proper identification of tumors in the brain. Brain magnetic resonance imaging (MRI) data has to be appropriately classified, and similarly, pulse signal analysis is required to evaluate the human heart operating condition. Several studies have used machine learning (ML) modeling to classify speech signals, but very few studies have explored the classification of audio signal attributes in the context of intelligent healthcare monitoring. The study thereby aims to introduce novel mathematical modeling to analyze and classify synthetic pulse audio signal attributes with cost-effective computation. The numerical modeling is composed of several functional blocks where deep neural network-based learning (DNNL) plays a crucial role during the training phase, and also it is further combined with a recurrent structure of long-short term memory (R-LSTM) feedback connections (FCs). The design approaches further experiment in a numerical computing environment in terms of accuracy and computational aspects. The classification outcome of the proposed approach shows that it attains approximately 85% accuracy, which is comparable to the baseline approaches and execution time.</p>

Multivariable sales prediction for filling stations via GA improved BiLSTM

Accurate sales prediction in filling stations is the basis to fill in the refined oil in time and avoid the out-of-stock as much as possible. Considering the defect of great “lag” in the general time series model, this paper summarizes the multiple factors that influence the oil sales and develops a multivariable long short-term memory (LSTM) neural network, with the hyper-parameters being improved by the genetic algorithm (GA). To further improve the prediction accuracy, the proposed LSTM neural network is generalized to bidirectional LSTM (BiLSTM), in which the impact of future factors on present sales can be taken into account by backward training. Finally, different LSTM structures and genetic algorithm parameters are tested to discuss their impact on prediction accuracy. Results demonstrated that genetic algorithm improved BiLSTM model is superior to extreme gradient boosting, ARIMA, and artificial neural network, having the highest accuracy of 89%.

Systematization of short-term forecasts of regional wave heights using a machine learning technique and long-term wave hindcast

Machine-learning techniques have been applied to wave forecasting to address the implementational and computational challenges of conventional numerical wave models. Most previous studies emphasized the accuracy of approaches based on in situ wave measurements, but such approaches are site-specific and lack the spatial and temporal fidelity required by regional wave climate forecasts. This study presents a novel systematization of forecasting regional wave heights at sufficient spatial resolution and coverage, using long-term wave hindcasts generated by numerical wave models as a training dataset for a long short-term memory (LSTM). This systematization not only ensures the accuracy of the model's forecasts but demonstrates the applicability of long-term wave hindcasts as input data. It involves thoroughly investigating the dependence of model accuracy on multi-task LSTM structures (training variables, input data record periods, and forecast lead times), and on regional wave climates (different wave sites and sea states). The resulting model performance is spatially heterogeneous in global wave forecasts, underlining the need for validation of wave forecasting models to global wave climates. This study reveals that machine-learning techniques that leverage validated long-term wave hindcasts can successfully forecast regional wave climates. As access to high-resolution regional wave hindcast data improves, accuracy will also increase.

A Deep Learning Model to Predict Evapotranspiration and Relative Humidity for Moisture Control in Tomato Greenhouses

The greenhouse industry achieves stable agricultural production worldwide. Various information and communication technology techniques to model and control the environment have been applied as data from environmental sensors and actuators in greenhouses are monitored in real time. The current study designed data-based, deep learning models for evapotranspiration (ET) and humidity in tomato greenhouses. Using time-series data and applying long short-term memory (LSTM) modeling, an ET prediction model was developed and validated in comparison with the Stanghellini model. Training with 20-day and testing with 3-day data resulted in RMSEs of 0.00317 and 0.00356 kgm−2 s−1, respectively. The standard error of prediction indicated errors of 5.76 and 6.45% in training and testing, respectively. Variables were used to produce a feature map using a two-dimensional convolution layer which was transferred to a subsequent layer and finally connected with the LSTM structure for modeling. The RMSE in humidity prediction using the test dataset was 2.87, indicating a performance better than conventional RNN-LSTM models. Irrigation plans and humidity control may be more accurately conducted in greenhouse cultivation using this model.