Long Short-term Memory Network Model Research Articles

Short-Term Air Traffic Flow Prediction Based on CEEMD-LSTM of Bayesian Optimization and Differential Processing

With the rapid development of China’s civil aviation, the flow of air traffic in terminal areas is also increasing. Short-term air traffic flow prediction is of great significance for the accurate implementation of air traffic flow management. To enhance the accuracy of short-term air traffic flow prediction, this paper proposes a short-term air traffic flow prediction model based on complementary ensemble empirical mode decomposition (CEEMD) and long short-term memory (LSTM) of the Bayesian optimization algorithm and data differential processing. Initially, the model performs CEEMD on the short-term air traffic flow series. Subsequently, to improve prediction accuracy, the data differencing is employed to stabilize the time series. Finally, the smoothed sequences are, respectively, input into the LSTM network model optimized by the Bayesian optimization algorithm for prediction. After data reconstruction, the final short-term flow prediction result is obtained. The model proposed in this paper is verified by using the data from Shanghai Pudong International Airport. The results show that the evaluation indexes of the prediction accuracy and fitting degree of the model, RMSE (Root Mean Square Error), MAE (Mean Absolute Error), and R2 (Coefficient of Determination), are 0.336, 0.239, and 97.535%, respectively. Compared to other classical time-series prediction models, the prediction accuracy is greatly improved, which can provide a useful reference for short-term air traffic flow prediction.

A novel deep learning rainfall–runoff model based on Transformer combined with base flow separation

ABSTRACT Precise long-term runoff prediction holds crucial significance in water resource management. Although the long short-term memory (LSTM) model is widely adopted for long-term runoff prediction, they encounter challenges such as error accumulation and low computational efficiency. To address these challenges, we utilized a novel method to predict runoff based on a Transformer and the base flow separation approach (BS-Former) in the Ningxia section of the Yellow River Basin. To evaluate the effectiveness of the Transformer model and its responsiveness to the base flow separation technique, we constructed LSTM and artificial neural network (ANN) models as benchmarks for comparison. The results show that Transformer outperforms the other models in terms of predictive performance and that base flow separation significantly improves the performance of the Transformer model. Specifically, the performance of BS-Former in predicting runoff 7 days in advance is comparable to that of the BS-LSTM and BS-ANN models with lead times of 4 and 2 days, respectively. In general, the BS-Former model is a promising tool for long-term runoff prediction.

Forecasting Model of Sustainable Financial Market Trend Based on Neural Network Algorithm

Recent advances in computer technology have resulted in the ongoing gathering of massive amounts of data and information. Because the financial market creates so much real-time data, including transaction records, we have a great potential to get important insights from analysing that data, especially in the banking industry. As a result, the goal of this work is to use the financial data that is now accessible to create a unique stock market prediction model. We employ the deep learning approach because of its exceptional capacity to learn from large datasets. This study proposes a hybrid approach that integrates the Archimedes optimisation algorithm (AOA) with a long short-term memory (LSTM) network. So far, heuristic-based trial and error has been extensively used to estimate the temporal window size and architectural components of long short-term memory networks. This paper investigates the temporal properties of financial market data by providing a systematic way to selecting the topology and time window size for the LSTM network. The experimental results demonstrate that the hybrid LSTM network and AOA model outperform the benchmark model.

Deep learning for predictive window operation modeling in open-plan offices

This study explores how the past, both short and long-term, affects the predictive window operation modeling in open-plan offices. To achieve this, the study proposes a deep learning method that uses long short-term memory (LSTM) artificial neural networks. The proposed model is an ensemble of LSTM networks that utilize relevant indoor environmental data from multiple sensory systems and weather data from a nearby weather station to predict the window state with different predictive horizons. The drawbacks of such networks are extensive training time and the lack of interpretability. Hence, Bayesian optimization is proposed for efficient hyperparameter tuning of the network. Furthermore, explainability through a model agnostic approach is utilized for feature importance ranking.The performance of the LSTM networks was compared to a dense feedforward neural network (DFNN) on an imbalanced dataset to evaluate the effect of the long-term past. Handling imbalanced window operation data was improved by employing class weight and early stopping, leading to a higher true positive rate, marginally better F1-score, and fewer network training epochs. The results indicate that a stateful bi-directional LSTM (Bi-LSTM) network models the data sequence more efficiently in aggregated comparison than other LSTM types and DFNN. However, the DFNN slightly outperformed LSTM networks in interval-by-interval comparison, with an F1-score of 0.88 when predicting the window state 10 min ahead, shifting to 0.82 for a 60-minute predictive horizon. Additionally, the study showed that the ensemble model performed better in terms of F1-score than the individual models.

A novel solar irradiance forecasting method based on multi-physical process of atmosphere optics and LSTM-BP model

Prediction of solar irradiance is crucial for minimizing energy costs and ensuring high power quality in electrical power grids that incorporate distributed solar photovoltaic generation. Traditional methods have focused mainly on time series prediction of irradiance or studies of the relationship between irradiance and environmental factors at a given moment. This paper presents a novel global horizontal irradiance forecasting method based on multi-physical process of atmosphere optics and LSTM-BP model. In the proposed method, cloud fraction, cloud albedo and aerosol optical thickness are selected as critical characteristic factors in the sunlight-atmosphere interaction. Long short-term memory (LSTM) is utilized to extract temporal features of the critical characteristic factors in various physical processes of atmospheric optics and predict their values. Subsequently, back propagation (BP) neural network is constructed to investigate the relationship between these characteristic factors and global horizontal irradiance under a given point in time. The global horizontal irradiance is predicted by combining the predicted values of critical characteristic factors and BP neural network for modelling irradiance decay process. To demonstrate the superior performance of the proposed model, the results obtained by the proposed method are compared with those of the LSTM network model for time series forecasting, in which six error evaluation indicators are used in the comparison. The proposed method for solar irradiance forecasting is verified in different time scales and under various weather conditions and shows better accuracy comparing with time series forecasting methods.

A Comparative Study of LSTM and Temporal Convolutional Network Models for Semisubmersible Platform Wave Runup Prediction

Abstract Wave runup prediction is necessary for offshore structure designs and early warnings. Data-driven methods based on machine learning have inspired reduced-order solutions for wave–structure interaction problems. This study provides the quantification of deep learning algorithms’ potential for wave runup prediction. Two prominent deep learning models were utilized to predict the wave runups along the fore column of semisubmersible under head seas. The long short-term memory (LSTM) and the temporal convolutional networks (TCNs) were comprehensively compared based on the datasets from a model test carried out in the deep ocean basin. The LSTM and TCN model structures were optimized to compare prediction accuracy and computational complexity reasonably. The results reveal that (1) both developed TCN and LSTM models had a satisfied prediction accuracy of over 90%. Their predictions were extended to 10 s into the future with accuracies over 80% and 45%, respectively. (2) With the noise-extended datasets, the LSTM model was robust to noises, while the TCN model showed better prediction performance on the extreme wave runup events. (3) The incident wave and dominant rotation provided the major information for wave runup prediction. TCN and LSTM models’ prediction accuracies were 91.5% and 89.3% based on the simplified input tensors composed of incident wave and pitch. The comparison showed the great potential of the TCN model to predict the nonlinear wave runup with less time and memory costs. The input tensors’ design and optimization based on physical understanding also play a significant role in the prediction performance.

Reduced-order modelling of unsteady cavitating flow around a Clark-Y hydrofoil

This paper proposes a novel approach that combines Proper Orthogonal Decomposition (POD) reduced-order system with Long Short-Term Memory (LSTM) neural network to predict flow velocity. Large Eddy Simulation (LES) is used to simulate the cavitating flow around a NACA66 hydrofoil. POD is adopted to reduce the dimensionality of the high-dimensional data. It was found that 66.81% of the flow field energy and dominant coherent structures can be captured with first eight POD modes. The LSTM network model was further used to predict the temporal data of the POD mode coefficients, and the error of the predicted coefficients was within an acceptable range. The reconstructed flow field agrees well with the real flow field and the cavitation development has also been well illustrated. This method provides a promising and efficient alternative for flow prediction and has potential for applications in fluid dynamics, aerospace engineering, and hydrodynamics.

Residual LSTM-based short duration forecasting of polarization current for effective assessment of transformers insulation

The empirical application of polarization and depolarization current (PDC) measurement of transformers facilitates the extraction of critical insulation-sensitive parameters. This technique, rooted in time-domain dielectric response analysis, forms the bedrock for parameterization and insulation modeling. However, the inherently time-consuming nature of polarization current measurements renders them susceptible to data corruption. This article explores deep-learning-based short-duration techniques for forecasting polarization current to address this limitation. By incorporating spatial shortcuts, the residual long short-term memory (LSTM) network facilitates the seamless propagation of spatial and temporal gradients. Furthermore, the relative forecasting assessment of the proposed residual LSTM model’s performance is made against traditional LSTM, attention LSTM, gated recurrent units (GRU), and convolutional neural network (CNN) models. Thus, optimal model selection strategies are evaluated based on their capability to capture extended dependencies and short-term information present in the data. In addition, the Monte Carlo dropout prediction is employed to estimate uncertainty in polarization current forecasts. The findings demonstrate that the proposed residual LSTM network model for polarization current forecasting yields the lowest error metrics and maintains prediction consistency over the testing duration. Thus, the proposed approach significantly reduces PDC measurement time, providing an effective means to develop proactive maintenance strategies for evaluating the insulation condition of transformers.

Comparative Evaluation of Deep Learning Techniques in Streamflow Monthly Prediction of the Zarrine River Basin

Predicting monthly streamflow is essential for hydrological analysis and water resource management. Recent advancements in deep learning, particularly long short-term memory (LSTM) and recurrent neural networks (RNN), exhibit extraordinary efficacy in streamflow forecasting. This study employs RNN and LSTM to construct data-driven streamflow forecasting models. Sensitivity analysis, utilizing the analysis of variance (ANOVA) method, also is crucial for model refinement and identification of critical variables. This study covers monthly streamflow data from 1979 to 2014, employing five distinct model structures to ascertain the most optimal configuration. Application of the models to the Zarrine River basin in northwest Iran, a major sub-basin of Lake Urmia, demonstrates the superior accuracy of the RNN algorithm over LSTM. At the outlet of the basin, quantitative evaluations demonstrate that the RNN model outperforms the LSTM model across all model structures. The S3 model, characterized by its inclusion of all input variable values and a four-month delay, exhibits notably exceptional performance in this aspect. The accuracy measures applicable in this particular context were RMSE (22.8), R2 (0.84), and NSE (0.8). This study highlights the Zarrine River’s substantial impact on variations in Lake Urmia’s water level. Furthermore, the ANOVA method demonstrates exceptional performance in discerning the relevance of input factors. ANOVA underscores the key role of station streamflow, upstream station streamflow, and maximum temperature in influencing the model’s output. Notably, the RNN model, surpassing LSTM and traditional artificial neural network (ANN) models, excels in accurately mimicking rainfall–runoff processes. This emphasizes the potential of RNN networks to filter redundant information, distinguishing them as valuable tools in monthly streamflow forecasting.

Research on Malware Detection Technology for Mobile Terminals Based on API Call Sequence

With the development of the Internet, the types and quantities of malware have grown rapidly, and how to identify unknown malware is becoming a new challenge. The traditional malware detection method based on fixed features is becoming more and more difficult. In order to improve detection accuracy and efficiency for mobile terminals, this paper proposed a malware detection method for mobile terminals based on application programming interface (API) call sequence, which was characterized by the API call sequence and used a series of feature preprocessing techniques to remove redundant processing of the API call sequence. Finally, the recurrent neural network method (RNN) was used to build the model and perform detection and verification. Furthermore, this paper constructed a malware detection model based on a two-way recurrent neural network and used the two-way long short-term memory network model (LSTM) to train the data set containing 5986 malware samples and 5065 benign software samples to obtain the final detection model and its parameters. Finally, the feature vector of the APK file to be detected was passed into the model and obtained the detection results. The experimental results indicated that the detection accuracy of this method can reach 93.68%.

Stabilization temperature prediction in carbon fiber production using empirical mode decomposition and long short-term memory network

Carbon fiber holds significant promise as a sustainable material with diverse applications. The production of carbon fiber involves a critical process known as oxidative stabilization. Optimizing this process necessitates accurate prediction of the stabilization temperature. However, predicting the stabilization temperature based on easily available heater power data encounters formidable challenges due to factors like nonlinearity, non-uniformity, time delay and time-varying effect. To address these issues, this study proposed an innovative method for predicting the stabilization temperature using empirical mode decomposition (EMD) and long short-term memory (LSTM) network. The distinctive aspect of this proposed method lies in the utilization of two EMD-based reconstruction approaches to preprocess the heater power training and testing data. This helps mitigate the nonlinearity between stabilization temperature and heater power output during the machine learning network training phase and alleviate the time-varying effect of heater power testing data relative to the training data during the network testing phase. Considering the hysteresis effect of heater power output on stabilization temperature, an LSTM network was integrated for accurate stabilization temperature prediction. The proposed EMD-LSTM network model was implemented for stabilization temperature predictions in six oxidation ovens of a carbon fiber production line. The average metrics for predicted results using this method are notably superior, with values of 0.0153 °C for root-mean-square error (RMSE), 0.3711 for relative RMSE (rRMSE), 0.0117 °C for mean absolute error (MAE), and 0.2838 for relative MAE (rMAE), surpassing the results obtained from the LSTM network model trained and tested using unreconstructed data. The comparative analysis further demonstrates the advantageous performance of the proposed EMD-LSTM network model compared to models based on five widely used machine learning networks. This study offers valuable insights for optimizing the oxidative stabilization process in carbon fiber production, thereby reducing waste heat and exhaust emissions, and promoting cleaner carbon fiber production.

PTS-LSTM: Temperature prediction for fused filament fabrication using thermal image time series

In-situ thermography for Fuse Filament Fabrication (FFF) processes reveals the dynamic thermal behavior during printing. The data collected are thermal image time series. Their infrared (IR) intensity is visual evidence of heat-affected zone (HAZ) temperatures, which can be leveraged to train deep learning models, e.g., Long Short-Term Memory (LSTM), for real-time temperature prediction and process monitoring. Nonetheless, the data collection method and printing path may pose challenges for data modeling. Typically, the IR camera has a fixed position while the HAZ moves per the predetermined printing path. Consequently, the HAZ shifts in images and the features extracted from these thermal images show a “periodic” behavior over time. Such periodic patterns do not reflect any useful information about HAZ temperatures. Instead, they are noise hiding and interrupting the true temperature information, thus must be removed before using the data to train an LSTM model for temperature prediction. This study integrates a time series model, i.e., ARIMA, with Stacked LSTM to build a pTS-LSTM model that eliminates noisy patterns and predicts temperatures during FFF printing. The case study results show the outperformance of pTS-LSTM over conventional LSTM and classic Recurrent Neural Network models. pTS-LSTM is demonstrated to be promising for in-situ process monitoring with low-quality thermal images. In FFF practices, pTS-LSTM will be a preferred option over the commonly used deep learning models for thermal-image-based temperature prediction.

Short-term power load comparison based on time series and neural networks considering multiple features

In order to achieve a certain balance between power supply and production demand, as well as to ensure the operation of social machinery and multiple markets, accurate power load forecasting is indispensable. A load prediction method based on the Convolutional Neural Network (CNN) - the Long Short-Term Memory (LSTM) and the CNN-gated Recurrent Unit (GRU) is proposed. The multi-feature load forecasting is constructed by the date factor, the weather factor, the load factor and the electricity price factor. The advantage of CNN in feature extraction of the data set is used to establish a high-dimensional relationship with load, optimize the input LSTM and GRU network model, train each group of the neural network model, and output load prediction value. Combining the load data of a certain area with its meteorological factors, the CNN-LSTM and CNN-GRU methods are tested and compared with the single-network models of Back Propagation (BP), LSTM and GRU. By comparing it with other popular algorithms, it is proved that the presented model has superior computational efficiency and prediction accuracy.

Aircraft parameter estimation using a stacked long short-term memory network and Levenberg-Marquardt method

To effectively estimate the unknown aerodynamic parameters from the aircraft’s flight data, this paper proposes a novel aerodynamic parameter estimation method incorporating a stacked Long Short-Term Memory (LSTM) network model and the Levenberg-Marquardt (LM) method. The stacked LSTM network model was designed to realize the aircraft dynamics modeling by utilizing a frame of nonlinear functional mapping based entirely on the measured input–output data of the aircraft system without requiring explicit postulation of the dynamics. The LM method combines the already-trained LSTM network model to optimize the unknown aerodynamic parameters. The proposed method is applied by using the real flight data, generated by ATTAS aircraft and a bio-inspired morphing Unmanned Aerial Vehicle (UAV). The investigation reveals that for the two different flight data, the designed stacked LSTM network structure can maintain the efficacy of the network prediction capability only by appropriately adjusting the dropout rates of its hidden layers without changing other network parameters (i.e., the initial weights, initial biases, number of hidden cells, time-steps, learning rate, and number of training iterations). Besides, the proposed method’s effectiveness and potential are demonstrated by comparing the estimated results of the ATTAS aircraft or the bio-inspired morphing UAV with the corresponding reference values or wind-tunnel results.

Predicting and explaining karst spring dissolved oxygen using interpretable deep learning approach

AbstractDissolved oxygen (DO) is one of the most important indicators of water quality and an essential measure for the aquatic organisms and the local ecosystem. DO concentrations in karst spring flow involves complex hydrological processes because of the heterogeneous nature of the karst system. In this study, an interpretable and explainable model that integrates the shapley additive explanations (SHAP) algorithm with the long short‐term memory network model (LSTM) is proposed to evaluate the contributions of karst spring discharge, precipitation, water temperature, and specific conductance to DO concentrations in karst spring flow. The hybrid model can predict the temporal fluctuations of DO levels and provide a robust characterization of DO behaviours. To demonstrate the applicability of the proposed model, we adopted the hydrological and meteorological data from Barton Springs, situated within a highly karstified segment of the Edwards aquifer characterized by extensive fractures and openings. The optimal prediction performance is achieved with a 14‐day time step, which is considered as the effective response time between various hydrological processes and DO concentrations at Barton Springs. It reveals that the influence of karst spring discharge, precipitation, water temperature, and specific conductance in previous 14 days collectively contribute to the current DO concentration in karst spring flow. The SHAP values of input features provide both local and global explanations, demonstrating the magnitude and the direction of each feature's impact on DO levels in karst spring flow. In descending order, the contributions of various hydrological processes to DO are ranked as follows: precipitation, karst spring discharge, temperature, and specific conductance. Precipitation and discharge exhibit positive SHAP values, indicating that increases in these hydrological processes contribute to higher DO levels in the karst flow. Water temperature and specific conductance have negative SHAP values, suggesting that higher water temperature and specific conductance will lead to decreased DO levels in the karst flow.

Application of recurrent neural networks for modeling and control of a quadruple‐tank system

AbstractThe quadruple tank (QT) system consists of four interacting tanks and can switch between the minimum and non‐minimum phase behavior with changes in the positions of pump valves and is considered a benchmark control problem. In the present study, long‐short term memory (LSTM), a type of recurrent neural networks (RNN) is designed for the benchmark QT system based on the model‐based control framework. Random input–output sequences are generated from the white box model of the QT system to train an LSTM network model. The LSTM network is tuned by adjusting its hyperparameters such as the number of hidden layers, hidden units, and epochs to minimize the prediction error on the test data. The trained model is cross validated both during and after training to avoid overfitting. Once a reasonably reliable model is obtained, another LSTM network is trained for use as a controller. The network architecture is constantly modified till the controller is able to track the test setpoints with minimum error. This procedure is repeated with a gated recurrent unit (GRU) network and the servo and regulatory response of both the network models and controller are evaluated in terms of standard performance measure namely root mean square error (RMSE), integral square error (ISE), and control effort (CE). It is observed that the controller designed based on RNN performs better than a conventional centralized controller.

Construction and Application of LSTM-Based Prediction Model for Tunnel Surrounding Rock Deformation

Tunnel surrounding rock deformation is a significant issue in tunnel construction and maintenance and has garnered attention from both domestic and international scholars. Traditional methods of predicting tunnel surrounding rock deformation involve fitting monitoring and measuring data, which is a laborious and resource-intensive process with low accuracy when predicting data with significant fluctuations. A deep learning approach can improve monitoring efficiency and accuracy while reducing labor costs. In this study, taking an actual tunnel project as an example, a long short-term memory (LSTM) network model was constructed based on the recurrent neural network algorithm with deep learning to model and analyze the tunnel monitoring and measurement data, and the model was used to analyze and predict the vault settlement of the tunnel. LSTM is a type of artificial neural network architecture that is commonly used in deep learning applications for sequence prediction tasks, such as natural language processing, speech recognition, and time-series forecasting. In predicting data with smaller fluctuations, the maximum error is 4.76 mm, the minimum error is 0.03 mm, the root mean square error is 2.64, and the coefficient of determination is 0.98. In predicting data with larger fluctuations, the maximum error is 8.32 mm, the minimum error is 0.13 mm, the root mean square error is 4.42, and the coefficient of determination is 0.88. The average error of the LSTM network model is 2.16 mm. With the growth of the prediction period, the prediction results become more and more stable and closer to the actual vault settlement, which provides a reliable reference for introducing the LSTM prediction method with deep learning to tunnel construction and promoting tunnel construction safety.

Ship Anomalous Behavior Detection Using Clustering and Deep Recurrent Neural Network

In this study, we propose a real-time ship anomaly detection method driven by Automatic Identification System (AIS) data. The method uses ship trajectory clustering classes as a normal model and a deep learning algorithm as an anomaly detection tool. The method is divided into three main steps: (1) quality maintenance of the original AIS data, (2) extraction of normal ship trajectory clusters using Density-Based Spatial Clustering of Applications with Noise (DBSCAN), in which a segmented improved Dynamic Time Warping (DTW) algorithm is used to measure the degree of trajectory similarity, (3) the clustering results are used as a normative model to train a Bi-directional Gated Recurrent Unit (BiGRU) recurrent neural network, which is used as a trajectory predictor to achieve real-time ship anomaly detection. Experiments were conducted using real AIS data from the port of Tianjin, China. The experimental results are manifold. Firstly, the data pre-processing process effectively improves the quality of raw AIS data. Secondly, the ship trajectory clustering model can accurately classify the traffic flow of different modes in the sea area. Moreover, the trajectory prediction result of the BiGRU model has the smallest error with the actual ship trajectory and has a better trajectory prediction performance compared with the Long Short-Term Memory Network model (LSTM) and Gated Recurrent Unit (GRU). In the final anomaly detection experiment, the detection accuracy and timeliness of the BiGRU model are also higher than LSTM and GRU. Therefore, the proposed method can achieve effective and timely detection of ship anomalous behaviors in terms of position, heading and speed during ship navigation, which provides insight to enhance the intelligence of marine traffic supervision and improve the safety of marine navigation.

End-to-end learning with interpretation on electrohysterography data to predict preterm birth

Prediction of preterm birth is a difficult task for clinicians. By examining an electrohysterogram, electrical activity of the uterus that can lead to preterm birth can be detected. Since signals associated with uterine activity are difficult to interpret for clinicians without a background in signal processing, machine learning may be a viable solution. We are the first to employ Deep Learning models, a long-short term memory and temporal convolutional network model, on electrohysterography data using the Term–Preterm Electrohysterogram database. We show that end-to-end learning achieves an AUC score of 0.58, which is comparable to machine learning models that use handcrafted features. Moreover, we evaluate the effect of adding clinical data to the model and conclude that adding the available clinical data to electrohysterography data does not result in a gain in performance. Also, we propose an interpretability framework for time series classification that is well-suited to use in case of limited data, as opposed to existing methods that require large amounts of data. Clinicians with extensive work experience as gynaecologist used our framework to provide insights on how to link our results to clinical practice and stress that in order to decrease the number of false positives, a dataset with patients at high risk of preterm birth should be collected. All code is made publicly available.

ResNext50 based convolution neural network-long short term memory model for plant disease classification

ResNext50 based convolution neural network-long short term memory model for plant disease classification