Long Short-term Memory Network Research Articles

Deep learning framework with Bayesian data imputation for modelling and forecasting groundwater levels

Although traditional physical models have been used to analyse groundwater systems, the emergence of novel machine learning models can improve the accuracy of the predictions. Deep learning has been prominent in environmental and climate change problems. In this paper, we present a framework for utilising deep learning models to predict groundwater levels based on nearby streamflow and rainfall data. We address the missing data problem using a Bayesian linear regression model within the deep learning framework. Our deep learning framework utilises models such as long-short term memory (LSTM) networks and convolutional neural networks (CNN) for multi-step ahead time series prediction. We examine the fluctuations in groundwater levels at various boreholes located near Middle Creek in New South Wales, Australia. We use the National Collaborative Research Infrastructure Strategy (NCRIS) groundwater database and utilise Bayesian linear regression to impute missing data. We investigate the accuracy of the selected models for individual and regional basins and univariate and multivariate strategies. Our results show that the LSTM-based regional model with multivariate strategy using rainfall data provided the best accuracy.

Indoor air quality prediction modeling for a naturally ventilated fitness building using RNN-LSTM artificial neural networks

Indoor air quality prediction modeling for a naturally ventilated fitness building using RNN-LSTM artificial neural networks

Face Manipulation Detection

The rise of digital manipulation techniques has led to the proliferation of deepfakes and other manipulated facial images, posing significant challenges to online trust and security. This paper proposes a novel deep learning model aimed at efficiently and effectively detecting face manipulations. The architecture combines the strengths of Efficient Net, a convolutional neural network (CNN) renowned for its accuracy and efficiency, with Long Short-Term Memory (LSTM) networks. Efficient Net is utilized for extracting high-level features from facial images, enabling the model to capture subtle inconsistencies that may indicate manipulation. These features serve as crucial inputs to the subsequent analysis performed by LSTM networks. LSTMs excel at capturing temporal dependencies within sequences of data, making them particularly well-suited for detecting manipulations in video sequences. By leveraging the power of CNNs for feature extraction and the sequential learning capabilities of LSTMs, the proposed hybrid approach aims to achieve superior performance in face manipulation detection. This combination allows the model to effectively analyse both spatial and temporal aspects of facial images, enhancing its ability to detect various forms of manipulation accurately

Short-Term Air Traffic Flow Prediction Based on CEEMD-LSTM of Bayesian Optimization and Differential Processing

With the rapid development of China’s civil aviation, the flow of air traffic in terminal areas is also increasing. Short-term air traffic flow prediction is of great significance for the accurate implementation of air traffic flow management. To enhance the accuracy of short-term air traffic flow prediction, this paper proposes a short-term air traffic flow prediction model based on complementary ensemble empirical mode decomposition (CEEMD) and long short-term memory (LSTM) of the Bayesian optimization algorithm and data differential processing. Initially, the model performs CEEMD on the short-term air traffic flow series. Subsequently, to improve prediction accuracy, the data differencing is employed to stabilize the time series. Finally, the smoothed sequences are, respectively, input into the LSTM network model optimized by the Bayesian optimization algorithm for prediction. After data reconstruction, the final short-term flow prediction result is obtained. The model proposed in this paper is verified by using the data from Shanghai Pudong International Airport. The results show that the evaluation indexes of the prediction accuracy and fitting degree of the model, RMSE (Root Mean Square Error), MAE (Mean Absolute Error), and R2 (Coefficient of Determination), are 0.336, 0.239, and 97.535%, respectively. Compared to other classical time-series prediction models, the prediction accuracy is greatly improved, which can provide a useful reference for short-term air traffic flow prediction.

An Efficient Water Quality Prediction and Assessment Method Based on the Improved Deep Belief Network—Long Short-Term Memory Model

The accuracy of water quality prediction and assessment has always been the focus of environmental departments. However, due to the high complexity of water systems, existing methods struggle to capture the future internal dynamic changes in water quality based on current data. In view of this, this paper proposes a data-driven approach to combine an improved deep belief network (DBN) and long short-term memory (LSTM) network model for water quality prediction and assessment, avoiding the complexity of constructing a model of the internal mechanism of water quality. Firstly, using Gaussian Restricted Boltzmann Machines (GRBMs) to construct a DBN, the model has a better ability to extract continuous data features compared to classical DBN. Secondly, the extracted time-series data features are input into the LSTM network to improve predicting accuracy. Finally, due to prediction errors, noise that randomly follows the Gaussian distribution is added to the assessment results based on the predicted values, and the probability of being at the current water quality level in the future is calculated through multiple evolutionary computations to complete the water quality assessment. Numerical experiments have shown that our proposed algorithm has a greater accuracy compared to classical algorithms in challenging scenarios.

On-demand design of dual-band electromagnetically induced transparency metamaterials based on improved convolutional neural network

Metamaterials (MMs) with dual electromagnetically induced transparency (EIT)-like windows possess more powerful capacity in Terahertz filtering than their single-EIT counterparts, but the design difficulty normally increases as the MM structure becomes more complicated. This work proposes a deep-learning method using a long short-term memory (LSTM) assisted convolutional neural network (LSTM-CNN) to simplify the design of dual-EIT MMs. Compared to the traditional CNN, the LSTM-CNN exhibits higher prediction accuracy for the geometric parameters of the MM according to the input spectrum, which can be ascribed to enhanced capability of extracting the spectral characteristics after adding the LSTM network. Further adding a feature-transforming network before the LSTM-CNN model to build a link between the MM performance and structure, a customizable MM design is realized. Two different dual-EIT metamaterials are quickly designed with this strategy, and the design accuracy has been fully demonstrated by simulations and experiments. The proposed deep learning method can achieve the rapid and efficient design of dual-EIT MMs, which offers a new pathway for the on-demand design of complex MM-based devices.

Predictive Modeling of Extreme Weather Forecasting Events: an LSTM Approach

For a variety of industries, including agriculture, water resource management, and flood forecasting, accurate rainfall prediction is crucial. The purpose of this research work is to improve rainfall forecast system by employing the Long Short-Term Memory (LSTM) based system. The LSTM utilized in the aforementioned study made predictions by using meteorological input variables such as temperature, humidity, and rainfall. Numerous elements affect the LSTM network's performance, such as the kind and volume of data, the suitability of the model architecture, and the tuning of hyperparameters. The dataset used for model training spans from January 2015 to December 2021 and includes rainfall data collected from the Zonal Agricultural Research Station (ZARS), Shenda Park, Kolhapur. Prior to model training, the input data undergoes rigorous preprocessing. This preprocessing involves data correction, achieved through moving averages, followed by feature scaling and normalization methods. These steps are crucial to align the dataset with the unique capabilities of the LSTM model. The total dataset has a R squared (R2) value 0.23517 and a mean squared error (MSE) value 92.1839, according to the simulated findings. These metrics affirm the robust performance of the LSTM model, suggesting a high probability of accurate rainfall predictions, particularly in non-linear and complex scenarios. Decision-makers in flood predictions, agriculture, and water resource management will find the knowledge gathered from this study to be useful. They emphasize how crucial it is to use cutting-edge techniques like LSTM to increase rainfall forecast accuracy and guide strategic planning in associated industries.

The application of artificial neural networks to forecast financial time series

Abstract The amount of information that is produced on a daily basis in the financial markets is vast and complex; consequently, the development of systems that simplify decision-making is an essential endeavor. In this article, several intelligent systems are proposed and tested to predict the closing price of the IBEX 35 index using more than ten years of historical data and five distinct architectures for neural networks. A multi-layer perceptron was the first step, followed by a simple recurrent neural network, a gated recurrent unit network and two long-short-term memory (LSTM) networks. The results of the analyses performed on these models have demonstrated a powerful capacity for prediction. Additionally, the findings of this research point to the fact that the application of intelligent systems can simplify the decision-making process in financial markets, which is a substantial advantage. Furthermore, by comparing the predicted outcome errors between the models, the LSTM presents the lowest error with a higher computational time in the training phase. The LSTM was able to accurately forecast the closing price of the day as well as the price for the following one and two days in advance. In conclusion, the empirical results demonstrated that these models could accurately predict financial data for trading purposes and that the application of intelligent systems, such as the LSTM network, represents a promising advancement in financial technology.

Implementation of a Long Short-Term Memory Neural Network-Based Algorithm for Dynamic Obstacle Avoidance.

Autonomous mobile robots are essential to the industry, and human-robot interactions are becoming more common nowadays. These interactions require that the robots navigate scenarios with static and dynamic obstacles in a safely manner, avoiding collisions. This paper presents a physical implementation of a method for dynamic obstacle avoidance using a long short-term memory (LSTM) neural network that obtains information from the mobile robot's LiDAR for it to be capable of navigating through scenarios with static and dynamic obstacles while avoiding collisions and reaching its goal. The model is implemented using a TurtleBot3 mobile robot within an OptiTrack motion capture (MoCap) system for obtaining its position at any given time. The user operates the robot through these scenarios, recording its LiDAR readings, target point, position inside the MoCap system, and its linear and angular velocities, all of which serve as the input for the LSTM network. The model is trained on data from multiple user-operated trajectories across five different scenarios, outputting the linear and angular velocities for the mobile robot. Physical experiments prove that the model is successful in allowing the mobile robot to reach the target point in each scenario while avoiding the dynamic obstacle, with a validation accuracy of 98.02%.

LSTM-based graph attention network for vehicle trajectory prediction

Vehicle Trajectory Prediction (VTP) is one of the key technologies for autonomous driving, which can improve the safety and collaboration of the autonomous driving system. The interaction behavior among vehicles in reality has an impact on VTP. However, many methods ignore the interaction among vehicles, which results in limited accuracy of prediction results. Therefore, we propose a Long Short-Term Memory (LSTM)-based Graph Attention Network (GAT) method for VTP, which encodes vehicle trajectory information with LSTM networks and represents vehicle interactions with GAT. Firstly, in order to capture the temporal relationship between positions and consider their influence, we use LSTM model to encode the position data. Meanwhile, to comprehensively model vehicle motion and use multidimensional feature representation, we employ another LSTM model to encode the motion data, including position, velocity and acceleration. Secondly, to learn distinct feature representation, we use one GAT module to process the LSTM position encoding features for capturing spatial relationships of position information. Another GAT module is employed to process the LSTM motion encoding features for fully considering multidimensional motion dynamics and spatial–temporal dependencies. Finally, the LSTM decoder receives all features and predicts the vehicle trajectory. The experimental results show that the proposed method demonstrates superior predictive performance by using the Next Generation Simulation (NGSIM) dataset.

A novel deep learning rainfall–runoff model based on Transformer combined with base flow separation

ABSTRACT Precise long-term runoff prediction holds crucial significance in water resource management. Although the long short-term memory (LSTM) model is widely adopted for long-term runoff prediction, it encounters challenges such as error accumulation and low computational efficiency. To address these challenges, we utilized a novel method to predict runoff based on a Transformer and the base flow separation approach (BS-Former) in the Ningxia section of the Yellow River Basin. To evaluate the effectiveness of the Transformer model and its responsiveness to the base flow separation technique, we constructed LSTM and artificial neural network (ANN) models as benchmarks for comparison. The results show that Transformer outperforms the other models in terms of predictive performance and that base flow separation significantly improves the performance of the Transformer model. Specifically, the performance of BS-Former in predicting runoff 7 days in advance is comparable to that of the BS-LSTM and BS-ANN models with lead times of 4 and 2 days, respectively. In general, the BS-Former model is a promising tool for long-term runoff prediction.

Real-time prediction of bottom-hole circulating temperature in geothermal wells using machine learning models

Drilling high-temperature geothermal wells presents technical and economic challenges. Real-time and precise estimation of bottom-hole circulating temperature (BHCT) during geothermal drilling is crucial for maximizing the working life of drill bits and temperature-limited bottom-hole assembly (BHA) components, thereby avoiding unplanned and unnecessary bit/BHA trips. This study introduces a novel machine learning (ML) model that accurately predicts BHCT in real time, enhancing proactive temperature management and mitigating thermally induced challenges during geothermal drilling operations.An integrated thermo-hydraulic model, validated using publicly available data from the Utah FORGE field, was used to generate a large dataset that simulates the transient BHCT under typical drilling conditions. This dataset was subsequently used to train and test the prediction capability of six ML models of the BHCT, including deep neural networks (DNN), support vector machines (SVR), random forests (RF), extreme gradient boosting regressor (XGBoost), ensemble (Stacked) algorithm, and long-short term memory (LSTM) networks. It was also used to generate an understanding of the non-linear relationship between the input features and the output variable (BHCT). Resultsdemonstrate the benefits of implementing the LSTM model for reliable prediction of the BHCT and capturing its full-scale transient behavior in geothermal wells. The LSTM model stands out for its excellent generalization capability, stability, and precision in real-time estimation of BHCT across various geothermal drilling conditions, including both stationary and dynamic scenarios. This is achieved without the high computational costs associated with complex numerical models, thus preventing temperature-related drilling problems. Among the other supervised ML models, the XGBoost model can still reliably simulate the BHCT with a mean absolute error (MAE) below 1 °C during geothermal drilling at constant drilling parameters, but will show overfitting when drilling parameters change.

Spatio-temporal prediction of total energy consumption in multiple regions using explainable deep neural network

Energy consumption forecasting is essential for energy system integration and management. However, existing studies mainly focus on temporal features of energy consumption, which neglects the spatial correlation of variables with time information. Capturing the spatio-temporal relationships helps to improve forecasting accuracy and further promote energy dispatch. To tackle this problem, an explainable Convolutional Neural Network-Long Short Term Memory forecasting model is employed to effectively predict the total energy consumption by capturing the spatial and temporal features of multivariate time series. In the model, the autoencoder is used to achieve the nonlinear dimensionality reduction and transfer the data to a low-dimensional space. Furthermore, a Convolutional Neural Network is used to extract more effective features from the decoded data, and long short-term memory is employed to identify the temporal dependencies between extracted features and total energy consumption. Shapley additive explanation is introduced to interpret the outputs of the black-box model. The superior performance of the proposed method with high accuracy and good adaptability is verified by the comparisons with conventional forecasting models. This method provides an insight into the regional energy consumption analyzing contributions of weather variables to energy consumption, which helps administers in understanding regional energy performance for enhancing energy efficiency.

ActiWise: Insight on Human Activity Recognition Using Deep Learning Approaches

In this study, we investigate the fusion of Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks for human activity recognition (HAR). By integrating hierarchical spatial features extracted by CNNs with LSTM networks' temporal modelling capabilities, our approach excels in discerning nuanced patterns from raw sensor data collected via wearable devices. Through rigorous experimentation and validation, our CNN+LSTM model demonstrates robust performance in accurately classifying a spectrum of human activities. This research advances HAR methodologies, shedding light on the synergistic interplay between spatial and temporal modelling in activity recognition, with implications across healthcare, sports analytics, and human-computer interaction domains. Index Terms- Human activity recognition, deep learning, Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM).

Predictive Analysis of COVID-19 Epidemic in Thailand: Evaluating Control Lockdown Measures using LSTM Networks

This study addresses the critical objective of evaluating the effectiveness of non-pharmaceutical lockdown measures implemented during COVID-19 outbreaks in Thailand. Assessing the outcome of these measures provides valuable insight that can inform and guide response to future outbreaks. Utilizing a closed-loop forecasting model built on Long Short-Term Memory (LSTM) networks, the research focuses on achieving precise daily forecasts of COVID-19 cases. The methodology involves optimizing hyperparameters through grid-search and incorporating training data from other countries that implemented similar measures. The LSTM, configured with an optimal number of hidden processing units, utilizes past lagged data of daily infected cases as predictors to generate multi-step-ahead predicted values, which are subsequently used as predictors in a recursive approach. As a result, the predicted cases closely align with measured data, facilitating the estimation of the effective reproduction number (Reff) to assess the performance of lockdown measures. The effectiveness of the lockdown measures is quantified at different time intervals: 51%, 41%, and 23% one day after implementation, increasing to 84%, 98%, and 34% after one week, and reaching 96%, 99%, and 73% at the endpoint of the first, second, and fourth waves of infection, respectively. Throughout these waves, the final Reff remains below 1, indicating ongoing but controllable COVID-19, demonstrating the efficacy of the implemented lockdown measures. It is noted that these results are based on specific LSTM model, as the effectiveness of lockdown measures may vary with alternative modeling approaches. Therefore, the findings should be interpreted in the context of this LSTM-framework.

Deep learning neural networks with input processing for vibration-based bearing fault diagnosis under imbalanced data conditions

Deep learning (DL) networks, such as convolutional neural networks (CNNs) and long short-term memory (LSTM), have gained popularity for bearing fault diagnosis utilizing raw vibration signals. However, their accuracy and stability are compromised when facing imbalanced real-world datasets. This research investigates the impact of imbalanced datasets and explores the potential of signal processing techniques on network inputs compared to the direct use of raw vibration signals. The DL techniques studied include LSTM, one-dimensional CNN, and two-dimensional (2D) CNN, and a novel hybrid 2DCNNLSTM algorithm, incorporating signal processing methods such as Fourier transform and continuous wavelet transform while maintaining nearly equal parameters and the same base architecture. The proposed hybrid 2DCNNLSTM algorithm combines the strengths of LSTM and CNN, allowing for improved bearing diagnosis by capturing both spatial and temporal information in vibration signals. The proposed 2DCNNLSTM algorithm also considers multi-channel input augmenting raw vibration signal, mean, and variance channels to extract meaningful features and enhance classification efficiency. The publicly available Case Western Reserve University benchmark-bearing test rig dataset with ten fault classes, the Paderborn University dataset with three fault classes, and NASA Centre for Intelligent Maintenance Systems bearing datasets with five fault classes are utilized to test the proposed deep learning networks’ accuracy, effectiveness, robustness, and stability. The studies reveal that the hybrid 2DCNNLSTM-based networks outperform both CNN and LSTM networks, even without input processing. Further, utilizing multi-channel input by augmenting the 2D raw signal with mean and variance value channels proves to be more efficient in handling imbalanced and complex datasets while employing a 2DCNNLSTM-based network.

Short-term photovoltaic energy generation for solar powered high efficiency irrigation systems using LSTM with Spatio-temporal attention mechanism

Solar irrigation systems should become more practical and efficient as technology advances. Automation and AI-based technologies can optimize solar energy use for irrigation while reducing environmental impacts and costs. These innovations have the potential to make agriculture more environmentally friendly and sustainable. Solar irrigation system implementation can be hampered by a lack of technical expertise in installation, operation, and maintenance. It must be technically and economically feasible to be practical and continuous. Due to weather and solar irradiation, photovoltaic power generation is difficult for high-efficiency irrigation systems. As a result, more precise photovoltaic output calculations could improve solar power systems. Customers should benefit from increased power plant versatility and high-quality electricity. As a result, an artificial intelligence-powered automated irrigation power-generation system may improve the existing efficiency. To predict high-efficiency irrigation system power outputs, this study proposed a spatial and temporal attention block-based long-short-term memory (LSTM) model. Using MSE, RMSE, and MAE, the results have been compared to pre-existing ML and a simple LSTM network. Moreover, it has been found that our model outperformed cutting-edge methods. MAPE was improved by 6–7% by increasing Look Back (LB) and Look Forward (LF). Future goals include adapting the technology for wind power production and improving the proposed model to harness customer behavior to improve forecasting accuracy.

Prediction of Parking Space Availability Using Improved MAT-LSTM Network

The prediction of parking space availability plays a crucial role in information systems providing parking guidance. However, controversy persists regarding the efficiency and accuracy of mainstream time series prediction methods, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). In this study, a comparison was made between a temporal convolutional network (TCN) based on CNNs and a long short-term memory (LSTM) network based on RNNs to determine an appropriate baseline for predicting parking space availability. Subsequently, a multi-head attention (MAT) mechanism was incorporated into an LSTM network, attempting to improve its accuracy. Experiments were conducted on three real and two synthetic datasets. The results indicated that the TCN achieved the fastest convergence, whereas the MAT-LSTM method provided the highest average accuracy, namely 0.0330 and 1.102 × 10−6, on the real and synthetic datasets, respectively. Furthermore, the improved MAT-LSTM model accomplished an increase of up to 48% in accuracy compared with the classic LSTM model. Consequently, we concluded that RNN-based networks are better suited for predicting long-time series. In particular, the MAT-LSTM method proposed in this study holds higher application value for predicting parking space availability with a higher accuracy.

Adaptive Kalman filter with LSTM network assistance for abnormal measurements

A classic state estimation method, the Kalman filter integrates prior information, system dynamics models, and measurement data to achieve posterior state estimations. However, measurements often encounter various unknown disturbances, leading to abnormal or inaccurate measurements and subsequently impacting the performance of Kalman filtering. In response to this challenge, this paper introduces a novel adaptive Kalman filter approach aided by posterior state estimations using the Long short-term memory (LSTM) networks. The proposed method begins by utilizing prior residuals to construct a chi-square distribution model, which facilitates the detection of abnormal measurements within data. Upon identifying abnormal measurements, an LSTM network is employed to generate alternative predictive measurements, replacing the original inaccurate measurement. This approach enhances the model’s capability to handle complex relationship and measurement uncertainties and improves filtering estimation accuracy. A noise covariance adjustment method is introduced in extreme cases where alternative predictive measurements alone are insufficient for filter requirements. This method mitigates the adverse effects of abnormal measurements on posterior estimation. Throughout the entire adaptive assistance process involving the LSTM network, deep learning maintains a stable structure of the filter while enhancing adaptability. This strategy ensures the filter’s resilience in dynamic environments with unknown factors by providing predicted measurements conforming to the chi-square distribution and corresponding measurement noise covariance. Ultimately, the efficacy of the proposed algorithm is validated through simulations and experiments involving vehicle positioning with inertial sensors.

Mixed Learning- and Model-Based Mass Estimation of Heavy Vehicles

This research utilized long short-term memory (LSTM) to oversee an RLS-based mass estimator based on longitudinal vehicle dynamics for heavy-duty vehicles (HDVs) instead of using the predefined rules. A multilayer LSTM network that analyzed parameters such as vehicle speed, longitudinal acceleration, engine torque, engine speed, and estimated mass from the RLS mass estimator was employed as the supervision method. The supervisory LSTM network was trained offline to recognize when the vehicle was operated so that the RLS estimator gave an estimate with the desired accuracy and the network was used as a reliability flag. High-fidelity simulation software was employed to collect data used to train and test the network. A threshold on the error percentage of the RLS mass estimator was used by the network to check the reliability of the algorithm. The preliminary findings indicate that the reliability of the RLS mass estimator could be predicted by using the LSTM network.