Simple Long Short-term Memory Research Articles

Competition of systematically optimized deep neural networks for the estimation of tensile behavior of aluminum-magnesium alloy

Aluminum-magnesium (Al-Mg) alloys are prevalently employed within the aerospace sector. This research engaged a suite of deep learning approaches, encompassing the Artificial Neural Network (ANN), Gated Recurrent Unit (GRU) networks, Long-Short Term Memory (LSTM), and simple Recurrent Neural Network (RNN) to evaluate their predictive efficacy regarding the tensile strength and stiffness of Al-Mg alloys obtained from molecular dynamics simulation. The Taguchi method was initially applied to refine the architecture of each deep neural network (DNN), followed by a comparative analysis of their optimized configurations. The findings of this investigation revealed that the refined simple RNN and LSTM models exhibited superior predictive accuracy for estimating the strength and stiffness of the alloy, respectively. Moreover, the study elucidated that DNNs equipped with memory capabilities outstripped traditional ANNs in forecasting the tensile properties of Al-Mg alloys.

Short-term photovoltaic energy generation for solar powered high efficiency irrigation systems using LSTM with Spatio-temporal attention mechanism

Solar irrigation systems should become more practical and efficient as technology advances. Automation and AI-based technologies can optimize solar energy use for irrigation while reducing environmental impacts and costs. These innovations have the potential to make agriculture more environmentally friendly and sustainable. Solar irrigation system implementation can be hampered by a lack of technical expertise in installation, operation, and maintenance. It must be technically and economically feasible to be practical and continuous. Due to weather and solar irradiation, photovoltaic power generation is difficult for high-efficiency irrigation systems. As a result, more precise photovoltaic output calculations could improve solar power systems. Customers should benefit from increased power plant versatility and high-quality electricity. As a result, an artificial intelligence-powered automated irrigation power-generation system may improve the existing efficiency. To predict high-efficiency irrigation system power outputs, this study proposed a spatial and temporal attention block-based long-short-term memory (LSTM) model. Using MSE, RMSE, and MAE, the results have been compared to pre-existing ML and a simple LSTM network. Moreover, it has been found that our model outperformed cutting-edge methods. MAPE was improved by 6–7% by increasing Look Back (LB) and Look Forward (LF). Future goals include adapting the technology for wind power production and improving the proposed model to harness customer behavior to improve forecasting accuracy.

COMPARATIVE STUDY OF TRANSFORMER- AND LSTM-BASED MACHINE LEARNING METHODS FOR TRANSIENT THERMAL FIELD RECONSTRUCTION

Solution reconstruction from limited number of measurements is useful in many areas of heat transfer applications. Unlike the standard problems, such reconstruction problems are ill-posed; thus, the nonuniqueness of solution and inherent instability severely complicates the modeling process. Consequently, more conventional inverse analysis methods to reconstruct solutions remain computationally intractable and lacking sufficient flexibility, especially when dealing with time-dependent problems. Aided by powerful graphical processing units (GPUs), machine learning (ML) methods rose in popularity due to their flexibility and ability to efficiently process large amounts of data. In recent years, the transformer-based ML models have gained recognition for their remarkable performance in natural language processing (NLP) tasks as well as time-series analysis, overshadowing the performance of the ML models conventionally used for sequence processing, such as the long short-term memory (LSTM) models. These achievements make transformer-based models seemingly ideal candidates for reconstructing full solutions from a few measurements. This article compares the performance of these novel transformer-based models with a simple LSTM model in reconstructing transient one-dimensional (1-D) and two-dimensional (2-D) thermal fields using sparse spatial measurements. Counterintuitively, the simple LSTM model achieves higher or comparable prediction accuracy compared to the complex transformer-based models while also exhibiting shorter or comparable training times, which may render transformerbased models a suboptimal choice for reconstructing transient solutions. Instead, more traditional sequence processing ML models, such as LSTM, might be preferred for this purpose.

Long Short-Term Memory-based Models for Sleep Quality Prediction from Wearable Device Time Series Data

Several studies suggest that sleep quality is associated with physical activities. Moreover, deep sleep time can be used to determine the sleep quality of an individual. In this work, we aim to find the association between physical activities and deep sleep time by modeling the time series data such as heart rate and a number of steps captured from a commercial wearable device. Our previous study demonstrates that deep learning-based time series modeling is well suited for our problem since the temporal patterns in the two physical parameters need to be captured to obtain more accurate results. We first preprocess our series data to have a time-step size of 10 minutes. To improve our previous effort in this modeling, we compare four different variants of Long Short-Term Memory (LSTM)-based models, ranging from single input to dual input models. Our result shows that the simple stacked LSTM model performs better for our data because the remaining models suffer from overfitting due to a larger number of the trained parameters.

A Privacy and Energy-Aware Federated Framework for Human Activity Recognition

Human activity recognition (HAR) using wearable sensors enables continuous monitoring for healthcare applications. However, the conventional centralised training of deep learning models on sensor data poses challenges related to privacy, communication costs, and on-device efficiency. This paper proposes a federated learning framework integrating spiking neural networks (SNNs) with long short-term memory (LSTM) networks for energy-efficient and privacy-preserving HAR. The hybrid spiking-LSTM (S-LSTM) model synergistically combines the event-driven efficiency of SNNs and the sequence modelling capability of LSTMs. The model is trained using surrogate gradient learning and backpropagation through time, enabling fully supervised end-to-end learning. Extensive evaluations of two public datasets demonstrate that the proposed approach outperforms LSTM, CNN, and S-CNN models in accuracy and energy efficiency. For instance, the proposed S-LSTM achieved an accuracy of 97.36% and 89.69% for indoor and outdoor scenarios, respectively. Furthermore, the results also showed a significant improvement in energy efficiency of 32.30%, compared to simple LSTM. Additionally, we highlight the significance of personalisation in HAR, where fine-tuning with local data enhances model accuracy by up to 9% for individual users.

Prediction of Wastewater Treatment Plant Effluent Water Quality Using Recurrent Neural Network (RNN) Models

Artificial Intelligence (AI) has recently emerged as a powerful tool with versatile applications spanning various domains. AI replicates human intelligence processes through machinery and computer systems, finding utility in expert systems, image and speech recognition, machine vision, and natural language processing (NLP). One notable area with limited exploration pertains to using deep learning models, specifically Recurrent Neural Networks (RNNs), for predicting water quality in wastewater treatment plants (WWTPs). RNNs are purpose-built for handling sequential data, featuring a feedback mechanism. However, standard RNNs may exhibit limitations in accommodating both short-term and long-term dependencies when addressing intricate time series problems. The solution to this challenge lies in adopting Long Short-Term Memory (LSTM) cells, known for their inherent memory management through a ‘forget gate’ mechanism. In general, LSTM architecture demonstrates superior performance. WWTP data represent a historical series influenced by fluctuating environmental conditions. This study employs simple RNNs and LSTM architecture to construct prediction models for effluent parameters, systematically assessing their performance through various training data scenarios and model architectures. The primary objective was to determine the most suitable WWTP dataset model. The study revealed that an epoch setting of 50 and a batch size of 100 yielded the lowest training time and root mean square error (RMSE) values for both RNN and LSTM models. Furthermore, when these models are applied to predict effluent parameters, they exhibit precise RMSE values for all parameters. The study results can be applied to detect potential upsets in WWTP operations.

Dynamic personalized human body energy expenditure: Prediction using time series forecasting LSTM models

Dynamic human energy expenditure (EE) consists of energy costs for resting metabolism, food digestion, physical activity, and thermoregulation. Currently, multiple models predict EE mainly concerning physical activity, thus, discarding other factors contributing to the dynamic variation of EE. This paper aimed to demonstrate that (i) a dynamic human body EE prediction requires the time series approach, (ii) personalization of input features and models can outperform the generalized approach. To achieve these objectives, data were collected from 3 sets of experiments with 6 test subjects wearing multiple sensors. The analysis of features' importance showed that the selection of features varies for activity-dominated and non-activity-dominated cases and also varies between individuals. Long-Short Term Memory (LSTM) networks were used to develop personalized models such as a simple LSTM, a convolutional LSTM (CNN-LSTM), and also an ensemble model combining CNN-LSTM with a Gradient Boosting algorithm (LSTM-LGBM). A personalized autoregressive linear model and a generalized approach of the LSTM-LGBM method were also developed to have a base of comparison. The results show that the personalized models provide good prediction accuracy, with the mean absolute percentage error (MAPE) mostly lying in the range of 5–15 %. The CNN-LSTM outperforms a simple LSTM model by 3–5 % in MAPE values, and the ensemble model outperforms the by 5–8 % the simple LSTM. The personalized modeling approach with LSTM has shown the potential to improve the prediction accuracy of dynamic EE and capture the non-activity-related effects such as thermoregulation and postprandial thermogenesis.

Applying and dissecting LSTM neural networks and regularized learning for dynamic inferential modeling

Deep learning models such as the long short-term memory (LSTM) network have been applied for dynamic inferential modeling. However, many studies apply LSTM as a black-box approach without examining the necessity and usefulness of the internal LSTM gates for inferential modeling. In this paper, we use LSTM as a state space realization and compare it with linear state space modeling and statistical learning methods, including N4SID, partial least squares, the Lasso, and support vector regression. Two case studies on an industrial 660 MW boiler and a debutanizer column process indicate that LSTM underperforms all other methods. LSTM is shown to be capable of outperforming linear methods for a simulated reactor process with severely excited nonlinearity in the data. By dissecting the sub-components of a simple LSTM model, the effectiveness of the LSTM gates and nonlinear activation functions is scrutinized.

RaKShA: A Trusted Explainable LSTM Model to Classify Fraud Patterns on Credit Card Transactions

Credit card (CC) fraud has been a persistent problem and has affected financial organizations. Traditional machine learning (ML) algorithms are ineffective owing to the increased attack space, and techniques such as long short-term memory (LSTM) have shown promising results in detecting CC fraud patterns. However, owing to the black box nature of the LSTM model, the decision-making process could be improved. Thus, in this paper, we propose a scheme, RaKShA, which presents explainable artificial intelligence (XAI) to help understand and interpret the behavior of black box models. XAI is formally used to interpret these black box models; however, we used XAI to extract essential features from the CC fraud dataset, consequently improving the performance of the LSTM model. The XAI was integrated with LSTM to form an explainable LSTM (X-LSTM) model. The proposed approach takes preprocessed data and feeds it to the XAI model, which computes the variable importance plot for the dataset, which simplifies the feature selection. Then, the data are presented to the LSTM model, and the output classification is stored in a smart contract (SC), ensuring no tampering with the results. The final data are stored on the blockchain (BC), which forms trusted and chronological ledger entries. We have considered two open-source CC datasets. We obtain an accuracy of 99.8% with our proposed X-LSTM model over 50 epochs compared to 85% without XAI (simple LSTM model). We present the gas fee requirements, IPFS bandwidth, and the fraud detection contract specification in blockchain metrics. The proposed results indicate the practical viability of our scheme in real-financial CC spending and lending setups.

Using Simple LSTM Models to Evaluate Effects of a River Restoration on Groundwater in Kushiro Wetland, Hokkaido, Japan

Wetland ecosystems with proper functioning provide various ecosystem services. Therefore, their conservation and restoration are of fundamental importance for sustainable development. This study used a deep learning model for groundwater level prediction to evaluate a wetland restoration project implemented in the Kushiro Wetland in Japan. The Kushiro Wetland had been degraded due to river improvement work. However, in 2010, a wetland restoration project was carried out to restore the meandering river channel, and a decade has passed since its completion. In this study, the wetland restoration project was evaluated by comparing the response of the groundwater level using a model that reproduced physical conditions with different characteristics before and after the restoration. At first, a deep learning model was created to predict groundwater levels pre- and post-restoration of a meandering river channel using observation data. Long short-term memory (LSTM) was used as the deep learning model. The most important aspect of this study was that LSTM was trained for each of the pre- and post-restoration periods when the hydrological and geological characteristics changed due to the river channel’s restoration. The trained LSTM model achieved high performance with a prediction error of the groundwater levels within 0.162 m at all observation points. Next, the LSTM models trained with the observation data of the post-restoration period were applied to evaluate the effectiveness of the meandering channel restoration. The results indicated that the meandering channel restoration improved hydrological processes in groundwater levels, i.e., their rainfall response and average groundwater water levels. Furthermore, the variable importance analysis of the explanatory variables in the LSTM model showed that river discharge and precipitation significantly contributed to groundwater level recovery in the Kushiro Wetland. These results indicated that the LSTM model could learn the differences in hydrological and geological characteristics’ changes due to channel restoration to groundwater levels. Furthermore, LSTM is a data-driven deep learning model, and by learning hydrological and geological conditions to identify factors that may affect groundwater levels, LSTM has the potential to become a powerful analysis method that can be used for environmental management and conservation issues.

DEEP LEARNING MODELS FOR NATURAL GAS DEMAND FORECASTING: A COMPARATIVE STUDY OF MLP, CNN, AND LSTM

This study aims to investigate the use of various deep learning techniques to predict future residential natural gas consumption in Italy, with a particular emphasis on the correlation between gas consumption and temperature. Four models were evaluated, including Multi-layer Perceptron (MLP), Convolutional Neural Network (CNN), Simple Long-Short Term Memory (LSTM), and Stack-LSTM, with the latter chosen due to its two-layer LSTM and potential to improve forecasting accuracy. Feature scaling was conducted with the MinMaxScaler method to ensure uniform values among variables. Statistical analysis was performed using Mean Squared Error (MSE), Mean Absolute Error (MAE), and R-squared accuracy metrics, with ANOVA tests and boxplots, used to visualize the distribution of accuracy metrics across test and full datasets. Results implied that the CNN and Stack-LSTM models were more effective in accurately predicting the target variable compared to the other models, as indicated by MSE and R-squared scores, as well as graphical comparisons of actual and predicted values. Finally, the research recommends the utilization of supplementary features in future research to increase the precision of forecasts.

Multi-step ahead wind power forecasting based on dual-attention mechanism

Accurate forecasting is essential for the economic benefits and efficient operation of intermittent wind power systems. Multi-step ahead wind power forecasting provides multiple benefits in the planning and operation of the power systems. This study proposes a deep learning model based on a dual-attention mechanism for multi-step ahead wind power forecasting. Both the attention mechanisms are applied over the encoder–decoder based sequence-to-sequence model consisting of long short-term memory (LSTM) blocks. The Bayesian optimization algorithm is applied to the proposed model to obtain the optimal combination of hyper-parameters. To evaluate its effectiveness, the proposed model has been compared with the persistence model, and state-of-the-art models such as simple LSTM, LSTM-attention, ensemble model, and neural basis expansion analysis for time series (N-BEATS) model. Furthermore, the performance of the attention mechanism with respect to impactful input features, such as wind speed and air pressure, was analyzed. The forecasting skill score of the proposed model was the highest among all other models in comparison, which indicates the effectiveness of the proposed model. Similarly, the proposed model outperformed the traditional methods in terms of other evaluating criteria, including mean absolute error (MAE), and root mean square error (RMSE) among others, hence, proving its efficacy. The average RMSE score of the proposed model for multi horizon forecasting was 0.04995, whereas that of N-BEATS was 0.0876, ensemble method was 0.1132 and LSTM-attention was around 0.101375. Similarly, the average forecasting skill score of proposed method was 0.6625, whereas that of N-BEATS was 0.3975 and LSTM-attention achieved 0.3775 skill score.

Hybrid voice activity detection system based on LSTM and auditory speech features

Voice Activity Detection (VAD), sometimes called as Speech Activity Detection, is the process of extracting speech regions in audio recordings including many type of sounds. Because undesired data causes both computational complexity and time wasting, most of speech based applications consider only speech part (region of interest) and ignore the rest. This is the main reason that makes usage of the VAD stands a preliminary stage in applications like automatic speech recognition (ASR), speaker identification/verification, speech enhancement, speaker diarization etc. In this study, a successful semi-supervised VAD system, which we named as “hybridVAD”, was proposed especially for the environment with high signal-to-noise ratio (SNR) with the manner of two-stage. At first, VAD decision was obtained from a relatively simple Long-Short Term Memory (LSTM) network trained by auditory speech features like energy, zero crossing rate (ZCR) and 13rd order-Mel Frequency Cepstral Coefficients (MFCC). After we applied a reasonable thresholding strategy to the same features to have second VAD decision, we combined both decisions with logical operators. The result was surprisingly showed that final VAD decision have low FEC and OVER errors, which are specifically critical for any speaker diarization system, mostly in the environments with high SNR.

Wind power forecast based on broad learning system and simplified long short term memory network

Due to its strong randomness and volatility, the modes of wind power are complex. After decomposing wind power time series into three subseries, the complex modes are deconstructed and each subseries maintain unique characteristics. Aiming at the characteristics of each subseries, a wind power forecast method based on Broad Learning System (BLS) and Simplified Long Short Term Memory (SLSTM) is proposed. Firstly, the decomposed subseries is analysed. The first layer subseries reflects the main amplitude changes caused by ramp events and the second represents the local fluctuations accompanying ramp events. Due to the randomness of ramp events, new modes appear frequently in the two subseries. The third layer subseries consists of continuous irregular jitter. Then, the BLS is introduced to forecast the first and second layer subseries. By adjusting its structure dynamically, various modes of ramp events can be learned from amplitude change and accompanying fluctuation. Additionally, according to the characteristics of the third layer subseries, continuous jitter needn't be transmitted in long-term memory. Setting the forget gate to 0, SLSTM is proposed to forecast the third layer subseries. Finally, the wind power data from the Belgian ELIA website is used to verify the superiority of the proposed method.

A Short-Term Traffic Flow Prediction Model Based on an Improved Gate Recurrent Unit Neural Network

With the increasing demand for intelligent transportation systems, short-term traffic flow prediction has become an important research direction. The memory unit of a Long Short-Term Memory (LSTM) neural network can store data characteristics over a certain period of time, hence the suitability of this network for time series processing. This paper uses an improved Gate Recurrent Unit (GRU) neural network to study the time series of traffic parameter flows. The LSTM short-term traffic flow prediction based on the flow series is first investigated, and then the GRU model is introduced. The GRU can be regarded as a simplified LSTM. After extracting the spatial and temporal characteristics of the flow matrix, an improved GRU with a bidirectional positive and negative feedback called the Bi-GRU prediction model is used to complete the short-term traffic flow prediction and study its characteristics. The Rectified Adaptive (RAdam) model is adopted to improve the shortcomings of the common optimizer. The cosine learning rate attenuation is also used for the model to avoid converging to the local optimal solution and for the appropriate convergence speed to be controlled. Furthermore, the scientific and reliable model learning rate is set together with the adaptive learning rate in RAdam. In this manner, the accuracy of network prediction can be further improved. Finally, an experiment of the Bi-GRU model is conducted. The comprehensive Bi-GRU prediction results demonstrate the effectiveness of the proposed method.

A novel seasonal segmentation approach for day-ahead load forecasting

Day-ahead load forecasting plays a crucial role in operation and management of power systems. Weather conditions have a significant impact on daily load profile, hence, it follows an almost similar pattern within a season. However, it varies markedly across the seasons. Existing literature on load forecasting adopts a very casual approach in considering the seasonality, based on either calendar month or some meteorological parameter, which is inconsistent and inaccurate, especially during transition periods, thus leading to high forecasting errors. This paper proposes a novel seasonal segmentation approach for day-ahead load forecasting that uses multiple bidirectional Long Short Term Memory (LSTM) networks. An index has been derived from various weather parameters that govern the selection of seasonal models for forecasting purposes. Weighted output from multiple seasonal models is also possible in special cases for better forecasting accuracy. The proposed seasonal segmentation approach avoids the need for frequent model retraining and results in a better forecast accuracy with a relatively simple LSTM structure. The performance of the proposed method has been validated and compared on the actual load data of Madhya Pradesh state (MP), India. The improved results suggest that the proposed approach can be applied reliably for load scheduling applications.

Long Short Term Memory Neural Network-Based Model Construction and Fne-Tuning for Air Quality Parameters Prediction

Abstract Air pollution has increased worries regarding health and ecosystems. Precise prediction of air quality parameters can assist in the effective action of air pollution control and prevention. In this work, a deep learning framework is proposed to predict parameters such as fine particulate matter and carbon monoxide. Long Short Term Memory (LSTM) neural network-based model that processes sequences in forward and backward direction to consider the influence of timesteps in both directions is employed. For further learning, unidirectional layers’ stacking is implemented. The performance of the model is optimized by fine-tuning hyperparameters, regularization techniques for overfitting resolution, and various merging options for the bidirectional input layer. The proposed model achieves good optimization and performs better than the simple LSTM and a Recurrent Neural Network (RNN) based model. Moreover, an attention-based mechanism is adopted to focus on more significant timesteps for prediction. The self-attention approach improves performance further and works well especially for longer sequences and extended time horizons. Experiments are conducted using real-world data collected, and results are evaluated using the mean square error loss function.

Hybrid Machine Learning Models for Forecasting Surgical Case Volumes at a Hospital

Recent developments in machine learning and deep learning have led to the use of multiple algorithms to make better predictions. Surgical units in hospitals allocate their resources for day surgeries based on the number of elective patients, which is mostly disrupted by emergency surgeries. Sixteen different models were constructed for this comparative study, including four simple and twelve hybrid models for predicting the demand for endocrinology, gastroenterology, vascular, urology, and pediatric surgical units. The four simple models used were seasonal autoregressive integrated moving average (SARIMA), support vector regression (SVR), multilayer perceptron (MLP), and long short-term memory (LSTM). The twelve hybrid models used were a combination of any two of the above-mentioned simple models, namely, SARIMA–SVR, SVR–SARIMA, SARIMA–MLP, MLP–SARIMA, SARIMA–LSTM, LSTM–SARIMA, SVR–MLP, MLP–SVR, SVR–LSTM, LSTM–SVR, MLP–LSTM, and LSTM–MLP. Data from the period 2012–2018 were used to build and test the models for each surgical unit. The results indicated that, in some cases, the simple LSTM model outperformed the others while, in other cases, there was a need for hybrid models. This shows that surgical units are unique in nature and need separate models for predicting their corresponding surgical volumes.

Rainfall-Runoff Short-Term Forecasting Method Based on LSTM

A rainfall-runoff forecasting method based on Long Short Term Memory (LSTM) was proposed in this study, which can extract the trend characteristics of runoff time series data through introducing daily rainfall data collected at related upstream stations and making use of the advantages of LSTM in saving long-term sequence feature information and avoid vanishing gradient, and identify the nonlinear mapping relationship in between, thus establishing a short-term runoff forecasting model. In this study, 24-hour and 5-day short-term forecasting models were established based on the runoff data collected at Danba Hydrologic Station in Dadu River Basin and historical rainfall collected at three upstream stations (Xiaojin, Dawei, and Fubian River). The experimental results showed that the forecasting models performed well during the inspection period. In 24-hour forecasting, RMSE was 98.016 and MAE was 45.709, which were 73.993 and 32.699, respectively in 5-day forecasting, indicating better performance and increased forecasting accuracy than simple LSTM model.

BS-LSTM: An Ensemble Recurrent Approach to Forecasting Soil Movements in the Real World

Machine learning (ML) proposes an extensive range of techniques, which could be applied to forecasting soil movements using historical soil movements and other variables. For example, researchers have proposed recurrent ML techniques like the long short-term memory (LSTM) models for forecasting time series variables. However, the application of novel LSTM models for forecasting time series involving soil movements is yet to be fully explored. The primary objective of this research is to develop and test a new ensemble LSTM technique (called “Bidirectional-Stacked-LSTM” or “BS-LSTM”). In the BS-LSTM model, forecasts of soil movements are derived from a bidirectional LSTM for a period. These forecasts are then fed into a stacked LSTM to derive the next period’s forecast. For developing the BS-LSTM model, datasets from two real-world landslide sites in India were used: Tangni (Chamoli district) and Kumarhatti (Solan district). The initial 80% of soil movements in both datasets were used for model training and the last 20% of soil movements in both datasets were used for model testing. The BS-LSTM model’s performance was compared to other LSTM variants, including a simple LSTM, a bidirectional LSTM, a stacked LSTM, a CNN-LSTM, and a Conv-LSTM, on both datasets. Results showed that the BS-LSTM model outperformed all other LSTM model variants during training and test in both the Tangni and Kumarhatti datasets. This research highlights the utility of developing recurrent ensemble models for forecasting soil movements ahead of time.