Vanilla Long Short-term Memory Research Articles

An Evaluation of Bidirectional Long Short-Term Memory Model for Estimating Monthly Rainfall in India

Objectives: Predicting the amount of rainfall is difficult due to its complexity and non-linearity. The objective of this study is to predict the average rainfall one month ahead using the all-India monthly average rainfall dataset from 1871 to 2016. Methods: This study proposed a Bidirectional Long Short-Term Memory (LSTM) model to predict the average monthly rainfall in India. The parameters of the models are determined using the grid search method. This study utilized the average monthly rainfall as an input, and the dataset consists of 1752 months of rainfall data prepared from thirty (30) meteorological sub-divisions in India. The model was compiled using the Mean Square Error (MSE) loss function and Adam optimizer. The models' performances were evaluated using statistical metrics such as Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE). Findings: This study discovered that the proposed Bidirectional LSTM model achieved an RMSE of 240.79 and outperformed an existing Recurrent Neural Network (RNN), Vanilla LSTM and Stacked LSTM by 8%, 4% and 2% respectively. The study also finds that increasing the input time step and increasing the number of cells in the hidden layer enhanced the prediction performance of the proposed model, and the Bidirectional LSTM converges at a lower epoch compared to RNN and LSTM models. Novelty: This study applied the Bidirectional LSTM for the first time in predicting all-India monthly average rainfall and provides a new benchmark for this dataset. Keywords: Deep Learning, LSTM, Rainfall prediction, Stacked LSTM, Bidirectional LSTM

FlightForecast: A Comparative Analysis of Stack LSTM and Vanilla LSTM Models for Flight Prediction

The Coronavirus was first reported in China in the city of Wuhan in December 2019, after a couple of months, it was widespread around the world. The whole world was in a state of lockdown. This hazardous disease affects the normal daily life of every individual and the tourism industry, especially the airline business was at a greater loss. Considering the airline business, this study contains data on commercial flights from 2019 to 2020. The conducted research analyzed the rise and fall of different flights in the lockdown period. The research is based on the variants of Long Short-Term Memory (LSTM) such as standard Recurrent Neural Network (RNN) and stack LSTM. The comparative research shows that the prediction of the stack LSTM model is better than the standard RNN keeping view of taking a considerable amount of time to train.

Investigating the applicability of deep learning and machine learning models in predicting the structural performance of V-shaped RC folded plates

Reinforced concrete folded plates (RC-FPs), which are a special class of shell structures, have recently become very popular in modern architectural and engineering applications because of the need for lightweight and aesthetic structures to cover large areas. However, it is known that studies on the structural performance of RC-FPs are insufficient. Therefore, this article presents a study on the development and comparison of different deep learning (DL) and machine learning (ML) models for the prediction of the structural performance of full-scale V-shaped RC-FPs produced from hybrid fiber-reinforced self-consolidating concrete (HFR-SCC) having different plate thicknesses (50, 60, 70, and 80 mm), fiber volumes (1.00% and 1.25%), and combinations (single, binary, and ternary). While vanilla long short-term memory (VLSTM) and bidirectional long short-term memory (BILSTM) are used as DL models, random forest (RF), extremely randomized trees (ERT), and adaptive boosting (AdaBoost) are preferred for ML models. To construct the models, the structural performance results of a total of 44 full-scale V-shaped RC-FPs subjected to four-point bending loading were adopted as the database. In addition to all these, the AdaBoost model is used to determine the relative feature importance of the input parameters. Based on the results, among the DL models, the BILSTM has the best ability to predict the structural performance values of V-shaped RC-FPs (such as R-squared values for maximum load-bearing capacity, cracking load, toughness, and deflection ductility are 0.934, 0.987, 0.972, and 0.812, respectively), while in ML models, this is valid for the ERT (such as R-square values are 0.917 for maximum load-bearing capacity, 0.936 for cracking load, 0.947 for toughness and 0.825 for deflection ductility). On the other hand, DL models predicted all other structural performance values better than ML models, except for deflection ductility. Besides, the most relative important input features for maximum load-bearing capacity and toughness values is plate thickness, whereas for cracking load and deflection ductility values compressive strength is important. In conclusion, it can be emphasized that the use of DL models can provide significant advantages in engineering applications, such as predicting the structural performance of V-shaped RC-FPs.

Wind Turbine Predictive Fault Diagnostics Based on a Novel Long Short-Term Memory Model

The operation and maintenance (O&M) issues of offshore wind turbines (WTs) are more challenging because of the harsh operational environment and hard accessibility. As sudden component failures within WTs bring about durable downtimes and significant revenue losses, condition monitoring and predictive fault diagnostic approaches must be developed to detect faults before they occur, thus preventing durable downtimes and costly unplanned maintenance. Based primarily on supervisory control and data acquisition (SCADA) data, thirty-three weighty features from operational data are extracted, and eight specific faults are categorised for fault predictions from status information. By providing a model-agnostic vector representation for time, Time2Vec (T2V), into Long Short-Term Memory (LSTM), this paper develops a novel deep-learning neural network model, T2V-LSTM, conducting multi-level fault predictions. The classification steps allow fault diagnosis from 10 to 210 min prior to faults. The results show that T2V-LSTM can successfully predict over 84.97% of faults and outperform LSTM and other counterparts in both overall and individual fault predictions due to its topmost recall scores in most multistep-ahead cases performed. Thus, the proposed T2V-LSTM can correctly diagnose more faults and upgrade the predictive performances based on vanilla LSTM in terms of accuracy, recall scores, and F-scores.

DeepEpi: Deep Learning Model for Predicting Gene Expression Regulation Based on Epigenetic Histone Modifications

Background: Histone modification is a vital element in gene expression regulation. The way in which these proteins bind to the DNA impacts whether or not a gene may be expressed. Although those factors cannot influence DNA construction, they can influence how it is transcribed. Objective: Each spatial location in DNA has its function, so the spatial arrangement of chromatin modifications affects how the gene can express. Also, gene regulation is affected by the type of histone modification combinations that are present on the gene and depends on the spatial distributional pattern of these modifications and how long these modifications read on a gene region. So, this study aims to know how to model Long-range spatial genome data and model complex dependencies among Histone reads. Methods: The Convolution Neural Network (CNN) is used to model all data features in this paper. It can detect patterns in histones signals and preserve the spatial information of these patterns. It also uses the concept of memory in long short-term memory (LSTM), using vanilla LSTM, Bi-Directional LSTM, or Stacked LSTM to preserve long-range histones signals. Additionally, it tries to combine these methods using ConvLSTM or uses them together with the aid of a self-attention. Results: Based on the results, the combination of CNN, LSTM with the self-attention mechanism obtained an Area under the Curve (AUC) score of 88.87% over 56 cell types. Conclusion: The result outperforms the present state-of-the-art model and provides insight into how combinatorial interactions between histone modification marks can control gene expression. The source code is available at https://github.com/RaniaHamdy/DeepEpi.

Bayesian Deep Learning Applied to LSTM Models for Predicting COVID-19 Confirmed Cases in Iraq

The COVID-19 pandemic has had a huge impact on populations around the world and has caused critical problems to medical systems. With the increasing number of COVID-19 infections, research has focused on forecasting the confirmed cases to make the right medical decisions. Despite the huge number of studies conducted to forecast the COVID-19 patients, the use of Deep Learning (DL) and Bayesian DL models are limited in this field in Iraq. Therefore, this research aims to predict the confirmed cases of COVID-19 in Iraq using classical DL models such as, Long-Short-Term-Memory (LSTM) and Bayesian LSTM models. In this study, Bayesian Deep Learning (BDL) using LSTM models was used to predict COVID-19 confirmed cases in Iraq. The motivation behind using BDL models is that they are capable to quantify model uncertainty and provide better results without overfitting or underfitting. A Monte Carlo (MC) Dropout, which is an approximation method, is added to the Bayesian-LSTM to create numerous predictions for each instance and evaluate epistemic uncertainty. To evaluate the performance of our proposed models, four evaluation measures (MSE, RMSE, R2, MAE) were used. Experimental results showed that the proposed models were efficient and provided an R2 of 0.93 and 0.92, for vanilla LSTM and Bayesian-LSTM, respectively. Furthermore, the two proposed models were optimized using ADAM and SGD optimizers, with the results revealing that optimizing with ADAM provided more accurate results. Thus, we believe that these models may assist the government in making critical decisions based on short-term predictions of confirmed cases in Iraq.

Analyzing the performance of long short‐term memory architectures for malware detection models

SummaryMalicious software forms a threat to many software‐intensive systems and as such several malware detection approaches have been introduced, often based on sequential data analysis. Long short‐term memory (LSTM) is an artificial recurrent neural network (RNN) architecture that is effective for sequential data analysis, however, no study has yet analyzed the performance of different LSTM architectures for the application of malware detection. In this article, we aim to evaluate and benchmark the performance of LSTM‐based malware detection approaches on specific LSTM architectures to provide insight into malware detection. Our method builds LSTM‐based malware prediction models and performs experiments using different LSTM architectures including Vanilla LSTM, stacked LSTM, bi‐directional LSTM, and CNN‐LSTM. We evaluated the performance of each of these architectures and different configurations. Our study, as a contribution, shows that Bidirectional LSTM with hyperparameter optimization is found to be overperforming other selected LSTM architectures. This study shows that different LSTM approaches and architectures are applicable to the malware detection problem. Quality attributes such as efficiency and accuracy, and the software system architecture adopted for the implementation impact the selection of the LSTM approach.

Long-Term Solar Irradiance Forecasting Using Multilinear Predictors

As the demand for crude oil is increasing every day, prices and pollution are both increasing in return, which has harmful effects on the environment. Thus, more attempts are being made to develop clean energy to rescue the planet and provide humanity with a cleaner energy source. As the renewable energy sector grows, new issues and challenges have emerged. Instability in electricity production from wind turbines, solar power plants, and dams creates challenges for energy transmission and storage systems. In order to achieve a more reliable and effective energy system, machine learning techniques have been used to forecast energy changes. Predicting the sun's Global Horizontal Irradiance (GHI) is one of the machine learning applications used in that sector. In this paper, many machine learning methods have been utilized, such as linear regression and long-short term memory (LSTM) methods to have long term GHI forecasting. Moreover, the significance of this paper is located in the way of prediction of the GHI irradiance prediction by using different levels of the linear regressors to find the best regressor level that provides the minimum error for the testing set based on cross-validation. Results showed that the regressor method provides a lower error compared with a single vanilla LSTM system for a shorter time computationally.

A Comparative Study of LSTM Variants in Prediction for Tesla’s Stock Price

Long short-term memory (LSTM) is widely used in the stock market to train the prediction model and forecast future stock prices. Applying the LSTM method to research may incur some problems and facilitate the improvement of the method. Therefore, many LSTM variants are put forward under different circumstances. This paper surveys four LSTM variants, including Vanilla, Stacked, Bi-directional, and CNN LSTM on two different data sets regarding Tesla's stock price. Two data sets mentioned in this paper represent different stock types. To be more specific, data set 1 refers to stocks with a single long-term trend, while data set 2 can be seen as an example of stocks with more complexity. The result shows that the Vanilla LSTM reaches the highest prediction accuracy on the data set without any irregular shift in the long-term trend. CNN LSTM also provides decent predictions for the stock price. Otherwise, the Stacked LSTM performs the best for stock prediction. Bi-LSTM and CNN LSTM are also suitable for stock forecasting in more complicated situations. The change in preference for model selection proves that a company's operation situation and market circumstances also influence the prediction performance of LSTM variants.

An air quality prediction model based on improved Vanilla LSTM with multichannel input and multiroute output

Long short-term memory (LSTM), especially vanilla LSTM (VLSTM), has been widely used in air quality prediction field. However, VLSTM has many more parameters, thereby making training slow and prediction performance unstable. The VLSTM network input data have not been selected for better efficiency. In this paper, we propose an air quality prediction model based on the improved VLSTM with multichannel input and multiroute output (IVLSTM-MCMR). The proposed model includes the IVLSTM and MCMR modules. The proposed IVLSTM module is developed by improving the VLSTM inner structure of VLSTM in order to reduce the number of parameters that help to accelerate the convergence. A new historical information usage approach is further proposed to obtain a stable training process. For the MCMR module, a multichannel data input model (MC) with an improved linear similarity dynamic time warping is introduced to choose the valid data as the input of IVLSTM. A multiroute output model (MR) is designed to integrate the results from MC, in which the results of different target stations with different features are output by different routes. We evaluate the proposed model with the collected data from Beijing, China, and the experimental results show that our model achieves improvements regarding the predication performance.

Spatio-temporal variation of Covid-19 health outcomes in India using deep learning based models

Deep learning methods have become the state of the art for spatio-temporal predictive analysis in a wide range of fields, including environmental management, public health, urban planning, pollution monitoring, and so on. Despite the fact that a variety of powerful deep learning-based models can address various problem-specific issues in different research domain, it has been found that no single optimal model can outperform everywhere. Now, in the last two years, various deep learning-based studies have provided a variety of best-performing techniques for predicting COVID-19 health outcomes. In this context, this study attempts to perform a case study that investigates the spatio-temporal variation in the performance of deep-learning-based methods for predicting COVID-19 health outcomes in India. Various widely applied deep learning models namely CNN (convolutional neural network), RNN (recurrent neural network), Vanilla LSTM (long short-term memory), LSTM Autoencoder, and Bidirectional LSTM are considered to investigate their spatio-temporal performance variation. The effectiveness of the models is assessed using various metrics based on COVID-19 mortality time-series from 36 states and union territories of India.

CBLSTM-AE: A Hybrid Deep Learning Framework for Predicting Energy Consumption

Multisource energy data, including from distributed energy resources and its multivariate nature, necessitate the integration of robust data predictive frameworks to minimise prediction error. This work presents a hybrid deep learning framework to accurately predict the energy consumption of different building types, both commercial and domestic, spanning different countries, including Canada and the UK. Specifically, we propose architectures comprising convolutional neural network (CNN), an autoencoder (AE) with bidirectional long short-term memory (LSTM), and bidirectional LSTM BLSTM). The CNN layer extracts important features from the dataset and the AE-BLSTM and LSTM layers are used for prediction. We use the individual household electric power consumption dataset from the University of California, Irvine to compare the skillfulness of the proposed framework to the state-of-the-art frameworks. Results show performance improvement in computation time of 56% and 75.2%, and mean squared error (MSE) of 80% and 98.7% in comparison with a CNN BLSTM-based framework (EECP-CBL) and vanilla LSTM, respectively. In addition, we use various datasets from Canada and the UK to further validate the generalisation ability of the proposed framework to underfitting and overfitting, which was tested on real consumers’ smart boxes. The results show that the framework generalises well to varying data and constraints, giving an average MSE of ∼0.09 across all datasets, demonstrating its robustness to different building types, locations, weather, and load distributions.

CNO-LSTM: A Chaotic Neural Oscillatory Long Short-Term Memory Model for Text Classification

Long Short-Term Memory (LSTM) networks are unique to exercise data in its memory cell with long-term memory as Natural Language Processing (NLP) tasks have inklings of intensive time and computational power due to their complex structures like magnitude language model Transformer required to pre-train and learn billions of data performing different NLP tasks. In this paper, a dynamic chaotic model is proposed for the objective of transforming neurons states in network with neural dynamic characteristics by restructuring LSTM as Chaotic Neural Oscillatory-Long-Short Term Memory (CNO-LSTM), where neurons in LSTM memory cells are weighed in substitutes by oscillatory neurons to speed up computational training of language model and improve text classification accuracy for real-world applications. From the implementation perspective, five popular datasets of general text classification including binary, multi classification and multi-label classification are used to compare with mainstream baseline models on NLP tasks. Results showed that the performance of CNO-LSTM, a simplified model structure and oscillatory neurons state in exercising different types of text classification tasks are above baseline models in terms of evaluation index such as Accuracy, Precision, Recall and F1. The main contributions are time reduction and improved accuracy. It achieved approximately 46.76% of the highest reduction training time and 2.55% accuracy compared with vanilla LSTM model. Further, it achieved approximately 35.86% in time reduction compared with attention model without oscillatory indicating that the model restructure has reduced GPU dependency to improve training accuracy.

Working Memory Connections for LSTM

Recurrent Neural Networks with Long Short-Term Memory (LSTM) make use of gating mechanisms to mitigate exploding and vanishing gradients when learning long-term dependencies. For this reason, LSTMs and other gated RNNs are widely adopted, being the standard de facto for many sequence modeling tasks. Although the memory cell inside the LSTM contains essential information, it is not allowed to influence the gating mechanism directly. In this work, we improve the gate potential by including information coming from the internal cell state. The proposed modification, named Working Memory Connection, consists in adding a learnable nonlinear projection of the cell content into the network gates. This modification can fit into the classical LSTM gates without any assumption on the underlying task, being particularly effective when dealing with longer sequences. Previous research effort in this direction, which goes back to the early 2000s, could not bring a consistent improvement over vanilla LSTM. As part of this paper, we identify a key issue tied to previous connections that heavily limits their effectiveness, hence preventing a successful integration of the knowledge coming from the internal cell state. We show through extensive experimental evaluation that Working Memory Connections constantly improve the performance of LSTMs on a variety of tasks. Numerical results suggest that the cell state contains useful information that is worth including in the gate structure.

A Stacked LSTM-Based Approach for Reducing Semantic Pose Estimation Error

Achieving high estimation accuracy is significant for semantic simultaneous localization and mapping (SLAM) tasks. Yet, the estimation process is vulnerable to several sources of error, including limitations of the instruments used to perceive the environment, shortcomings of the employed algorithm, environmental conditions, or other unpredictable noise. In this article, a novel stacked long short-term memory (LSTM)-based error reduction approach is developed to enhance the accuracy of semantic SLAM in presence of such error sources. Training and testing data sets were constructed through simulated and real-time experiments. The effectiveness of the proposed approach was demonstrated by its ability to capture and reduce semantic SLAM estimation errors in training and testing data sets. Quantitative performance measurement was carried out using the absolute trajectory error (ATE) metric. The proposed approach was compared with vanilla and bidirectional LSTM networks, shallow and deep neural networks, and support vector machines. The proposed approach outperforms all other structures and was able to significantly improve the accuracy of semantic SLAM. To further verify the applicability of the proposed approach, it was tested on real-time sequences from the TUM RGB-D data set, where it was able to improve the estimated trajectories.

Deep learning‐based household electric energy consumption forecasting

With the advent of various electronic products, the household electric energy consumption is continuously increasing, and therefore it becomes very important to predict the household electric energy consumption accurately. Energy prediction models also have been developed for decades with advanced machine learning technologies. Meanwhile, the deep learning models are still actively under study, and many newer models show the state-of-the-art performance. Therefore, it would be meaningful to conduct the same experiment with these new models. Here, the authors predict the household electric energy consumption using deep learning models, known to be suitable for dealing with time-series data. Specifically, vanilla long short-term memory (LSTM), sequence to sequence, and sequence to sequence with attention mechanism are used to predict the electric energy consumption in the household. As a result, the vanilla LSTM shows the best performance on the root-mean-square error metric. However, from a graphical point of view, it seems that the sequence-to-sequence model predicts the energy consumption patterns best and the vanilla LSTM does not follow the pattern well. Also, to achieve the best performance of each deep learning model, vanilla LSTM, sequence to sequence, and sequence to sequence with attention mechanism should observe past 72, 72, and 24 h, respectively.

Statistical Explorations and Univariate Timeseries Analysis on COVID-19 Datasets to Understand the Trend of Disease Spreading and Death.

“Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2)”, the novel coronavirus, is responsible for the ongoing worldwide pandemic. “World Health Organization (WHO)” assigned an “International Classification of Diseases (ICD)” code—“COVID-19”-as the name of the new disease. Coronaviruses are generally transferred by people and many diverse species of animals, including birds and mammals such as cattle, camels, cats, and bats. Infrequently, the coronavirus can be transferred from animals to humans, and then propagate among people, such as with “Middle East Respiratory Syndrome (MERS-CoV)”, “Severe Acute Respiratory Syndrome (SARS-CoV)”, and now with this new virus, namely “SARS-CoV-2”, or human coronavirus. Its rapid spreading has sent billions of people into lockdown as health services struggle to cope up. The COVID-19 outbreak comes along with an exponential growth of new infections, as well as a growing death count. A major goal to limit the further exponential spreading is to slow down the transmission rate, which is denoted by a “spread factor (f)”, and we proposed an algorithm in this study for analyzing the same. This paper addresses the potential of data science to assess the risk factors correlated with COVID-19, after analyzing existing datasets available in “ourworldindata.org (Oxford University database)”, and newly simulated datasets, following the analysis of different univariate “Long Short Term Memory (LSTM)” models for forecasting new cases and resulting deaths. The result shows that vanilla, stacked, and bidirectional LSTM models outperformed multilayer LSTM models. Besides, we discuss the findings related to the statistical analysis on simulated datasets. For correlation analysis, we included features, such as external temperature, rainfall, sunshine, population, infected cases, death, country, population, area, and population density of the past three months—January, February, and March in 2020. For univariate timeseries forecasting using LSTM, we used datasets from 1 January 2020, to 22 April 2020.

Efficient online learning with improved LSTM neural networks

We introduce efficient online learning algorithms based on the Long Short Term Memory (LSTM) networks that employ the covariance information. In particular, we introduce the covariance of the present and one-time step past input vectors into the gating structure of the LSTM networks. Additionally, we include the covariance of the output vector, and we learn their weight matrices to improve the learning performance of the LSTM networks where we also provide their updates. We reduce the number of system parameters through the weight matrix factorization where we convert the LSTM weight matrices into two smaller matrices in order to achieve high learning performance with low computational complexity. Moreover, we apply the introduced approach to the Gated Recurrent Unit (GRU) architecture. In our experiments, we illustrate significant performance improvements achieved by our methods on real-life datasets with respect to the vanilla LSTM and vanilla GRU networks.

Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network

Because of their effectiveness in broad practical applications, LSTM networks have received a wealth of coverage in scientific journals, technical blogs, and implementation guides. However, in most articles, the inference formulas for the LSTM network and its parent, RNN, are stated axiomatically, while the training formulas are omitted altogether. In addition, the technique of “unrolling” an RNN is routinely presented without justification throughout the literature. The goal of this tutorial is to explain the essential RNN and LSTM fundamentals in a single document. Drawing from concepts in Signal Processing, we formally derive the canonical RNN formulation from differential equations. We then propose and prove a precise statement, which yields the RNN unrolling technique. We also review the difficulties with training the standard RNN and address them by transforming the RNN into the “Vanilla LSTM”11The nickname “Vanilla LSTM” symbolizes this model’s flexibility and generality (Greff et al., 2015). network through a series of logical arguments. We provide all equations pertaining to the LSTM system together with detailed descriptions of its constituent entities. Albeit unconventional, our choice of notation and the method for presenting the LSTM system emphasizes ease of understanding. As part of the analysis, we identify new opportunities to enrich the LSTM system and incorporate these extensions into the Vanilla LSTM network, producing the most general LSTM variant to date. The target reader has already been exposed to RNNs and LSTM networks through numerous available resources and is open to an alternative pedagogical approach. A Machine Learning practitioner seeking guidance for implementing our new augmented LSTM model in software for experimentation and research will find the insights and derivations in this treatise valuable as well.

Cultural Heritage and the Intelligent Internet of Things

Fourth Industrial Revolution technologies, such as artificial intelligence, big data, the Internet of Things (IoT), and virtual reality, have disrupted legacy methods of operations and have led to progress in many industries worldwide. These technologies also affect the cultural and national heritage. IoT generates large volumes of streaming data; therefore, advanced data analytics using big data analytics and artificial neural networks is an important research topic. In this study, IoT sensor data was collected at the restored Woljeong Bridge, which was originally built in the eighth century, or AD 760, during the Silla Dynasty (57 BC--AD 935) in South Korea. We empirically evaluate a recurrent neural network with recurrent units, including a long short-term memory (LSTM) unit and a gated recurrent unit (GRU). Additionally, we evaluate hybrid deep-learning models (convolution neural networks [CNN]-LSTM and CNN-GRU) to build a prediction model, facilitating the preventive conservation of an invaluable cultural and national heritage site. The experimental results show that the LSTM unit is an effective and robust model. When comparing the hybrid models (i.e., the joint CNN-LSTM and CNN-GRU architectures), we found that the vanilla LSTM and GRU models had superior time-series prediction capabilities.