Well-known Recurrent Neural Network Research Articles

Linear matrix genetic programming as a tool for data-driven black-box control-oriented modeling in conditions of limited access to training data

In the paper, a new evolutionary technique called Linear Matrix Genetic Programming (LMGP) is proposed. It is a matrix extension of Linear Genetic Programming and its application is data-driven black-box control-oriented modeling in conditions of limited access to training data. In LMGP, the model is in the form of an evolutionarily-shaped program which is a sequence of matrix operations. Since the program has a hidden state, running it for a sequence of input data has a similar effect to using well-known recurrent neural networks such as Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU). To verify the effectiveness of the LMGP, it was compared with different types of neural networks. The task of all the compared techniques was to reproduce the behavior of a nonlinear model of an underwater vehicle. The results of the comparative tests are reported in the paper and they show that the LMGP can quickly find an effective and very simple solution to the given problem. Moreover, a detailed comparison of models, generated by LMGP and LSTM/GRU, revealed that the former are up to four times more accurate than the latter in reproducing vehicle behavior.

Heteroscedasticity effects as component to future stock market predictions using RNN-based models.

Heteroscedasticity effects are useful for forecasting future stock return volatility. Stock volatility forecasting provides business insight into the stock market, making it valuable information for investors and traders. Predicting stock volatility is a crucial task and challenging. This study proposes a hybrid model that predicts future stock volatility values by considering the heteroscedasticity element of the stock price. The proposed model is a combination of Generalized Autoregressive Conditional Heteroskedasticity (GARCH) and a well-known Recurrent Neural Network (RNN) algorithm Long Short-Term Memory (LSTM). This proposed model is referred to as GARCH-LSTM model. The proposed model is expected to improve prediction accuracy by considering heteroscedasticity elements. First, the GARCH model is employed to estimate the model parameters. After that, the ARCH effect test is used to test the residuals obtained from the model. Any untrained heteroscedasticity element must be found using this step. The hypothesis of the ARCH test yielded a p-value less than 0.05 indicating there is valuable information remaining in the residual, known as heteroscedasticity element. Next, the dataset with heteroscedasticity is then modelled using an LSTM-based RNN algorithm. Experimental results revealed that hybrid GARCH-LSTM had the lowest MAE (7.961), RMSE (10.466), MAPE (0.516) and HMAE (0.005) values compared with a single LSTM. The accuracy of forecasting was also significantly improved by 15% and 13% with hybrid GARCH-LSTM in comparison to single LSTMs. Furthermore, the results reveal that hybrid GARCH-LSTM fully exploits the heteroscedasticity element, which is not captured by the GARCH model estimation, outperforming GARCH models on their own. This finding from this study confirmed that hybrid GARCH-LSTM models are effective forecasting tools for predicting stock price movements. In addition, the proposed model can assist investors in making informed decisions regarding stock prices since it is capable of closely predicting and imitating the observed pattern and trend of KLSE stock prices.

Two-Stage UA-GAN for Precipitation Nowcasting

Short-term rainfall prediction by radar echo map extrapolation has been a very hot area of research in recent years, which is also an area worth studying owing to its importance for precipitation disaster prevention. Existing methods have some shortcomings. In terms of image indicators, the predicted images are not clear enough and lack small-scale details, while in terms of precipitation accuracy indicators, the prediction is not accurate enough. In this paper, we proposed a two-stage model (two-stage UA-GAN) to achieve more accurate prediction echo images with more details. For the first stage, we used the Trajectory Gated Recurrent Unit (TrajGRU) model to carry out a pre-prediction, which proved to have a good ability to capture spatiotemporal movement of rain field. In the second stage, we proposed a spatiotemporal attention enhanced Generative Adversarial Networks (GAN) model with a U-Net structure and a new deep residual attention module in order to carry out the refinement and improvement of the first-stage prediction. Experimental results showed that our model outperforms the optical-flow based method Real-Time Optical Flow by Variational Methods for Echoes of Radar (ROVER), and some well-known Recurrent Neural Network (RNN)-based models (TrajGRU, PredRNN++, ConvGRU, Convolutional Long Short-Term Memory (ConvLSTM)) in terms of both image detail indexes and precipitation accuracy indexes, and is visible to the naked eye to have better accuracy and more details.

A novel approach for COVID-19 Infection forecasting based on multi-source deep transfer learning

COVID-19 is a contagious disease; so, predicting its future infections in a provincial region requires the consideration of the related data (i.e., rates of infection, mortality and recovery, etc.) over a period of time. Clearly, the COVID-19 data of a particular provincial region can be easily modelled as a time-series. However, predicting the future COVID-19 infections in a particular region is quite challenging when the availability of COVID-19 dataset of the province is of little quantity. Accordingly, ML models when deployed for such tasks usually results in low infection prediction accuracy. To overcome such issues of low variance and high bias in a model due to data scarcity, multi-source transfer learning (MSTL) along with deep learning may be quite useful and effective. Therefore, this paper proposes a novel technique based on multi-source deep transfer learning (MSDTL) to efficiently forecast the future COVID-19 infections in the provinces with insufficient COVID-19 data. The proposed approach is a novel contribution as it considers the fact that future COVID-19 transmission in a region also depends on its population density and economic conditions (GDP) for accurate forecasting of the infections to tackle the pandemic efficiently. The importance of this feature selection is experimentally proved in this paper. Our proposed approach employs the well-known recurrent neural network architecture, the Long-short term memory (LSTM), a popular deep-learning model for history-dependent tasks. A comparative analysis has been performed with existing state-of-art algorithms to portray the efficiency of LSTM. Thus, formation of MSDTL approach enhances the predictive precision capability of the LSTM. We evaluate the proposed methodology over the COVID-19 dataset from sixty-two provinces belonging to different nations. We then empirically evaluate the performance of the proposed approach using two different evaluation metrics, viz. The mean absolute percentage error and the coefficient of determination. We show that our proposed MSDTL based approach is better in terms of the accuracy of the future infection prediction, and produces improvements up to 96% over its without-TL counterpart.

Real-Time Parameter Estimation of an Electrochemical Lithium-Ion Battery Model Using a Long Short-Term Memory Network

An electrochemical lithium-ion battery model is well known to be suited for effectively describing the microstructure evolution in charging and discharging processes of a lithium-ion battery with practically realizable complexity. This paper presents a neural network-based parameter estimation scheme to identify the parameters of an electrochemical lithium-ion battery model in a near-optimal and real-time manner in order to consistently observe the electrochemical states of batteries. The network is first trained to learn the dynamics of the electrochemical lithium-ion battery model, and then, it is applied to estimate the parameters with available finite-time measurements of voltage, current, temperature, and state of charge. In order to efficiently learn the dynamic characteristics of a lithium-ion battery, a well-known recurrent neural network, called a long short-term memory model, is employed with other techniques such as batch normalization, dropout, and stochastic gradient descent with warm restarts for learning speed enhancement and regularization. Using synthetic and experimental data, we show that the proposed estimation scheme works well, finding parameters and recovering the voltage profiles within the root-mean-square error of 0.43% and 26 mV, respectively, even with measurements obtained within a sufficiently short interval of time.

COVID-19 in Bangladesh: A Deeper Outlook into The Forecast with Prediction of Upcoming Per Day Cases Using Time Series.

A global pandemic on March 11th of 2020, which was initially renowned by the World Health Organization (WHO) revealed the coronavirus (the COVID-19 epidemic). Coronavirus was flown in -December 2019 in Wuhan, Hubei region in China. Currently, the situation is enlarged by the infection in more than 200 countries all over the world. In this situation it was rising into huge forms in Bangladesh too. Modulated with a public dataset delivered by the IEDCR health authority, we have produced a sustainable prognostic method of COVID-19 outbreak in Bangladesh using a deep learning model. Throughout the research, we forecasted up to 30 days in which per day actual prediction was confirmed, death and recoveries number of people. Furthermore, we illustrated that long short-term memory (LSTM) demands the actual output trends among time series data analysis with a controversial study that exceeds random forest (RF) regression and support vector regression (SVR), which both are machine learning (ML) models. The current COVID-19 outbreak in Bangladesh has been considered in this paper. Here, a well-known recurrent neural network (RNN) model in order to referred by the LSTM network that has forecasted COVID-19 cases on per day infected scenario of Bangladesh from May 15th of 2020 till June 15th of 2020. Added with a comparative study that drives into the LSTM, SVR, RF regression which is processed by the RMSE transmission rate. In all respects, in Bangladesh the gravity of COVID-19 has become excessive nowadays so that depending on this situation public health sectors and common people need to be aware of this situation and also be able to get knowledge of how long self-lockdown will be maintained. So far, to the best of our knowledge LSTM based time series analysis forecasting infectious diseases is a well-done formula

State-of-Health Estimation of Li-ion Batteries in Electric Vehicle Using IndRNN under Variable Load Condition

In electric vehicles (EVs), battery management systems (BMS) carry out various functions for effective utilization of stored energy in lithium-ion batteries (LIBs). Among numerous functions performed by the BMS, estimating the state of health (SOH) is an essential and challenging task to be accomplished at regular intervals. Accurate estimation of SOH ensures battery reliability by computing remaining lifetime and forecasting its failure conditions to avoid battery risk. Accurate estimation of SOH is challenging, due to uncertain operating conditions of EVs and complex non-linear electrochemical characteristics demonstrated by LIBs. In most of the existing studies, standard charge/discharge patterns with numerous assumptions are considered to accelerate the battery ageing process. However, such patterns and assumptions fail to reflect the real world operating condition of EV batteries, which is not appropriate for BMS of EVs. In contrast, this research work proposes a unique SOH estimation approach, using an independently recurrent neural network (IndRNN) in a more realistic manner by adopting the dynamic load profile condition of EVs. This research work illustrates a deep learning-based data-driven approach to estimate SOH by analyzing their historical data collected from LIBs. The IndRNN is adapted due to its ability to capture complex non-linear characteristics of batteries by eliminating the gradient problem and allowing the neural network to learn long-term dependencies among the capacity degradations. Experimental results indicate that the IndRNN based model is able to predict a battery’s SOH accurately with root mean square error (RMSE) reduced to 1.33% and mean absolute error (MAE) reduced to 1.14%. The maximum error (MAX) produced by IndRNN throughout the testing process is 2.5943% which is well below the acceptable SOH error range of ±5% for EVs. In addition, to demonstrate effectiveness of the IndRNN attained results are compared with other well-known recurrent neural network (RNN) architectures such as long short-term memory (LSTM) and gated recurrent unit (GRU). From the comparison of results, it is clearly evident that IndRNN outperformed other RNN architectures with the highest SOH accuracy rate.

Towards a real-time processing framework based on improved distributed recurrent neural network variants with fastText for social big data analytics

Big data generated by social media stands for a valuable source of information, which offers an excellent opportunity to mine valuable insights. Particularly, User-generated contents such as reviews, recommendations, and users’ behavior data are useful for supporting several marketing activities of many companies. Knowing what users are saying about the products they bought or the services they used through reviews in social media represents a key factor for making decisions. Sentiment analysis is one of the fundamental tasks in Natural Language Processing. Although deep learning for sentiment analysis has achieved great success and allowed several firms to analyze and extract relevant information from their textual data, but as the volume of data grows, a model that runs in a traditional environment cannot be effective, which implies the importance of efficient distributed deep learning models for social Big Data analytics. Besides, it is known that social media analysis is a complex process, which involves a set of complex tasks. Therefore, it is important to address the challenges and issues of social big data analytics and enhance the performance of deep learning techniques in terms of classification accuracy to obtain better decisions.In this paper, we propose an approach for sentiment analysis, which is devoted to adopting fastText with Recurrent neural network variants to represent textual data efficiently. Then, it employs the new representations to perform the classification task. Its main objective is to enhance the performance of well-known Recurrent Neural Network (RNN) variants in terms of classification accuracy and handle large scale data. In addition, we propose a distributed intelligent system for real-time social big data analytics. It is designed to ingest, store, process, index, and visualize the huge amount of information in real-time. The proposed system adopts distributed machine learning with our proposed method for enhancing decision-making processes. Extensive experiments conducted on two benchmark data sets demonstrate that our proposal for sentiment analysis outperforms well-known distributed recurrent neural network variants (i.e., Long Short-Term Memory (LSTM), Bidirectional Long Short-Term Memory (BiLSTM), and Gated Recurrent Unit (GRU)). Specifically, we tested the efficiency of our approach using the three different deep learning models. The results show that our proposed approach is able to enhance the performance of the three models. The current work can provide several benefits for researchers and practitioners who want to collect, handle, analyze and visualize several sources of information in real-time. Also, it can contribute to a better understanding of public opinion and user behaviors using our proposed system with the improved variants of the most powerful distributed deep learning and machine learning algorithms. Furthermore, it is able to increase the classification accuracy of several existing works based on RNN models for sentiment analysis.

2D recurrent neural networks: a high-performance tool for robust visual tracking in dynamic scenes

This paper proposes a novel method for robust visual tracking of arbitrary objects, based on the combination of image-based prediction and position refinement by weighted correlation. The effectiveness of the proposed approach is demonstrated on a challenging set of dynamic video sequences, extracted from the final of triple jump at the London 2012 Summer Olympics. A comparison is made against five baseline tracking systems. The novel system shows remarkable superior performances with respect to the other methods, in all considered cases characterized by changing background, and a large variety of articulated motions. The novel architecture, from here onward named 2D Recurrent Neural Network (2D-RNN), is derived from the well-known recurrent neural network model and adopts nearest neighborhood connections between the input and context layers in order to store the temporal information content of the video. Starting from the selection of the object of interest in the first frame, neural computation is applied to predict the position of the target in each video frame. Normalized cross-correlation is then applied to refine the predicted target position. 2D-RNN ensures limited complexity, great adaptability and a very fast learning time. At the same time, it shows on the considered dataset fast execution times and very good accuracy, making this approach an excellent candidate for automated analysis of complex video streams.

Simulated annealing algorithm-based Elman network for dynamic system identification

One of the well-known recurrent neural networks is the Elman network. Recently, it has been used in applications of system identification. The network has feedforward and feedback connections. It can be trained essentially as a feedforward network by means of the basic backpropagation algorithm, but its feedback connections have to be kept constant. For training success, it is important to select the correct values for the feedback connections. However, finding these values manually can be a lengthy trial-and-error process. This paper investigates the use of the simulated annealing (SA) algorithm to obtain the weight values of both the feedforward and feedback connections of Elman networks used for dynamic system identification. The SA algorithm is an efficient random search procedure, which can simultaneously obtain the optimal weight values of both the feedforward and feedback connections.

Stock Price Prediction Based on Procedural Neural Networks

We present a spatiotemporal model, namely, procedural neural networks for stock price prediction. Compared with some successful traditional models on simulating stock market, such as BNN (backpropagation neural networks, HMM (hidden Markov model) and SVM (support vector machine)), the procedural neural network model processes both spacial and temporal information synchronously without slide time window, which is typically used in the well‐known recurrent neural networks. Two different structures of procedural neural networks are constructed for modeling multidimensional time series problems. Learning algorithms for training the models and sustained improvement of learning are presented and discussed. Experiments on Yahoo stock market of the past decade years are implemented, and simulation results are compared by PNN, BNN, HMM, and SVM.

Training Elman and Jordan networks for system identification using genetic algorithms

Two of the well-known recurrent neural networks are the Elman network and the Jordan network. Recently, modifications have been made to these networks to facilitate their applications in dynamic systems identification. Both the original and the modified networks have trainable feedforward connections. However, in order that they can be trained essentially as feedforward networks by means of the simple backpropagation algorithm, their feedback connections have to be kept constant. For the training to converge, it is important to select correct values for the feedback connections, but finding these values manually can be a lengthy trial-and-error process. This paper describes the use of genetic algorithms (GAs) to train the Elman and Jordan networks for dynamic systems identification. The GA is an efficient, guided, random search procedure which can simultaneously obtain the optimal weights of both the feedforward and feedback connections.