Standard Recurrent Neural Networks Research Articles

Enhancing stock market prediction accuracy with recurrent deep learning models: A case study on the CAC40 index

This paper explores the application of Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks for stock price prediction over a 10-day horizon. The study aims to compare the predictive performance of these two deep learning architectures within the context of financial forecasting. Utilizing historical stock data from the CAC40 dataset, which represents a capitalization-weighted measure of the 40 most significant stocks on the Euronext Paris, we train and evaluate RNN and LSTM models to forecast future stock prices. Our results demonstrate the superior performance of LSTM networks in capturing the intricate temporal dependencies inherent in stock price data. Compared to standard RNNs, LSTM models exhibit higher accuracy and provide more reliable forecasts over the 10-day prediction period. The specialized memory cells and gating mechanisms in LSTM networks enable them to effectively identify both short-term changes and long-term patterns in stock prices, thus outperforming traditional RNN architectures. This enhanced ability to model the complex dynamics of stock market data underscores the potential of LSTM networks to improve investment decision-making, risk management, and the overall efficiency of financial markets. The insights gained from this study contribute to the growing body of knowledge on the application of deep learning in finance and investment, offering valuable guidance for practitioners and researchers seeking to harness the power of advanced algorithms for stock market prediction and optimization.

Movie Recommendation System Using Optimized RNN Approach.

This paper proposes a movie recommendation system that utilizes an optimized Recurrent Neural Network (RNN) approach. The proposed system is designed to provide users with personalized movie recommendations based on their previous movie preferences and the sentiments. The system works by taking user input, analyzing their movie preferences using content-based filtering techniques, and generating a list of recommended movies. The RNN architecture used in this system is optimized using a combination of techniques such as dropout regularization, early stopping, and parameter tuning. The proposed optimization techniques aim to reduce overfitting, improve convergence speed, and increase the model's overall accuracy. To evaluate the effectiveness of the proposed approach, we conducted experiments on the Movie Lens dataset. The results indicate that the optimized RNN-based movie recommendation system outperforms other existing recommendation systems such as collaborative filtering, content-based filtering, and standard RNN models. Furthermore, the proposed system achieved a significant improvement in accuracy and provided highly personalized recommendations to users. Overall, the proposed movie recommendation system using optimized RNN approach is a promising solution for providing personalized movie recommendations to users. It can be implemented in various platforms such as movie streaming websites, social media, and other movie-related platforms to improve the user experience and increase engagement

Enhancing continuous time series modelling with a latent ODE-LSTM approach

Due to their dynamic properties such as irregular sampling rate and high-frequency sampling, Continuous Time Series (CTS) are found in many applications. Since CTS with irregular sampling rate are difficult to model with standard Recurrent Neural Networks (RNNs), RNNs have been generalised to have continuous-time hidden dynamics defined by a Neural Ordinary Differential Equation (Neural ODE), leading to the ODE-RNN model. Another approach that provides a better modelling is that of the Latent ODE model, which constructs a continuous-time model where a latent state is defined at all times. The Latent ODE model uses a standard RNN as the encoder and a Neural ODE as the decoder. However, since the RNN encoder leads to difficulties with missing data and ill-defined latent variables, a Latent ODE-RNN model has recently been proposed that uses a ODE-RNN model as the encoder instead.Both the Latent ODE and Latent ODE-RNN models are difficult to train due to the vanishing and exploding gradients problem. To overcome this problem, the main contribution of this paper is to propose and illustrate a new model based on a new Latent ODE using an ODE-LSTM (Long Short-Term Memory) network as an encoder - the Latent ODE-LSTM model. To limit the growth of the gradients, the Norm Gradient Clipping strategy was embedded on the Latent ODE-LSTM model.The performance evaluation of the new Latent ODE-LSTM (with and without Norm Gradient Clipping) for modelling CTS with regular and irregular sampling rates is then demonstrated. Numerical experiments show that the new Latent ODE-LSTM performs better than Latent ODE-RNNs and can avoid the vanishing and exploding gradients during training.Code implementations developed in this work are available at github.com/CeciliaCoelho/LatentODELSTM.

FlightForecast: A Comparative Analysis of Stack LSTM and Vanilla LSTM Models for Flight Prediction

The Coronavirus was first reported in China in the city of Wuhan in December 2019, after a couple of months, it was widespread around the world. The whole world was in a state of lockdown. This hazardous disease affects the normal daily life of every individual and the tourism industry, especially the airline business was at a greater loss. Considering the airline business, this study contains data on commercial flights from 2019 to 2020. The conducted research analyzed the rise and fall of different flights in the lockdown period. The research is based on the variants of Long Short-Term Memory (LSTM) such as standard Recurrent Neural Network (RNN) and stack LSTM. The comparative research shows that the prediction of the stack LSTM model is better than the standard RNN keeping view of taking a considerable amount of time to train.

Gated recurrent unit network: A promising approach to corporate default prediction

AbstractThis paper presents a promising approach using gated recurrent unit (GRU) network to predict bankruptcy based on the whole sequence of financial statements of the companies listed on an unregulated market. This approach contrasts with the traditional literature where default prediction is usually tackled with methods that do not fully account for a company's history. The GRU network can be used to model long‐term dependencies thanks to its update and reset gates, which prevent the vanishing gradient problem and decide how much of the past information is relevant for predicting a default. This aspect may be of utmost importance in alternative markets where the signal detection problem is particularly strong. The performance of the GRU network is compared against the performance of other standard machine learning methods including Cox proportional hazards, gradient‐boosted Cox proportional hazards model, extreme gradient boosting, random survival forest, and standard recurrent neural networks on the basis of a broad selection of performance metrics. The GRU network not only outperforms standard machine learning methods in out‐of‐sample forecasts but also seems to be more robust in terms of in‐ versus out‐of‐sample performance.

Prediction of Wastewater Treatment Plant Effluent Water Quality Using Recurrent Neural Network (RNN) Models

Artificial Intelligence (AI) has recently emerged as a powerful tool with versatile applications spanning various domains. AI replicates human intelligence processes through machinery and computer systems, finding utility in expert systems, image and speech recognition, machine vision, and natural language processing (NLP). One notable area with limited exploration pertains to using deep learning models, specifically Recurrent Neural Networks (RNNs), for predicting water quality in wastewater treatment plants (WWTPs). RNNs are purpose-built for handling sequential data, featuring a feedback mechanism. However, standard RNNs may exhibit limitations in accommodating both short-term and long-term dependencies when addressing intricate time series problems. The solution to this challenge lies in adopting Long Short-Term Memory (LSTM) cells, known for their inherent memory management through a ‘forget gate’ mechanism. In general, LSTM architecture demonstrates superior performance. WWTP data represent a historical series influenced by fluctuating environmental conditions. This study employs simple RNNs and LSTM architecture to construct prediction models for effluent parameters, systematically assessing their performance through various training data scenarios and model architectures. The primary objective was to determine the most suitable WWTP dataset model. The study revealed that an epoch setting of 50 and a batch size of 100 yielded the lowest training time and root mean square error (RMSE) values for both RNN and LSTM models. Furthermore, when these models are applied to predict effluent parameters, they exhibit precise RMSE values for all parameters. The study results can be applied to detect potential upsets in WWTP operations.

A Hybrid Optimization Approach for Neural Machine Translation Using LSTM+RNN with MFO for Under Resource Language (Telugu)

NMT (Neural Machine Translation) is an innovative approach in the field of machine translation, in contrast to SMT (statistical machine translation) and Rule-based techniques which has resulted annotable improvements. This is because NMT is able to overcome many of the shortcomings that are inherent in the traditional approaches. The Development of NMT has grown tremendously in the recent years but NMT performance remain under optimal when applied to low resource language pairs like Telugu, Tamil and Hindi. In this work a proposedmethod fortranslating pairs (Telugu to English) is attempted, an optimal approach which enhancesthe accuracy and execution time period.A hybrid method approach utilizing Long short-term memory (LSTM) and traditional Recurrent Neural Network (RNN) are used for testing and training of the dataset. In the event of long-range dependencies, LSTM will generate more accurate results than a standard RNN would endure and the hybrid technique enhances the performance of LSTM. LSTM is used during the encoding and RNN is used in decoding phases of NMT. Moth Flame Optimization (MFO) is utilized in the proposed system for the purpose of providing the encoder and decoder model with the best ideal points for training the data.

Lithium battery life prediction based on deep learning

Li-ion batteries have the advantages of high efficiency, high energy density and long life, having developed rapidly in recent years. However, it is tough to accurately predict the decline trend of Li-ion battery capacity, which limits the further improvement of their service life and safety. In this study, a method of using wavelet denoising to preprocess the data and predicting the life of Li-ion battery based on whale optimization algorithm combined with long short-term memory network (WOA-LSTM) is proposed. In this paper, two sets of data sets B0005 and B0006 of NASA 's public data set are used. The original battery capacity data is subjected to wavelet transform and noise reduction to remove noise and redundant information. The calculation results of SNR and RMSE are 48.1119 and 0.006225, respectively. Then LSTM (Long short-term memory), RNN (Recurrent Neural Network), GRU (Gated Recurrent Unit) and WOA-LSTM are used to predict the remaining useful life of the data set RUL (Remaining Useful Lifetime, RUL), and the data error is compared, showing that in the results of MAE, RMSE and MAPE three prediction error indicators. The prediction results of WOA-LSTM in B0005 and B0006 show the minimum prediction errors, which are 0.0563,0.0710,0.0415 in B0005 data set and 0.0583,0.0831,0.0454 in B0006 data set. Compared with the standard LSTM model, RNN model and GRU model, the error indexes of the model are decreased, which are 7 %, 4 % and 3 % respectively, which has great advantages. This method can provide a reliable predictive analysis method for battery design and fault diagnosis.

An RNN-Based Performance Identification Model for Multi-Agent Containment Control Systems

In the containment control problem of multi-agent systems (MASs), the convergence of followers is always a potential threat to the security of system operations. From the perspective of system topology, the inherently non-linear properties of the algebraic connectivity of the follower2follower (F2F) network, combined with the influence of the leader2follower (L2F) topology on the system, make it difficult to design the convergence positions of the followers through mere mathematical analysis. Therefore, in the background of temporary networking tasks for large-scale systems, to achieve the goal of forecasting the performance of the whole system when networking is only completed with local information, this paper investigates the application and effectiveness of recurrent neural networks (RNNs) in the containment control system performance identification, thus improving the efficiency of system networking while ensuring system security. Two types of identification models based on two types of neural networks (NNs), MLP and standard RNN are developed, according to the range of information required for performance identification. Evaluation of the models is carried out by means of the coefficient of determination (R2) as well as the root-mean-square error (RMSE). The results show that each model may produce a better forecasting accuracy than the other models in specific cases, with models based on the standard RNN possessing smaller errors. With the proposed method, model identification can be achieved, but in-depth development of the model in further studies is still necessary to the extent the accuracy of the model.

Analysis of sentiment analysis model based on deep learning

A traditional yet important topic in the study of natural language processing is sentiment analysis. Deep learning models have gradually taken over as one of the primary techniques for resolving sentiment analysis issues over the last ten years. Common deep learning models targeting sentiment analysis tasks include both recurrent and convolutional neural networks, as well as the BERT model. The current research examines the classificational accuracy of numerous deep learning models with diverse structural types in order to compare their performance in sentiment analysis. Results from the experiments suggest that the pre-training BERT model achieves the highest accuracy, while the convolutional neural network appears to sustain better results on sentiment analysis than standard recurrent neural networks.

Time to Die 2: Improved in-game death prediction in Dota 2

Competitive video game playing, an activity called esports, is increasingly popular to the point that there are now many professional competitions held for a variety of games. These competitions are broadcast in a professional manner similar to traditional sports broadcasting. Esports games are generally fast paced, and due to the virtual nature of these games, camera positioning can be limited. Therefore, knowing ahead of time where to position cameras, and what to focus a broadcast and associated commentary on, is a key challenge in esports reporting. This gives rise to moment-to-moment prediction within esports matches which can empower broadcasters to better observe and process esports matches. In this work we focus on this moment-to-moment prediction and in particular present techniques for predicting if a player will die within a set number of seconds for the esports title Dota 2. A player death is one of the most consequential events in Dota 2. We train our model on ‘telemetry’ data gathered directly from the game itself, and position this work as a novel extension of our previous work on the challenge. We use an enhanced dataset covering 9,822 Dota 2 matches. Since the publication of our previous work, new dataset parsing techniques developed by the WEAVR project enable the model to track more features, namely player status effects, and more importantly, to operate in real time. Additionally, we explore two new enhancements to the original model: one data-based extension and one architectural. Firstly we employ learnt embeddings for categorical features, e.g. which in game character a player has selected, and secondly we explicitly model the temporal element of our telemetry data using recurrent neural networks. We find that these extensions and additional features all aid the predictive power of the model achieving an F1 score of 0.54 compared to 0.17 for our previous model (on the new data). We improve this further by experimenting with the length of the time-series in the input data and find using 15 time steps further improves the F1 score to 0.62. This compares to F1 of 0.1 for a standard RNN on the same task. Additionally a deeper analysis of the Time to Die model is carried out to assess its suitability as a broadcast aid.

SpaRCe: Improved Learning of Reservoir Computing Systems Through Sparse Representations.

"Sparse" neural networks, in which relatively few neurons or connections are active, are common in both machine learning and neuroscience. While, in machine learning, "sparsity" is related to a penalty term that leads to some connecting weights becoming small or zero, in biological brains, sparsity is often created when high spiking thresholds prevent neuronal activity. Here, we introduce sparsity into a reservoir computing network via neuron-specific learnable thresholds of activity, allowing neurons with low thresholds to contribute to decision-making but suppressing information from neurons with high thresholds. This approach, which we term "SpaRCe," optimizes the sparsity level of the reservoir without affecting the reservoir dynamics. The read-out weights and the thresholds are learned by an online gradient rule that minimizes an error function on the outputs of the network. Threshold learning occurs by the balance of two opposing forces: reducing interneuronal correlations in the reservoir by deactivating redundant neurons, while increasing the activity of neurons participating in correct decisions. We test SpaRCe on classification problems and find that threshold learning improves performance compared to standard reservoir computing. SpaRCe alleviates the problem of catastrophic forgetting, a problem most evident in standard echo state networks (ESNs) and recurrent neural networks in general, due to increasing the number of task-specialized neurons that are included in the network decisions.

Probabilistic traffic breakdown forecasting through Bayesian approximation using variational LSTMs

This paper proposes a framework for short-term traffic breakdown probability calculation using a Variational LSTM neural network model. Considering that traffic breakdown is a stochastic event, this forecast framework was devised to produce distributions as outputs, which cannot be achieved using standard deterministic recurrent neural networks. Therefore, the model counts on the robustness of neural networks but also includes the stochastic characteristics of highway capacity. The framework consists of three main steps: (i) build and train a probabilistic speed forecasting neural network, (ii) forecast speed distributions with the trained model using Monte Carlo (MC) dropout, and therefore perform Bayesian approximation, and (iii) establish a speed threshold that characterizes breakdown occurrence and calculate breakdown probabilities based on the speed distributions. The proposed framework produced an efficient control over traffic breakdown occurrence, can deal with many independent variables or features, and can be combined with traffic management strategies.

Improving Performance of Recurrent Neural Networks Using Simulated Annealing for Vertical Wind Speed Estimation

An accurate vertical wind speed (WS) data estimation is required to determine the potential for wind farm installation. In general, the vertical extrapolation of WS at different heights must consider different parameters from different locations, such as wind shear coefficient, roughness length, and atmospheric conditions. The novelty presented in this article is the introduction of two steps optimization for the Recurrent Neural Networks (RNN) model to estimate WS at different heights using measurements from lower heights. The first optimization of the RNN is performed to minimize a differentiable cost function, namely, mean squared error (MSE), using the Broyden-Fletcher-Goldfarb-Shanno algorithm. Secondly, the RNN is optimized to reduce a non-differentiable cost function using simulated annealing (RNN-SA), namely mean absolute error (MAE). Estimation of WS vertically at 50 m height is done by training RNN-SA with the actual WS data a 10–40 m heights. The estimated WS at height of 50 m and the measured WS at 10–40 heights are further used to train RNN-SA to obtain WS at 60 m height. This procedure is repeated continuously until the WS is estimated at a height of 180 m. The RNN-SA performance is compared with the standard RNN, Multilayer Perceptron (MLP), Support Vector Machine (SVM), and state of the art methods like convolutional neural networks (CNN) and long short-term memory (LSTM) networks to extrapolate the WS vertically. The estimated values are also compared with real WS dataset acquired using LiDAR and tested using four error metrics namely, mean squared error (MSE), mean absolute percentage error (MAPE), mean bias error (MBE), and coefficient of determination (). The numerical experimental results show that the MSE values between the estimated and actual WS at 180 m height for the RNN-SA, RNN, MLP, and SVM methods are found to be 2.09, 2.12, 2.37, and 2.63, respectively.

Cyberattack Detection in Smart Grids based on Reservoir Computing

This paper proposes a data-driven cyberattack detection method in smart grids based on reservoir computing (RC). It has been discovered that standard recurrent neural networks such as long short-term memory (LSTM) and gated recurrent unit (GRU) achieve high classification performance on the attack and contingency scenario. However, those models require large computational time for the learning process, and hence their re-training during daily operation of the grid is impractical. To overcome this challenge, this paper adopts echo state network (ESN) as a specific architecture of RC, which is known to be a fast learning framework. Numerical experiments with standard datasets show that the proposed method greatly reduces the computational time with low performance degradation.

Spatio-temporal prediction of temperature in fluidized bed biomass gasifier using dynamic recurrent neural network method

• Long-short term memory model is proposed to predict temperatures in fluidized bed gasifier. • The proposed model is validated with pilot-scale plant data resolved in space and time. • Multi-step ahead temperature progressions of freeboard, fluidized bed and outlet gas, are predicted. • Gated recurrent unit and long short-term memory models are compared. In this study, a long short-term memory (LSTM) based dynamic recurrent neural network model is proposed for multi-step ahead temperature predictions in a pilot-scale fluidized bed biomass gasifier (FBG). The LSTM model predicts not only the temporal but also the spatial distribution of temperature by considering the temperature of each region of the FBG (fluidized bed, freeboard and outlet gas) as a separate target parameter. The proposed model is validated by comparing simulation data with experimental observations acquired during operation of the FBG. The validation results reveal that the proposed LSTM model is capable of accurately (MAE<6) predicting 1-min-ahead temperature of all the FBG regions. The LSTM model is further challenged for temperature predictions at farther future points (3 min and 5 min ahead) to test the prediction limits of the LSTM model. For 5 min ahead predictions, the proposed LSTM-based prediction model is also compared with other state-of-the-art dynamic neural network methods that include the standard recurrent neural network (S-RNN) and its advanced variant, the gated recurrent unit (GRU). The comparative findings for far future predictions show that LSTM has the highest accuracy, and also exhibit that GRU does not have universally faster convergence than LSTM.

Geographic Named Entity Recognition by Employing Natural Language Processing and an Improved BERT Model

Toponym recognition, or the challenge of detecting place names that have a similar referent, is involved in a number of activities connected to geographical information retrieval and geographical information sciences. This research focuses on recognizing Chinese toponyms from social media communications. While broad named entity recognition methods are frequently used to locate places, their accuracy is hampered by the many linguistic abnormalities seen in social media posts, such as informal sentence constructions, name abbreviations, and misspellings. In this study, we describe a Chinese toponym identification model based on a hybrid neural network that was created with these linguistic inconsistencies in mind. Our method adds a number of improvements to a standard bidirectional recurrent neural network model to help with location detection in social media messages. We demonstrate the results of a wide-ranging evaluation of the performance of different supervised machine learning methods, which have the natural advantage of avoiding human design features. A set of controlled experiments with four test datasets (one constructed and three public datasets) demonstrates the performance of supervised machine learning that can achieve good results on the task, significantly outperforming seven baseline models.

SGORNN: Combining scalar gates and orthogonal constraints in recurrent networks

Recurrent Neural Network (RNN) models have been applied in different domains, producing high accuracies on time-dependent data. However, RNNs have long suffered from exploding gradients during training, mainly due to their recurrent process. In this context, we propose a variant of the scalar gated FastRNN architecture, called Scalar Gated Orthogonal Recurrent Neural Networks (SGORNN). SGORNN utilizes orthogonal matrices at the recurrent step. Our experiments evaluate SGORNN using two recently proposed orthogonal parametrizations for the recurrent weights of an RNN. We present a constraint on the scalar gates of SGORNN, which is easily enforced at training time to provide a probabilistic generalization gap which grows linearly with the length of sequences processed. Next, we provide bounds on the gradients of SGORNN to show the impossibility of exponentially exploding gradients through time. Our experimental results on the addition problem confirm that our combination of orthogonal and scalar gated RNNs are able to outperform other orthogonal RNNs and LSTM on long sequences. We further evaluate SGORNN on the HAR-2 classification task, where it improves upon the accuracy of several models using far fewer parameters than standard RNNs. Finally, we evaluate SGORNN on the Penn Treebank word-level language modeling task, where it again outperforms its related architectures and shows comparable performance to LSTM using far less parameters. Overall, SGORNN shows higher representation capacity than the other orthogonal RNNs tested, suffers from less overfitting than other models in our experiments, benefits from a decrease in parameter count, and alleviates exploding gradients during backpropagation through time.

Optimized long short-term memory (LSTM) network for performance prediction in unconventional reservoirs

Unconventional resources play an increasingly important role in the global energy supply. Performance prediction is crucial for adjusting development methods in unconventional reservoirs to ensure high production. However, daily production prediction of tight gas wells relies on many factors and complex variation patterns exist, which makes it difficult to predict using conventional methods. Therefore, this study proposes a hyper-parameters optimized long short-term memory (LSTM) network to effectively forecast daily gas production. Bayesian optimization is adopted to optimize the essential configuration of the LSTM. The model is trained by five features (wellhead tubing pressure, wellhead casing pressure, reservoir temperature, water production and gas rate) for seventy-five tight gas wells in the Ordos Basin, China, and is then validated on a test dataset with six wells in the same region. The average mean square error (MSE) of the test dataset is as low as 0.006, which indicates high prediction accuracy. Compared with cases with different input features, the five-feature case provides the most stable result for time-series performance prediction. We then comprehensively evaluate the daily gas production prediction of the LSTM network, including verifying it with the auto-regression integrated moving average (ARIMA), the artificial neural network (ANN), and the standard recurrent neural network (RNN). The results demonstrate that the optimized LSTM network can handle sequential data with time-series dependencies to improve the accuracy of performance prediction.

Influence-aware memory architectures for deep reinforcement learning in POMDPs

AbstractDue to its perceptual limitations, an agent may have too little information about the environment to act optimally. In such cases, it is important to keep track of the action-observation history to uncover hidden state information. Recent deep reinforcement learning methods use recurrent neural networks (RNN) to memorize past observations. However, these models are expensive to train and have convergence difficulties, especially when dealing with high dimensional data. In this paper, we propose influence-aware memory, a theoretically inspired memory architecture that alleviates the training difficulties by restricting the input of the recurrent layers to those variables that influence the hidden state information. Moreover, as opposed to standard RNNs, in which every piece of information used for estimating Q values is inevitably fed back into the network for the next prediction, our model allows information to flow without being necessarily stored in the RNN’s internal memory. Results indicate that, by letting the recurrent layers focus on a small fraction of the observation variables while processing the rest of the information with a feedforward neural network, we can outperform standard recurrent architectures both in training speed and policy performance. This approach also reduces runtime and obtains better scores than methods that stack multiple observations to remove partial observability.