Application Of Recurrent Neural Networks Research Articles

Recurrent Neural Networks: A Comprehensive Review of Architectures, Variants, and Applications

Recurrent neural networks (RNNs) have significantly advanced the field of machine learning (ML) by enabling the effective processing of sequential data. This paper provides a comprehensive review of RNNs and their applications, highlighting advancements in architectures, such as long short-term memory (LSTM) networks, gated recurrent units (GRUs), bidirectional LSTM (BiLSTM), echo state networks (ESNs), peephole LSTM, and stacked LSTM. The study examines the application of RNNs to different domains, including natural language processing (NLP), speech recognition, time series forecasting, autonomous vehicles, and anomaly detection. Additionally, the study discusses recent innovations, such as the integration of attention mechanisms and the development of hybrid models that combine RNNs with convolutional neural networks (CNNs) and transformer architectures. This review aims to provide ML researchers and practitioners with a comprehensive overview of the current state and future directions of RNN research.

Estimating forest aboveground biomass in tropical forests with Landsat time-series data and recurrent neural network

ABSTRACT The insensitivity and saturation of sensor signals significantly limit the application of Landsat data on estimating biomass in tropical forests with a high level of biomass. The previous studies with single-date Landsat data have lower accuracies in tropical forests compared to boreal and temperate forests. Landsat time-series data provide a promising opportunity to improve the accuracy by enhancing the relationship between Landsat spectral reflectance and forest aboveground biomass with disturbance and recovery dynamics. Compared to the single-date image, Landsat time-series data can capture abrupt spectral changes (e.g. harvesting and fire) and show the regrowth process in forested pixels. However, very limited studies take advantage of Landsat time-series data to estimate aboveground biomass in tropical forests. Recurrent neural networks (RNNs) are powerful deep learning techniques to capture time dependencies in sequence data. However, the application of RNNs in estimating forest biomass has not been explored yet. Therefore, we integrate the long short-term memory network (LSTM) and the fully connected neuron network (FNN) to establish an RNN-FNN model for estimating forest biomass with Landsat time-series imagery and airborne LiDAR data. We compared the proposed model with the commonly used Random Forest model and linear regression model which are implemented with single-date data. The results show that the RNN-FNN model can deal with Landsat time-series sequence data to enhance the relationship between Landsat spectral reflectance and forest aboveground biomass. The proposed model achieves the R2 of 0.63 and RMSE of 25.5 Mg/ha, which significantly outperformed the Random Forest model and linear regression model with Landsat single-date data. This study demonstrates the value of RNN and Landsat time-series imagery in estimating forest biomass for tropical forests.

Advancements in intrusion detection: A lightweight hybrid RNN-RF model.

Computer networks face vulnerability to numerous attacks, which pose significant threats to our data security and the freedom of communication. This paper introduces a novel intrusion detection technique that diverges from traditional methods by leveraging Recurrent Neural Networks (RNNs) for both data preprocessing and feature extraction. The proposed process is based on the following steps: (1) training the data using RNNs, (2) extracting features from their hidden layers, and (3) applying various classification algorithms. This methodology offers significant advantages and greatly differs from existing intrusion detection practices. The effectiveness of our method is demonstrated through trials on the Network Security Laboratory (NSL) and Canadian Institute for Cybersecurity (CIC) 2017 datasets, where the application of RNNs for intrusion detection shows substantial practical implications. Specifically, we achieved accuracy scores of 99.6% with Decision Tree, Random Forest, and CatBoost classifiers on the NSL dataset, and 99.8% and 99.9%, respectively, on the CIC 2017 dataset. By reversing the conventional sequence of training data with RNNs and then extracting features before applying classification algorithms, our approach provides a major shift in intrusion detection methodologies. This modification in the pipeline underscores the benefits of utilizing RNNs for feature extraction and data preprocessing, meeting the critical need to safeguard data security and communication freedom against ever-evolving network threats.

Hybrid RNN-LSTM Networks for Enhanced Intrusion Detection in Vehicle CAN Systems

Electric vehicles (EVs) use electric motors for propulsion, relying on electric energy stored in batteries or other energy storage devices. The standard communication protocol used in EVs is the Control Area Network (CAN), a communication protocol widely used in the automotive industry for networking and communication between various components within a vehicle. CAN protocol, designed without any care about protection, as automotive systems become more connected, the vulnerability to cyber threats, including intrusion attacks. The most common intrusion attacks on EVs are Denial of Service (DoS), Fuzzy, and Impersonation Attacks. These become a significant challenge due to the imperative need for robust Intrusion Detection Systems (IDS) in CAN networks. This paper explores the application of advanced deep learning techniques, specifically Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM) networks, to enhance the effectiveness of intrusion detection in the EV domain. We will use a hybrid Deep-learning model to improve the analysis. First, we will apply the RNN model, and the output will come as input for the second model, LSTM. Our proposed hybrid model achieved an accuracy of 93%. The outcomes of this research contribute to advancing cybersecurity measures in vehicular networks, ensuring the integrity and safety of connected vehicles. The applicability of RNN and LSTM techniques in the context of CAN networks demonstrates their potential to evolve as integral components of next-generation intrusion detection systems, fostering a secure and resilient automotive ecosystem.

Exploring Flip Flop memories and beyond: training Recurrent Neural Networks with key insights.

Training neural networks to perform different tasks is relevant across various disciplines. In particular, Recurrent Neural Networks (RNNs) are of great interest in Computational Neuroscience. Open-source frameworks dedicated to Machine Learning, such as Tensorflow and Keras have produced significant changes in the development of technologies that we currently use. This work contributes by comprehensively investigating and describing the application of RNNs for temporal processing through a study of a 3-bit Flip Flop memory implementation. We delve into the entire modeling process, encompassing equations, task parametrization, and software development. The obtained networks are meticulously analyzed to elucidate dynamics, aided by an array of visualization and analysis tools. Moreover, the provided code is versatile enough to facilitate the modeling of diverse tasks and systems. Furthermore, we present how memory states can be efficiently stored in the vertices of a cube in the dimensionally reduced space, supplementing previous results with a distinct approach.

Optimization and Improvement of Recurrent Neural Networks in Stock Price Prediction

Recurrent Neural Networks (RNNs) are a significant branch of deep learning, particularly suitable for handling data with temporal dependencies. In the financial domain, stock price prediction is a highly focused problem. Traditional stock price prediction methods, often based on linear models, fail to fully capture the nonlinear dynamics of stock prices. RNNs, with their memory capabilities, can capture the long-term dependencies of stock prices, hence offering great application potential in stock price prediction. However, RNNs also face challenges when processing stock price data, as stock prices are influenced by numerous factors and are highly complex and uncertain. Additionally, stock price data often contains noise and outliers, impacting the model’s predictive performance. To address these issues, incremental improvements in prediction accuracy and model generalization capabilities are achieved through optimizing the model structure and improving data preprocessing methods. As technology advances and algorithms continue to innovate, the application of RNNs in stock price prediction will become more widespread and in-depth.

A model-free method based on RDPG for fiber diameter steady control

Optical fibers have made outstanding contributions to optical instruments such as micro-channel plates (MCPs) and micro-pore optics (MPOs). As the stricter performance requirements for the MCP and MPO, traditional diameter control methods are incapable. Additionally, the partial observation issue of fiber drawing process further limits the outcome of the existing control approaches. Innovative control algorithms are vital for the diameter precision enhancement. We investigate the significance of recurrent deterministic policy gradient (RDPG) on diameter control. RDPG consists of deep deterministic policy gradient (DDPG) and recurrent neural network (RNN) where DDPG is a model-free reinforcement learning algorithm for continuous control and RNN for addressing partial observation. The impact of RNN variants selection on control performance is further discussed. Results illustrate that the application of RNN substantially solves the partial observation and varying RNN variants play a positive role in diameter control, but bidirectional gated recurrent unit (BiGRU) reaches the best accuracy. Comparison experiment results indicate that RDPG is able to learn reasonable control strategies and achieves significantly more accurate and robust diameter control in contrast with the existing methods.

Predictive Data Analysis: Leveraging RNN and LSTM Techniques for Time Series Dataset

Recurrent neural networks (RNNs) and long short-term memory (LSTM) models have demonstrated tremendous effectiveness in modeling time series data due to the ability they possess to capture temporal dependencies. Stock market data is an example of time series data. Stock price data is very volatile and dynamic therefore one of the most turbulent areas to invest in might be the stock market. The choice to buy or sell stocks is heavily influenced by external circumstances and statistical analysis of previous stock performance. Stock price index forecasting has been a major area of research for many years and many machine learning and deep learning methods have been proposed to simplify this hard process, but little success has been discovered so far.In this paper, the application of RNNs and LSTMs on a stock price dataset to predict future values is explored. We start with Recurrent neural networks (RNNs) to predict the values but one of the major challenges with it is the "vanishing gradient" problem, which makes it difficult for the network to learn long-term dependencies in the data. To overcome this long short-term memory (LSTM) was used which eliminates the vanishing gradient problem. Data were normalized and divided into time steps to determine the relationship between past values and future values to make accurate predictions. The outcomes show that the MLS LSTM method significantly outperforms existing algorithms, achieving a prediction accuracy of 98.1% on the training data set and a 91.97% accuracy on the testing data set.

Seizure Detection Via Time Series Classification Using Modified Metaheuristic Optimized Recurrent Networks

Epilepsy, colloquially termed seizure disorder, constitutes a neurological condi-tion characterized by unpredictable and sudden episodes of heightened electrical activity in the brain. The inherent risks associated with seizures, particularly in their nascent stages, stem from their challenging predictability, often leading to individuals endangering themselves through falls or lapses into unconsciousness in precarious settings. Additionally, the potential for neurological damage further compounds the complexity of managing this condition. This research endeavors to explore the viability of Recurrent Neural Networks (RNNs) for the early detection of seizures using electroencephalogram (EEG) recordings derived from real-world scenarios. Acknowledging the pivotal role of network topology and training parameters in achieving optimal performance, this study employs meta-heuristic algorithms to fine-tune performance through judicious hyperparameter selection. Notably, a modified iteration of a recently introduced algorithm is introduced herein. The application of RNNs to seizure detection yields promising outcomes, with constructed models achieving an accuracy surpassing 99%. The results obtained underscore the potential of employing metaheuristic-optimized RNNs in seizure detection as a means of substantially enhancing the quality of life for individuals afflicted by epilepsy. The demonstrated accuracy of these models suggests a robust and reliable methodology, paving the way for future advancements in the realm of neurotechnological interventions. This investigation contributes valuable insights into the intersection of machine learning, neuroscience, and healthcare, offering a foundation for further research and technological developments aimed at ameliorating the challenges posed by epilepsy.

An Overview of the Stability Analysis of Recurrent Neural Networks With Multiple Equilibria.

The stability analysis of recurrent neural networks (RNNs) with multiple equilibria has received extensive interest since it is a prerequisite for successful applications of RNNs. With the increasing theoretical results on this topic, it is desirable to review the results for a systematical understanding of the state of the art. This article provides an overview of the stability results of RNNs with multiple equilibria including complete stability and multistability. First, preliminaries on the complete stability and multistability analysis of RNNs are introduced. Second, the complete stability results of RNNs are summarized. Third, the multistability results of various RNNs are reviewed in detail. Finally, future directions in these interesting topics are suggested.

A multiscale recurrent neural network model for predicting energy production from geothermal reservoirs

Optimization of energy production from geothermal reservoirs requires reliable prediction of energy production performance under alternative operation and development scenarios. Traditionally, reservoir simulation models are used for the evaluation and screening of alternative production and development plans. However, simulation models require extensive data collection and modeling efforts and are time-consuming to build, run, and update. Data-driven predictive models, on the other hand, can serve as efficient prediction tools that can be used for decision support and management of daily operations and surveillance activities. Data-driven models become particularly attractive when a reservoir simulation model for a field does not exist and/or is difficult to build. Machine learning (ML)-based data-driven models that have recently become popular in several fields exploit statistical patterns and relations in training data to generate predictions. As such, they tend to perform better in interpolation problems (that is, prediction within the training data range) than when they are used to extrapolate beyond the training data. Production data from geothermal reservoirs tend to exhibit short-term variabilities as well as long-term trends, such as monotonically declining production temperatures. Capturing both short-term features and long-term trends with ML-based models is not trivial. We evaluate the use of recurrent neural networks (RNN) for the prediction of energy production from geothermal reservoirs. RNN is a class of ML architectures that are used to represent and predict sequential/dynamic data. Thus, it can be challenging to apply RNN to problems where long-term trends must be captured and extrapolation beyond the training data range is needed. We introduce the multiscale RNN architecture to extend the application of RNN to detect and predict both short-term variabilities and long-term trends in geothermal data. The developed architecture consists of a long-term component to only capture low-frequency data patterns, and a short-term component to detect features with higher frequency and more nonlinearity. The final prediction is obtained by combining the long-term and short-term predictions. Both synthetic and field data are used to evaluate the presented multiscale RNN model. The prediction performance of the multiscale RNN is compared against those obtained from the regular RNN and the autoregressive (AR) model. The results suggest that the multiscale architecture improves the long-term prediction performance of the regular RNN and enhances its robustness against noise.

Recurrent neural networks for dynamical systems: Applications to ordinary differential equations, collective motion, and hydrological modeling.

Classical methods of solving spatiotemporal dynamical systems include statistical approaches such as autoregressive integrated moving average, which assume linear and stationary relationships between systems' previous outputs. Development and implementation of linear methods are relatively simple, but they often do not capture non-linear relationships in the data. Thus, artificial neural networks (ANNs) are receiving attention from researchers in analyzing and forecasting dynamical systems. Recurrent neural networks (RNNs), derived from feed-forward ANNs, use internal memory to process variable-length sequences of inputs. This allows RNNs to be applicable for finding solutions for a vast variety of problems in spatiotemporal dynamical systems. Thus, in this paper, we utilize RNNs to treat some specific issues associated with dynamical systems. Specifically, we analyze the performance of RNNs applied to three tasks: reconstruction of correct Lorenz solutions for a system with a formulation error, reconstruction of corrupted collective motion trajectories, and forecasting of streamflow time series possessing spikes, representing three fields, namely, ordinary differential equations, collective motion, and hydrological modeling, respectively. We train and test RNNs uniquely in each task to demonstrate the broad applicability of RNNs in the reconstruction and forecasting the dynamics of dynamical systems.

Explicit Duration Recurrent Networks.

Recurrent neural networks (RNNs) can be used to operate over sequences of vectors and have been successfully applied to a variety of problems. However, it is hard to use RNNs to model the variable dwell time of the hidden state underlying an input sequence. In this article, we interpret the typical RNNs, including original RNN, standard long short-term memory (LSTM), peephole LSTM, projected LSTM, and gated recurrent unit (GRU), using a slightly extended hidden Markov model (HMM). Based on this interpretation, we are motivated to propose a novel RNN, called explicit duration recurrent network (EDRN), analog to a hidden semi-Markov model (HSMM). It has a better performance than conventional LSTMs and can explicitly model any duration distribution function of the hidden state. The model parameters become interpretable and can be used to infer many other quantities that the conventional RNNs cannot obtain. Therefore, EDRN is expected to extend and enrich the applications of RNNs. The interpretation also suggests that the conventional RNNs, including LSTM and GRU, can be made small modifications to improve their performance without increasing the parameters of the networks.

Application and Evaluation of Sports Event Management Method Based on Recurrent Neural Network

The methods of sports event management are the life cycle management method, which can be divided into four stages: start-up planning, plan preparation, real-time control, and followup evaluation. The administrative measures management law includes administrative orders, instructions, regulations, and systems of administrative organizations at all levels. The system management law has four characteristics: mandatory, authoritative, stable, and preventive. The current management mode of sports events is mainly that the government is deeply involved in the operation of the entire event, and relevant personnel are selected from various government departments to form an organizing committee, and the resources related to the event are controlled and regulated by the government. However, in our country, there is currently no unified sports event management standard. Therefore, this paper proposes the application and evaluation of sports event management method based on recurrent neural network (RNN). The comments of sports events are extracted from the network, classified with an RNN, and finally, an improvement plan is obtained through evaluation. The main work of this paper is as follows: (1) the development status of sports event management at home and abroad and the application of RNN are introduced, and RNN is used in the evaluation and classification of sports event management. (2) We propose a sentiment classification model GCNN-GRU that fuses local feature extraction. Aiming at the defect that the basic model is easy to lose key phrase information, a CNN with a gating mechanism is used to extract and filter local features. The classification experiment results show that the proposed GCNN-GRU has the best classification effect on the Chinese sentiment dataset.

Times-series data augmentation and deep learning for construction equipment activity recognition

Automated, real-time, and reliable equipment activity recognition on construction sites can help to minimize idle time, improve operational efficiency, and reduce emissions. Previous efforts in activity recognition of construction equipment have explored different classification algorithms anm accelerometers and gyroscopes. These studies utilized pattern recognition approaches such as statistical models (e.g., hidden-Markov models); shallow neural networks (e.g., Artificial Neural Networks); and distance algorithms (e.g., K-nearest neighbor) to classify the time-series data collected from sensors mounted on the equipment. Such methods necessitate the segmentation of continuous operational data with fixed or dynamic windows to extract statistical features. This heuristic and manual feature extraction process is limited by human knowledge and can only extract human-specified shallow features. However, recent developments in deep neural networks, specifically recurrent neural network (RNN), presents new opportunities to classify sequential time-series data with recurrent lateral connections. RNN can automatically learn high-level representative features through the network instead of being manually designed, making it more suitable for complex activity recognition. However, the application of RNN requires a large training dataset which poses a practical challenge to obtain from real construction sites. Thus, this study presents a data-augmentation framework for generating synthetic time-series training data for an RNN-based deep learning network to accurately and reliably recognize equipment activities. The proposed methodology is validated by generating synthetic data from sample datasets, that were collected from two earthmoving operations in the real world. The synthetic data along with the collected data were used to train a long short-term memory (LSTM)-based RNN. The trained model was evaluated by comparing its performance with traditionally used classification algorithms for construction equipment activity recognition. The deep learning framework presented in this study outperformed the traditionally used machine learning classification algorithms for activity recognition regarding model accuracy and generalization.

Efficient neural decoding of self-location with a deep recurrent network.

Place cells in the mammalian hippocampus signal self-location with sparse spatially stable firing fields. Based on observation of place cell activity it is possible to accurately decode an animal’s location. The precision of this decoding sets a lower bound for the amount of information that the hippocampal population conveys about the location of the animal. In this work we use a novel recurrent neural network (RNN) decoder to infer the location of freely moving rats from single unit hippocampal recordings. RNNs are biologically plausible models of neural circuits that learn to incorporate relevant temporal context without the need to make complicated assumptions about the use of prior information to predict the current state. When decoding animal position from spike counts in 1D and 2D-environments, we show that the RNN consistently outperforms a standard Bayesian approach with either flat priors or with memory. In addition, we also conducted a set of sensitivity analysis on the RNN decoder to determine which neurons and sections of firing fields were the most influential. We found that the application of RNNs to neural data allowed flexible integration of temporal context, yielding improved accuracy relative to the more commonly used Bayesian approaches and opens new avenues for exploration of the neural code.

Training recurrent neural networks robust to incomplete data: Application to Alzheimer's disease progression modeling.

Disease progression modeling (DPM) using longitudinal data is a challenging machine learning task. Existing DPM algorithms neglect temporal dependencies among measurements, make parametric assumptions about biomarker trajectories, do not model multiple biomarkers jointly, and need an alignment of subjects' trajectories. In this paper, recurrent neural networks (RNNs) are utilized to address these issues. However, in many cases, longitudinal cohorts contain incomplete data, which hinders the application of standard RNNs and requires a pre-processing step such as imputation of the missing values. Instead, we propose a generalized training rule for the most widely used RNN architecture, long short-term memory (LSTM) networks, that can handle both missing predictor and target values. The proposed LSTM algorithm is applied to model the progression of Alzheimer's disease (AD) using six volumetric magnetic resonance imaging (MRI) biomarkers, i.e., volumes of ventricles, hippocampus, whole brain, fusiform, middle temporal gyrus, and entorhinal cortex, and it is compared to standard LSTM networks with data imputation and a parametric, regression-based DPM method. The results show that the proposed algorithm achieves a significantly lower mean absolute error (MAE) than the alternatives with p < 0.05 using Wilcoxon signed rank test in predicting values of almost all of the MRI biomarkers. Moreover, a linear discriminant analysis (LDA) classifier applied to the predicted biomarker values produces a significantly larger area under the receiver operating characteristic curve (AUC) of 0.90 vs. at most 0.84 with p < 0.001 using McNemar's test for clinical diagnosis of AD. Inspection of MAE curves as a function of the amount of missing data reveals that the proposed LSTM algorithm achieves the best performance up until more than 74% missing values. Finally, it is illustrated how the method can successfully be applied to data with varying time intervals. This paper shows that built-in handling of missing values in training an LSTM network benefits the application of RNNs in neurodegenerative disease progression modeling in longitudinal cohorts.

Applications of Recurrent Neural Networks in Environmental Factor Forecasting: A Review

Analysis and forecasting of sequential data, key problems in various domains of engineering and science, have attracted the attention of many researchers from different communities. When predicting the future probability of events using time series, recurrent neural networks (RNNs) are an effective tool that have the learning ability of feedforward neural networks and expand their expression ability using dynamic equations. Moreover, RNNs are able to model several computational structures. Researchers have developed various RNNs with different architectures and topologies. To summarize the work of RNNs in forecasting and provide guidelines for modeling and novel applications in future studies, this review focuses on applications of RNNs for time series forecasting in environmental factor forecasting. We present the structure, processing flow, and advantages of RNNs and analyze the applications of various RNNs in time series forecasting. In addition, we discuss limitations and challenges of applications based on RNNs and future research directions. Finally, we summarize applications of RNNs in forecasting.

A deep neural network approach for learning intrinsic protein-RNA binding preferences.

The complexes formed by binding of proteins to RNAs play key roles in many biological processes, such as splicing, gene expression regulation, translation and viral replication. Understanding protein-RNA binding may thus provide important insights to the functionality and dynamics of many cellular processes. This has sparked substantial interest in exploring protein-RNA binding experimentally, and predicting it computationally. The key computational challenge is to efficiently and accurately infer protein-RNA binding models that will enable prediction of novel protein-RNA interactions to additional transcripts of interest. We developed DLPRB (Deep Learning for Protein-RNA Binding), a new deep neural network (DNN) approach for learning intrinsic protein-RNA binding preferences and predicting novel interactions. We present two different network architectures: a convolutional neural network (CNN), and a recurrent neural network (RNN). The novelty of our network hinges upon two key aspects: (i) the joint analysis of both RNA sequence and structure, which is represented as a probability vector of different RNA structural contexts; (ii) novel features in the architecture of the networks, such as the application of RNNs to RNA-binding prediction, and the combination of hundreds of variable-length filters in the CNN. Our results in inferring accurate RNA-binding models from high-throughput in vitro data exhibit substantial improvements, compared to all previous approaches for protein-RNA binding prediction (both DNN and non-DNN based). A more modest, yet statistically significant, improvement is achieved for in vivo binding prediction. When incorporating experimentally-measured RNA structure, compared to predicted one, the improvement on in vivo data increases. By visualizing the binding specificities, we can gain biological insights underlying the mechanism of protein RNA-binding. The source code is publicly available at https://github.com/ilanbb/dlprb. Supplementary data are available at Bioinformatics online.

Multilayered Echo State Machine: A Novel Architecture and Algorithm.

In this paper, we present a novel architecture and learning algorithm for a multilayered echo state machine (ML-ESM). Traditional echo state networks (ESNs) refer to a particular type of reservoir computing (RC) architecture. They constitute an effective approach to recurrent neural network (RNN) training, with the (RNN-based) reservoir generated randomly, and only the readout trained using a simple computationally efficient algorithm. ESNs have greatly facilitated the real-time application of RNN, and have been shown to outperform classical approaches in a number of benchmark tasks. In this paper, we introduce a novel criteria for integrating multiple layers of reservoirs within the ML-ESM. The addition of multiple layers of reservoirs are shown to provide a more robust alternative to conventional RC networks. We demonstrate the comparative merits of this approach in a number of applications, considering both benchmark datasets and real world applications.