Use Of Recurrent Neural Networks Research Articles

A physics-inspired neural network for short-wave radiation parameterization

Abstract Radiation parameterization schemes are crucial components of weather and climate models, but they are also known to be computationally intensive. In recent decades, researchers have attempted to emulate these schemes using neural networks, with more attention to convolutional neural networks. However, in this paper, we explore the potential of recurrent neural networks (RNNs) for predicting solar heating rates. Our architecture was trained and tested using long-term hindcast data from the Pechora Sea region, with the conventional RRTMG scheme serving as a shortwave parameterization. Our findings show that the RNN offers rapid learning, fast inference, and excellent data fitting. We also present preliminary results demonstrating the use of RNNs for operational weather forecasting, which achieved a significant speedup in parameterization and reduced the overall forecast time by 40 %.

Time-efficient detection of false position attack in 5G and beyond vehicular networks

5G-V2X-enabled transportation systems rely on seamless cooperation between vehicles, infrastructure, and pedestrians, facilitated by the exchange of real-time position and state information among these entities. Misbehaving vehicles try injecting bogus messages into the network, thereby compromising its reliability and security. This work proposes an efficient Deep Learning (DL)-based Misbehavior Detection System (MDS) that leverages the use of Recurrent Neural Networks (RNNs) to analyze message consistency and detect bogus information in 5G-V2X networks. We highlight the significance of incorporating historical data to analyze consistency. Additionally, we emphasize the importance of evaluating the computational overhead induced by MDSs in V2X networks, given the critical need for low latency communication. We validate our work through both experimental and theoretical studies and compare it to existing works. The obtained results show the effectiveness of our work, achieving an accuracy of 95% in detecting false information injection attacks. Additionally, we guarantee a low required time/computing complexity of O(1), thereby avoiding significant overhead impacting the end-to-end latency when exchanging CAMs.

Effective text classification using BERT, MTM LSTM, and DT

Text classification plays a critical role in managing large volumes of electronically produced texts. As the number of such texts increases, manual analysis becomes impractical, necessitating an intelligent approach for processing information. Deep learning models have witnessed widespread application in text classification, including the use of recurrent neural networks like Many to One Long Short-Term Memory (MTO LSTM). Nonetheless, this model is limited by its reliance on only the last token for text labelling. To overcome this limitation, this study introduces a novel hybrid model that combines Bidirectional Encoder Representations from Transformers (BERT), Many To Many Long Short-Term Memory (MTM LSTM), and Decision Templates (DT) for text classification. In this new model, the text is first embedded using the BERT model and then trained using MTM LSTM to approximate the target at each token. Finally, the approximations are fused using DT. The proposed model is evaluated using the well-known IMDB movie review dataset for binary classification and Drug Review Dataset for multiclass classification. The results demonstrate superior performance in terms of accuracy, recall, precision, and F1 score compared to previous models. The hybrid model presented in this study holds significant potential for a wide range of text classification tasks and stands as a valuable contribution to the field.

Automatic detection of cell-cycle stages using recurrent neural networks.

Mitosis is the process by which eukaryotic cells divide to produce two similar daughter cells with identical genetic material. Research into the process of mitosis is therefore of critical importance both for the basic understanding of cell biology and for the clinical approach to manifold pathologies resulting from its malfunctioning, including cancer. In this paper, we propose an approach to study mitotic progression automatically using deep learning. We used neural networks to predict different mitosis stages. We extracted video sequences of cells undergoing division and trained a Recurrent Neural Network (RNN) to extract image features. The use of RNN enabled better extraction of features. The RNN-based approach gave better performance compared to classifier based feature extraction methods which do not use time information. Evaluation of precision, recall, and F-score indicates the superiority of the proposed model compared to the baseline. To study the loss in performance due to confusion between adjacent classes, we plotted the confusion matrix as well. In addition, we visualized the feature space to understand why RNNs are better at classifying the mitosis stages than other classifier models, which indicated the formation of strong clusters for the different classes, clearly confirming the advantage of the proposed RNN-based approach.

The verification of periodicity with the use of recurrent neural networks

Abstract The ability to automatically and robustly self-verify periodicity present in time-series astronomical data is becoming more important as data sets rapidly increase in size. The age of large astronomical surveys has rendered manual inspection of time-series data less practical. Previous efforts in generating a false alarm probability to verify the periodicity of stars have been aimed towards the analysis of a constructed periodogram. However, these methods feature correlations with features that do not pertain to periodicity, such as light-curve shape, slow trends, and stochastic variability. The common assumption that photometric errors are Gaussian and well determined is also a limitation of analytic methods. We present a novel machine learning based technique which directly analyses the phase-folded light curve for its false alarm probability. We show that the results of this method are largely insensitive to the shape of the light curve, and we establish minimum values for the number of data points and the amplitude to noise ratio.

BiBitcoin Price Prediction using Recurrent Neural Networks and Long Short-Term Memory

Bitcoin's decentralized nature has made it a popular mode of payment for buyers and sellers, but its highly volatile nature poses a challenge for investors. This study aims to predict future Bitcoin prices using a combination of Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM) algorithms. The non-stationary nature of Bitcoin prices is addressed using RNN, which is particularly useful for analyzing sequential data.The dataset used in this research is sourced from the Kraken exchange and includes various factors that are believed to influence Bitcoin prices, such as transaction volume, hash rate, and Google search trends. The data is preprocessed and cleaned to ensure accuracy, and then fed into the RNN and LSTM models for training and testing.The study's use of RNN and LSTM algorithms demonstrates the effectiveness of these methods in predicting Bitcoin prices, particularly in the context of sequential data. The results of the study, provide insights into potential future trends in Bitcoin prices and identify key indicators that significantly influence Bitcoin prices. The findings of this research have important implications for investors and traders looking to make informed decisions in the cryptocurrency market, as well as for researchers seeking to improve our understanding of Bitcoin's price dynamics. By predicting future prices, the study provides insights that can mitigate the risks associated with Bitcoin's volatility, making it a more viable investment option.

Recurrent neural network for the dynamics of Zika virus spreading

<abstract> <p>Recurrent Neural Networks (RNNs), a type of machine learning technique, have recently drawn a lot of interest in numerous fields, including epidemiology. Implementing public health interventions in the field of epidemiology depends on efficient modeling and outbreak prediction. Because RNNs can capture sequential dependencies in data, they have become highly effective tools in this field. In this paper, the use of RNNs in epidemic modeling is examined, with a focus on the extent to which they can handle the inherent temporal dynamics in the spread of diseases. The mathematical representation of epidemics requires taking time-dependent variables into account, such as the rate at which infections spread and the long-term effects of interventions. The goal of this study is to use an intelligent computing solution based on RNNs to provide numerical performances and interpretations for the SEIR nonlinear system based on the propagation of the Zika virus (SEIRS-PZV) model. The four patient dynamics, namely susceptible patients S(y), exposed patients admitted in a hospital E(y), the fraction of infective individuals I(y), and recovered patients R(y), are represented by the epidemic version of the nonlinear system, or the SEIR model. SEIRS-PZV is represented by ordinary differential equations (ODEs), which are then solved by the Adams method using the Mathematica software to generate a dataset. The dataset was used as an output for the RNN to train the model and examine results such as regressions, correlations, error histograms, etc. For RNN, we used 100% to train the model with 15 hidden layers and a delay of 2 seconds. The input for the RNN is a time series sequence from 0 to 5, with a step size of 0.05. In the end, we compared the approximated solution with the exact solution by plotting them on the same graph and generating the absolute error plot for each of the 4 cases of SEIRS-PZV. Predictions made by the model appeared to be become more accurate when the mean squared error (MSE) decreased. An increased fit to the observed data was suggested by this decrease in the MSE, which suggested that the variance between the model's predicted values and the actual values was dropping. A minimal absolute error almost equal to zero was obtained, which further supports the usefulness of the suggested strategy. A small absolute error shows the degree to which the model's predictions matches the ground truth values, thus indicating the level of accuracy and precision for the model's output.</p> </abstract>

Financial Risk Prediction Model in the Context of Big Data - Corporate Financial Risk Control Based on LSTM Deep Neural Networks

Abstract This paper is based on the use of recurrent neural networks and LSTM deep neural networks to obtain the financial risk prediction feature sequence in the context of big data. The financial risk prediction feature sequence is used as the input value of the input gate of the LSTM deep neural network model after data filtering, normalization and loss function optimization, and then the financial risk prediction for the output gate of the LSTM deep neural network model. Considering the availability of data, small and medium-sized enterprises listed in A-share companies in the Wind database are selected as sample enterprises, and evaluation indexes are constructed and detected at the same time so as to complete the experimental design of enterprise financial risk prediction in the context of big data. The prediction of enterprise financial risk is empirically analyzed using simulation analysis and statistical analysis. The results show that in the model performance analysis, the average value of ten years of data, the highest value is still the result obtained by LSTM training, 0.761, compared with other models of LSTM deep neural network in static financial risk prediction in the overall best performance. In the case study of Yibai Pharmaceutical, the minimum value of the rate of return, return on total assets, and return on assets were -10.02%, 2.56%, -20.72%, which reflects the fact that the private enterprises still have large profitability space to be mined. This study helps investors or financial institutions such as funds to find out the possible financial risk crisis of listed companies as early as possible to avoid the parties from incurring large financial losses.

Neural network models for seabed stability: a deep learning approach to wave-induced pore pressure prediction

Wave cyclic loading in submarine sediments can lead to pore pressure accumulation, causing geohazards and compromising seabed stability. Accurate prediction of long-term wave-induced pore pressure is essential for disaster prevention. Although numerical simulations have contributed to understanding wave-induced pore pressure response, traditional methods lack the ability to simulate long-term and real oceanic conditions. This study proposes the use of recurrent neural network (RNN) models to predict wave-induced pore pressure based on in-situ monitoring data. Three RNN models (RNN, LSTM, and GRU) are compared, considering different seabed depths, and input parameters. The results demonstrate that all three RNN models can accurately predict wave-induced pore pressure data, with the GRU model exhibiting the highest accuracy (absolute error less than 2 kPa). Pore pressure at the previous time step and water depth are highly correlated with prediction, while wave height, wind speed, and wind direction show a secondary correlation. This study contributes to the development of wave-induced liquefaction early warning systems and offers insights for utilizing RNNs in geological time series analysis.

Development of recurrent neural networks for price forecasting at cryptocurrency exchanges

The study focuses on improving the quality of using recurrent neural networks (RNNs) to predict cryptocurrency prices. The formula of the target variable for the model based on the arithmetic mean is developed, which allows us to better take into account the dynamics of cryptocurrency exchanges. The factors affecting this variable were grouped into features based on the volume of daily cryptocurrency trading, the volatility of the relevant prices, and the pre-calculated and selected signals of technical indicators. As part of the study, an algorithm for processing daily data was developed for the model. The results obtained made it possible to create a holistic model for forecasting stock prices. Two recurrent neural networks were trained: one with a long short-term memory (LSTM) and the other with a recurrent gate unit (GRU). To determine the efficiency of the models, the analysis was carried out using two key indicators: the Sortino coefficient, which measures the relative risk/reward for each additional unit of unwanted volatility, and the Sharpe ratio, which measures the return on assets, subtracting the free risk. As a result, it was found that both models have similar results in terms of accuracy (~69 %). Still, the GRU-based model showed significantly better values of the Sortino coefficients (3.13) and Sharpe’s coefficient (2.45), which allows us to conclude that it is effective on cryptocurrency exchanges. At the same time, the LSTM model requires more parameters for training than the GRU model with an identical structure, which leads to a longer training time. The obtained scientific and practical results are aimed at more efficient use of recurrent neural networks in price forecasting on cryptocurrency exchanges

Stock price forecasting using PSO hypertuned neural nets and ensembling

The stock market is a platform that allows individuals and organizations to buy stocks of publicly listed companies. It is imperative for investors and traders to utilize the platform to buy and sell stocks efficiently, but they must also determine when to do it in order to maximize profits. As trading involves holding stocks for shorter periods, projecting the future direction of a stock’s price becomes essential. In recent years, deep neural network-based trading strategies have been researched and implemented to identify when a stock’s price will increase or decrease. The main issues in implementing such solutions are that they need to deal with the noisy nature of the stock market and the problem of overfitting. The objective of the paper is to put forth an approach that utilizes deep learning techniques to predict price movements in the Nifty 50 index. The paper will explore the use of Recurrent Neural Networks (RNNs) for the given task. The paper will also look into applying metaheuristic algorithms to further improve the results of the prediction models. In this approach, RNNs, including Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), are utilized to predict the movement of the index. The models are trained on a unique and efficient feature set that takes into consideration the stock price of large market capitalization companies present in the National Stock Exchange (NSE). Our findings show that ensembled architectures produce better results than individual models. An LSTM and GRU ensembled architecture produced an accuracy of 56.66% and a precision of 0.4734. Particle swarm optimization (PSO) was put forth as a method to hypertune the models to improve their performance. The LSTM, hypertuned with PSO, produced an accuracy of 57.64% and a precision of 0.2882. To further enhance the model’s stock price prediction performance, the LSTM and GRU ensembled architecture was ensembled with the PSO hypertuned LSTM architecture to produce a model that gives the highest accuracy of 57.72%. The proposed ensemble approach outperforms the other cutting edge techniques used to forecast how the stock price of the NSE will move. Additionally, the ensemble method increased precision from 0.2882 to 0.5485, demonstrating that ensembling and the PSO algorithm combine to produce models with superior performance. Based on the results, combining PSO hyper parameter optimized models with ensembling provides a good approach towards price movement predictions and also shows the potential of using this approach in other Artificial Intelligence (AI) fields to improve the performance of deep learning models.

Loss-of-Control Prediction of a Quadcopter Using Recurrent Neural Networks

Loss of control (LOC) is a prevalent cause of drone crashes. Onboard prevention systems should be designed requiring low computing power, for which data-driven techniques provide a promising solution. This study proposes the use of recurrent neural networks (RNNs) for LOC prediction. Four architectures were trained in order to identify which RNN configuration is most suitable and if this model can predict LOC for changing aerodynamic characteristics, wind conditions, quadcopter types, and LOC events. One-hundred and seventy-two real-world LOC events were conducted using a 53 g Tiny Whoop, a 73 g URUAV UZ85, and a 265 g GEPRC CineGO quadcopter. For these flights, LOC was initiated by demanding an excessive yaw rate (2000 deg/s), which provokes an unrecoverable upset and subsequent crash. All RNNs were trained using only onboard sensor measurements. It was found that the commanded rotor values provided the clearest early warning signals for LOC because these values showed saturation before LOC. Moreover, all four architectures could correctly and reliably predict the impending LOC event 2 s before it actually occurred. Furthermore, to investigate generality of the methodology, the predictors were successfully applied to flight data in which the quadcopter mass, blade diameter, and blade count were varied.

Earthquake Prediction for the Düzce Province in the Marmara Region Using Artificial Intelligence

By definition, an earthquake is a naturally occurring event. This natural event may be a disaster that causes significant damage, loss of life, and other economic effects. The possibility of predicting a natural event such as an earthquake will minimize the negative effects mentioned above. In this study, data collection, processing, and data evaluation regarding earthquakes were carried out. Earthquake forecasting was performed using the RNN (recurrent neural network) method. This study was carried out using seismic data with a magnitude of 3.0 and above of the Düzce Province between 1990 and 2022. In order to increase the learning potential of the method, the b and d values of earthquakes were calculated. The detection of earthquakes within a specific time interval in the Marmara region of Turkey, the classification of earthquake-related seismic data using artificial neural networks, and the generation of predictions for the future highlight the importance of this study. Our results demonstrated that the prediction performance could be significantly improved by incorporating the b and d coefficients of earthquakes, as well as the data regarding the distance between the Moon and the Earth, along with the use of recurrent neural networks (RNNs).

Use of recurrent neural networks considering maintenance to predict urban road performance in Beijing, China.

A correct understanding of the pavement performance change law forms the premise of the scientific formulation of maintenance decisions. This paper aims to develop a predictive model taking into account the costs of different types of maintenance works that reflects the continuous true usage performance of the pavement. The model proposed in this study was trained on a dataset containing five-year maintenance workdata on urban roads in Beijing with pavement performance indicators for the corresponding years. The same roads were matched and combined to obtain a set of sequences of pavement performance changes with the features of the current year; with the recurrent-neural-network-based long short-term memory (LSTM) network and gate recurrent unit (GRU) network, the prediction accuracy of highway pavement performance on the test set was significantly increased. The prediction result indicates that the generalization ability of the improved recurrent neural network model is satisfactory, with the R2 achieving 0.936, and of the two models the GRU model is more efficient, with an accuracy thatreaches almost the same level as LSTMbut with the training convergence time reduced to 25 s. This study demonstrates that data generated by the work of maintenance units can be used effectivelyin the prediction of pavement performance. This article is part of the theme issue 'Artificial intelligence in failure analysis of transportation infrastructure and materials'.

Diversity of Models Based on Sequence to Sequence for Neural Machine Translation Tasks

One of the most critical areas of Natural Language Processing study is machine translation, which realizes the translation process from source language to target language. After a rapid development, neural machine translation has become mainstream. Most of the tasks of natural language generation are implemented based on sequence to sequence model. The current technology of machine translation is the use of recurrent neural networks and attention mechanisms, which is essentially a set of encoding and decoding techniques. In order to handle the issue of excessive information compression introduced by decoder, attention mechanism is proposed. Transformer is also built with attention, which not only computes faster, but also achieves better results on translation tasks. In this paper, a series of comparison of these models and description of the performance on the dataset were shown.

Exploring the Use of Recurrent Neural Networks for Time Series Forecasting

Exploring the Use of Recurrent Neural Networks for Time Series Forecasting

Exploring the Use of Recurrent Neural Networks for Time Series Forecasting

Exploring the Use of Recurrent Neural Networks for Time Series Forecasting

Recurrent Neural Networks for Artifact Correction in HRV Data During Physical Exercise


 
 
 In this paper, we propose the use of recurrent neural networks (RNNs) for artifact correction and analysis of heart rate variability (HRV) data. HRV can be a valuable metric for determining the function of the heart and the autonomic nervous system. When measured during exercise, motion artifacts present a significant challenge. Several methods for artifact correction have previously been proposed, none of them applying machine learning, and each presenting some limitations regarding an accurate representation of HRV metrics. RNNs offer the ability to capture patterns that might otherwise not be detected, yielding predictions where no prior physiological assumptions are needed.
 A hyperparameter search has been carried out to determine the best network configuration and the most important hyperparameters. The approach was tested on two extensive multi-subject data sets, one from a recreational bicycle race and the other from a laboratory experiment. The results demonstrate that RNNs outperform by order of magnitude existing methods with respect to the calculation of derived HRV metrics. However, they are not able to accurately fill in individual missing RR intervals in sequence. Future research should pursue improvements in the prediction of RR interval lengths and reduction in necessary training data.
 
 

Deep SCMA receiver: A low‐complexity joint decoder and channel estimator for SCMA over time‐varying channels using RNNs

AbstractSparse code multiple access (SCMA) is a promising non‐orthogonal multiple access scheme for cellular Internet of things (IoT) due to its ability to support massive connectivity, grant‐free transmission and scalability. Inspired by the recent developments of deep learning for physical layer communications, we present a design of an uplink SCMA receiver using deep learning. We propose the use of recurrent neural networks (RNNs) for joint channel estimation and multiuser data detection of uplink SCMA under time‐varying Rayleigh channel using a single deep learning structure. The use of RNNs enables the receiver to learn the time correlation between the received samples with a very low pilot density and with low complexity. Compared with the conventional SCMA receiver, the simulation results show that the proposed deep learning‐based receiver can achieve BER performance similar to that of the conventional SCMA receivers (such as sparse pilot channel estimator and message‐passing algorithm detector) with very low pilot density and with much lower complexity. Moreover, the proposed SCMA receiver shows good resilience to small changes in the receiver speed (second‐order channel statistics) which enables the proposed deep learning receiver to work over a reasonable range of receiver speeds without parameter tuning. Fine tuning the network parameters to capture the variations of the channel, during online transmission using small data sets and small training period, is also checked and provides additional BER benefits.

Event‐based robust performance guarantee for nonaffine plants via system identification

AbstractIn this paper, by making use of recurrent neural networks (RNNs), a novel event‐based robust optimal controller is designed for a category of continuous‐time (CT) nonlinear nonaffine systems with disturbances. For tackling the optimal‐robust problem transformation and the performance guarantee, a system identifier is developed to reconstruct the system, and an event‐triggered critic‐only strategy is investigated to replace the traditional time‐based actor‐critic structure. The RNN is adopted to reconstruct the nonaffine system as an identifier for obtaining the affine model. It simplifies the system structure and reduces computational burdens. In addition, the stability of the unknown nominal system learned by the RNN is proved. Then, using the obtained system identifier and the adaptive critic learning structure, the proposed event‐triggered robust control with performance guarantee is realized. At last, the effectiveness of the given design method is verified by two simulation examples.