Analysis Of Recurrent Neural Networks Research Articles

Stability Analysis of Recurrent Neural Networks With Time-Varying Delay by Flexible Terminal Interpolation Method.

This brief studies the stability problem of recurrent neural networks with time-varying delay. Based on one tunable parameter α , a flexible terminal interpolation method is proposed to change the interval with fixed terminals as 2k+1-3 ones with flexible terminals. Associated with the flexible subintervals, a novel Lyapunov-Krasovskii functional with more delay information is constructed. In order to estimate the Lyapunov-Krasovskii functional, a quadratic reciprocally convex inequality is proposed, which covers some existing ones as its special cases. Based on these ingredients, a new stability criterion is derived in the form of linear matrix inequalities. A comprehensive comparison of results is given to illustrate the newly proposed stability criterion.

Stability Analysis of Recurrent Neural Networks With Time-Varying Delay Based on a Flexible Negative-Determination Quadratic Function Method.

This brief investigates the stability problem of recurrent neural networks (RNNs) with time-varying delay. First, by introducing some flexibility factors, a flexible negative-determination quadratic function method is proposed, which contains some existing methods and has less conservatism. Second, some integral inequalities and the flexible negative-determination quadratic function method are used to give an accurate upper bound of the Lyapunov-Krasovskii functional (LKF) derivative. As a result, a less conservative stability criterion of delayed RNNs is derived, whose effectiveness and superiority are finally illustrated through two numerical examples.

A comparative analysis of recurrent neural network and support vector machine for binary classification of spam short message service

Over the years, communication through Short Message Service (SMS) has been a primary tool for mobile subscribers. SMS has varied applications in health, industry, finances, education and social networking among others. The growth of mobile devices and SMS usage has consequently increased the attack surface for cyber-criminals culminating to the proliferation of malicious activities introduced using SMS spam, phishing, spyware, malware etc. Ham messages are normal messages people trade with one another and are usually not unwanted by the recipient, while spam messages are unsolicited junk and redundant messages that may be sent to a large number of people at once and are usually unwanted. Various spam detection models have been developed using various traditional machine and deep learning techniques. However, most studies where comparison between deep and traditional machine learning algorithms is done, have unfortunately omitted K-Nearest Neighbors and Support Vector Machine (SVM) which are empirically deemed as the most popular traditional machine learning algorithms. In this study, therefore, we develop a deep learning model based on Recurrent Neural Network (RNN) for Spam and Ham SMS classification and compare its performance against SVM model for the same University of California (UCl) SMS dataset. The results show that RNN has a slightly higher training and validation accuracy of 0.98 compared to SVM at 0.94, however, the false positive rate of SVM is marginally lower. Exploring application of deep learning with better optimization algorithms such as RNN improves accuracy, reduces computational complexities i.e. memory consumption and speed, and thus minimizing false positive rates. For future work, we suggest the use of varied performance metrics to validate the model in a distributed dataset environment.

An Overview of the Stability Analysis of Recurrent Neural Networks With Multiple Equilibria.

The stability analysis of recurrent neural networks (RNNs) with multiple equilibria has received extensive interest since it is a prerequisite for successful applications of RNNs. With the increasing theoretical results on this topic, it is desirable to review the results for a systematical understanding of the state of the art. This article provides an overview of the stability results of RNNs with multiple equilibria including complete stability and multistability. First, preliminaries on the complete stability and multistability analysis of RNNs are introduced. Second, the complete stability results of RNNs are summarized. Third, the multistability results of various RNNs are reviewed in detail. Finally, future directions in these interesting topics are suggested.

Analysis of recurrent neural networks via property-directed verification of surrogate models

This paper presents a property-directed approach to verifying recurrent neural networks (RNNs). To this end, we learn a deterministic finite automaton as a surrogate model from a given RNN using active automata learning. This model may then be analyzed using model checking as a verification technique. The term property-directed reflects the idea that our procedure is guided and controlled by the given property rather than performing the two steps separately. We show that this not only allows us to discover small counterexamples fast, but also to generalize them by pumping toward faulty flows hinting at the underlying error in the RNN. We also show that our method can be efficiently used for adversarial robustness certification of RNNs.

Design and analysis of recurrent neural networks for ultrafast optical pulse nonlinear propagation.

In this work, we analyze different types of recurrent neural networks (RNNs) working under several different parameters to best model the nonlinear optical dynamics of pulse propagation. Here we studied the propagation of picosecond and femtosecond pulses under distinct initial conditions going through 13 m of a highly nonlinear fiber and demonstrated the application of two RNNs returning error metrics such as normalized root mean squared error (NRMSE) as low as 9%. Those results were further extended for a dataset outside the initial pulse conditions used on the RNN training, and the best-proposed network was still able to achieve a NRMSE below 14%. We believe that this study can contribute to a better understanding of building RNNs employed for modeling nonlinear optical pulse propagation and of how the peak power and nonlinearity affect the prediction error.

Time-resolved trajectory of glucose lowering medications and cardiovascular outcomes in type 2 diabetes: a recurrent neural network analysis

AimTreatment algorithms define lines of glucose lowering medications (GLM) for the management of type 2 diabetes (T2D), but whether therapeutic trajectories are associated with major adverse cardiovascular events (MACE) is unclear. We explored whether the temporal resolution of GLM usage discriminates patients who experienced a 4P-MACE (heart failure, myocardial infarction, stroke, death for all causes).MethodsWe used an administrative database (Veneto region, North-East Italy, 2011–2018) and implemented recurrent neural networks (RNN) with outcome-specific attention maps. The model input included age, sex, diabetes duration, and a matrix of GLM pattern before the 4P-MACE or censoring. Model output was discrimination, reported as area under receiver characteristic curve (AUROC). Attention maps were produced to show medications whose time-resolved trajectories were the most important for discrimination.ResultsThe analysis was conducted on 147,135 patients for training and model selection and on 10,000 patients for validation. Collected data spanned a period of ~ 6 years. The RNN model efficiently discriminated temporal patterns of GLM ending in a 4P-MACE vs. those ending in an event-free censoring with an AUROC of 0.911 (95% C.I. 0.904–0.919). This excellent performance was significantly better than that of other models not incorporating time-resolved GLM trajectories: (i) a logistic regression on the bag-of-words encoding all GLM ever taken by the patient (AUROC 0.754; 95% C.I. 0.743–0.765); (ii) a model including the sequence of GLM without temporal relationships (AUROC 0.749; 95% C.I. 0.737–0.761); (iii) a RNN model with the same construction rules but including a time-inverted or randomised order of GLM. Attention maps identified the time-resolved pattern of most common first-line (metformin), second-line (sulphonylureas) GLM, and insulin (glargine) as those determining discrimination capacity.ConclusionsThe time-resolved pattern of GLM use identified patients with subsequent cardiovascular events better than the mere list or sequence of prescribed GLM. Thus, a patient’s therapeutic trajectory could determine disease outcomes.

Robustness analysis and training of recurrent neural networks using dissipativity theory

Abstract Neural networks are widely applied in control applications, yet providing safety guarantees for neural networks is challenging due to their highly nonlinear nature. We provide a comprehensive introduction to the analysis of recurrent neural networks (RNNs) using robust control and dissipativity theory. Specifically, we consider H 2 {\mathcal{H}_{2}} -performance and the ℓ 2 {\ell _{2}} -gain to quantify the robustness of dynamic RNNs with respect to input perturbations. First, we analyze the robustness of RNNs using the proposed robustness certificates and then, we present linear matrix inequality constraints to be used in training of RNNs to enforce robustness. Finally, we illustrate in a numerical example that the proposed approach enhances the robustness of RNNs.

<i>L</i><sub>2 </sub>Induced Norm Analysis for Nonnegative Input Signals and Its Application to Stability Analysis of Recurrent Neural Networks

<i>L</i><sub>2 </sub>Induced Norm Analysis for Nonnegative Input Signals and Its Application to Stability Analysis of Recurrent Neural Networks

Comparative Analysis of Recurrent Neural Networks in Stock Price Prediction for Different Frequency Domains

Investors in the stock market have always been in search of novel and unique techniques so that they can successfully predict stock price movement and make a big profit. However, investors continue to look for improved and new techniques to beat the market instead of old and traditional ones. Therefore, researchers are continuously working to build novel techniques to supply the demand of investors. Different types of recurrent neural networks (RNN) are used in time series analyses, especially in stock price prediction. However, since not all stocks’ prices follow the same trend, a single model cannot be used to predict the movement of all types of stock’s price. Therefore, in this research we conducted a comparative analysis of three commonly used RNNs—simple RNN, Long Short Term Memory (LSTM), and Gated Recurrent Unit (GRU)—and analyzed their efficiency for stocks having different stock trends and various price ranges and for different time frequencies. We considered three companies’ datasets from 30 June 2000 to 21 July 2020. The stocks follow different trends of price movements, with price ranges of $30, $50, and $290 during this period. We also analyzed the performance for one-day, three-day, and five-day time intervals. We compared the performance of RNN, LSTM, and GRU in terms of R2 value, MAE, MAPE, and RMSE metrics. The results show that simple RNN is outperformed by LSTM and GRU because RNN is susceptible to vanishing gradient problems, while the other two models are not. Moreover, GRU produces lesser errors comparing to LSTM. It is also evident from the results that as the time intervals get smaller, the models produce lower errors and higher reliability.

A New Result on Stability Analysis of Recurrent Neural Networks with Time-Varying Delay Based on an Extended Delay-Dependent Integral Inequality

A New Result on Stability Analysis of Recurrent Neural Networks with Time-Varying Delay Based on an Extended Delay-Dependent Integral Inequality

Formula omitted] induced norm analysis of discrete-time LTI systems for nonnegative input signals and its application to stability analysis of recurrent neural networks

In this paper, we focus on the “positive” l2 induced norm of discrete-time linear time-invariant systems where the input signals are restricted to be nonnegative. To cope with the nonnegativity of the input signals, we employ copositive programming as the mathematical tool for the analysis. Then, by applying an inner approximation to the copositive cone, we derive numerically tractable semidefinite programming problems for the upper and lower bound computation of the “positive” l2 induced norm. This norm is typically useful for the stability analysis of feedback systems constructed from an LTI system and nonlinearities where the nonlinear elements provide only nonnegative signals. As a concrete example, we illustrate the usefulness of the “positive” l2 induced norm for the stability analysis of recurrent neural networks with activation functions being rectified linear units.

Prescribed convergence analysis of recurrent neural networks with parameter variations

Recurrent neural networks are designed to be convergent to the desired equilibrium point for their applications. Network parameter variations lead network states to other different points. So this paper discusses the prescribed convergence problem of recurrent neural networks with parameter variations. Firstly, we recurrent neural networks’ equilibrium point variation principles when parameters are changed. Then we design one track controller to make recurrent neural networks be convergent to the prescribed equilibrium for known parameter variations. Next, we present one adaptive controller to lead network states to the desired equilibrium for unknown parameter variations. At last, two examples are given for validating the presented methods.

Comparison Between Non-Linear Autoregressive and Non-Linear Autoregressive with Exogeneous Inputs Models for Predicting Cardiac Ischemic Beats

The prediction accuracy and generalization ability of neural models for forecasting Myocardial Ischemic Beats depends on type and architecture of employed network model. This paper presents the comparison analysis of recurrent neural network (RNN) architectures with embedded memory, Non-linear Autoregressive (NAR) and Non-linear Autoregressive with Exogeneous inputs (NARX) models for forecasting Ischemic Beats in ECG. Numerous architectures of the NAR and NARX models are verified for prediction and the performances are evaluated in terms of MSE. The performances of NAR and NARX models are validated by using ECG signals acquired from MIT-BIH database. The results have depicted that the NARX architecture with 2 neurons in hidden layer and 1 delay line outperformed with least Mean Square Error (MSE) of 0.0001 for detecting the ischemic beats in ECG.

Maturation of nasal microbiota and antibiotic exposures during early childhood: a population-based cohort study

ObjectivesLittle is known about maturation of the airway microbiota during early childhood and the consequences of early-life antibiotic exposure. MethodsIn a population-based birth cohort of 902 healthy Finnish children, we applied deep neural network models to investigate the relationship between the nasal microbiota (measured by 16S rRNA gene sequencing at up to three time points) and child age during the first 24 months. We also performed stratified analyses according to antibiotic exposure during the age period 0–2 months. ResultsThe dense deep neural network analysis successfully modelled the relationship between 232 bacterial genera and child age with a mean absolute error of 4.3 (95%CI 4.0–4.7) months. Similarly, the recurrent neural network analysis also successfully modelled the relationship between 215 genera and child age with a mean absolute error of 0.45 (95%CI 0.42–0.47) months. Among the genera, Staphylococcus spp. and members of the Corynebacteriaceae decreased with age, while Dolosigranulum and Moraxella increased with age in the first 2 years of life (all false discovery rate (FDR) = 0.001). In children without early-life antibiotic exposure, Dolosigranulum increased with age (FDR = 0.001). By contrast, in those with early-life antibiotic exposure, Haemophilus increased with age (FDR = 0.002). ConclusionsIn this prospective birth cohort of healthy children, we demonstrated the development of the nasal microbiota, with shifts in specific genera constituting maturation, in the first 2 years of life. Antibiotic exposures during early infancy were related to different age-discriminatory bacteria.

Analysis of Recurrent Neural Network and Predictions

This paper analyzes the operation principle and predicted value of the recurrent-neural-network (RNN) structure, which is the most basic and suitable for the change of time in the structure of a neural network for various types of artificial intelligence (AI). In particular, an RNN in which all connections are symmetric guarantees that it will converge. The operating principle of a RNN is based on linear data combinations and is composed through the synthesis of nonlinear activation functions. Linear combined data are similar to the autoregressive-moving average (ARMA) method of statistical processing. However, distortion due to the nonlinear activation function in RNNs causes the predicted value to be different from the predicted ARMA value. Through this, we know the limit of the predicted value of an RNN and the range of prediction that changes according to the learning data. In addition to mathematical proofs, numerical experiments confirmed our claims.

Finite-Time $$L_\infty $$ Performance State Estimation of Recurrent Neural Networks with Sampled-Data Signals

This paper, by proposing a sampled-data control scheme, we investigate the finite-time $$L_\infty $$ performance state estimation of recurrent neural networks. By constructing a novel Lyapunov functional, new stability and stabilization conditions are derived. By utilizing integral inequality techniques, sufficient LMI conditions are derived to ensure the finite-time stability of considered neural networks. Furthermore, finite-time observer gain analysis of recurrent neural networks is set up to measure its disturbance tolerance capability in the fixed time interval. Numerical examples are given to verify the effectiveness of the proposed approach.

On Complete Stability of Recurrent Neural Networks With Time-Varying Delays and General Piecewise Linear Activation Functions.

This paper addresses the problem of complete stability of delayed recurrent neural networks with a general class of piecewise linear activation functions. By applying an appropriate partition of the state space and iterating the defined bounding functions, some sufficient conditions are obtained to ensure that an n -neuron neural network is completely stable with exactly ∏i=1n(2Ki-1) equilibrium points, among which ∏i=1nKi equilibrium points are locally exponentially stable and the others are unstable, where Ki (i=1,…,n) are non-negative integers which depend jointly on activation functions and parameters of neural networks. The results of this paper include the existing works on the stability analysis of recurrent neural networks with piecewise linear functions as special cases and hence can be considered as the improvement and extension of the existing stability results in the literature. A numerical example is provided to illustrate the derived theoretical results.

Multistability and instability analysis of recurrent neural networks with time-varying delays

This paper provides new theoretical results on the multistability and instability analysis of recurrent neural networks with time-varying delays. It is shown that such n-neuronal recurrent neural networks have exactly (4k+3)k0 equilibria, (2k+2)k0 of which are locally exponentially stable and the others are unstable, where k0 is a nonnegative integer such that k0≤n. By using the combination method of two different divisions, recurrent neural networks can possess more dynamic properties. This method improves and extends the existing results in the literature. Finally, one numerical example is provided to show the superiority and effectiveness of the presented results.

Analysis of Recurrent Neural Networks for Probabilistic Modeling of Driver Behavior

The validity of any traffic simulation model depends on its ability to generate representative driver acceleration profiles. This paper studies the effectiveness of recurrent neural networks in predicting the acceleration distributions for car following on highways. The long short-term memory recurrent networks are trained and used to propagate the simulated vehicle trajectories over 10-s horizons. On the basis of several performance metrics, the recurrent networks are shown to generally match or outperform baseline methods in replicating driver behavior, including smoothness and oscillatory characteristics present in real trajectories. This paper reveals that the strong performance is due to the ability of the recurrent network to identify recent trends in the ego-vehicle's state, and recurrent networks are shown to perform as, well as feedforward networks with longer histories as inputs.