Backpropagation Through Time Research Articles

Robust Adaptive Gradient-Descent Training Algorithm for Recurrent Neural Networks in Discrete Time Domain

For a recurrent neural network (RNN), its transient response is a critical issue, especially for real-time signal processing applications. The conventional RNN training algorithms, such as backpropagation through time (BPTT) and real-time recurrent learning (RTRL), have not adequately addressed this problem because they suffer from low convergence speed. While increasing the learning rate may help to improve the performance of the RNN, it can result in unstable training in terms of weight divergence. Therefore, an optimal tradeoff between RNN training speed and weight convergence is desired. In this paper, a robust adaptive gradient-descent (RAGD) training algorithm of RNN is developed based on a novel RNN hybrid training concept. It switches the training patterns between standard real-time online backpropagation (BP) and RTRL according to the derived convergence and stability conditions. The weight convergence and L(2)-stability of the algorithm are derived via the conic sector theorem. The optimized adaptive learning maximizes the training speed of the RNN for each weight update without violating the stability and convergence criteria. Computer simulations are carried out to demonstrate the applicability of the theoretical results.

Quasi-stable EEG synchrony in resting and working brain

We examine the efficiency of Recurrent Neural Networks in forecasting the spatiotemporal dynamics of high dimensional and reduced order complex systems using Reservoir Computing (RC) and Backpropagation through time (BPTT) for gated network architectures. We highlight advantages and limitations of each method and discuss their implementation for parallel computing architectures. We quantify the relative prediction accuracy of these algorithms for the long-term forecasting of chaotic systems using as benchmarks the Lorenz-96 and the Kuramoto–Sivashinsky (KS) equations. We find that, when the full state dynamics are available for training, RC outperforms BPTT approaches in terms of predictive performance and in capturing of the long-term statistics, while at the same time requiring much less training time. However, in the case of reduced order data, large scale RC models can be unstable and more likely than the BPTT algorithms to diverge. In contrast, RNNs trained via BPTT show superior forecasting abilities and capture well the dynamics of reduced order systems. Furthermore, the present study quantifies for the first time the Lyapunov Spectrum of the KS equation with BPTT, achieving similar accuracy as RC. This study establishes that RNNs are a potent computational framework for the learning and forecasting of complex spatiotemporal systems.

Modelling and Predictive Control of a Multivariable Process Using Recurrent Neural Networks

Industrial processes are naturally multivariable in nature, which exhibit non-linear behaviour and complex dynamic properties. The multivariable four-tank system has attracted recent attention as it exhibits elegantly complex dynamics, which includes interaction, transmission zero, and non-minimum phase characteristics that emerge from a simple cascade of tanks. In this paper, a neural model predictive controller (NMPC) is compared with a decentralized PI controller designed for a four-tank system. The approach adopts the approximation capabilities of a recurrent Elman network for modelling purposes. The learning process is performed with a modification of a back propagation through time (BPTT) algorithm. The algorithms are simulated using MATLAB and the performances of NMPC and decentralized PI control are compared based on integral square error (ISE) values for set point tracking and load changes. The performance comparison shows that NMPC is a suitable controller for four-tank system.

Continuous-Time Adaptive Critics

A continuous-time formulation of an adaptive critic design (ACD) is investigated. Connections to the discrete case are made, where backpropagation through time (BPTT) and real-time recurrent learning (RTRL) are prevalent. Practical benefits are that this framework fits in well with plant descriptions given by differential equations and that any standard integration routine with adaptive step-size does an adaptive sampling for free. A second-order actor adaptation using Newton's method is established for fast actor convergence for a general plant and critic. Also, a fast critic update for concurrent actor-critic training is introduced to immediately apply necessary adjustments of critic parameters induced by actor updates to keep the Bellman optimality correct to first-order approximation after actor changes. Thus, critic and actor updates may be performed at the same time until some substantial error build up in the Bellman optimality or temporal difference equation, when a traditional critic training needs to be performed and then another interval of concurrent actor-critic training may resume.

Learning grammatical structure with Echo State Networks

Echo State Networks (ESNs) have been shown to be effective for a number of tasks, including motor control, dynamic time series prediction, and memorizing musical sequences. However, their performance on natural language tasks has been largely unexplored until now. Simple Recurrent Networks (SRNs) have a long history in language modeling and show a striking similarity in architecture to ESNs. A comparison of SRNs and ESNs on a natural language task is therefore a natural choice for experimentation. Elman applies SRNs to a standard task in statistical NLP: predicting the next word in a corpus, given the previous words. Using a simple context-free grammar and an SRN with backpropagation through time (BPTT), Elman showed that the network was able to learn internal representations that were sensitive to linguistic processes that were useful for the prediction task. Here, using ESNs, we show that training such internal representations is unnecessary to achieve levels of performance comparable to SRNs. We also compare the processing capabilities of ESNs to bigrams and trigrams. Due to some unexpected regularities of Elman's grammar, these statistical techniques are capable of maintaining dependencies over greater distances than might be initially expected. However, we show that the memory of ESNs in this word-prediction task, although noisy, extends significantly beyond that of bigrams and trigrams, enabling ESNs to make good predictions of verb agreement at distances over which these methods operate at chance. Overall, our results indicate a surprising ability of ESNs to learn a grammar, suggesting that they form useful internal representations without learning them.

Multifeedback-Layer Neural Network

The architecture and training procedure of a novel recurrent neural network (RNN), referred to as the multifeedback-layer neural network (MFLNN), is described in this paper. The main difference of the proposed network compared to the available RNNs is that the temporal relations are provided by means of neurons arranged in three feedback layers, not by simple feedback elements, in order to enrich the representation capabilities of the recurrent networks. The feedback layers provide local and global recurrences via nonlinear processing elements. In these feedback layers, weighted sums of the delayed outputs of the hidden and of the output layers are passed through certain activation functions and applied to the feedforward neurons via adjustable weights. Both online and offline training procedures based on the backpropagation through time (BPTT) algorithm are developed. The adjoint model of the MFLNN is built to compute the derivatives with respect to the MFLNN weights which are then used in the training procedures. The Levenberg-Marquardt (LM) method with a trust region approach is used to update the MFLNN weights. The performance of the MFLNN is demonstrated by applying to several illustrative temporal problems including chaotic time series prediction and nonlinear dynamic system identification, and it performed better than several networks available in the literature.

Recurrent neural networks in the context of SQL attacks

In the paper we present a new approach based on application of neural networks to detect SQLattacks. SQL attacks are those attacks that take advantage of using SQL statements to beperformed. The problem of detection of this class of attacks is transformed to time seriesprediction problem. SQL queries are used as a source of events in a protected environment. Todifferentiate between normal SQL queries and those sent by an attacker, we divide SQL statementsinto tokens and pass them to our detection system, which predicts the next token, taking intoaccount previously seen tokens. In the learning phase tokens are passed to recurrent neuralnetwork (RNN) trained by backpropagation through time (BPTT) algorithm. Teaching data areshifted by one token forward in time with relation to input. The purpose of the testing phase is topredict the next token in the sequence. All experiments were conducted on Jordan and Elmannetworks using data gathered from PHP Nuke portal. The experimental results show that theJordan network outperforms the Elman network predicting correctly queries of the length up toten.

Backpropagation Algorithms for a Broad Class of Dynamic Networks

This paper introduces a general framework for describing dynamic neural networks--the layered digital dynamic network (LDDN). This framework allows the development of two general algorithms for computing the gradients and Jacobians for these dynamic networks: backpropagation-through-time (BPTT) and real-time recurrent learning (RTRL). The structure of the LDDN framework enables an efficient implementation of both algorithms for arbitrary dynamic networks. This paper demonstrates that the BPTT algorithm is more efficient for gradient calculations, but the RTRL algorithm is more efficient for Jacobian calculations.

Identification and Control of Bioreactor using Recurrent Networks

A nonlinear model predictive control (NMPC) strategy based on recurrent neural networks (RNN) is proposed for a single‐input single‐output system (SISO) to control the uncertain nonlinear process. The automatic configuration and modeling of the networks is carried out using a recurrent Elman network using back propagation through time (BPTT) with MATLAB. Identification of the process is performed with a RNN based nonlinear autoregressive with exogenous input (NARX) model and the incorporation of the developed model in the formulation of NMPC is presented. Further, the results of the NMPC is compared with a well tuned IMC based PI controller, which shows a better performance based on the rise time and settling time of the proposed NMPC scheme for the control of an unstable bioreactor.

An adaptive recurrent fuzzy system for nonlinear identification

This paper describes the architecture and training procedure of a recurrent fuzzy system (RFS). The RFS is composed of a fuzzy inference system (FIS) and a delayed feedback connection. The recurrent property comes from feeding the FIS output back to the FIS input via an adjustable feedback parameter. Both the on-line and off-line training procedures based on the backpropagation-through-time (BPTT) algorithm have been investigated. The adjoint model of the RFS is obtained and used to compute the gradients. It is shown that the off-line training is insufficient to adapt to changes in system dynamics. So, an on-line training procedure is derived. In this procedure, a first in first out stack is used to store a certain history of the input-output data to perform a truncated BPTT algorithm. A quasi-Newton optimization method with a line search algorithm is used to adjust the RFS parameters. The performance of the developed RFS is demonstrated by applying to the identification of nonlinear dynamic systems. The simulation studies show that the proposed identification model has the ability to learn dynamics of highly nonlinear systems and compensate system uncertainties. The results are promising for the further application in the area of control and modeling.

Prediction of Chaotic Time Series Based on the Recurrent Predictor Neural Network

Chaos limits predictability so that the long-term prediction of chaotic time series is very difficult. The main purpose of this paper is to study a new methodology to model and predict chaotic time series based on a new recurrent predictor neural network (RPNN). This method realizes long-term prediction by making accurate multistep predictions. This RPNN consists of nonlinearly operated nodes whose outputs are only connected with the inputs of themselves and the latter nodes. The connections may contain multiple branches with time delays. An extended algorithm of self-adaptive back-propagation through time (BPTT) learning algorithm is used to train the RPNN. In simulation, two performance measures [root-mean-square error (RMSE) and prediction accuracy (PA)] show that the proposed method is more effective and accurate for multistep prediction. It can identify the systems characteristics quite well and provide a new way to make long-term prediction of the chaotic time series.

Artificial neural network model of the mapping between electromyographic activation and trajectory patterns in free-arm movements.

Artificial neural networks (ANNs) have been used to identify the relationship between electromyographic (EMG) activity and arm kinematics during the execution of motor tasks. Although considerable work has been devoted to showing that ANNs perform this mapping, there has been little work to explore any relationship with physiological properties of the neuromuscular systems. A back-propagation through time (BPTT) ANN was used to map the EMG of five selected muscles (pectoralis major (PM), anterior deltoid (AD), posterior deltoid (PD), biceps brachii (BB) and triceps brachii (TB)) on arm kinematics in seven normal subjects performing three-dimensional unrestrained grasping movements. To investigate the physiological validity of the BPTT-ANN, inputs were artificially altered, and the predicted outputs were analysed. Results show that the BPTT-ANN performed the mapping correctly (root mean square (RMS) error between target and predicted outputs averaged across subject test sets was 0.092 +/- 0.015). Moreover, it provided insights into the roles of muscles in performing the movement (average indexes measuring the output alteration with respect to the target were 0.070 +/- 0.027, 0.356 +/- 0.172, 0.568 +/- 0.413, 0.510 +/- 0.268, 0.681 +/- 0.430 for PM, AD, PD, BB, TB, respectively, in the movement forward phase, and 0.077 +/- 0.015, 0.179 +/- 0.147, 0.291 +/- 0.247, 0.671 +/- 0.054, 0.232 +/- 0.097 in the return phase).

Comparison of four gradient-learning algorithms for neural network Wiener models

A gradient-based approach to training neural network Wiener models is presented. Calculation of the gradient or approximate gradient for the series-parallel and parallel Wiener models by the backpropagation, the sensitivity method (SM) and the backpropagation through time (BPTT) is considered in a unified framework. Four different recursive learning algorithms are derived, analysed and compared. For the truncated BPTT, it is shown that the determination of the number of unfolding time steps can be made on the basis of an impulse response function of sensitivity models. Analysis of the computational complexity of these algorithms shows that, contrary to the other recurrent neural network models, computation of the gradient in parallel Wiener models with the sensitivity method or backpropagation through time requires only a little more computational burden than the backpropagation. A simulated data example and a real data example of a laboratory two-tank system are also included to make comparison of different methods and their effectiveness and practical feasibility are shown.

Neural network approximation to nonlinear dynamics by velocity error backpropagation

AbstractThis paper presents a new type of recurrent neural network (RNN) and its learning algorithm for nonlinear dynamics, called “Velocity‐Error Backpropagation (VEBP).”In VEBP, learning is performed in two steps: (a) The velocity vector field of reference trajectories is approximated by a feedforward neural network (NN) with biconnection layers by backpropagating the velocity errors directly. (b) The RNN is constructed by adding integrators and output feedback loops to the trained feedforward NN.VEBP has some advantages over “backpropagation through time (BPTT),” the conventional learning method for RNNs. The effectiveness of the presented RNN and its learning algorithm is demonstrated by simulation results for some examples of nonlinear dynamics. © 2002 Wiley Periodicals, Inc. Electr Eng Jpn, 140(2): 26–35, 2002; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/eej.10011

Neural Network Application To Optimal Control Of Nonlinear Systems

This paper presents the derivation of the numerical algorithm for optimal control of nonlinear multivariable systems with control and state vectors constraints. The algorithm derivation is based on the backpropagationthrough-time (BPTT) algorithm which is used as a learning algorithm for recurrent neural networks. This approach is not based on Lagrange multiplier techniques and the calculus of variations. The derived algorithm is used for the control of the cooperative work of two robots with two degrees of freedom. The main problem is the determination of the control vectors of robots for the transfer of rigid load from the initial state to the final one in a fixed time while maintaining constant distance between robot hands and avoiding the cross-section of the robot hands.

Recurrent Neural Networks for Predicting Outcomes after Liver Transplantation: Representing Temporal Sequence of Clinical Observations

This paper investigates a version of recurrent neural network with the backpropagation through time (BPTT) algorithm for predicting liver transplant graft failure based on a time series sequence of clinical observations. The objective is to improve upon the current approaches to liver transplant outcome prediction by developing a more complete model that takes into account not only the preoperative risk assessment, but also the early postoperative history. A 6-fold cross-validation procedure was used to measure the performance of the networks. The data set was divided into a learning set and a test set by maintaining the same proportion of positive and negative cases in the original set. The effects of network complexity on overfitting were investigated by constructing two types of networks with different numbers of hidden units. For each type of network, 10 individual networks were trained on the learning set and used to form a committee. The performance of the networks was measured exhaustively with respect to both the entire training and test sets. The networks were capable of learning the time series problem and achieved good performances of 90% correct classification on the learning set and 78% on the test set. The prediction accuracy increases as more information becomes progressively available after the operation with the daily improvement of 10% on the learning set and 5% on the test set. Recurrent neural networks trained with BPTT algorithm are capable of learning to represent temporal behavior of the time series prediction task. This model is an improvement upon the current model that does not take into account postoperative temporal information.

速度誤差逆伝搬学習アルゴリズムによる非線形力学系のニューラルネットワーク近似

This paper presents a new type of recurrent neural network (RNN) and its learning algorithm for nonlinear dynamics, called “Velocity-Error Backpropagation (VEBP).” In VEBP, learning is performed in two steps: (a) The velocity vector field of reference trajectories is approximated by a feedforward neural network (NN) with biconnection layers by backpropagating the velocity errors directly. (b) The RNN is constructed by adding integrators and output feedback loops to the trained feedforward NN. VEBP has some advantages over “backpropagation through time (BPTT),” the conventional learning method for RNNs. The effectiveness of the presented RNN and its learning algorithm is demonstrated by simulation results for some examples of nonlinear dynamics. © 2002 Wiley Periodicals, Inc. Electr Eng Jpn, 140(2): 26–35, 2002; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/eej.10011

Universal learning network and its application to robust control

Universal learning networks (ULNs) and robust control system design are discussed, ULNs provide a generalized framework to model and control complex systems. They consist of a number of interconnected nodes where the nodes may have any continuously differentiable nonlinear functions in them and each pair of nodes can be connected by multiple branches with arbitrary time delays. Therefore, physical systems which can be described by differential or difference equations and also their controllers can be modeled in a unified way. So, ULNs constitute a superset of neural networks or fuzzy neural networks. In order to optimize the systems, a generalized learning algorithm is derived for the ULNs, in which both the first order derivatives (gradients) and the higher order derivatives are incorporated. The derivatives are calculated by using forward or backward propagation schemes. These algorithms for calculating the derivatives are extended versions of back propagation through time (BPTT) and real time recurrent learning (RTRL) by Williams in the sense that generalized nonlinear functions and higher order derivatives are dealt with. As an application of ULNs, the higher order derivative, one of the distinguished features of ULNs, is applied to realizing a robust control system in this paper. In addition, it is shown that the higher order derivatives are effective tools to realize sophisticated control of nonlinear systems. Other features of ULNs such as multiple branches with arbitrary time delays and using a priori information will be discussed in other papers.

Universal learning network and its application to chaos control

Universal Learning Networks (ULNs) are proposed and their application to chaos control is discussed. ULNs provide a generalized framework to model and control complex systems. They consist of a number of inter-connected nodes where the nodes may have any continuously differentiable nonlinear functions in them and each pair of nodes can be connected by multiple branches with arbitrary time delays. Therefore, physical systems, which can be described by differential or difference equations and also their controllers, can be modeled in a unified way, and so ULNs may form a super set of neural networks and fuzzy neural networks. In order to optimize the ULNs, a generalized learning algorithm is derived, in which both the first order derivatives (gradients) and the higher order derivatives are incorporated. The derivatives are calculated by using forward or backward propagation schemes. These algorithms for calculating the derivatives are extended versions of Back Propagation Through Time (BPTT) and Real Time Recurrent Learning (RTRL) of Williams in the sense that generalized node functions, generalized network connections with multi-branch of arbitrary time delays, generalized criterion functions and higher order derivatives can be deal with. As an application of ULNs, a chaos control method using maximum Lyapunov exponent of ULNs is proposed. Maximum Lyapunov exponent of ULNs can be formulated by using higher order derivatives of ULNs, and the parameters of ULNs can be adjusted so that the maximum Lyapunov exponent approaches the target value. From the simulation results, it has been shown that a fully connected ULN with three nodes is able to display chaotic behaviors.

A NEW STOCHASTIC LEARNING ALGORITHM FOR NEURAL NETWORKS

A new stochastic learning algorithm using Gaussian white noise sequence, referred to as Subconscious Noise Reaction (SNR), is proposed for a class of discrete-time neural networks with time-dependent connection weights. Unlike the back-propagation-through-time (BTT) algorithm, SNR does not require the synchronous transmission of information backward along connection weights, while it uses only ubiquitous noise and local signals, which are correlated against a single performance functional, to achieve simple sequential (chronologically ordered) updating of connection weights. The algorithm is derived and analyzed on the basis of a functional derivative formulation of the gradient descent method in conjunction with stochastic sensitivity analysis techniques using the variational approach.