Recurrent Neural Nets Research Articles

A signal processing framework based on dynamic neural networks with application to problems in adaptation, filtering, and classification

We present a coherent neural net based framework for solving various signal processing problems. It relies on the assertion that time-lagged recurrent networks possess the necessary representational capabilities to act as universal approximators of nonlinear dynamical systems. This applies to system identification, time-series prediction, nonlinear filtering, adaptive filtering, and temporal pattern classification. We address the development of models of nonlinear dynamical systems, in the form of time-lagged recurrent neural nets, which can be used without further training. We employ a weight update procedure based on the extended Kalman filter (EKF). Against the tendency for a net to forget earlier learning as it processes new examples, we develop a technique called multistream training. We demonstrate our framework by applying it to 4 problems. First, we show that a single time-lagged recurrent net can be trained to produce excellent one-time-step predictions for two different time series and also to be robust to severe errors in the input sequence. Second, we model stably a complex system containing significant process noise. The remaining two problems are drawn from real-world automotive applications. One involves input-output modeling of the dynamic behavior of a catalyst-sensor system which is exposed to an operating engine's exhaust stream, the other the real-time and continuous detection of engine misfire.

Recurrent neural nets as dynamical Boolean systems with application to associative memory

Discrete-time/discrete-state recurrent neural networks are analyzed from a dynamical Boolean systems point of view in order to devise new analytic and design methods for the class of both single and multilayer recurrent artificial neural networks. With the proposed dynamical Boolean systems analysis, we are able to formulate necessary and sufficient conditions for network stability which are more general than the well-known but restrictive conditions for the class of single layer networks: (1) symmetric weight matrix with (2) positive diagonal and (3) asynchronous update. In terms of design, we use a dynamical Boolean systems analysis to construct a high performance associative memory. With this Boolean memory, we can guarantee that all fundamental memories are stored, and also guarantee the size of the basin of attraction for each fundamental memory.

Fast sigmoidal networks via spiking neurons.

We show that networks of relatively realistic mathematical models for biological neurons in principle can simulate arbitrary feedforward sigmoidal neural nets in a way that has previously not been considered. This new approach is based on temporal coding by single spikes (respectively by the timing of synchronous firing in pools of neurons) rather than on the traditional interpretation of analog variables in terms of firing rates. The resulting new simulation is substantially faster and hence more consistent with experimental results about the maximal speed of information processing in cortical neural systems. As a consequence we can show that networks of noisy spiking neurons are "universal approximators" in the sense that they can approximate with regard to temporal coding any given continuous function of several variables. This result holds for a fairly large class of schemes for coding analog variables by firing times of spiking neurons. This new proposal for the possible organization of computations in networks of spiking neurons systems has some interesting consequences for the type of learning rules that would be needed to explain the self-organization of such networks. Finally, the fast and noise-robust implementation of sigmoidal neural nets by temporal coding points to possible new ways of implementing feedforward and recurrent sigmoidal neural nets with pulse stream VLSI.

Dynamic rectification of data via recurrent neural nets and the extended Kalman filter

AbstractThe presence of autocorrelated measurement errors and/or measurement bias in process measurements poses serious problems in the rectification of data taken from dynamic processes. The proposed procedure to resolve these problems involves the use of recurrent neural networks (RNN) and the extended Kalman filter (EKF). By interpreting RNNs within a nonlinear state‐space context, a state‐augmented EKF can be used to optimally estimate both the states of the RNNs and noise and bias models. RNN models can be identified off‐line and utilized for data rectification within the extended Kalman filter in process environments in which badly autocorrelated measurement errors exist in the data. The same technique is also used to estimate measurement bias present in both process input and output variables. This approach has the advantage that models developed from “first principles” are not required and that rectification can be performed solely on the basis of the contaminated dynamic data.

Systems Combining Linearity and Saturations, and Relations of “Neural Nets”

Abstract This paper deals with control systems consisting of linearly interconnected integrators (or delay lines) and scalar nonlinearities. For linear systems with saturating sensors, we mention results on observability and minimal realization. When saturations appear in actuators, questions of control become of interest, and we describe stabilization techniques. If there are feedback loops containing the nonlinearities, “recurrent neural nets” are obtained, and we discuss various issues relating to their computational power and identifiability of parameters. Parts of the work surveyed here were jointly pursued with Francesca Albertini, Renee Schwarzschild, Hava Siegelmann, Hector Sussmann, and Yudi Yang.

Homotopy analysis of recurrent neural nets

Homotopy analysis of recurrent neural nets