Realization theory of recurrent neural networks and rational systems

Abstract

In this paper, we show that, under mild assumptions, input-output behavior of a continuous-time recurrent neural network (RNN) can be represented by a rational or polynomial non-linear system. The assumptions concern the activation function of RNNs, and they are satisfied by many classical activation functions. We also present an algorithm for constructing the polynomial and rational system. This embedding of RNNs into rational systems can be useful for stability, identifiability, and realization theory for RNNs, as these problems have been studied for polynomial/rational systems. In particular, we use this embedding for deriving necessary conditions for realizability of an input-output map by RNN, and for deriving sufficient conditions for minimality of an RNN.