Dynamics of Neural Networks - Some Qualitative Properties

Abstract

All neural networks, both natural and artificial, are characterized by two kinds of dynamics. The first one is concerned with what we would call "learning dynamics", in fact the sequential (discrete time) dynamics of the choice of synaptic weights. The second one is the intrinsic dynamics of the neural network viewed as a dynamical system after the weights have been established via learning. The paper deals with the second kind of dynamics. Since the emergent computational capabilities of a recurrent neural network can be achieved provided it has suitable dynamical properties when viewed as a system with several equilibria, the paper deals with those qualitative properties connected to the achievement of such dynamical properties, more precisely the gradient like behavior. In the case of the neural networks with delays, these aspects are reformulated in accordance with the state of the art of the theory of delay dynamical systems.