The “off line learning approximation” in continuous time neural networks: An adiabatic theorem

Abstract

In this paper we consider the justification of the “Off Line Approximation” in continuous time neural networks from a rigorous mathematical point of view. In real time models, the behavior of a network is characterized by two distinct dynamics evolving according different time scales, the weight dynamics which are the “slow” and the activation dynamics which are the “fast.” The “off-line approximation” assumes that during the learning process, neural activities are in their steady states. Such an approximation is a common dogma often used to provide an analysis of network behavior. In this paper we consider convergent networks and prove that this approximation is valid on the time scale 1/ε where ε is the learning rate parameter which controls the learning velocity. We apply these results to prove the stability of Hebbian learning in a two-layered neural network which can be seen as a continuous time version of the self-organizing Kohonen's model.