Mathematical Description of Differential Hebbian Plasticity and its Relation to Reinforcement Learning

Abstract

The human brain consists of more than a billion nerve cells, the neurons, each having several thousand connections, the synapses. These connections are not fixed but change all the time. In order to describe synaptic plasticity, different mathematical rules have been proposed most of which follow Hebb"s postulate. Donald Hebb suggested in 1949 that synapses only change if pre-synaptic activity, i.e. the activity of a synapse that converges to the neuron, and post-synaptic activity, i.e. activity of the neuron itself, correlate with each other. A general descriptive framework, however, is yet missing for this influential class of plasticity rules. In addition, the description of the dynamics of the synaptic connections under Hebbian plasticity is limited either to the plasticity of only one synapse or to simple, stationary activity patterns. In spite of this, Hebbian plasticity has been applied to different fields, for instance to classical conditioning. However, the extension to operant conditioning and to the closely related reinforcement learning is problematic. So far reinforcement learning can not be implemented directly at a neuron as the plasticity of converging synapses depends on information that needs to be computed by many neurons. In this thesis we describe the plasticity of a single plastic synapse by introducing a new theoretical framework for its analysis based on their auto- and cross-correlation terms. With this framework we are able to compare and draw conclusions about the stability of several different rules. This makes it also possible to specifically construct Hebbian plasticity rules for various systems. For instance, an additional plasticity modulating factor is sufficient to eliminate the auto-correlation contribution. Along these lines we also generalize two already existing models, a fact which leads to a novel so-called Variable Output Trace (VOT) plasticity rule that will be of further importance. In a next step we extend our analysis to many plastic synapses where we develop a complete analytical solution which characterizes the dynamics of synaptic connections even for non-stationary activity. This allows us to predict the synaptic development of symmetrical differential Hebbian plasticity. In the last part of this thesis, we present a general setup with which any Hebbian plasticity rule with a negative auto-correlation can be used to emulate temporal difference learning, a widely used reinforcement learning algorithm. Specifically we use differential Hebbian plasticity with a modulating factor and the VOT plasticity rule developed in the first part to prove their asymptotic equivalence to temporal difference learning and additionally investigate the practicability of these realizations. With the results developed in this thesis, it is possible to relate different Hebbian rules and their properties to each other. It is also possible for the first time to calculate plasticity analytically for many synapses with continuously changing activity. This is of relevance for all behaving systems (machines, animals) whose interaction with their environment leads to widely varying neural activation.