A network model that can learn reward timing using reinforced expression of synaptic plasticity

Abstract

Recent experimental results indicate that cells within theprimary visual cortex can learn to predict the time ofrewards associated with visual cues [1]. In this work, dif-ferent visual cues were paired with rewards at specific tem-poral offsets. Before training, neurons in visual cortexwere active only during the duration of the visual cue.After sufficient training neurons developed persistentactivity for a time period correlated with the timing ofreward.Recurrent connections in a neural network can be con-structed to set a desired network time constant that is dif-ferent from the time constants of the constituent neurons.However, it is not known how such a network can learnthe appropriate recurrent weights. A plasticity model thatis able to accomplish this must be sensitive to the timingof reward events that, at least initially, occur seconds afterthe activity in the network returns to its basal level. Inorder to learn the appropriate dynamics, this networkneeds to solve a temporal credit assignment problem. Inour model plasticity is an ongoing process changing therecurrent synaptic weights as a function of their activity; inthe absence of a reward signal this plasticity rapidlydecays. External reward signals allow permanent expres-sion of preceding plasticity events, reinforcing only thosewhich predict the reward. As a result, the network dynam-ics are altered and it develops time constants correlatedwith the timing of different rewards. As in other reinforce-ment learning models the reward signal is inhibited by thenetwork activity to produce a stable activity pattern.We have implemented these ideas in both abstract passiveintegrator networks and in more realistic integrate and firenetworks and obtained results that are qualitatively simi-lar to the experimental results. Further, we examine theimplications of different possible biophysical mecha-nisms and propose experiments to test which specificmechanism are involved.Support: NSF CRCNS grant number 0515285.