Dopamine transients do not act as model-free prediction errors during associative learning

Melissa J Sharpe,Etienne J P Maes,Lauren E Mueller,Geoffrey Schoenbaum,Chun Yun Chang,Hannah M Batchelor,Yael Niv

doi:10.1038/s41467-019-13953-1

Abstract

Dopamine neurons are proposed to signal the reward prediction error in model-free reinforcement learning algorithms. This term represents the unpredicted or ‘excess’ value of the rewarding event, value that is then added to the intrinsic value of any antecedent cues, contexts or events. To support this proposal, proponents cite evidence that artificially-induced dopamine transients cause lasting changes in behavior. Yet these studies do not generally assess learning under conditions where an endogenous prediction error would occur. Here, to address this, we conducted three experiments where we optogenetically activated dopamine neurons while rats were learning associative relationships, both with and without reward. In each experiment, the antecedent cues failed to acquire value and instead entered into associations with the later events, whether valueless cues or valued rewards. These results show that in learning situations appropriate for the appearance of a prediction error, dopamine transients support associative, rather than model-free, learning.

Highlights

Dopamine neurons are proposed to signal the reward prediction error in model-free reinforcement learning algorithms
To the best of our knowledge, there is currently no report that directly tests whether a transient increase in the activity of midbrain dopamine neurons functions to assign value directly to cues when it is delivered in contexts where a prediction error might normally be expected to appear during associative learning
The use of a design that isolates dopamine transients when rats are learning about neutral information is advantageous because it dissociates the endogeonous value signal elicited by a motivationally-significant reward from any value that may be inherent in the dopamine transient itself

Summary

Introduction

Dopamine neurons are proposed to signal the reward prediction error in model-free reinforcement learning algorithms. To the best of our knowledge, there is currently no report that directly tests whether a transient increase in the activity of midbrain dopamine neurons functions to assign value directly to cues when it is delivered in contexts where a prediction error might normally be expected to appear during associative learning While exceptions exist, these contexts commonly have at least two events arranged in a relatively precise relationship to the error signal—one whose onset co-occurs with the onset of the error and another whose onset is antecedent to the error. We used conditioned reinforcement to assess cue value independent of the associative predictions about reward, and in each experiment we found that the dopamine transient supported the development of associative representations without endowing the antecedent cues with value

Methods

Results

Conclusion