Chapter 11 - Does the Dopaminergic Error Signal Act Like a Cached-Value Prediction Error?

Abstract

The phasic dopamine error signal is currently argued to be synonymous with the prediction error inherent in Sutton and Barto’s (1987, 1998) model-free reinforcement learning algorithm. This limits the role phasic dopamine can play in learning because this signal drives learning only in response to changes in value and not in response to more detailed departures from expectations. For example, the model-free error signal will drive learning when you expect $5 and get $10, but not when you expect apples and receive oranges. This conceptualization likely reflected the fact that this model was considered the most evolved model of reinforcement learning at this time. However, this model departed in theoretically consequential ways from its predecessors. Earlier models of reinforcement learning (e.g., Bush & Mosteller, 1951; Rescorla & Wagner, 1972) described the prediction error as providing the catalyst for learning, where such learning presumably referenced associations between events currently being represented elsewhere in the brain. Here, we discuss the evolution of these models of reinforcement learning insofar as they pertain to an interpretation of the dopamine prediction error. Then, we focus on whether the dopamine prediction error is restricted like that described in Sutton and Barto (1987, 1998) or whether this signal may function more broadly as a catalyst for learning (Rescorla & Wagner, 1972). Based on recent data, we argue that while the phasic dopamine signal conforms to a model-free account in some circumstances, it is not restricted to a model-free learning signal. Instead, the phasic dopaminergic error functions more generally to signal violations of expectancies to drive real-world associations between events. Such an interpretation constitutes a step back to earlier models of reinforcement learning, in which the prediction error provides the catalyst for learning, rather than a signal, which endows antecedent cues with value.