Dopamine enhances model-free credit assignment through boosting of retrospective model-based inference.

Lorenz Deserno,Rani Moran,Ying Lee,Raymond J Dolan,Peter Dayan,Jochen Michely

doi:10.7554/elife.67778

Abstract

Dopamine is implicated in representing model-free (MF) reward prediction errors a as well as influencing model-based (MB) credit assignment and choice. Putative cooperative interactions between MB and MF systems include a guidance of MF credit assignment by MB inference. Here, we used a double-blind, placebo-controlled, within-subjects design to test an hypothesis that enhancing dopamine levels boosts the guidance of MF credit assignment by MB inference. In line with this, we found that levodopa enhanced guidance of MF credit assignment by MB inference, without impacting MF and MB influences directly. This drug effect correlated negatively with a dopamine-dependent change in purely MB credit assignment, possibly reflecting a trade-off between these two MB components of behavioural control. Our findings of a dopamine boost in MB inference guidance of MF learning highlight a novel DA influence on MB-MF cooperative interactions.

Highlights

Dual system theories of reinforcement learning (RL) propose that behaviour is controlled by a prospective, model-based (MB), planning system and a retrospective, model-free (MF), value-caching system (Daw and Dayan, 2014; Daw et al, 2005; Dolan and Dayan, 2013)
We previously showed that a positive effect of reward at the non-informative destination on choice repetition, implicating a preferential guidance of MFCA towards the ghost-nominated vehicle guided by retrospective MB inference
We show that enhancing DA boosted the guidance of MF credit assignment by retrospective

Summary

Introduction

Dual system theories of reinforcement learning (RL) propose that behaviour is controlled by a prospective, model-based (MB), planning system and a retrospective, model-free (MF), value-caching system (Daw and Dayan, 2014; Daw et al, 2005; Dolan and Dayan, 2013). Studies, motivated by classical psychological dual control distinctions regarding goal and habit (Balleine and Dickinson, 1998; Dickinson, 1985; Dolan and Dayan, 2013), focused on how these systems operated separately, with subsequent work indicating their influences being combined only close to the point of choice (Daw et al, 2011) Even in the former and other experiments (involving the well described two-step task), human functional neuroimaging showed that activity in the ventral striatum and the prefrontal cortex reflected MB values as well as MF RPEs (Daw et al., 2011; for replication see Deserno et al, 2015b, 2015a). More recent evidence has raised the possibility that this may occur during rest periods (Antonov et al, 2021; Liu et al, 2021a)

Objectives

Methods

Results

Conclusion