Task complexity interacts with state-space uncertainty in the arbitration between model-based and model-free learning

Geon Yeong Park,Sang Wan Lee,John P O′Doherty,Dongjae Kim

doi:10.1038/s41467-019-13632-1

Abstract

It has previously been shown that the relative reliability of model-based and model-free reinforcement-learning (RL) systems plays a role in the allocation of behavioral control between them. However, the role of task complexity in the arbitration between these two strategies remains largely unknown. Here, using a combination of novel task design, computational modelling, and model-based fMRI analysis, we examined the role of task complexity alongside state-space uncertainty in the arbitration process. Participants tended to increase model-based RL control in response to increasing task complexity. However, they resorted to model-free RL when both uncertainty and task complexity were high, suggesting that these two variables interact during the arbitration process. Computational fMRI revealed that task complexity interacts with neural representations of the reliability of the two systems in the inferior prefrontal cortex.

Highlights

It has previously been shown that the relative reliability of model-based and model-free reinforcement-learning (RL) systems plays a role in the allocation of behavioral control between them
When interrogating the fitted model, we found an effect of uncertainty and complexity on the weighting between model-based RL (MB) and model-free SARSA learner (MF) control (Fig. 6c; two-way repeated measures ANOVA; p < 1e-4 for the main effect of both state-transition uncertainty and task complexity; p = 0.039 for the interaction effect; full statistics are shown in Supplementary Table 4)
These behavioral results were supported by evidence that a region of the brain previously implicated in the arbitration process, the inferior lateral prefrontal cortex (ilPFC) encodes signals related to the reliability of the predictions of the two systems that would support an uncertainty-based arbitration mechanism, but that activity in this region is better accounted for by an arbitration model that incorporates the effects of task complexity into the arbitration process

Summary

Introduction

It has previously been shown that the relative reliability of model-based and model-free reinforcement-learning (RL) systems plays a role in the allocation of behavioral control between them. 1234567890():,; It has been suggested that two distinct mechanisms exist for controlling instrumental actions: a model-free RL system that learns values for actions based on the history of rewards obtained on those actions[1,2,3] and a model-based RL (MB) system that computes action values flexibly based on its knowledge about state-action-state transitions incorporated into an internal model of the structure of the world[4,5]. These two systems have different relative advantages and disadvantages. Another important element of this trade-off is the relative computational cost of engaging in model-based control

Objectives

Methods

Results

Conclusion