Adaptive behaviour and feedback processing integrate experience and instruction in reinforcement learning

Anne-Marike Schiffer,Kayla Siletti,Florian Waszak,Nick Yeung

doi:10.1016/j.neuroimage.2016.08.057

Abstract

In any non-deterministic environment, unexpected events can indicate true changes in the world (and require behavioural adaptation) or reflect chance occurrence (and must be discounted). Adaptive behaviour requires distinguishing these possibilities. We investigated how humans achieve this by integrating high-level information from instruction and experience. In a series of EEG experiments, instructions modulated the perceived informativeness of feedback: Participants performed a novel probabilistic reinforcement learning task, receiving instructions about reliability of feedback or volatility of the environment. Importantly, our designs de-confound informativeness from surprise, which typically co-vary. Behavioural results indicate that participants used instructions to adapt their behaviour faster to changes in the environment when instructions indicated that negative feedback was more informative, even if it was simultaneously less surprising. This study is the first to show that neural markers of feedback anticipation (stimulus-preceding negativity) and of feedback processing (feedback-related negativity; FRN) reflect informativeness of unexpected feedback. Meanwhile, changes in P3 amplitude indicated imminent adjustments in behaviour. Collectively, our findings provide new evidence that high-level information interacts with experience-driven learning in a flexible manner, enabling human learners to make informed decisions about whether to persevere or explore new options, a pivotal ability in our complex environment.

Highlights

Humans and other animals use their ability to predict which action will lead to which outcome to choose appropriate actions and monitor their success
In our EEG measures, we focused in particular on the feedback-related negativity (FRN) component as a marker of feedback processing, the stimulus preceding negativity (SPN) as a correlate of the anticipation of feedback, and the P3 as an index of feedback evaluation for immediate updating of action plans
The difference was not significant. These findings are relevant in interpreting analyses of the FRN, which is usually described as a correlate of frequencybased unexpectedness

Summary

Introduction

Humans and other animals use their ability to predict which action will lead to which outcome to choose appropriate actions and monitor their success. Occurrence of unexpected events can indicate incorrect or failed actions. In non-deterministic environments, unexpected events can happen for fundamentally different reasons: They may indicate true changes in the world and require adaptation, but sometimes they may instead reflect chance occurrence and should be discounted. An agent needs to determine whether or not unexpected events indicate that a change in the environment has occurred. The agent must assess and integrate the event's informative value. Within this framework, the informative value of an unexpected event would be high, for example, if volatility in the environment was known to be high: unexpected n Corresponding author at: Department of Experimental Psychology, University of Oxford, OX13UD Oxford, UK

Objectives

Methods

Results

Discussion

Conclusion