Active learning using uncertainty: behavioral evidence and neural correlates in rats

Abstract

Listen

Translate article

Event Abstract Back to Event Active learning using uncertainty: behavioral evidence and neural correlates in rats Statistical learning theory proposes that ?active learners? use their uncertainty estimates to optimally set their learning rate so as to learn more when uncertain and less when certain. In the machine learning literature it is well established that active learning based on informative data points speeds up learning. However, little is known about whether these principles apply to animal learning. Here we test the hypothesis that rats are active learners and use uncertainty estimates to learn optimally. To study this issue we used an odor-mixture categorization task performed by rats and examined the trial-by-trial updating of behavioral strategy. As animals learn to perform a categorization task they use reinforcement feedback to establish the decision-boundary, yet this boundary may be continually updated during on-going performance after overt learning asymptoted. Indeed, we found that animals were dynamically adjusting their decision strategy even after extensive training. For difficult decisions (those near the category boundary) the outcome is very informative about location of the decision boundary, while the outcome of easy decisions (those far from the decision boundary) reveals little about the boundary. Accordingly, the decision boundary should be adjusted more following difficult trials with high uncertainty than for trials with no uncertainty. Indeed, we found that rats biased their decisions toward the more recently rewarded direction as if their decision boundary was shifted. This bias, however, was only observed for difficult decisions suggesting that the category boundary and not the choice-bias was being updated. Moreover, the magnitude of this bias was proportional to the uncertainty of the previous decision, as predicted. Therefore on-going learning appears to depend on a graded prediction error signal combining reward feedback and uncertainty estimates. These behavioral data could be quantitatively explained by an "active" delta learning rule, where reward predictions are computed based on decision uncertainty. We also show how a trial-by-trial uncertainty estimate can be naturally computed in this class of models (see also Shenoy et al, COSYNE 2008). To understand the neural basis of this process we recorded neurons in the orbitofrontal cortex of rats performing the olfactory categorization task. We found a population of neurons whose firing rate during the reward anticipation period closely resembled the expected signal of decision uncertainty. About half of these neurons also carried information about the reward outcome of the previous trial based on a regression analysis. This type of activity closely matches the learning signal predicted by active learning models. Taken together the behavioral and computational results show that rats are "active learners", combining reward feedback and decision uncertainty estimates to update their decision strategy. Moreover orbitofrontal cortex neurons carry information that is relevant for such active learning process. Conference: Computational and systems neuroscience 2009, Salt Lake City, UT, United States, 26 Feb - 3 Mar, 2009. Presentation Type: Poster Presentation Topic: Poster Presentations Citation: (2009). Active learning using uncertainty: behavioral evidence and neural correlates in rats. Front. Syst. Neurosci. Conference Abstract: Computational and systems neuroscience 2009. doi: 10.3389/conf.neuro.06.2009.03.284 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 04 Feb 2009; Published Online: 04 Feb 2009. Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Google Google Scholar PubMed Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.