Inferring human intrinsic rewards through inverse reinforcement learning

Abstract

Event Abstract Back to Event Inferring human intrinsic rewards through inverse reinforcement learning Constantin A. Rothkopf1, 2* 1 Goethe University, Frankfurt Institute for Advanced Studies, Germany 2 University Osnabrück, Institute of Cognitive Science, Germany In a large variety of situations one would like to have an expressive and accurate model of observed behavior. Here we show the advantage of expressing behavior as a combination of concurrent goals in the context of optimal control, which has the distinct advantage of expressing behavioral goals as reward functions. We show that, in such a setting, a specific formulation of inverse reinforcement learning can be derived that allows the recovery of reward weights, which quantify how much individual component tasks contribute to the overall behavior. By parameterizing the contributions of individual modules through their respective Q-functions one does not require explicit transition functions for the recovery of the reward weights. We show how to recover the component reward weights for individual tasks and demonstrate through simulations that good estimates can be obtained already with minimal amounts of observation. We apply this framework to a sequence of experiments involving human participants in a multiple objective navigation task in a virtual environment. Participants were given different task specifications leading do different walking behavior. It is shown that the recovered intrinsic reward weights reflect the given instructions on a trial by trial and individual subject's basis, but that subjects have systematic biases that lead them to assign rewards to tasks that they were not instructed to do. Furthermore, we show how the modular framework predicts behavior in novel configurations and novel task combinations. References C. A. Rothkopf, C. Dimitrakakis: 'Preference elicitation and inverse reinforcement learning', 22nd European Conference on Machine Learning (ECML), September 5-9, 2011 C. A. Rothkopf: 'Modular models of task based visually guided behavior', Ph. D. thesis, University of Rochester. Department of Brain and Cognitive Sciences, Department of Computer Science, 2008 Keywords: biological costs, inverse reinforcement learning, navigation, reinforcement learning Conference: Bernstein Conference 2012, Munich, Germany, 12 Sep - 14 Sep, 2012. Presentation Type: Poster Topic: Attention, reward, decision making Citation: Rothkopf CA (2012). Inferring human intrinsic rewards through inverse reinforcement learning. Front. Comput. Neurosci. Conference Abstract: Bernstein Conference 2012. doi: 10.3389/conf.fncom.2012.55.00050 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 25 May 2012; Published Online: 12 Sep 2012. * Correspondence: Dr. Constantin A Rothkopf, Goethe University, Frankfurt Institute for Advanced Studies, Frankfurt, Germany, constantin.rothkopf@cogsci.tu-darmstadt.de Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Constantin A Rothkopf Google Constantin A Rothkopf Google Scholar Constantin A Rothkopf PubMed Constantin A Rothkopf Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.