An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users.

Adam Bignold,Francisco Cruz,Cameron Foale,Peter Vamplew,Richard Dazeley

doi:10.3390/biomimetics6010013

Abstract

Interactive reinforcement learning methods utilise an external information source to evaluate decisions and accelerate learning. Previous work has shown that human advice could significantly improve learning agents’ performance. When evaluating reinforcement learning algorithms, it is common to repeat experiments as parameters are altered or to gain a sufficient sample size. In this regard, to require human interaction every time an experiment is restarted is undesirable, particularly when the expense in doing so can be considerable. Additionally, reusing the same people for the experiment introduces bias, as they will learn the behaviour of the agent and the dynamics of the environment. This paper presents a methodology for evaluating interactive reinforcement learning agents by employing simulated users. Simulated users allow human knowledge, bias, and interaction to be simulated. The use of simulated users allows the development and testing of reinforcement learning agents, and can provide indicative results of agent performance under defined human constraints. While simulated users are no replacement for actual humans, they do offer an affordable and fast alternative for evaluative assisted agents. We introduce a method for performing a preliminary evaluation utilising simulated users to show how performance changes depending on the type of user assisting the agent. Moreover, we describe how human interaction may be simulated, and present an experiment illustrating the applicability of simulating users in evaluating agent performance when assisted by different types of trainers. Experimental results show that the use of this methodology allows for greater insight into the performance of interactive reinforcement learning agents when advised by different users. The use of simulated users with varying characteristics allows for evaluation of the impact of those characteristics on the behaviour of the learning agent.

Highlights

Reinforcement learning (RL) is a machine learning technique that allows artificial intelligence to learn from experience
We introduced a general methodology for evaluating interactive reinforcement learning by employing simulated users as a substitute for actual humans
While simulated users are not a replacement for real human testing, it was demonstrated that evaluations using simulated users could show detailed insights into how the agent is expected to act under certain interaction conditions

Summary

Introduction

Reinforcement learning (RL) is a machine learning technique that allows artificial intelligence to learn from experience. RL agents attempt to refine their behaviour through interaction with the environment [1]. Using a trial and error approach, an RL agent can observe how performed actions affect the agent’s state and the reward obtained. In this regard, the sequence of actions an agent chooses to take, given the information it has learned about the problem, is known as the agent’s policy. The agent learns the steps that lead to an intended outcome by reinforcing the desired behaviour with a reward value [2]

Methods

Results

Conclusion