Joint Estimation of Expertise and Reward Preferences From Human Demonstrations

Abstract

When a robot learns from human examples, most approaches assume that the human partner provides examples of optimal behavior. However, there are applications in which the robot learns from nonexpert humans. We argue that the robot should learn not only about the human's objectives, but also about their expertise level. The robot could then leverage this joint information to reduce or increase the frequency at which it provides assistance to its human's partner or be more cautious when learning new skills from novice users. Similarly, by taking into account the human's expertise, the robot would also be able to infer a human's true objectives even when the human fails to properly demonstrate these objectives due to a lack of expertise. In this article, we propose to jointly infer the expertise level and the objective function of a human given observations of their (possibly) nonoptimal demonstrations. Two inference approaches are proposed. In the first approach, inference is done over a finite discrete set of possible objective functions and expertise levels. In the second approach, the robot optimizes over the space of all possible hypotheses and finds the objective function and the expertise level that best explain the observed human behavior. We demonstrate our proposed approaches both in simulation and with real user data.