Perception-based generalization in model-based reinforcement learning

Abstract

In recent years, the advances in robotics have allowed for robots to venture into places too dangerous for humans. Unfortunately, the terrain in which these robots are being deployed may not be known by humans in advance, making it difficult to create motion programs robust enough to handle all scenarios that the robot may encounter. For this reason, research is being done to add learning capabilities to improve the robot's ability to adapt to its environment. Reinforcement learning is well suited for these robot domains because often the desired outcome is known, but the best way to achieve this outcome is unknown. In a real world domain, a reinforcement-learning agent has to learn a great deal from experience. Therefore, it must be sample-size efficient. To do so, it must balance the amount of exploration that is needed to properly model the environment with the need to use the information that it has already obtained to complete its original task. In robot domains, the exploration process is especially costly in both time and energy. Therefore, it is important to make the best possible use of the robot's limited opportunities for exploration without degrading the robot's performance. This dissertation discusses a specialization of the standard Markov Decision Process (MDP) framework that allows for easier transfer of experience between similar states and introduces an algorithm that uses this new framework to perform more efficient exploration in robot-navigation problems. It then develops methods for an agent to determine how to accurately group similar states. One proposed technique clusters states by their observed outcomes. To make it possible to extrapolate observed outcomes to as-yet unvisited states, a second approach uses perceptual information such as the output of an image-processing system to group perceptually similar states with the hope that they will also be related in terms of outcomes. However, there are many different percepts from which a robot could obtain state groupings. To address this issue, a third algorithm is presented that determines how to group states when the agent has multiple, possibly conflicting, inputs from which to choose. Robot experiments of all algorithms proposed are included to demonstrate the improvements that can be obtained by using the approaches presented.