Robust and explorative behavior in model-based Bayesian reinforcement learning

Abstract

This paper considers a tractable simplified problem of model-based Bayesian reinforcement learning (BRL) in terms of real-world samples, computational complexity, and target uncertainties. Robust control and adaptive control are two of the most successful and tractable conventional control design theories against uncertainties in various domain, while they have contrasting ideas. We show that both theories can be explained in a unified manner by approximation model-based BRL algorithms. We propose a forward search tree with robust solutions as a simplified tractable problem, which explicitly includes both theories at the same time. While the structure of the problem has been already seen in a branch and bound method, we provide a novel analysis of the behaviors resulting from it. Through a simple example, we compare the solutions of the proposed problem with the optimal BRL solution and the two conventional approaches, and discuss the interpretation.