Posterior Sampling-Based Reinforcement Learning for Control of Unknown Linear Systems

Abstract

We propose a posterior sampling-based learning algorithm for the linear quadratic (LQ) control problem with unknown system parameters. The algorithm is called posterior sampling-based reinforcement learning for LQ regulator (PSRL-LQ) where two stopping criteria determine the lengths of the dynamic episodes in posterior sampling. The first stopping criterion controls the growth rate of episode length. The second stopping criterion is triggered when the determinant of the sample covariance matrix is less than half of the previous value. We show under some conditions on the prior distribution that the expected (Bayesian) regret of PSRL-LQ accumulated up to time $T$ is bounded by $\tilde{O}(\sqrt{T})$ . Here, $\tilde{O}(\cdot)$ hides constants and logarithmic factors. Numerical simulations are provided to illustrate the performance of PSRL-LQ.