Model-Free Reinforcement Learning of Minimal-Cost Variance Control

Abstract

This letter proposes two reinforcement learning (RL) algorithms for solving a class of coupled algebraic Riccati equations (CARE) for linear stochastic dynamic systems with unknown state and input matrices. The CARE are formulated for a minimal-cost variance (MCV) control problem that aims to minimize the variance of a cost function while keeping its mean at an acceptable range using a noisy infinite-horizon full-state feedback linear quadratic regulator (LQR). We propose two RL algorithms where the input matrix can be estimated at the very first iteration. This, in turn, frees up significant amount of computational complexity in the intermediate steps of the learning phase by avoiding repeated matrix inversion of a high-dimensional data matrix. The overall complexity is shown to be less than RL for both stochastic and deterministic LQR. Additionally, the disturbance noise entering the model is not required to satisfy any condition for ensuring efficiency of either RL algorithms. Simulation examples are presented to illustrate the effectiveness of the two designs.