Reinforcement learning for optimal tracking and regulation: A unified framework

Abstract

Reinforcement learning (RL) has been widely used to design feedback controllers for both discrete-time and continuous-time dynamical systems. This technique allows for the design of a class of adaptive controllers that learn optimal control solutions forward in time, and without knowing the full system dynamics. Integral reinforcement learning (IRL) and off-policy RL algorithms for continuous-time (CT) systems, and Q-learning and heuristic dynamic programing for discrete-time (DT) systems have been successfully used to learn the optimal control solutions, online in real time. The application of these methods, however, has been mostly limited to the design of optimal regulators. Nevertheless, in practice it is often required to force the states or outputs of the system to track a reference (desired) trajectory. A unified framework for both tracking and regulation problems is defined here and it is shown here how we can develop online model-free RL algorithms to solve the unified tracking and regulation control problem for both CT and DT systems.