Constrained and Stabilizing Stacked Adaptive Dynamic Programming and a Comparison with Model Predictive Control

Abstract

Model predictive control (MPC) is in many applications the de facto approach to optimal control. It typically provides an optimal input (sequence) for a finite-horizon of given running costs. Another approach, called dynamic programming (DP), is based on the Hamilton-Jacobi-Bellman formalism and usually seeks optimal inputs over an infinite horizon of running costs. Unlike MPC, DP is much less computationally tractable and typically requires state space discretization which leads to the so-called curse of dimensionality. Adaptive dynamic programming (ADP), an approach based on reinforcement learning, seeks to address the difficulties of DP by introducing approximation models for the optimal cost function and control policies. In a variant of ADP called stacked ADP (sADP), control policies are optimized over a finite stack of value function approximants, thus making it somewhat similar to MPC. First, similarities and differences between a variant of ADP and MPC are discussed. Second, MPC stability results are transferred to ADP and state and input constraints are considered. The work is concluded by a case study.