Error analysis of conventional discrete and gradient dynamic programming

Abstract

An asymptotic error analysis of the conventional discrete dynamic programming (DDP) method is presented, and upper bounds of the error in the control policy (i.e., the difference of the estimated and true optimal control) at each operation period are computed. This error is shown to be of the order of the state discretization interval (ΔS), a result with significant implications in the optimization of multistate systems where the “curse of dimensionality” restricts the number of states to a relatively small number. The error in the optimal cost varies with ΔS2. The analysis provides useful insights into the effects of state discretization on calculated control and cost functions, the comparability of results from different discretizations, and criteria about the required number of nodes. In an effort to reduce the discretization error in the case of smooth cost functions, a new discrete dynamic programming method, termed gradient dynamic programming (GDP), is proposed. GDP uses a piecewise Hermite interpolation of the cost‐to‐go function, at each stage, which preserves the values of the cost‐to‐go function and of its first derivatives at the discretization nodes. The error in the control policy is shown to be of the order of (ΔS)3 and the error in the cost to vary with ΔS4. Thus as ΔS decreases, GDP converges to the true optimum much more rapidly than DDP. Another major advantage of the new methodology is that it facilitates the use of Newton‐type iterative methods in the solution of the nonlinear optimization problems at each stage. The linear convergence of DDP and the superlinear convergence of GDP are illustrated in an example.