Online Synchronous Policy Iteration Method for Optimal Control

Abstract

In this chapter, we discuss an online algorithm based on policy iteration (PI) for learning the continuous-time (CT) optimal control solution for nonlinear systems with infinite horizon costs. We present an online adaptive algorithm implemented as an actor/critic structure which involves simultaneous continuous-time adaptation of both actor and critic neural networks. We call this “synchronous” PI. A persistence of excitation condition is shown to guarantee convergence of the critic to the actual optimal value function. Novel tuning algorithms are given for both critic and actor networks, with extra terms in the actor tuning law being required to guarantee closed-loop dynamical stability. The convergence to the optimal controller is proven, and stability of the system is also guaranteed. Simulation examples show the effectiveness of the new algorithm.