Policy Iteration Approach Research Articles

Data-based robust optimal control of continuous-time affine nonlinear systems with matched uncertainties

In this paper, the robust optimal control of continuous-time affine nonlinear systems with matched uncertainties is investigated by using a data-based integral policy iteration approach. It is a natural extension of the traditional optimal control design, under the framework of adaptive dynamic programming (ADP) method, to robust optimal control of nonlinear systems with matched uncertainties. In theoretical aspect, by increasing a feedback gain to the optimal controller of the nominal system, the robust controller of the matched uncertain system is obtained, which also achieves optimality with a newly well-defined cost function. When regarding the implementation, the data-based integral policy iteration algorithm is used to solve the Hamilton–Jacobi–Bellman equation corresponding to the nominal system with completely unknown dynamics information. Then, the actor-critic technique based on neural networks and least squares implementation method are employed to facilitate deriving the optimal control law iteratively, so that the closed-form expression of the robust optimal controller is available. Additionally, two simulation examples with application backgrounds are presented to illustrate the effectiveness of the established robust optimal control scheme. In summary, it is important to note that the result developed in this paper broadens the application scope of ADP-based optimal control approach to more general nonlinear systems possessing dynamical uncertainties.

Policy Gradient Approach of Event‐Based Optimization and Its Online Implementation

AbstractIn the theory of event‐based optimization (EBO), the decision making is triggered by events, which is different from the traditional state‐based control in Markov decision processes (MDP). In this paper, we propose a policy gradient approach of EBO. First, an equation of performance gradient in the event‐based policy space is derived based on a fundamental quantity called Q‐factors of EBO. With the performance gradient, we can find the local optimum of EBO using the gradient‐based algorithm. Compared to the policy iteration approach in EBO, this policy gradient approach does not require restrictive conditions and it has a wider application scenario. The policy gradient approach is further implemented based on the online estimation of Q‐factors. This approach does not require the prior information about the system parameters, such as the transition probability. Finally, we use an EBO model to formulate the admission control problem and demonstrate the main idea of this paper. Such online algorithm provides an effective implementation of the EBO theory in practice.

Temporal Difference-Based Policy Iteration for Optimal Control of Stochastic Systems

In this paper, a unified policy iteration approach is presented for the optimal control problem of stochastic system with discounted average cost and continuous state space. The approach consists o...

A Neural-Network based Flight Controller for UASs

This paper presents a nonlinear gust-attenuation controller based on constrained neural-network (NN) theory. The controller aims to achieve sufficient stability and handling quality for a fixed-wing unmanned aerial system (UAS) in a gusty environment when control inputs are subjected to constraints. Constraints in inputs emulate situations where aircraft actuators fail requiring the aircraft to be operated with fail-safe capability. The proposed controller enables gust-attenuation property and stabilizes the aircraft dynamics in a gusty environment. The proposed flight controller is obtained by solving the Hamilton-Jacobi-Isaacs (HJI) equations based on an policy iteration (PI) approach. Performance of the controller is evaluated using a high-fidelity six degree-of-freedom Shadow UAS model. Simulations show that our controller demonstrates great performance improvement in a gusty environment, especially in angle-of-attack (AOA), pitch and pitch rate. Comparative studies are conducted with the proportional-integral-derivative (PID) controllers, justifying the efficiency of our controller and verifying its suitability for integration into the design of flight control systems for forced landing of UASs.

A policy iteration approach to online optimal control of continuous-time constrained-input systems

This paper is an effort towards developing an online learning algorithm to find the optimal control solution for continuous-time (CT) systems subject to input constraints. The proposed method is based on the policy iteration (PI) technique which has recently evolved as a major technique for solving optimal control problems. Although a number of online PI algorithms have been developed for CT systems, none of them take into account the input constraints caused by actuator saturation. In practice, however, ignoring these constraints leads to performance degradation or even system instability. In this paper, to deal with the input constraints, a suitable nonquadratic functional is employed to encode the constraints into the optimization formulation. Then, the proposed PI algorithm is implemented on an actor–critic structure to solve the Hamilton–Jacobi–Bellman (HJB) equation associated with this nonquadratic cost functional in an online fashion. That is, two coupled neural network (NN) approximators, namely an actor and a critic are tuned online and simultaneously for approximating the associated HJB solution and computing the optimal control policy. The critic is used to evaluate the cost associated with the current policy, while the actor is used to find an improved policy based on information provided by the critic. Convergence to a close approximation of the HJB solution as well as stability of the proposed feedback control law are shown. Simulation results of the proposed method on a nonlinear CT system illustrate the effectiveness of the proposed approach.

Policy iteration for bounded-parameter POMDPs

POMDP is considered as a basic model for decision making under uncertainty. As a generalization of the exact POMDP model, the bounded-parameter POMDP (BPOMDP) provides only upper and lower bounds on the state-transition probabilities, observation probabilities and rewards, which is particularly suitable for characterizing the situations where the underlying model is imprecisely given or time-varying. This paper presents the optimistic criterion for optimality for solving BPOMDPs, under which the optimistically optimal value function is defined. By representing a policy explicitly as a finite-state controller, we propose a policy iteration approach that is shown to converge to an $$\epsilon$$ -optimal policy under the optimistic optimality criterion.

Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics

This paper presents a novel policy iteration approach for finding online adaptive optimal controllers for continuous-time linear systems with completely unknown system dynamics. The proposed approach employs the approximate/adaptive dynamic programming technique to iteratively solve the algebraic Riccati equation using the online information of state and input, without requiring the a priori knowledge of the system matrices. In addition, all iterations can be conducted by using repeatedly the same state and input information on some fixed time intervals. A practical online algorithm is developed in this paper, and is applied to the controller design for a turbocharged diesel engine with exhaust gas recirculation. Finally, several aspects of future work are discussed.

Average control of Markov decision processes with Feller transition probabilities and general action spaces

This paper studies the average control problem of discrete-time Markov Decision Processes (MDPs for short) with general state space, Feller transition probabilities, and possibly non-compact control constraint sets A(x). Two hypotheses are considered: either the cost function c is strictly unbounded or the multifunctions Ar(x)={a∈A(x):c(x,a)≤r} are upper-semicontinuous and compact-valued for each real r. For these two cases we provide new results for the existence of a solution to the average-cost optimality equality and inequality using the vanishing discount approach. We also study the convergence of the policy iteration approach under these conditions. It should be pointed out that we do not make any assumptions regarding the convergence and the continuity of the limit function generated by the sequence of relative difference of the α-discounted value functions and the Poisson equations as often encountered in the literature.

Q-learning and policy iteration algorithms for stochastic shortest path problems

We consider the stochastic shortest path problem, a classical finite-state Markovian decision problem with a termination state, and we propose new convergent Q-learning algorithms that combine elements of policy iteration and classical Q-learning/value iteration. These algorithms are related to the ones introduced by the authors for discounted problems in Bertsekas and Yu (Math. Oper. Res. 37(1):66-94, 2012). The main difference from the standard policy iteration approach is in the policy evaluation phase: instead of solving a linear system of equations, our algorithm solves an optimal stopping problem inexactly with a finite number of value iterations. The main advantage over the standard Q-learning approach is lower overhead: most iterations do not require a minimization over all controls, in the spirit of modified policy iteration. We prove the convergence of asynchronous deterministic and stochastic lookup table implementations of our method for undiscounted, total cost stochastic shortest path problems. These implementations overcome some of the traditional convergence difficulties of asynchronous modified policy iteration, and provide policy iteration-like alternative Q-learning schemes with as reliable convergence as classical Q-learning. We also discuss methods that use basis function approximations of Q-factors and we give an associated error bound.

Combined Fixed Point and Policy Iteration for Hamilton--Jacobi--Bellman Equations in Finance

Implicit methods for Hamilton--Jacobi--Bellman (HJB) partial differential equations give rise to highly nonlinear discretized algebraic equations. The classic policy iteration approach may not be efficient in many circumstances. In this article, we derive sufficient conditions to ensure convergence of a combined fixed point policy iteration scheme for the solution of discretized equations. Numerical examples are included for a singular stochastic control problem arising in insurance (a guaranteed minimum withdrawal benefit), where the underlying risky asset follows a jump diffusion, and an American option assuming a regime switching process.

Geodesic Gaussian kernels for value function approximation

The least-squares policy iteration approach works efficiently in value function approximation, given appropriate basis functions. Because of its smoothness, the Gaussian kernel is a popular and useful choice as a basis function. However, it does not allow for discontinuity which typically arises in real-world reinforcement learning tasks. In this paper, we propose a new basis function based on geodesic Gaussian kernels, which exploits the non-linear manifold structure induced by the Markov decision processes. The usefulness of the proposed method is successfully demonstrated in simulated robot arm control and Khepera robot navigation.

Policy iteration based feedback control

It is well known that stochastic control systems can be viewed as Markov decision processes (MDPs) with continuous state spaces. In this paper, we propose to apply the policy iteration approach in MDPs to the optimal control problem of stochastic systems. We first provide an optimality equation based on performance potentials and develop a policy iteration procedure. Then we apply policy iteration to the jump linear quadratic problem and obtain the coupled Riccati equations for their optimal solutions. The approach is applicable to linear as well as nonlinear systems and can be implemented on-line on real world systems without identifying all the system structure and parameters.

Average optimality for continuous-time Markov decision processes with a policy iteration approach

This paper deals with the average expected reward criterion for continuous-time Markov decision processes in general state and action spaces. The transition rates of underlying continuous-time jump Markov processes are allowed to be unbounded, and the reward rates may have neither upper nor lower bounds. We give conditions on the system's primitive data and under which we prove the existence of the average reward optimality equation and an average optimal stationary policy. Also, under our conditions we ensure the existence of ϵ-average optimal stationary policies. Moreover, we study some properties of average optimal stationary policies. We not only establish another average optimality equation on an average optimal stationary policy, but also present an interesting “martingale characterization” of such a policy. The approach provided in this paper is based on the policy iteration algorithm. It should be noted that our way is rather different from both the usually “vanishing discounting factor approach” and the “optimality inequality approach” widely used in the previous literature.

Pavement Resurfacing Planning for Highway Networks: Parametric Policy Iteration Approach

This paper presents a modeling framework for planning pavement resurfacing activities on highway networks in the case of continuous pavement state, discrete time, and infinite horizon. Optimal resurfacing policies that minimize discounted life-cycle costs are obtained by solving a multidimensional dynamic program, where travelers’ route choices and the agency’s resource allocation decisions are considered simultaneously. To reduce computational difficulty, policy iteration is used together with a parametric function approximation technique. Numerical examples show that the proposed approach solves the planning problem efficiently. The effect of various factors (travelers’ route choice, agency budget, etc.) on the optimal policies is also analyzed.

A Dynamic Nash Game Model of Oil Market Disruption and Strategic Stockpiling

In this paper we present and analyze a Nash dynamic game model for investigating public and private sector oil inventory policies in unstable world oil markets. Conceptual results about the model include simple, verifiable conditions for uniqueness and stability of solutions, along with characterizations of optimal policies. We also present results and discuss computational issues that arise in determining subgame perfect infinite-horizon equilibria, using a steady-state policy iteration approach.

Capacity Planning for Large Multilocation Systems: Approximate and Incomplete Dynamic Programming Approaches

Methods for planning capacity expansion typically have been restricted to problems with many locations in a static environment or a few locations in a dynamic environment. Two approaches are developed here for dynamic capacity planning problems with many locations. The first is an approximate approach based on an equivalent annual cost measure, and the second is a procedure for systematic improvement of the approximate solution. The method for improvement is called “incomplete dynamic programming” since it consists of an approximation to the first cycle of the dynamic programming policy iteration approach. Computational results are reported for tests of the methods against dynamic programming solutions for small problems. Applications are made to two versions of a large-scale problem of planning capacity expansion for India's nitrogenous fertilizer industry, and results are compared with those for other approaches.

RAPID CONVERGENCE TECHNIQUES FOR MARKOV DECISION PROCESSES*

When a person is working with large scale Markov Decision Processes, he normally uses the policy iteration approach developed by Howard [1] and modified by White [3]. White's modification makes use of the method of successive approximations. Computational experience has shown that for many processes, the rate of convergence of the successive approximation is very slow. In this paper, techniques for speeding convergence are discussed. Numerical examples and computational experience which show the relative merits of the various approaches are presented.

Structured markovian decision problems

AbstractStructured finite action‐finite state space discounted Markovian decision problems are analyzed. Any problem of a general class is shown to be equivalent to a “separated” problem with decomposable problem structure. A modified policy iteration approach is developed for this decomposable reformulation. Both analytic and computer evaluations of the decomposition algorithm's effectiveness are presented.