Policy Iteration Research Articles

Heuristic algorithm for nested Markov decision process: Solution quality and computational complexity

A nested Markov decision process (NMDP) is a multi-level MDP consisting of an outer MDP and several inner MDPs. These MDPs are dependent on each other such that, each state of the outer MDP induces a unique inner MDP. We propose for the first time an algorithm to solve an infinite horizon nested Markov decision process under the average reward criterion (NMDP-HA). The algorithm incorporates the policy iteration method, which is composed of the value-determination operation and the policy improvement routine. To evaluate the solution quality and computational complexity of the NMDP-HA, we develop a specialized enumerative algorithm adapted from a completely observable MDP equivalent of the NMDP problem. The proposed NMDP-HA is illustrated with several numerical examples and our results for the problem instances evaluated indicate that the heuristic algorithm can find the same optimal solution in a fraction of the time the total enumeration algorithm uses to exhaustively search the entire solution space. For the cases, where the optimal solution is not found, the percentage deviation from the optimal solution is less than 5%.

Input-output data based tracking control under DoS attacks

ABSTRACT This paper investigates the secure optimal tracking control problem for cyber-physical systems, in which the controller-actuator channel is jammed by denial-of-service attacks. Without leveraging model knowledge, learning-based tracking control algorithms are proposed by utilizing the measured input-output data. The impact of malicious attacks on tracking performance is discussed. More specifically, an augmented system consisted of the system model and the reference model is derived. Then the states of the augmented system are rewritten by utilizing the input, output, and reference trajectory, following which both the Bellman equation and algebraic Riccati equation are given. The conditions of existence and uniqueness of the solution to the algebraic Riccati equation are derived. Furthermore, both policy iteration and value iteration learning-based tracking control algorithms are provided for cyber-physical systems against denial-of-service attacks, and the convergence of the algorithms is also proved. Finally, a DC motor system and a numerical example are given to illustrate the effectiveness of the proposed control algorithms.

Solution for Pursuit-Evasion Game of Agents by Adaptive Dynamic Programming

The paper studies a novel method for real-time solutions of the two-player pursuit-evasion game. The min-max principle is adopted to confirm the Nash equilibrium of the game. As agents in the game can form an Internet of Things (IoT) system, the real-time control law of each agent is obtained by taking a linear-quadratic cost function in adaptive dynamic programming. By introducing the Lyapunov function, we consider the scenario when capture occurs. Since most actual systems are continuous, the policy iteration algorithm is used to make the real-time policy converge to the analytical solution of the Nash equilibrium. Furthermore, we employ the value function approximation method to calculate the neural network parameters without directly solving the Hamilton–Jacobi–Isaacs equation. Simulation results depict the method’s feasibility in different scenarios of the pursuit-evasion game.

Learning equilibrium mean‐variance strategy

AbstractWe study a dynamic mean‐variance portfolio optimization problem under the reinforcement learning framework, where an entropy regularizer is introduced to induce exploration. Due to the time–inconsistency involved in a mean‐variance criterion, we aim to learn an equilibrium policy. Under an incomplete market setting, we obtain a semi‐analytical, exploratory, equilibrium mean‐variance policy that turns out to follow a Gaussian distribution. We then focus on a Gaussian mean return model and propose a reinforcement learning algorithm to find the equilibrium policy. Thanks to a thoroughly designed policy iteration procedure in our algorithm, we prove the convergence of our algorithm under mild conditions, despite that dynamic programming principle and the usual policy improvement theorem failing to hold for an equilibrium policy. Numerical experiments are given to demonstrate our algorithm. The design and implementation of our reinforcement learning algorithm apply to a general market setup.

Overcoming the Long Horizon Barrier for Sample-Efficient Reinforcement Learning with Latent Low-Rank Structure

The practicality of reinforcement learning algorithms has been limited due to poor scaling with respect to the problem size, as the sample complexity of learning an ε-optimal policy is Ω(|S||A|H/ ε2) over worst case instances of an MDP with state space S, action space A, and horizon H. We consider a class of MDPs for which the associated optimal Q* function is low rank, where the latent features are unknown. While one would hope to achieve linear sample complexity in |S| and |A| due to the low rank structure, we show that without imposing further assumptions beyond low rank of Q*, if one is constrained to estimate the Q function using only observations from a subset of entries, there is a worst case instance in which one must incur a sample complexity exponential in the horizon H to learn a near optimal policy. We subsequently show that under stronger low rank structural assumptions, given access to a generative model, Low Rank Monte Carlo Policy Iteration (LR-MCPI) and Low Rank Empirical Value Iteration (LR-EVI) achieve the desired sample complexity of Õ((|S|+|A|)poly (d,H)/ε2) for a rank d setting, which is minimax optimal with respect to the scaling of |S|, |A|, and ε. In contrast to literature on linear and low-rank MDPs, we do not require a known feature mapping, our algorithm is computationally simple, and our results hold for long time horizons. Our results provide insights on the minimal low-rank structural assumptions required on the MDP with respect to the transition kernel versus the optimal action-value function.

Optimal entanglement distribution policies in homogeneous repeater chains with cutoffs

We study the limits of bipartite entanglement distribution using a chain of quantum repeaters that have quantum memories. To generate end-to-end entanglement, each node can attempt the generation of an entangled link with a neighbor, or perform an entanglement swapping measurement. A maximum storage time, known as cutoff, is enforced on the memories to ensure high-quality entanglement. Nodes follow a policy that determines when to perform each operation. Global-knowledge policies take into account all the information about the entanglement already produced. Here, we find global-knowledge policies that minimize the expected time to produce end-to-end entanglement. Our methods are based on Markov decision processes and value and policy iteration. We compare optimal policies to a policy in which nodes only use local information. We find that the advantage in expected delivery time provided by an optimal global-knowledge policy increases with increasing number of nodes and decreasing probability of successful swapping.

A Novel Data-Based Fault-Tolerant Control Method for Multicontroller Linear Systems via Distributed Policy Iteration

This article presents a novel data-based fault-tolerant control method for multicontroller linear systems via distributed policy iteration. The traditional fault-tolerant control methods based on policy iteration may cause a huge-computational burden under the situation of high-dimension control laws. In order to solve this problem, a novel distributed policy iteration method is presented, where only one iterative control law is updated in each iteration, to realize the fault-tolerant control of multicontroller linear systems. The main contributions can be highlighted as follows: 1) a novel data-based distributed policy iteration method is presented to reduce the computational burden; 2) the fault-tolerant control method is presented via designing fault compensators; and 3) the developed data-based method only requires the partial system information. First, the model-based fault-tolerant control via distributed policy iteration and fault compensation is provided. Based on the model-based method, a data-based fault-tolerant control method is presented. Finally, numerical experiments are given to show the performance of the presented method.

Model-Based Policy Iterations for Nonlinear Systems via Controlled Hamiltonian Dynamics

The infinite-horizon optimal control problem for nonlinear systems is studied. In the context of model-based, iterative learning strategies we propose an alternative definition and construction of the <i>temporal difference error</i> arising in Policy Iteration strategies. In such architectures the error is computed via the evolution of the Hamiltonian function (or, possibly, of its integral) along the trajectories of the closed-loop system. Herein the temporal difference error is instead obtained via two subsequent steps: first the dynamics of the underlying costate variable in the Hamiltonian system is steered by means of a (virtual) control input in such a way that the stable invariant manifold becomes externally attractive. Then, the <i>distance-from-invariance</i> of the manifold, induced by approximate solutions, yields a natural candidate measure for the <i>policy evaluation</i> step. The <i>policy improvement</i> phase is then performed by means of standard gradient descent methods that allows to correctly update the weights of the underlying functional approximator. The above architecture then yields an <i>iterative (episodic) learning</i> scheme based on a scalar, constant <i>reward</i> at each iteration, the value of which is insensitive to the length of the episode, as in the original spirit of Reinforcement Learning strategies for discrete-time systems. Finally, the theory is validated by means of a numerical simulation involving an automatic flight control problem.

Policy Iteration Based Approximate Dynamic Programming Toward Autonomous Driving in Constrained Dynamic Environment

In the area of autonomous driving, it typically brings great difficulty in solving the motion planning problem since the vehicle model is nonlinear and the driving scenarios are complex. Particularly, most of the existing methods cannot be generalized to dynamically changing scenarios with varying surrounding vehicles. To address this problem, this development here investigates the framework of integrated decision and control. As part of the modules, static path planning determines the reference candidates ahead, and then the optimal path-tracking controller realizes the specific autonomous driving task. An innovative and effective constrained finite-horizon approximate dynamic programming (ADP) algorithm is herein presented to generate the desired control policy for effective path tracking. With the generalized policy neural network that maps from the state to the control input, the proposed algorithm preserves the high effectiveness for the motion planning problem towards changing driving environments with varying surrounding vehicles. Moreover, the algorithm attains the noteworthy advantage of alleviating the typically heavy computational loads with the mode of offline training and online execution. As a result of the utilization of multi-layer neural networks in conjunction with the actor-critic framework, the constrained ADP method is capable of handling complex and multidimensional scenarios. Finally, various simulations have been carried out to show that the constrained ADP algorithm is effective.

Inverse Reinforcement Q-Learning Through Expert Imitation for Discrete-Time Systems.

In inverse reinforcement learning (RL), there are two agents. An expert target agent has a performance cost function and exhibits control and state behaviors to a learner. The learner agent does not know the expert's performance cost function but seeks to reconstruct it by observing the expert's behaviors and tries to imitate these behaviors optimally by its own response. In this article, we formulate an imitation problem where the optimal performance intent of a discrete-time (DT) expert target agent is unknown to a DT Learner agent. Using only the observed expert's behavior trajectory, the learner seeks to determine a cost function that yields the same optimal feedback gain as the expert's, and thus, imitates the optimal response of the expert. We develop an inverse RL approach with a new scheme to solve the behavior imitation problem. The approach consists of a cost function update based on an extension of RL policy iteration and inverse optimal control, and a control policy update based on optimal control. Then, under this scheme, we develop an inverse reinforcement Q-learning algorithm, which is an extension of RL Q-learning. This algorithm does not require any knowledge of agent dynamics. Proofs of stability, convergence, and optimality are given. A key property about the nonunique solution is also shown. Finally, simulation experiments are presented to show the effectiveness of the new approach.

Cooperative Assist-as-Needed Control for Robotic Rehabilitation: A Two-Player Game Approach

This paper presents an adaptive optimal control strategy for developing assist-as-needed robotic rehabilitation. The primary goal is to encourage patient participation and increase the effectiveness of training sessions by minimizing robot intervention while following a predefined path. To achieve this, the problem is modeled as a two-player non-zero-sum game, with cooperative and individual objectives for the human and robot specified as different cost functions. Policy iteration techniques are adopted for learning the optimal solution online. The shared autonomy feature is specifically achieved through seamless adaptation of the robot's autonomy according to its estimation of human intention. The performance of the proposed approach is illustrated in several simulations and experimental studies.

Relaxed Actor-Critic With Convergence Guarantees for Continuous-Time Optimal Control of Nonlinear Systems

This paper presents the Relaxed Continuous-Time Actor-critic (RCTAC) algorithm, a method for finding the nearly optimal policy for nonlinear continuous-time (CT) systems with known dynamics and infinite horizon, such as the path-tracking control of vehicles. RCTAC has several advantages over existing adaptive dynamic programming algorithms for CT systems. It does not require the ``admissibility" of the initialized policy or the input-affine nature of controlled systems for convergence. Instead, given any initial policy, RCTAC can converge to an admissible, and subsequently nearly optimal policy for a general nonlinear system with a saturated controller. RCTAC consists of two phases: a warm-up phase and a generalized policy iteration phase. The warm-up phase minimizes the square of the Hamiltonian to achieve admissibility, while the generalized policy iteration phase relaxes the update termination conditions for faster convergence. The convergence and optimality of the algorithm are proven through Lyapunov analysis, and its effectiveness is demonstrated through simulations and real-world path-tracking tasks.

Human-in-the-loop distributed cooperative tracking control with applications to autonomous ground vehicles: A data-driven mixed iteration approach

This paper proposes a data-driven learning algorithm for the human-in-the-loop cooperative tracking control of multi-agent systems with completely unknown dynamics. The core role of the human is to supervise a team of agents via sending a control command to only the leader vehicle, where the leader is a non-autonomous agent that has a nonzero but bounded control input receiving the control command from the human operator. To effectively reduce communication frequency among vehicles, a distributed observer only using the relative input–output information is projected to approximate the relative state information of agents. Then, taking the transient state into account instead of the steady-state, a value iteration (VI) algorithm is proposed to solve the optimal output trajectory tracking control without an initial stabilizing control policy but sacrificing the convergence rate. Under the VI algorithm, a mixed iteration (MI) algorithm by making the full advantages of both VI and the traditional policy iteration algorithms is developed, which not only guarantees the convergence rate but also removes the requirement of an initial stabilizing control policy. Finally, a practical multi-robot network is provided to verify and compare the advantages of the proposed theoretical algorithm.

Value-Gradient Based Formulation of Optimal Control Problem and Machine Learning Algorithm

Optimal control problem is typically solved by first finding the value function through the Hamilton–Jacobi equation (HJE) and then taking the minimizer of the Hamiltonian to obtain the control. In this work, instead of focusing on the value function, we propose a new formulation for the gradient of the value function (value-gradient) as a decoupled system of partial differential equations in the context of a continuous-time deterministic discounted optimal control problem. We develop an efficient iterative scheme for this system of equations in parallel by utilizing the fact that they share the same characteristic curves as the HJE for the value function. For the theoretical part, we prove that this iterative scheme converges linearly in sense for some suitable exponent in a weight function. For the numerical method, we combine a characteristic line method with machine learning techniques. Specifically, we generate multiple characteristic curves at each policy iteration from an ensemble of initial states and compute both the value function and its gradient simultaneously on each curve as the labeled data. Then supervised machine learning is applied to minimize the weighted squared loss for both the value function and its gradients. Experimental results demonstrate that this new method not only significantly increases the accuracy but also improves the efficiency and robustness of the numerical estimates, particularly with less characteristics data or fewer training steps.

Distributed output data-driven optimal robust synchronization of heterogeneous multi-agent systems

This work presents an output-feedback policy learning algorithm underlining input–output system data for distributed robust optimal synchronization of heterogeneous multi-agent systems. The output-feedback synchronization problem in the context of this work is formulated via robust output regulation and reinforcement learning modeling the interactions among agents by a zero-sum game. The proposed learning and control structure only requires the local system data for each agent and distributed output data among communicating neighbors. We utilize system-level synchysis for the continuous-time state reconstruction for the distributed learning with convergence and stability proofs under the proposed output-feedback policy for solving the zero-sum game. We further show that policy learning is assured under the proposed data criteria relating to input–output data only rather than any inter-immediate gains from policy iterations. Based on the cooperative robust output regulation, this work gains robustness after the learning is complete and establishes an output data-driven distributed optimal robust synchronization without knowing accurate system dynamics. A numerical example shows the effectiveness of the proposed learning algorithm.

Optimal [formula omitted] tracking control of nonlinear systems with zero-equilibrium-free via novel adaptive critic designs

In this paper, a novel adaptive critic control method is designed to solve an optimal H∞ tracking control problem for continuous nonlinear systems with nonzero equilibrium based on adaptive dynamic programming (ADP). To guarantee the finiteness of a cost function, traditional methods generally assume that the controlled system has a zero equilibrium point, which is not true in practical systems. In order to overcome such obstacle and realize H∞ optimal tracking control, this paper proposes a novel cost function design with respect to disturbance, tracking error and the derivative of tracking error. Based on the designed cost function, the H∞ control problem is formulated as two-player zero-sum differential games, and then a policy iteration (PI) algorithm is proposed to solve the corresponding Hamilton–Jacobi–Isaacs (HJI) equation. In order to obtain the online solution to the HJI equation, a single-critic neural network structure based on PI algorithm is established to learn the optimal control policy and the worst-case disturbance law. It is worth mentioning that the proposed adaptive critic control method can simplify the controller design process when the equilibrium of the systems is not zero. Finally, simulations are conducted to evaluate the tracking performance of the proposed control methods.

Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems

This article studies the adaptive optimal stationary control of continuous-time linear stochastic systems with both additive and multiplicative noises, using reinforcement learning techniques. Based on policy iteration, a novel off-policy reinforcement learning algorithm, named optimistic least-squares-based policy iteration, is proposed, which is able to find iteratively near-optimal policies of the adaptive optimal stationary control problem directly from input/state data without explicitly identifying any system matrices, starting from an initial admissible control policy. The solutions given by the proposed optimistic least-squares-based policy iteration are proved to converge to a small neighborhood of the optimal solution with probability one, under mild conditions. The application of the proposed algorithm to a triple inverted pendulum example validates its feasibility and effectiveness.

Approximating the Stationary Bellman Equation by Hierarchical Tensor Products

We treat infinite horizon optimal control problems by solving the associated stationary Hamilton-Jacobi-Bellman (HJB) equation numerically to compute the value function and an optimal feedback law. The dynamical systems under consideration are spatial discretizations of non linear parabolic partial differential equations (PDE), which means that the HJB is non linear and suffers from the curse of dimensionality. Its non linearity is handled by the Policy Iteration algorithm, where the problem is reduced to a sequence of linear, hyperbolic PDEs. These equations remain the computational bottleneck due to their high dimensions. By the method of characteristics these linearized HJB equations can be reformulated via the Koopman operator in the spirit of dynamic programming. The resulting operator equations are solved using a minimal residual method. To overcome numerical infeasability we use low rank hierarchical tensor product approximation/tree-based tensor formats, in particular tensor trains (TT tensors), and multi-polynomials, together with high dimensional quadrature, e.g. Monte-Carlo. By controlling a destabilized version of viscous Burgers and a diffusion equation with unstable reaction term numerical evidences are given.

Route optimization of vacant taxicab considering sequential dependence in abstract grid network based on quadtree

ABSTRACT In this paper, we propose a method of route optimization of vacant taxicab based on the abstract grid network. Firstly, an abstract grid network is built by the quadtree technology with the vacant trajectory data embedded. Secondly, the Markov decision process framework considering the sequential dependence is used to model the problem. Finally, we solve this problem by the policy iteration and compare the method by taking the GPS trajectory data in different hours of 797 taxicabs in a typical working day in Shenzhen, comparing that with the optimal strategy without considering sequential dependence, MNP, local hotspot and random walk algorithm by simulation, respectively. The results show that: The use of quadtree can reduce the sparseness of matching probability from 80.46% to 30.18%. The average revenue per unit distance has increased by 6.10%, 8.75%, 33.24% and 60.06%. The vacant driving rate has decreased by 5.33%, 6.15%, 18.86%, and 23.18%.

Composite robust control of uncertain nonlinear systems with unmatched disturbances using policy iteration

In this paper, the composite robust control problem of uncertain nonlinear systems with unmatched disturbances is investigated. In order to improve the robust control performance, the integral sliding mode control method is considered together with H∞ control for nonlinear systems. By designing a disturbance observer with a new structure, the estimations of disturbances can be obtained with small errors, which are used to construct sliding mode control policy and avoid high gains. On the basis of ensuring the accessibility of specified sliding surface, the guaranteed cost control problem of nonlinear sliding mode dynamics is considered. To overcome the difficulty of robust control design caused by nonlinear characteristics, a modified policy iteration method based on sum of squares is proposed to solve the H∞ control policy of the nonlinear sliding mode dynamics. Finally, the effectiveness of the proposed robust control method is verified by simulation tests.