Policy Iteration Research Articles

GE-DDRL: Graph Embedding and Deep Distributional Reinforcement Learning for Reliable Shortest Path: A Universal and Scale Free Solution

This paper studies the reliable shortest path (RSP) problem in stochastic transportation networks. State-of-the-art RSP solutions usually target one specific RSP problem; moreover, the corresponding algorithm’s computational complexity scales at least linearly with the size of the underlying transportation network. While in this paper, we propose a graph embedding and deep distributional reinforcement learning (GE-DDRL) method, which serves as a universal and scale-free solution to the RSP problem. GE-DDRL uses deep distributional reinforcement learning (DDRL) to estimate the full travel-time distribution of a given routing policy, and improves the given routing policy with the generalized policy iteration (GPI) scheme. Further, in order to achieve the generalization ability to new destination nodes, we employ one of the canonical graph embedding techniques (Skip-Gram) to compress the nodes’ representation into <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$d$</tex-math> </inline-formula> -dimensional real-valued vectors. With the properly compressed node features, GE-DDRL is able to generalize its estimation of the routing policy’s travel-time distribution to untrained destination nodes, and hence achieve the ‘all-to-all’ navigation functionality. To the best of our knowledge, GE-DDRL serves as the first RSP planner, which applies simultaneously to almost all RSP objectives and in the meanwhile, is scale free with the size of the transportation network in terms of the online decision-making time and memory complexity. Experimental results and comparisons with state of the arts show the efficacy and efficiency of GE-DDRL in a range of transportation networks.

Stochastic Dynamic Information Flow Tracking game using supervised learning for detecting advanced persistent threats

Advanced persistent threats (APTs) are organized prolonged cyberattacks by sophisticated attackers with the intent of stealing critical information. Although APT activities are stealthy and evade detection by traditional detection tools, they interact with the system components to make progress in the attack. These interactions lead to information flows that are recorded in the form of a system log. Dynamic Information Flow Tracking (DIFT) has been shown to be an effective way to detect APTs using information flows. A DIFT-based detection mechanism dynamically performs security analysis on the information flows to detect possible attacks. However, wide range security analysis using DIFT results in a significant increase in performance overhead and high rates of false-positives and false-negatives. In this paper, we model the strategic interaction between APT and DIFT as a non-cooperative stochastic game. The game unfolds on a state space constructed from an information flow graph (IFG) that is extracted from the system log. The objective of the APT in the game is to choose transitions in the IFG to find an optimal path in the IFG from an entry point of the attack to an attack target. On the other hand, the objective of DIFT is to dynamically select nodes in the IFG to perform security analysis for detecting APT. Our game model has imperfect information as the players are unaware of the actions of the opponent. We consider two scenarios of the game (i) the false-positive and false-negative rates of DIFT (i.e., transition probabilities of the game) are known and (ii) the false-positive and false-negative rates are unknown. For case (i), we propose a value iteration-based algorithm and prove that the solution converges to the optimal solution (Nash equilibrium). Case (ii) translates to an incomplete information game with unknown transition probabilities. For case (ii), we propose a supervised learning-based algorithm, referred to as Hierarchical Supervised Learning (HSL) algorithm. HSL integrates a neural network, to predict the value vector of the game, with a policy iteration algorithm to compute an approximate equilibrium. We implemented our algorithms for cases (i) and (ii) on real attack datasets for nation state and ransomware attacks and validated the performance of our approach. We compared the performance of the HSL algorithm when the transition probabilities are unknown with instances with known transition probabilities and demonstrated that HSL algorithm converges to a solution close to optimal (i.e., optimal value vector) while the value vector obtained using greedy does not converge to optimal for 44.4% of the states and the mean absolute error is almost 200 times that of the HSL.

Quantum logic gate synthesis as a Markov decision process

Reinforcement learning has witnessed recent applications to a variety of tasks in quantum programming. The underlying assumption is that those tasks could be modeled as Markov decision processes (MDPs). Here, we investigate the feasibility of this assumption by exploring its consequences for single-qubit quantum state preparation and gate compilation. By forming discrete MDPs, we solve for the optimal policy exactly through policy iteration. We find optimal paths that correspond to the shortest possible sequence of gates to prepare a state or compile a gate, up to some target accuracy. Our method works in both the absence and presence of noise and compares favorably to other quantum compilation methods, such as the Ross–Selinger algorithm. This work provides theoretical insight into why reinforcement learning may be successfully used to find optimally short gate sequences in quantum programming.

Adaptively optimal energy management for integrated hydrogen energy systems

AbstractIntegrated hydrogen energy systems (IHESs) have become attractive alternatives to cope with the depletion of fossil fuels and increasingly severe climate change problems. This paper proposes an adaptively optimal energy scheduling method based on deep deterministic policy gradient (DDPG) to improve the operational efficiency of IHES. The optimal scheduling problem is formulated as a Markov decision process problem with action space, environmental states, and action‐value function. The DDPG‐based optimal energy management algorithm with actor‐critic structure is proposed based on policy gradients and neural networks. Through actor‐critic network training and policy iteration, the energy management scheme can be adaptively optimized according to the dynamic responses of IHES. The benefits of the proposed algorithm are analysed through time‐domain simulations, and the scheduling robustness under different uncertain conditions is verified.

NN-based reinforcement-learning optimal sliding mode control for drag-free and attitude of spacecraft with state constraints

This paper investigates a tracking control issue for a class of drag-free spacecraft with state constraints caused by the optical assembly. Firstly, a nonlinear kinematics and dynamics relative equation of coupling system of position and attitude is devised by a 6-degree of freedom (DOF) model of rigid body. Secondly, an optimal control method with a terminal functional is designed to meet state constraints. To solve the Hamilton-Jacobi-Bellman (HJB) equation generated by the optimal control of the nonlinear model, a policy iteration ideal is presented and the final numerical solution of the iteration is obtained by a critic neural network (NN). Thirdly, to enhance the robustness of the closed-loop system, an optimal sliding mode control is proposed, which can ensure the optimal and robustness performance simultaneously. At last, the simulation results demonstrate the performance of the proposed methods and the contrast of precision and robustness of two methods are showcased.

ADP based output-feedback fault-tolerant tracking control for underactuated AUV with actuators faults

In this work, the output-feedback fault-tolerant tacking control issue for underactuated autonomous underwater vehicle (AUV) with actuators faults is investigated. Firstly, an output-feedback error tacking system is constructed based on the theoretical model of underactuated AUV with actuators faults. Then, an adaptive dynamic programming (ADP) based fault-tolerant control controller is developed. In our proposed control scheme, a neural-network observer is designed to approximate the system states with actuators faults. An online policy iteration algorithm is designed with critic network and action network in order to improve the tracking accuracy. Based on Lyapunov stability theorem, the stability of the error tracking system is guaranteed by the proposed controller. At last, the simulation results show that the underactuated AUV achieves better tracking performance.

Reinforcement Learning-Based Model-Free Controller for Feedback Stabilization of Robotic Systems.

This article presents a reinforcement learning (RL) algorithm for achieving model-free control of robotic applications. The RL functions are adapted with the least-square temporal difference (LSTD) learning algorithms to develop a model-free state feedback controller by establishing linear quadratic regulator (LQR) as a baseline controller. The classical least-square policy iteration technique is adapted to establish the boundary conditions for complexities incurred by the learning algorithm. Furthermore, the use of exact and approximate policy iterations estimates the parameters of the learning functions for a feedback policy. To assess the operation of the proposed controller, the trajectory tracking and balancing control problems of unmanned helicopters and balancer robotic applications are solved for real-time experiment. The results showed the robustness of the proposed approach in achieving trajectory tracking and balancing control.

Linear-Like Policy Iteration Based Optimal Control for Continuous-Time Nonlinear Systems

We propose a novel strategy to construct optimal controllers for continuous-time nonlinear systems by means of linear-like techniques, provided that the optimal value function is differentiable and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">quadratic-like</i> . This assumption covers a wide range of cases and holds locally around an equilibrium under mild assumptions. The proposed strategy does not require solving the Hamilton-Jacobi-Bellman equation, that is a nonlinear partial differential equation, which is known to be hard or impossible to solve. Instead, the Hamilton-Jacobi-Bellman equation is replaced with an easy-solvable state-dependent Lyapunov matrix equation. We exploit a linear-like factorization of the underlying nonlinear system and a policy-iteration algorithm to yield a linear-like policy-iteration for nonlinear systems. The proposed control strategy solves optimal nonlinear control problems in an asymptotically exact, yet still linear-like manner. We prove optimality of the resulting solution and illustrate the results via four examples.

Continuous-Time Stochastic Policy Iteration of Adaptive Dynamic Programming

Continuous-Time Stochastic Policy Iteration of Adaptive Dynamic Programming

Data-Driven Policy Iteration for Nonlinear Optimal Control Problems.

The design of optimal control laws for nonlinear systems is tackled without knowledge of the underlying plant and of a functional description of the cost function. The proposed data-driven method is based only on real-time measurements of the state of the plant and of the (instantaneous) value of the reward signal and relies on a combination of ideas borrowed from the theories of optimal and adaptive control problems. As a result, the architecture implements a policy iteration strategy in which, hinging on the use of neural networks, the policy evaluation step and the computation of the relevant information instrumental for the policy improvement step are performed in a purely continuous-time fashion. Furthermore, the desirable features of the design method, including convergence rate and robustness properties, are discussed. Finally, the theory is validated via two benchmark numerical simulations.

Impact of agile, condition-based maintenance strategy on cost efficiency of production systems

Maintenance plays an increasingly important role in the life of production companies, as professional maintenance is an important prerequisite for the reliable operation of resources. A well-chosen maintenance strategy can make a major contribution to increased efficiency of production processes. The main goal of this research is to propose a novel optimization approach to define optimal maintenance strategy that ensures the efficient operation of the production process while reducing maintenance costs. The developed optimization method is based on Howard’s policy iteration and describes the objective of the planning as a Markov decision process. The novelty and the scientific contribution of the presented study is the application of Howard’s policy iteration methodology in a Markov decision process for agile, condition-based maintenance strategy optimization. As the results of the numerical analysis of the scenarios shows, the implementation of an optimized maintenance strategy based on the proposed approach can significantly increase the maintenance efficiency of the production process. The main reason for this is that the level and type of maintenance is always implemented depending on the current state of the system components, which reduces both the maintenance cost and the losses due to production downtime.

Non‐zero‐sum games of discrete‐time Markov jump systems with unknown dynamics: An off‐policy reinforcement learning method

AbstractThis article concentrates on the non‐zero‐sum games problem of discrete‐time Markov jump systems without requiring the system dynamics information. First, the multiplayer non‐zero‐sum games problem can be converted to solve a set of coupled game algebraic Riccati equations, which is difficult to be solved directly. Then, to obtain the optimal control policies, a model‐based algorithm adapting the policy iteration approach is proposed. However, the model‐based algorithm relies on system dynamics information, which has the limitations in practice. Subsequently, an off‐policy reinforcement learning algorithm is given to get rid of the dependence on system dynamics information, which only uses the information of system states and inputs. Moreover, the proof of convergence and Nash equilibrium are also given. Finally, a numerical example is given to demonstrate the effectiveness of the proposed algorithms.

Deep reinforcement learning for approximate policy iteration: convergence analysis and a post-earthquake disaster response case study

Deep reinforcement learning for approximate policy iteration: convergence analysis and a post-earthquake disaster response case study

Two-player zero-sum game based neural critic tracking control for UAUV with unknown disturbance via backstepping method

In this work, a new tracking control scheme is developed for underactuated autonomous underwater vehicle(UAUV) with unknown disturbance. First, an error tracking system is constructed using backstepping method, which transforms the tracking control problem into the two-player zero-sum game problem. One player is the optimal controller which is designed to make the UAUV track the desired trajectory. The other player is the unknown disturbance that represent the overall damping effect of the UAUV. Then, the single critic network based online policy iteration algorithm is designed to get the optimal control law and the worst-case disturbance, which can achieve the near-optimal control performance and relax the requirements for initial admissible control conditions. In order to improve the converge velocity of tracking error, the weight update law are designed. Furthermore, a discount coefficient is introduced into the performance index due to the nonlinearity and the complexity of UAUV. In addition, the stability of the UAUV is analyzed based on the Lyapunov stability theory the system is guaranteed to be uniformly ultimately bounded (UUB). Finally, the effectiveness of the proposed method is demonstrated through the real-time simulations on the UAUV model.

Optimal containment control for a class of heterogeneous multi‐agent systems with actuator faults

AbstractThis article investigates the optimal containment control problem for a class of heterogeneous multi‐agent systems with time‐varying actuator faults and unmatched disturbances based on adaptive dynamic programming. Since there exist unknown input signals in each leader, distributed observers are utilized to estimate trajectories in the convex hull spanned by leaders. The containment control problem is then transformed into an optimal tracking problem. To compensate for the actuator faults and unmatched disturbances, a novel performance index function is designed. We prove that the optimal control policy can ensure that the tracking error system of each follower is uniformly ultimately bounded. The online policy iteration algorithm is implemented using critic neural networks to obtain the optimal control policy. A numerical example is provided to demonstrate the effectiveness of the proposed control policy.

Fully cooperative games with state and input constraints using reinforcement learning based on control barrier functions

AbstractThis paper provides a novel safe reinforcement learning (RL) control algorithm to solve safe optimal problems for fully cooperative (FC) games of discrete‐time multiplayer nonlinear systems with state and input constraints. The FC game is a special case of nonzero‐sum (NZS) games, where all players cooperate to accomplish a common task. The algorithm is proposed based on the policy iteration (PI) framework utilizing only the measured data along the system trajectories in the environment. Different from most works about PI, an effective method of obtaining initial safe and stable control policies is given here. In addition, control barrier functions (CBFs) and an input constraint function are introduced to augment reward functions. And the monotonically nonincreasing property of the iterative value function in the PI algorithm maintains the safe set forward invariant. Then, the neural networks are employed to approximate the system dynamics, the iterative control policies, and the iterative value function, respectively. Furthermore, the proposed algorithm is supported by theoretical proofs that guarantee both safety and convergence. Finally, the effectiveness and safety of the algorithm are illustrated by the results of the simulation.

Data‐driven policy iteration algorithm for continuous‐time stochastic linear‐quadratic optimal control problems

AbstractThis paper studies a continuous‐time stochastic linear‐quadratic (SLQ) optimal control problem on infinite‐horizon. Combining the Kronecker product theory with an existing policy iteration algorithm, a data‐driven policy iteration algorithm is proposed to solve the problem. In contrast to most existing methods that need all information of system coefficients, the proposed algorithm eliminates the requirement of three system matrices by utilizing data of a stochastic system. More specifically, this algorithm uses the collected data to iteratively approximate the optimal control and a solution of the stochastic algebraic Riccati equation (SARE) corresponding to the SLQ optimal control problem. The convergence analysis of the obtained algorithm is given rigorously, and a simulation example is provided to illustrate the effectiveness and applicability of the algorithm.

Adaptive optimal control of continuous-time nonlinear affine systems via hybrid iteration

In this paper, a novel successive approximation framework, named hybrid iteration (HI), is proposed to fill up the performance gap between two well-known dynamic programming algorithms, namely policy iteration (PI) and value iteration (VI). Using HI, an approximated optimal control policy can be learned without prior knowledge of an initial admissible control policy required by PI. Additionally, the HI algorithm converges to the optimal solution much faster than VI, and thus requires tremendously less number of learning iterations and CPU-time, compared to VI. Initially, we develop a model-based HI algorithm, and then extend it to a data-driven HI algorithm which learns the optimal control policy without any information of the physics of the system. Simulation results demonstrate the efficacy of the proposed HI algorithm.

Policy-Iteration-Based Finite-Horizon Approximate Dynamic Programming for Continuous-Time Nonlinear Optimal Control.

The Hamilton-Jacobi-Bellman (HJB) equation serves as the necessary and sufficient condition for the optimal solution to the continuous-time (CT) optimal control problem (OCP). Compared with the infinite-horizon HJB equation, the solving of the finite-horizon (FH) HJB equation has been a long-standing challenge, because the partial time derivative of the value function is involved as an additional unknown term. To address this problem, this study first-time bridges the link between the partial time derivative and the terminal-time utility function, and thus it facilitates the use of the policy iteration (PI) technique to solve the CT FH OCPs. Based on this key finding, the FH approximate dynamic programming (ADP) algorithm is proposed leveraging an actor-critic framework. It is shown that the algorithm exhibits important properties in terms of convergence and optimality. Rather importantly, with the use of multilayer neural networks (NNs) in the actor-critic architecture, the algorithm is suitable for CT FH OCPs toward more general nonlinear and complex systems. Finally, the effectiveness of the proposed algorithm is demonstrated by conducting a series of simulations on both a linear quadratic regulator (LQR) problem and a nonlinear vehicle tracking problem.

Data-Driven Decentralized Control for Large-Scale Systems With Sparsity and Communication Delays

This article studies the decentralized control of large-scale systems with sparsity and communication delays. The large-scale system is defined over a directed connected graph and the information structure is partially nested. Based on the decomposition of the noise history, the optimal problem of the overall large-scale system can be decomposed into independent subproblems. Hence, the data-driven decentralized control method is investigated to find the optimal controllers using adaptive dynamic programming (ADP), which could release the dependence on the knowledge of model. In addition, state feedback and output feedback policy iteration algorithms are developed, respectively. Rigorous stability analysis shows that the proposed algorithms can stabilize the large-scale systems asymptotically. Finally, the effectiveness of the proposed theoretical methods is demonstrated by the application of heavy duty vehicle (HDV) platooning.