Policy Iteration Research Articles

Constrained event-driven policy iteration design for nonlinear discrete time systems

This paper studies an adaptive critic design (ACD) solution based on the event-driven mechanism for the constrained state and input system with external disturbance. At first, the barrier function is presented to construct the transformation of the constrained state system, and then, a suitable value function with non-quadratic utility function is designed according to the auxiliary system to acquire the control sequence in the infinity domain. After that, the comprehensive problem is transited into solving the Hamilton–Jacobi-Isaacs (HJI) equation. Then, the optimal control policy pair is obtained by policy iteration (PI) via double event-driven scheme. The three critic, action and disturbance neural networks (NNs) are built up to manipulate the data online concurrently to approximate the control pair sequence, and the ultimately uniformly bounded (UUB) proof of the estimation error generated by NNs is given. The last but not least, two sets of contrast simulations are analyzed to demonstrate the effectiveness of the proposed algorithm, which makes the model reduce the data transmission density and decrease the update times of control policies under the premise of system stability and ideal performance.

Discounted linear Q-learning control with novel tracking cost and its stability

In this article, in order to achieve optimal tracking control of unknown linear discrete systems, a model-free scheme based on Q-learning is established online. First, we introduce an innovative performance index function, so as to eliminate the tracking error and avert the calculation for stable control policies of the reference trajectory. Taking value iteration and policy iteration into consideration, the corresponding model-based approaches are derived. Then, the Q-function is developed and the model-free algorithm utilizing Q-learning is given for the sake of dealing with the linear quadratic tracking (LQT) problem online without relying on system dynamics information. In addition, novel stability analysis based on Q-learning is provided for the discounted LQT control issue and the probing noise is demonstrated that it does not result in any excitation noise bias. Finally, by means of conducting numerical simulation, the proposed Q-learning algorithm is demonstrated to be effective and practicable.

Policy Iteration Method for Time-Dependent Mean Field Games Systems with Non-separable Hamiltonians

We introduce two algorithms based on a policy iteration method to numerically solve time-dependent Mean Field Game systems of partial differential equations with non-separable Hamiltonians. We prove the convergence of such algorithms in sufficiently small time intervals with Banach fixed point method. Moreover, we prove that the convergence rates are linear. We illustrate our theoretical results by numerical examples, and we discuss the performance of the proposed algorithms.

Contractivity of Bellman operator in risk averse dynamic programming with infinite horizon

The paper deals with a risk averse dynamic programming problem with infinite horizon. First, the required assumptions are formulated to have the problem well defined. Then the Bellman equation is derived, which may be also seen as a standalone reinforcement learning problem. The fact that the Bellman operator is contraction is proved, guaranteeing convergence of various solution algorithms used for dynamic programming as well as reinforcement learning problems, which we demonstrate on the value iteration and the policy iteration algorithms.

GPI-Based design for partially unknown nonlinear two-player zero-sum games

This article focuses on obtaining the solution of two-player zero-sum games (ZSGs) where the saddle point may not exist. From the standpoints of the upper performance index function (PIF) and lower PIF separately, we seek the solution of two-player differential ZSGs without assuming the strict existence of saddle point. To achieve the objective, we apply the generalized policy iteration (GPI) technique to design the controller. With neural network approximators, the system trajectory data are used to update the approximation of the PIFs. It is established that the upper (lower) PIF sequence is convergent along the iteration axis under the GPI scheme. When the two function sequences converge to equal values, the saddle point is attained, and vice versa. Numerical simulation results conformed to the suitability of the designed algorithm.

Predictive Modelling of a Honeypot System Based on a Markov Decision Process and a Partially Observable Markov Decision Process

A honeypot is used to attract and monitor attacker activities and capture valuable information that can be used to help practice good cybersecurity. Predictive modelling of a honeypot system based on a Markov decision process (MDP) and a partially observable Markov decision process (POMDP) is performed in this paper. Analyses over a finite planning horizon and an infinite planning horizon for a discounted MDP are respectively conducted. Four methods, including value iteration (VI), policy iteration (PI), linear programming (LP), and Q-learning, are used in the analyses over an infinite planning horizon for the discounted MDP. The results of the various methods are compared to evaluate the validity of the created MDP model and the parameters in the model. The optimal policy to maximise the total expected reward of the states of the honeypot system is achieved, based on the MDP model employed. In the modelling over an infinite planning horizon for the discounted POMDP of the honeypot system, the effects of the observation probability of receiving commands, the probability of attacking the honeypot, the probability of the honeypot being disclosed, and transition rewards on the total expected reward of the honeypot system are studied.

Inexact GMRES Policy Iteration for Large-Scale Markov Decision Processes

Policy iteration enjoys a local quadratic rate of contraction, but its iterations are computationally expensive for Markov decision processes (MDPs) with a large number of states. In light of the connection between policy iteration and the semismooth Newton method and taking inspiration from the inexact variants of the latter, we propose inexact policy iteration, a new class of methods for large-scale finite MDPs with local contraction guarantees. We then design an instance based on the deployment of the generalized minimal residual method (GMRES) for the approximate policy evaluation step, which we call inexact GMRES policy iteration. Finally, we demonstrate the superior practical performance of inexact GMRES policy iteration on an MDP with 10000 states, where it achieves a × 5.8 and × 2.2 speedup with respect to policy iteration and optimistic policy iteration, respectively.

DRL-Based Slice Admission Using Overbooking in 5G Networks

In 5G-and-beyond networks, the concept of Network Slicing is used to support multiple independent and co-existing logical networks on a physical network infrastructure. The infrastructure provider (InP) owns the set of virtual and physical resources that are used to support the tenant slice requests. Each slice request specifies a service level agreement (SLA) that contains the required slice-level resources (computation and communication) and the revenue provided by the tenant. Due to limited resources, the InP cannot accommodate all requests made by the tenants. In general, it has been found that tenants tend to overestimate their resource demands (e.g., for 5G Core computation) to reduce possible SLA violations. In this paper, we consider two major slice types: Elastic (low priority and low revenue) and Inelastic (high priority and high revenue). We apply the concept of overbooking, where the InP accepts more slices while considering slice priorities in order to maximize the overall revenue and utilization. We consider a multi-tenant environment and propose a slice admission system named PRLOV, which is based on Reinforcement Learning (RL) and prediction methods. The system predicts the future resource demands of Elastic slices and applies an opportunistic overbooking technique to overbook InP for accepting more slices. In addition, the admission decision is formulated as an Markov Decision Process (MDP) problem and solved using standard RL techniques (Policy Iteration, Q-Learning, DQN). The performance of our proposed work is compared against three other heuristics (Basic, Prediction, PRL) that do not use overbooking. Data traces from the Materna data center networks were used for prediction purposes. The important performance metrics measures include InP total revenue, the acceptance rate of respective slices and overall resource utilization for different slices. The results show that the proposed work significantly outperforms the other mechanisms in terms of revenue gain and resource utilization. Simulation results show that PRLOV provides a revenue gain of 6%, 26%, and 102% compared to the PRL, Prediction and Basic scheme.

Link Analysis for Solving Multiple-Access MDPs With Large State Spaces

Wireless communication networks can be well-modeled by Markov Decision Processes (MDPs). While traditional dynamic programming algorithms such as value and policy iteration have lower complexity than brute force strategies, they still suffer from complexity issues for large state spaces. In this paper, the development of moderate complexity algorithms with high performance is sought for wireless network control. To this end, the approximate value function can be computed by projecting the original value function into a lower-dimensional subspace with the careful choice of basis vectors using tools from graph signal processing (GSP). Although GSP theory mainly focuses on undirected graphs, the transition graphs of MDPs for wireless networks are generally directed. For this reason, graph symmetrization based on the co-link method is considered due to its ability to preserve multi-hop dependencies in the graph. Given the multiple-access model under consideration, key properties of the transition probability matrix are exploited to determine the optimal subspace for the approximate value function with low complexity. The numerical results for a multiple-access channel show that the proposed algorithm can find the optimal basis vectors with high accuracy, and furthermore, the approach is robust to changes in system parameters. It is also shown that the projected equation method outperforms the state aggregation technique in producing higher accuracy with a lower runtime complexity.

Data-Driven Optimal Synchronization Control for Leader-Follower Multiagent Systems

In this article, we develop data-driven optimal synchronization control architectures for leader-follower multiagent systems with additive disturbances and unknown system matrices. To minimize output synchronization error, algebraic Riccati equations (AREs) are derived, and unique feedback gains are determined by policy iteration. On that basis, two data-driven optimal synchronization control algorithms are developed without relying on the dynamics of the system, which guarantee output synchronization while minimizing synchronization errors and rejecting disturbances. The first algorithm uses the output synchronization error data to perform online data-driven learning (DDL), while the second algorithm uses the input data to perform DDL, where both data sample requirements are transformed into rank conditions. We have presented rigorous theoretical analyses of our proposed algorithms, which demonstrate that if an initial control protocol can make the system achieve output synchronization under mild conditions, our proposed two algorithms can take advantage of the data from reaching synchronization to optimize the closed-loop performance. Finally, a numerical example is provided to emphasize the effectiveness of our methods.

Learning-Based Adaptive Optimal Control for Flotation Processes Subject to Input Constraints

This article presents a learning-based adaptive optimal control approach for flotation processes subject to input constraints and disturbances using adaptive dynamic programming (ADP) along with double-loop iteration. First, the principle of the operational pattern is adopted to preset reagents’ addition based on the feeding condition. Then, this article leverages a deep learning model, which is composed of multiple neural layers to detect flotation indexes directly from the raw froth images. After that, the tracking error between the detected flotation indexes and the reference values can be minimized by using ADP-based double-loop iteration. Particularly, a policy-iteration (PI) method is utilized for the proposed learning-based ADP algorithm. In the inner loop, the optimal control problem is formulated as a linear quadratic regulator (LQR) problem using the low-gain feedback design method. In the outer loop, the design parameters, i.e., weighting matrices, are tuned automatically to satisfy the input constraints. Finally, the analytical results demonstrate that the proposed scheme can guarantee asymptotic tracking in the presence of actuator saturation and disturbances.

Reinforcement Learning Control for Discrete-time Systems over Fading Channels

This paper studies the average cost minimization problem for a class of discrete-time systems suffering from fading channels and additive noise. An approximate generalized policy iteration algorithm is proposed to search for the optimal control policy for the considered problem without resorting to the system matrices. The estimation errors of the kernel matrices are proven to be small and the proposed model-free reinforcement learning algorithm is proven to be convergent. Some numerical examples are provided to illustrate the obtained results.

Learning-Based Adaptive Optimal Output Regulation of Discrete-Time Linear Systems

In this paper, we address the problem of model-free optimal output regulation of discrete-time systems that aims at achieving asymptotic tracking and disturbance rejection without the knowledge of the system parameters. Insights from reinforcement learning and adaptive dynamic programming are used to solve this problem. An interesting discovery is that the model-free discrete-time output regulation differs from the continuous-time counterpart in terms of the persistent excitation condition required to ensure the uniqueness and convergence of the policy iteration. In this work, we carefully establish the persistent excitation condition to ensure the uniqueness and convergence properties of the policy iteration.

Distributed Randomized Multiagent Policy Iteration in Reinforcement Learning

Distributed Randomized Multiagent Policy Iteration in Reinforcement Learning

Actor–Critic or Critic–Actor? A Tale of Two Time Scales

We revisit the standard formulation of tabular actor-critic algorithm as a two time-scale stochastic approximation with value function computed on a faster time-scale and policy computed on a slower time-scale. This emulates policy iteration. We observe that reversal of the time scales will in fact emulate value iteration and is a legitimate algorithm. We provide a proof of convergence and compare the two empirically with and without function approximation (with both linear and nonlinear function approximators) and observe that our proposed critic-actor algorithm performs on par with actor-critic in terms of both accuracy and computational effort.

Cooperative Learning of Multi-Agent Systems Via Reinforcement Learning

In many specific scenarios, accurateand practical cooperative learning is a commonly encountered challenge in multi-agent systems. Thus, the current investigation focuses on cooperative learning algorithms for multi-agent systems and underpins an alternate data-based neural network reinforcement learning framework. To achieve the data-based learning optimization, the proposed cooperative learning framework, which comprises two layers, introduces a virtual learning objective. The followers learn the behaviors of the virtual objects in the first layer based on the adaptive neural networks (NNs). Specifically, the actor and critic NNs are applied to acquire cooperative behaviors and assess this layer's long-term utility function. Then another layer realizes the tracking performance between the virtual objects and the leader by introducing the local data-based performance index. Then, we formulate a resulting deterministic optimization problem and resolve it effectively with the policy iteration algorithm. This intuitive cooperative learning algorithm also preserves good robustness properties and eliminates the dependence on the prior knowledge of the multi-agent system model in the solution process. Finally, a multi-robot formation system demonstrates this promising development's practical appeal and highly effective outcome.

On the Optimality and Convergence Properties of the Iterative Learning Model Predictive Controller

In this technical article, we analyze the performance improvement and optimality properties of the learning model predictive control (LMPC) strategy for linear deterministic systems. The LMPC framework is a policy iteration scheme where closed-loop trajectories are used to update the control policy for the next execution of the control task. We show that, when a linear independence constraint qualification (LICQ) condition holds, the LMPC scheme guarantees strict iterative performance improvement and optimality, meaning that the closed-loop cost evaluated over the entire task converges asymptotically to the optimal cost of the infinite-horizon control problem. Compared to previous works, this sufficient LICQ condition can be easily checked, it holds for a larger class of systems and it can be used to adaptively select the prediction horizon of the controller, as demonstrated by a numerical example.

Reinforcement Learning with Partial Parametric Model Knowledge

We adapt reinforcement learning methods for continuous control to bridge the gap between complete ignorance and perfect knowledge of the environment. Our method, Partial Knowledge Least Squares Policy Iteration (PLSPI), takes inspiration from both model-free RL and model-based control. It uses incomplete information from a partial model and retains RL's data-driven adaption towards optimal performance. The linear quadratic regulator provides a case study; numerical experiments demonstrate the effectiveness and resulting benefits of the proposed method.

DNN-Approximations of Control Laws for Nonlinear Systems with Polytopic Constraints

This work focuses on deep artificial feed-forward neural networks as parametric function approximators in optimal control of discrete-time nonlinear but affine-in-control systems subject to polytopic constraints on the continuous states and control inputs. The neural networks are either considered as approximators of optimal control laws or as approximators of the corresponding cost functions. In both cases, an approach is developed for determining the approximated optimal control inputs without violating the constraints. A simple approximate policy iteration algorithm exploiting both approaches is finally presented and illustrated for a numerical example.

Optimal Control and Adaptive Learning for Stabilization of a Quadrotor-type Unmanned Aerial Vehicle via Approximate Dynamic Programming

The development of an optimal controller for stabilization of a quadrotor system using an adaptive critic structure based on policy iteration schemes is proposed in this paper. This approach is inserted in the context of Approximate Dynamic Programming and it is used to solve optimal decision problems on-line, without requiring complete knowledge of the system dynamics model to be controlled. The main feature of the adaptive critic design method that allows for on-line implementation is that it solves the Bellman optimality equation in a forward-in-time fashion, whereas traditional dynamic programming requires a backward-in-time procedure. This feedback control design technique is able to tune the controller parameters on-line in the presence of variations in plant dynamics and external disturbances using data measured along the system trajectories. Computational simulation results based on a quadrotor model demonstrate the effectiveness of the proposed control scheme.