Policy Iteration Approach Research Articles

Policy Iteration Q -Learning for Linear Itô Stochastic Systems With Markovian Jumps and Its Application to Power Systems.

This article addresses the solution of continuous-time linear Itô stochastic systems with Markovian jumps using an online policy iteration (PI) approach grounded in Q -learning. Initially, a model-dependent offline algorithm, structured according to traditional optimal control strategies, is designed to solve the algebraic Riccati equation (ARE). Employing Lyapunov theory, we rigorously derive the convergence of the offline PI algorithm and the admissibility of the iterative control law through mathematical analysis. This article represents the first attempt to tackle these technical challenges. Subsequently, to address the limitations inherent in the offline algorithm, we introduce a novel online Q -learning algorithm tailored for Itô stochastic systems with Markovian jumps. The proposed Q -learning algorithm obviates the need for transition probabilities and system matrices. We provide a thorough stability analysis of the closed-loop system. Finally, the effectiveness and applicability of the proposed algorithms are demonstrated through a simulation example, underpinned by the theorems established herein.

Non‐zero‐sum games of discrete‐time Markov jump systems with unknown dynamics: An off‐policy reinforcement learning method

AbstractThis article concentrates on the non‐zero‐sum games problem of discrete‐time Markov jump systems without requiring the system dynamics information. First, the multiplayer non‐zero‐sum games problem can be converted to solve a set of coupled game algebraic Riccati equations, which is difficult to be solved directly. Then, to obtain the optimal control policies, a model‐based algorithm adapting the policy iteration approach is proposed. However, the model‐based algorithm relies on system dynamics information, which has the limitations in practice. Subsequently, an off‐policy reinforcement learning algorithm is given to get rid of the dependence on system dynamics information, which only uses the information of system states and inputs. Moreover, the proof of convergence and Nash equilibrium are also given. Finally, a numerical example is given to demonstrate the effectiveness of the proposed algorithms.

Model-free policy iteration approach to NCE-based strategy design for linear quadratic Gaussian games

This paper presents a novel completely model-free method to solve continuous-time linear quadratic (LQ) mean field games (MFGs). We consider the setting where agents can only access their own states and are coupled via their individual infinite-horizon discounted costs. For such games, a set of Nash certainty equivalence-based strategies, determined by two gain matrices and the aggregate population effect, i.e. mean field (MF) state, are developed, providing an ɛ-Nash equilibrium under certain assumptions. Note that the gain matrices are solved from two algebraic Riccati equations (AREs), respectively, and the MF state is given by the linear ordinary differential equation (ODE). All the solutions of these three equations require the exact model of system dynamics. To implement the NCE-based strategy with completely unknown system dynamics, the proposed approach develops a single-agent policy iteration (PI) and a single-agent MF generator to solve the above parameters. Specifically, first, model-based iterative equations with guaranteed convergence are developed to approximate two gain matrices. Then, the requirement of dynamical model is removed from the iteration process via the integral reinforcement learning (IRL) technique. A model-free algorithm is conducted by using measured data from a selected agent as reinforcement signals over a certain time interval. Finally, based on the obtained gain matrices, the MF state is computed offline depending on samples collected from the same agent. A numerical example is given to verify the effectiveness of the proposed method.

H∞-Based Minimal Energy Adaptive Control With Preset Convergence Rate.

This work studies the H∞ -based minimal energy control with a preset convergence rate (PCR) problem for a class of disturbed linear time-invariant continuous-time systems with matched external disturbance. This problem aims to design an optimal controller so that the energy of the control input satisfies a predetermined requirement. Moreover, the closed-loop system asymptotic stability with PCR is ensured simultaneously. To deal with this problem, a modified game algebraic Riccati equation (MGARE) is proposed, which is different from the game algebraic Riccati equation in the traditional H∞ control problem due to the state cost being lost. Therefore, a unique positive-definite solution of the MGARE is theoretically analyzed with its existing conditions. In addition, based on this formulation, a novel approach is proposed to solve the actuator magnitude saturation problem with the system dynamics being exactly known. To relax the requirement of the knowledge of system dynamics, a model-free policy iteration approach is proposed to compute the solution of this problem. Finally, the effectiveness of the proposed approaches is verified through two simulation examples.

Optimal synchronization control for heterogeneous multi‐agent systems: Online adaptive learning solutions

AbstractThis paper presents an online adaptive learning solution to the optimal synchronization control problem of heterogeneous multi‐agent systems via a novel distributed policy iteration approach. For the leader‐follower multi‐agents, the dynamics of all the followers are heterogeneous with leader disturbance. To make the output of each follower synchronize with the leader's output, we propose a synchronization control protocol where the stability conditions for selecting the feedback gains are given. Next, with a minimization of the output tracking errors, we optimize the feedback gains for the synchronization control protocol, and the unique solutions for those feedback gains are learned via a novel distributed policy iteration approach. We prove that the proposed online adaptive learning solution will converge to the optimal control solution under some mild conditions. Finally, an illustrative numerical example is provided to show the effectiveness of the approach.

Online battery scheduling for grid-connected photo-voltaic systems

Abstract We study the problem of determining optimum policy for managing battery energy storage system (BESS) in grid-connected photo-voltaic (PV) systems, where the stochastic electricity demands from the load are met from three sources: grid, PV energy, and BESS. BESS is used either to store excess energy generated from PV systems for later use, or to purchase energy from the grid when the time-of-use (TOU) pricing is lower. The objective is to identify the optimum charging/discharging schedule of BESS so that the long-term cost of energy purchased from the grid is minimized. The stochastic variabilities in loads and PV energy are captured by employing probabilistic models of periodic stochastic process with parameters estimated using historical data. The optimization problem is formulated under the framework of periodic discounted Markov decision process (MDP), and the problem formulation includes the aging effects of batteries and solar panels. The online optimization problem is solved by adopting a policy iteration approach tailored for periodic MDP. The proposed online scheduling algorithm provides periodic policies for a period of 24-hour, where the system model is updated every day based on load and PV energy from the previous day in a rolling horizon fashion. Simulation results demonstrate that the proposed algorithm can achieve a 41.6% reduction in annual utility bills compared to conventional systems without PV and BESS, thus ascertaining the values of installing BESS and PV systems.

Cooperative adaptive optimal output regulation of nonlinear discrete-time multi-agent systems

This paper studies a cooperative adaptive optimal output regulation problem for a class of strict-feedback nonlinear discrete-time (DT) multi-agent systems (MASs) with partially unknown dynamics. Adaptive distributed observer, reinforcement learning (RL) and output regulation techniques are integrated to compute an adaptive near-optimal tracker for each follower. First, under the output regulation theory, the cooperative adaptive optimal output regulation problem is decomposed into a feedforward control design problem which can be addressed by solving nonlinear regulator equations, and an adaptive optimal feedback control problem. Different from general adaptive optimal stabilization problem of nonlinear DT systems, the value function of the optimal output regulation problem is positive semidefinite. This paper analyzes the stability of the closed-loop system without relying on the Lyapunov stability theory and also proposes a policy iteration (PI) approach to approximate the value function with the convergence proof. Then, an adaptive distributed observer is designed for each follower so that they can estimate the state of the leader. A model-state-input structure is developed to find the solutions to regulator equations for each follower and a critic–actor structure is employed to solve the optimal feedback control problem using the measured data based on the neural network (NN) and RL. It is shown that using the estimate values, the tracking errors are uniformly ultimately bounded. A simulation example is given to show the effectiveness of the proposed method.

Policy Iteration Approach to the Infinite Horizon Average Optimal Control of Probabilistic Boolean Networks

This article studies the optimal control of probabilistic Boolean control networks (PBCNs) with the infinite horizon average cost criterion. By resorting to the semitensor product (STP) of matrices, a nested optimality equation for the optimal control problem of PBCNs is proposed. The Laurent series expression technique and the Jordan decomposition method derive a novel policy iteration-type algorithm, where finite iteration steps can provide the optimal state feedback law, which is presented. Finally, the intervention problem of the probabilistic Ara operon in E. coil, as a biological application, is solved to demonstrate the effectiveness and feasibility of the proposed theoretical approach and algorithms.

Financial intermediation and capital reallocation

To understand the link between financial intermediation activities and the real economy, we build a general equilibrium model in which agency frictions in the financial sector affect the efficiency of capital reallocation across firms and generate aggregate economic fluctuations. We develop a recursive policy iteration approach to fully characterize the nonlinear equilibrium dynamics and the off-steady-state crisis behavior. In our model, adverse shocks to agency frictions exacerbate capital misallocation and manifest themselves as variations in total factor productivity at the aggregate level. Our model endogenously generates countercyclical volatility in the aggregate time series and countercyclical dispersion in the marginal product of capital and asset returns in the cross-section.

Suboptimal Control for Nonlinear Systems with Disturbance via Integral Sliding Mode Control and Policy Iteration

In this paper, we consider a suboptimal control problem for nonlinear systems with unmatched disturbance via integral sliding mode control and policy iteration. Firstly, the unmatched disturbance is estimated by a nonlinear disturbance observer. Secondly, the integral sliding mode controller based on the disturbance estimation is designed to guarantee the reachability of the sliding-mode surface. Furthermore, the policy iteration approach based on actor-critic neural networks is used to design the suboptimal controller, which can achieve the desired performance. Finally, the effectiveness of the proposed approach is demonstrated through simulation examples.

H∞ Control for Discrete-Time Multi-Player Systems via Off-Policy Q-Learning

This paper presents a novel off-policy game Q-learning algorithm to solve $H_\infty $ control problem for discrete-time linear multi-player systems with completely unknown system dynamics. The primary contribution of this paper lies in that the Q-learning strategy employed in the proposed algorithm is implemented in an off-policy policy iteration approach other than on-policy learning, since the off-policy learning has some well-known advantages over the on-policy learning. All of players struggle together to minimize their common performance index meanwhile defeating the disturbance that tries to maximize the specific performance index, and finally they reach the Nash equilibrium of game resulting in satisfying disturbance attenuation condition. For finding the solution of the Nash equilibrium, $H_\infty $ control problem is first transformed into an optimal control problem. Then an off-policy Q-learning algorithm is put forward in the typical adaptive dynamic programming (ADP) and game architecture, such that control policies of all players can be learned using only measured data. More importantly, the rigorous proof of no bias of solution to the Nash equilibrium by using the proposed off-policy game Q-learning algorithm is presented. Comparative simulation results are provided to verify the effectiveness and demonstrate the advantages of the proposed method.

Off-Policy Q-Learning for Anti-Interference Control of Multi-Player Systems

Abstract This paper develops a novel off-policy game Q-learning algorithm to solve the anti-interference control problem for discrete-time linear multi-player systems using only data without requiring system matrices to be known. The primary contribution of this paper lies in that the Q-learning strategy employed in the proposed algorithm is implemented in an off-policy policy iteration approach other than on-policy learning due to the well-known advantages of off-policy Q-learning over on-policy Q-learning. All of the players work hard together for the goal of minimizing their common performance index meanwhile defeating the disturbance that tries to maximize the specific performance index, and finally they reach the Nash equilibrium of the game resulting in satisfying disturbance attenuation condition. In order to find the solution to the Nash equilibrium, the anti-interference control problem is first transformed into an optimal control problem. Then an off-policy Q-learning algorithm is proposed in the framework of typical adaptive dynamic programming (ADP) and game architecture, such that control policies of all players can be learned using only measured data. Comparative simulation results are provided to verify the effectiveness of the proposed method.

A dual sourcing inventory model for modal split transport: Structural properties and optimal solution

Shifting freight volumes from road to rail transport increases the economic performances of freight logistics. However, compared to road transport, rail transport generally lacks the flexibility in delivery quantity and frequency, and exhibits economies of scale in its shipment volume. This often leads to high inventory levels in the destination after deliveries. We generalize the tailored base-surge dual sourcing inventory model by introducing a fixed cost in rail transport, adding an extra decision in its delivery frequency, and relaxing the assumption of the base stock control of road transport, to support firms’ modal split transport optimization. The objective is to optimize the controls of the two transport modes and the corresponding inventory management at the destination, which minimize the combined average transport and inventory costs per period in the steady state. Using stochastic dynamic programming, we find that when the delivery quantity and frequency of rail transport is fixed, the optimal shipment volume via the road transport indeed follows a base stock control. This allows to solve the relevant Bellman equation via an efficient policy iteration approach. We also find that the total cost is convex in the delivery quantity of rail transport, and a bi-section search can be applied. Finally, we analyze the sensitivity and robustness of our model using values suggested by a consumer goods firm.

An average optimal control approach to the set stabilization problem for boolean control networks

AbstractThis paper addresses the set stabilization problem for deterministic Boolean control networks (BCNs). An optimal control approach is investigated to solve the problems by using the semi‐tensor product of matrices, where a policy iteration algorithm for the set stabilization problem is deduced. Finally, the intervention problem of a cAMP receptor protein is addressed in the framework of the set stabilization problem. The problem is solved to validate the effectiveness of the proposed policy iteration approach for a practical application.

Online model-free controller for flexible wing aircraft: a policy iteration-based reinforcement learning approach

The aerodynamic model of flexible wing aircraft is highly nonlinear with continuously time-varying dynamics under kinematic constraints. The nonlinearities stem from the aerodynamic forces and continuous deformations in the flexible wing. In spite of the various experimental attempts and theoretical setups that were made to model these dynamics, an accurate formulation was not achieved. The control paradigms of the aircraft are concerned with the electro-mechanical coupling between the pilot and the wing. It is challenging to design a flight controller for such aircraft while complying with these constraints. In this paper, innovative machine learning technique is employed to design a robust online model-free control scheme for flexible wing aircraft. The controller maintains internal asymptotic stability for the aircraft in real-time using selected set of measurements or states in uncertain dynamical environment. It intelligently incorporates the varying dynamics, geometric parameters, and physical constraints of the aircraft into optimal control strategies. The adaptive learning structure employs a policy iteration approach, taking advantage of Bellman optimality principles, to converge to an optimal control solution for the problem. Artificial neural networks are adopted to implement the adaptive learning algorithm in real-time without prior knowledge of the aerodynamic model of the aircraft. The control scheme is generalized and shown to function effectively for different pilot/wing control mechanisms. It also demonstrated its ability to overcome the undesired stability problems caused by coupling the pilot’s dynamics with the flexible wing’s frame of motion.

Adaptive Optimal Control for a Class of Nonlinear Systems: The Online Policy Iteration Approach.

This paper studies the online adaptive optimal controller design for a class of nonlinear systems through a novel policy iteration (PI) algorithm. By using the technique of neural network linear differential inclusion (LDI) to linearize the nonlinear terms in each iteration, the optimal law for controller design can be solved through the relevant algebraic Riccati equation (ARE) without using the system internal parameters. Based on PI approach, the adaptive optimal control algorithm is developed with the online linearization and the two-step iteration, i.e., policy evaluation and policy improvement. The convergence of the proposed PI algorithm is also proved. Finally, two numerical examples are given to illustrate the effectiveness and applicability of the proposed method.

Data-driven adaptive dynamic programming for partially observable nonzero-sum games via Q-learning method

ABSTRACTThis paper concerns with a class of discrete-time linear nonzero-sum games with the partially observable system state. As is known, the optimal control policy for the nonzero-sum games relies on the full state measurement which is hard to fulfil in partially observable environment. Moreover, to achieve the optimal control, one needs to know the accurate system model. To overcome these deficiencies, this paper develops a data-driven adaptive dynamic programming method via Q-learning method using measurable input/output data without any system knowledge. First, the representation of the unmeasurable inner system state is built using historical input/output data. Then, based on the representation state, a Q-function-based policy iteration approach with convergence analysis is introduced to approximate the optimal control policy iteratively. A neural network (NN)-based actor-critic framework is applied to implement the developed data-driven approach. Finally, two simulation examples are provided to demonstrate the effectiveness of the developed approach.

Optimal control of Boolean control networks with average cost: A policy iteration approach

This paper deals with the infinite horizon optimal control problem for deterministic Boolean control networks (BCNs) with average cost. Based on the semi-tensor product of matrices and Jordan decomposition technique, a nested optimality equation for the average infinite horizon problem of BCNs is presented. By resorting to Laurent series expression, a novel policy iteration algorithm, which can find the optimal state feedback controller in finite iteration steps, is proposed. Finally, as a practical application, the optimal intervention problem of Ara operon in E. coil is addressed.

Policy Iteration Approach to Control Residual Gas Fraction in IC Engines Under the Framework of Stochastic Logical Dynamics

This brief investigates the cycle-to-cycle transient behavior of the residual gas fraction (RGF) in terms of systems theory and proposes a multivalued logic-based optimal control strategy for the attenuation of RGF fluctuation. First, an in-cylinder pressure-based method for measuring RGF is provided, and the stochastic properties of the RGF are examined based on statistical data obtained by conducting experiments on a full-scale internal combustion engine test bench. Based on the observation of the experiment, a stochastic logical transient model is proposed to represent the cycle-to-cycle transient behavior of the RGF. Then, an optimal feedback control law, which targets the rejection of the RGF fluctuation, is derived using the policy iteration algorithm. Finally, the experimental result is demonstrated to show the effectiveness of the proposed optimal control law.

Least squares approximate policy iteration for learning bid prices in choice-based revenue management

We consider the revenue management problem of capacity control under customer choice behavior. An exact solution of the underlying stochastic dynamic program is difficult because of the multi-dimensional state space and, thus, approximate dynamic programming (ADP) techniques are widely used. The key idea of ADP is to encode the multi-dimensional state space by a small number of basis functions, often leading to a parametric approximation of the dynamic program’s value function. In general, two classes of ADP techniques for learning value function approximations exist: mathematical programming and simulation. So far, the literature on capacity control largely focuses on the first class.In this paper, we develop a least squares approximate policy iteration (API) approach which belongs to the second class. Thereby, we suggest value function approximations that are linear in the parameters, and we estimate the parameters via linear least squares regression. Exploiting both exact and heuristic knowledge from the value function, we enforce structural constraints on the parameters to facilitate learning a good policy. We perform an extensive simulation study to investigate the performance of our approach. The results show that it is able to obtain competitive revenues compared to and often outperforms state-of-the-art capacity control methods in reasonable computational time. Depending on the scarcity of capacity and the point in time, revenue improvements of around 1% or more can be observed. Furthermore, the proposed approach contributes to simulation-based ADP, bringing forth research on numerically estimating piecewise linear value function approximations and their application in revenue management environments.