Iterative Control Policy Research Articles

Novel generalized policy iteration for efficient evolving control of nonlinear systems

In this article, we construct a novel generalized policy iteration framework to address optimal regulation problems for discrete-time nonlinear systems in a more efficient way. Relevant properties are investigated for the framework, including monotonicity and convergence of the iterative value function sequence as well as the admissibility of the iterative control policy. Additionally, an innovative approach is developed to seek an initial admissible control policy for the framework with an adjustable searching speed. Based on these, an evolving control algorithm is presented with stability guarantee. This algorithm employs iterative control policies for system control during the computation of the optimal control policy, as opposed to waiting for the generation of the optimal control policy before implementing control. Eventually, two simulation experiments are conducted with real-world physical backgrounds, in order to illustrate the performance of the proposed strategy.

Convergence and stability analysis of value iteration Q-learning under non-discounted cost for discrete-time optimal control

This paper presents a theoretical analysis of the value iteration Q-learning with non-discounted costs. The analysis focuses on two main aspects: the convergence of the iterative Q-function and the stability of the system under the final iterative control policy. Unlike previous theoretical results on Q-learning, our analysis takes into account the effect of approximation errors, leading to a more comprehensive investigation. We first discuss the effect of approximation errors on the iterative Q-function update. Then, considering the presence of approximation errors in each iteration, we analyze the convergence of the iterative Q-function. Furthermore, we establish a sufficient condition, also accounting for the approximation errors, to ensure the stability of the system under the final iterative control policy. Finally, two simulation cases are conducted to validate the presented convergence and stability results.

Convergence and Stability of Optimal Regulation via Generalized N-Step Value Gradient Learning.

In this article, the generalized N -step value gradient learning (GNSVGL) algorithm, which takes a long-term prediction parameter λ into account, is developed for infinite horizon discounted near-optimal control of discrete-time nonlinear systems. The proposed GNSVGL algorithm can accelerate the learning process of adaptive dynamic programming (ADP) and has a better performance by learning from more than one future reward. Compared with the traditional N -step value gradient learning (NSVGL) algorithm with zero initial functions, the proposed GNSVGL algorithm is initialized with positive definite functions. Considering different initial cost functions, the convergence analysis of the value-iteration-based algorithm is provided. The stability condition for the iterative control policy is established to determine the value of the iteration index, under which the control law can make the system asymptotically stable. Under such a condition, if the system is asymptotically stable at the current iteration, then the iterative control laws after this step are guaranteed to be stabilizing. Two critic neural networks and one action network are constructed to approximate the one-return costate function, the λ -return costate function, and the control law, respectively. It is emphasized that one-return and λ -return critic networks are combined to train the action neural network. Finally, via conducting simulation studies and comparisons, the superiority of the developed algorithm is confirmed.

Optimal Learning Output Tracking Control: A Model-Free Policy Optimization Method With Convergence Analysis.

Optimal learning output tracking control (OLOTC) in a model-free manner has received increasing attention in both the intelligent control and the reinforcement learning (RL) communities. Although the model-free tracking control has been achieved via off-policy learning and Q -learning, another popular RL idea of direct policy learning, with its easy-to-implement feature, is still rarely considered. To fill this gap, this article aims to develop a novel model-free policy optimization (PO) algorithm to achieve the OLOTC for unknown linear discrete-time (DT) systems. The iterative control policy is parameterized to directly improve the discounted value function of the augmented system via the gradient-based method. To implement this algorithm in a model-free manner, a model-free two-point policy gradient (PG) algorithm is designed to approximate the gradient of discounted value function by virtue of the sampled states and the reference trajectories. The global convergence of model-free PO algorithm to the optimal value function is demonstrated with the sufficient quantity of samples and proper conditions. Finally, numerical simulation results are provided to validate the effectiveness of the present method.

Policy Optimization Adaptive Dynamic Programming for Optimal Control of Input-Affine Discrete-Time Nonlinear Systems

In this article, a policy optimization adaptive dynamic programming (POADP) method is developed for optimal control of discrete-time unknown nonlinear systems, where the iterative control policy is parameterized to optimize the iterative <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -function directly. The relaxed condition for the learning rate is given to guarantee the convergence of the present algorithm. Furthermore, the Polyak–Łojasiewicz inequality is introduced to analyze the optimality, i.e., the iterative <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -function converges to the optimum within a given computational threshold under a finite iteration, and the rate of convergence (i.e., the required minimum number of iterations) for the developed POADP method is also illustrated. To ease real implementations, the iterative <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -function and the iterative control policy are approximated by employing an actor–critic structure. Then, an experiment-based method is developed to obtain the initial weights of actor–critic structure. Finally, numerical simulation results of two examples are provided to validate the effectiveness of the POADP algorithm.

Discounted Near-Optimal Control of Affine Systems via a Progressive Cost Evolution Formulation

In this essay, we develop a combined value iteration (CVI) framework to address discounted optimal control problems for discrete-time affine nonlinear systems. First, generated by novel value iteration (NVI), the admissibility is investigated for the iterative control policy. Note that the relaxation factor leads to the adjustable convergence speed. Second, the constraint condition for the discount factor is established to guarantee the admissibility of the iterative control policy derived from stabilizing value iteration (SVI). In addition, the monotonicity is discussed for the iterative cost function sequence. More importantly, CVI is constructed based on NVI and SVI. Third, produced by CVI, the system stability under the evolving control sequence is ensured by the introduction of the attraction region. In the end, a numerical example is involved to confirm the related theoretical results.

Dynamic event-triggered control for discrete-time nonlinear Markov jump systems using policy iteration-based adaptive dynamic programming

This paper investigates a dynamic event-triggered optimal control problem of discrete-time (DT) nonlinear Markov jump systems (MJSs) via exploring policy iteration (PI) adaptive dynamic programming (ADP) algorithms. The performance index function (PIF) defined in each subsystem is updated by utilizing an online PI algorithm, and the corresponding control policy is derived via solving the optimal PIF. Then, we adopt neural network (NN) techniques, including an actor network and a critic network, to estimate the iterative PIF and control policy. Moreover, the designed dynamic event-triggered mechanism (DETM) is employed to avoid wasting additional resources when the estimated iterative control policy is updated. Finally, based on the Lyapunov difference method, it is proved that the system stability and the convergence of all signals can be guaranteed under the developed control scheme. A simulation example for DT nonlinear MJSs with two system modes is presented to demonstrate the feasibility of the control design scheme.

Safe reinforcement learning for affine nonlinear systems with state constraints and input saturation using control barrier functions

This paper provides a novel safe reinforcement learning (RL) control algorithm to solve safe optimal problems for discrete-time affine nonlinear systems, while the safety and convergence of the control algorithm are proven. The algorithm is proposed based on an adjusted policy iteration (PI) framework using only the measured data along the system trajectories in the environment. The adjusted PI algorithm combines with the system predictive information. Unlike most PI algorithms, an effective method of obtaining an initial safe and stable control policy is given here. In addition, control barrier functions (CBFs) and an input constraint function are introduced to augment reward functions. And the monotonically nonincreasing property of the iterative value function maintains the safe set forward invariant in the PI framework. Moreover, the safety and convergence of the proposed algorithm are proven in theory. Then, the design and implementation of the proposed algorithm are presented based on the identifier-actor-critic structure, where neural networks are employed to approximate the system dynamics, the iterative control policy, and the iterative value function, respectively. Finally, the simulation results illustrate the effectiveness and safety of the proposed algorithm.

Constrained Decoupling Adaptive Dynamic Programming for A Partially Uncontrollable Time-Delayed Model of Energy Systems

The lack of feedback-loop design and uncontrollable state reconstruction, such as load demand and renewable generation, are key factors to hinder the development of energy control. Adaptive dynamic programming is an efficient method to break through the bottleneck. However, most current works assume that the evolution of uncontrollable state satisfies Markovian property. This paper proposes a time-delayed adaptive dynamic programming framework for optimal energy control, utilizing more comprehensive historical data for state reconstruction. The uncontrollable state is reconstructed by an attention mechanism-based Transformer network based on online measured data. Then, a self-learning iterative algorithm is developed to obtain the optimal state feedback control policy under a discounted performance index function. The problem of solving iterative control policy, which is a mixed-integer nonlinear programming problem with an enormous computational burden, is decomposed into three sub-problems, i.e. nonlinear programming with no discrete variable, which is solved by the Projected Newton method, mixed-integer quadratic programming, and quadratic programming. Numerical results illustrate that the developed control algorithm minimizes the electricity cost in the long term and avoids peak load.

Value‐iteration‐based affine nonlinear optimal control involving admissibility discussion

AbstractIn this article, the admissibility analysis of the value‐iteration‐based control policy is investigated for affine nonlinear dynamics. First, a series of admissible control criteria are discussed for nonlinear and linear systems. With the help of these criteria, it is more convenient to determine whether an iterative control policy is admissible or not compared with the previous work. Second, once the iterative control policy is established to be admissible by the criteria, the stability of the closed‐loop systems using the control law can be guaranteed. In addition, under certain conditions, the admissibility and asymptotic stability similar to policy iteration can be obtained. Finally, two numerical examples with the physical background are afforded to validate the effectiveness of presented theoretical results.

Policy Gradient Adaptive Critic Designs for Model-Free Optimal Tracking Control With Experience Replay

A model-free optimal tracking controller is designed for discrete-time nonlinear systems through policy gradient adaptive critic designs (PGACDs) with experience replay (ER). By using system transformation, optimal tracking control problems are converted into optimal regulation problems. An off-policy PGACD algorithm is developed to minimize the iterative <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -function and improve the tracking control performance. The proposed method is realized based on the critic network and the actor network (AN), which are applied to approximate the iterative <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -function and the iterative control policy, respectively. Then, the policy gradient technique is introduced to derive a novel weight updating law of the AN explicitly by using measured system data only. The convergence of the iteration is established through theoretical analysis, and the uniform ultimate boundedness is demonstrated for the closed-loop system under the PGACD-based controller by using Lyapunov’s direct method. To guarantee the stability and increase the data usage efficiency of the learning process, an ER-based learning framework is designed to improve the realizability of the proposed method. Finally, simulation results of two examples are provided to demonstrate the performance of the off-policy PGACD algorithm.

Generalized value iteration for discounted optimal control with stability analysis

Abstract In this work, the generalized value iteration with a discount factor is developed for optimal control of discrete-time nonlinear systems, which is initialized with a positive definite value function rather than zero. The convergence analysis of the discounted value function sequence is provided. The condition for the discount factor is given to guarantee the stability of the controlled plants. With this operation, the iterative control policy that asymptotically stabilizes the closed-loop system can be determined. The introduction of a discount factor has eased some conditions that the system dynamics and the initialization of the generalized value iteration need to fulfill. It is not required that the initial control policy is stabilizing. In the iteration process, under some conditions, if the system is asymptotically stable at the current iteration, then it can be guaranteed that the iterative control policies after this current iteration step also are stabilizing. It is convenient and practical to evaluate the asymptotic stability of the closed-loop system using the iterative control policy. A numerical example with physical background is carried out to validate the present results.

Value iteration based integral reinforcement learning approach for H∞ controller design of continuous-time nonlinear systems

Abstract In this paper, a novel integral reinforcement learning approach is developed based on value iteration (VI) for designing the H∞ controller of continuous-time (CT) nonlinear systems. First, the VI learning mechanism is introduced to solve the zero-sum game problems, which is equivalent to the Hamilton–Jacobi–Isaacs (HJI) equation arising in H∞ control problems. Since the proposed method is based on VI learning mechanism, it does not require the admissible control for the implementation, and thus satisfies a more general initial condition than the works based on policy iteration (PI). The iterative property of the value function is analysed with an arbitrary initial positive function, and the H∞ controller can be derived as the iteration converges. For the implementation of the proposed method, three neural networks are introduced to approximate the iterative value function, the iterative control policy and the iterative disturbance policy, respectively. To verify the effectiveness of the VI based method, a linear case and a nonlinear case are presented, respectively.

An integrated control method for designing non-blocking supervisors using Petri nets

This article presents an efficient integrated approach on designing a non-blocking supervisor for the most general classes of Petri nets, called G-systems that allow multiple resource acquisitions. This work mainly focuses on developing a deadlock prevention policy with a polynomial computational complexity. First, an extraction algorithm of liveness requirement constraints is presented according to the concept of resource partial orders. By considering the different resource requirements of various processes, monitors are added for the uncontrolled G-system on the basis of those precise linear inequality constraints. Afterward, we explore an iterative control policy by utilizing the traditional mathematical programming method, which can ensure the liveness of the resultant controlled system. Comparing with the existing deadlock control policies reported in the literature, the proposed method can achieve a non-blocking controlled G-system with simple structure and high computational efficiency. Finally, a...

WiFlix: Adaptive Video Streaming in Massive MU-MIMO Wireless Networks

We consider the problem of simultaneous on-demand streaming of stored video to multiple users in a multicell wireless network where multiple unicast streaming sessions are run in parallel and share the same frequency band. Each streaming session is formed by the sequential transmission of video “chunks,” such that each chunk arrives into the corresponding user playback buffer within its playback deadline. We formulate the problem as a network utility maximization (NUM) where the objective is to fairly maximize users’ video streaming quality of experience (QoE) and then derive an iterative control policy using Lyapunov optimization, which solves the NUM problem up to any level of accuracy and yields an online protocol with control actions at every iteration decomposing into two layers interconnected by the users’ request queues : 1) a video streaming adaptation layer reminiscent of dynamic adaptive streaming over HTTP (DASH), implemented at each user node; and 2) a transmission scheduling layer where a max-weight scheduler is implemented at each base station. The proposed chunk request scheme is a pull strategy where every user opportunistically requests video chunks from the neighboring base stations and dynamically adapts the quality of its requests based on the current size of the request queue. For the transmission scheduling component, we first describe the general max-weight scheduler and then particularize it to a wireless network where the base stations have multiuser multiple-input multiple-output (MU-MIMO) beamforming capabilities. We exploit the channel hardening effect of large-dimensional MIMO channels (massive MIMO) and devise a low complexity user selection scheme to solve the underlying combinatorial problem of selecting user subsets for downlink beamforming, which can be easily implemented and run independently at each base station. Furthermore, through simulations, we show that deploying MU-MIMO significantly improves video streaming performance and also that the proposed cross-layer approach is able to serve users more fairly than a baseline scheme representative of current systems running independently designed protocol layers.

Neuro-optimal tracking control for a class of discrete-time nonlinear systems via generalized value iteration adaptive dynamic programming approach

In this paper, a novel value iteration adaptive dynamic programming (ADP) algorithm, called "generalized value iteration ADP" algorithm, is developed to solve infinite horizon optimal tracking control problems for a class of discrete-time nonlinear systems. The developed generalized value iteration ADP algorithm permits an arbitrary positive semi-definite function to initialize it, which overcomes the disadvantage of traditional value iteration algorithms. Convergence property is developed to guarantee that the iterative performance index function will converge to the optimum. Neural networks are used to approximate the iterative performance index function and compute the iterative control policy, respectively, to implement the iterative ADP algorithm. Finally, a simulation example is given to illustrate the performance of the developed algorithm.

Policy to cope with deadlocks and livelocks for flexible manufacturing systems using the max′‐controlled new smart siphons

Both deadlocks and livelocks can result in the serious problems in running process of a flexible manufacturing system (FMS). This study proposes an iterative control policy for an FMS modelled with Petri nets on the basis of a combination of revised mixed integer programming and the concept of max′-controlled siphon, which can not only solve the smart siphons associated with deadlocks and livelocks in Petri nets directly, but also make them max′-controlled. Accordingly, an original Petri net system with deadlocks and livelocks can be turned into the live controlled one with a simple structure, and meanwhile no smart siphons can be found in it. It lays foundations for further analysis and control on deadlocks and livelocks. Compared with the existing methods in the literature, the proposed one is more general and effective. A theoretical analysis and several examples are given to demonstrate its efficiency and practical potentials.

Stable iterative adaptive dynamic programming algorithm with approximation errors for discrete-time nonlinear systems

In this paper, a novel iterative adaptive dynamic programming (ADP) algorithm is developed to solve infinite horizon optimal control problems for discrete-time nonlinear systems. When the iterative control law and iterative performance index function in each iteration cannot be accurately obtained, it is shown that the iterative controls can make the performance index function converge to within a finite error bound of the optimal performance index function. Stability properties are presented to show that the system can be stabilized under the iterative control law which makes the present iterative ADP algorithm feasible for implementation both on-line and off-line. Neural networks are used to approximate the iterative performance index function and compute the iterative control policy, respectively, to implement the iterative ADP algorithm. Finally, two simulation examples are given to illustrate the performance of the present method.

A New Self-Learning Optimal Control Scheme for Discrete-Time Nonlinear Systems Using Policy Iterative Adaptive Dynamic Programming

In this paper, a new self-learning method using policy iterative adaptive dynamic programming (ADP) is developed to obtain the optimal control scheme of discrete-time nonlinear systems. The iterative ADP algorithm permits an arbitrary admissible control law to initialize the iterative algorithm. It is the first time that the properties of the policy iterative ADP are established for the discrete-time situation. It proves that the iterative performance index function is non-increasingly convergent to the optimal solution of the Hamilton-Jacobi-Bellman (HJB) equation. It also proves that any of the iterative control policy can stabilize the nonlinear systems. Neural networks are used to approximate the performance index function and compute the optimal control policy, respectively, for facilitating the implementation of the iterative ADP algorithm. Finally, a simulation example is given to illustrate the performance of the present method.