Policy Iteration Research Articles

Safe reinforcement learning for affine nonlinear systems with state constraints and input saturation using control barrier functions

This paper provides a novel safe reinforcement learning (RL) control algorithm to solve safe optimal problems for discrete-time affine nonlinear systems, while the safety and convergence of the control algorithm are proven. The algorithm is proposed based on an adjusted policy iteration (PI) framework using only the measured data along the system trajectories in the environment. The adjusted PI algorithm combines with the system predictive information. Unlike most PI algorithms, an effective method of obtaining an initial safe and stable control policy is given here. In addition, control barrier functions (CBFs) and an input constraint function are introduced to augment reward functions. And the monotonically nonincreasing property of the iterative value function maintains the safe set forward invariant in the PI framework. Moreover, the safety and convergence of the proposed algorithm are proven in theory. Then, the design and implementation of the proposed algorithm are presented based on the identifier-actor-critic structure, where neural networks are employed to approximate the system dynamics, the iterative control policy, and the iterative value function, respectively. Finally, the simulation results illustrate the effectiveness and safety of the proposed algorithm.

Reinforcement learning approach to the control of heavy material handling manipulators for agricultural robots

In this paper, we consider the optimal control problem of heavy material handling manipulators for agricultural robots. Unlike the existing results on agricultural robots, the robot parameters may be unknown for the designer in this paper. To learn the linear quadratic control gain under unknown robot parameters, two reinforcement learning algorithms, i.e., policy iteration (PI) algorithm and value iteration (VI) algorithm, are proposed. Then, through combining the advantages of PI algorithm and VI algorithm, i.e., satisfactory convergence rate and without the restriction on feasibility initial control policy, respectively, a hybrid iteration (HI) algorithm is proposed, which can both achieve a satisfactory convergence rate and remove restrictions on feasibility initial control policy. It is shown that the convergence of the proposed HI algorithm can be achieved in theory. Finally, a simulation example is given to show that our designed HI algorithm can achieve a satisfactory simulation time.

Model-free optimal control of discrete-time systems with additive and multiplicative noises

This paper investigates the optimal control problem for a class of discrete-time stochastic systems subject to additive and multiplicative noises. An algebraic Riccati equation is established which gives the form of the solution to the problem. To obtain the optimal control gain iteratively, an offline policy iteration is presented with convergence proof. A model-free reinforcement learning algorithm is proposed to learn the optimal admissible control policy using the system states and inputs without resorting to the system matrices. It is proven that the estimation error of the kernel matrix is bounded and the iterative control gain is admissible. Compared with the existing work, this paper considers the model-free controller learning problem for stochastic systems suffering from both additive and multiplicative noises using reinforcement learning. The proposed algorithm is illustrated through a numerical example, which shows that our algorithm outperforms other policy iteration algorithms.

Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors.

In reinforcement learning (RL), function approximation errors are known to easily lead to the Q -value overestimations, thus greatly reducing policy performance. This article presents a distributional soft actor-critic (DSAC) algorithm, which is an off-policy RL method for continuous control setting, to improve the policy performance by mitigating Q -value overestimations. We first discover in theory that learning a distribution function of state-action returns can effectively mitigate Q -value overestimations because it is capable of adaptively adjusting the update step size of the Q -value function. Then, a distributional soft policy iteration (DSPI) framework is developed by embedding the return distribution function into maximum entropy RL. Finally, we present a deep off-policy actor-critic variant of DSPI, called DSAC, which directly learns a continuous return distribution by keeping the variance of the state-action returns within a reasonable range to address exploding and vanishing gradient problems. We evaluate DSAC on the suite of MuJoCo continuous control tasks, achieving the state-of-the-art performance.

Reversible Markov decision processes and the Gaussian free field

A Markov decision problem is called reversible if the stationary controlled Markov chain is reversible under every stationary Markovian strategy. A natural application in which such problems arise is in the control of Metropolis–Hastings type dynamics. We characterize all discrete time reversible Markov decision processes with finite state and action spaces. We show that the policy iteration algorithm for finding an optimal policy can be significantly simplified in Markov decision problems of this type. We also highlight the relation between the finite time evolution of the accrual of reward and the Gaussian free field associated to the controlled Markov chain.

Dual Dynamic Programming for the Mean Standard Deviation Canadian Traveller Problem

This article studies the mean standard deviation (mean-std) Canadian traveller problem (CTP). Different from the canonical CTP, which aims at minimizing the traveller's expected travel time, while considering edge breakdown probabilities, we introduce the reliability version of CTP, which tries to find a routing policy with the minimal linear combination of the travel time's mean and standard deviation. With the recent development of internet-of-things (IoT) technology, the transportation network's edges' travel-time statistics, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</i> . <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e</i> ., mean and standard deviation, as well as the traversal probabilities, are available to the end users. With those information, we propose a dual dynamic programming (DDP) method, which simultaneously estimates the first-order and the second-order moments of a given decision-list (DL) policy, and thereby makes improvements towards to the optimal one through the generalized policy iteration (GPI) scheme. We construct an open source benchmark environment to evaluate the performance of different mean-std CTP solutions, and show that the DDP method outperforms state of the arts in a range of transportation networks.

DROP: Deep relocating option policy for optimal ride-hailing vehicle repositioning

In a ride-hailing system, an optimal relocation of vacant vehicles can significantly reduce fleet idling time and balance the supply–demand distribution, enhancing system efficiency and promoting driver satisfaction and retention. Model-free deep reinforcement learning (DRL) has been shown to dynamically learn the relocating policy by actively interacting with the intrinsic dynamics in large-scale ride-hailing systems. However, the issues of sparse reward signals and unbalanced demand and supply distribution place critical barriers in developing effective DRL models. Conventional exploration strategy (e.g., the ϵ-greedy) may barely work under such an environment because of dithering in low-demand regions distant from high-revenue regions. This study proposes the deep relocating option policy (DROP) that supervises vehicle agents to escape from oversupply areas and effectively relocate to potentially underserved areas. We propose to learn the Laplacian embedding of a time-expanded relocation graph, as an approximation representation of the system relocation policy. The embedding generates task-agnostic signals, which in combination with task-dependent signals, constitute the pseudo-reward function for generating DROPs. We present a hierarchical learning framework that trains a high-level relocation policy and a set of low-level DROPs. We note that DROP is a general method and can be incorporated into existing model-free RL advances for further improvements in ride-hailing applications. The effectiveness of our approach is demonstrated using a custom-built high-fidelity simulator with real-world trip record data. We report that DROP significantly improves baseline value and policy iteration algorithms and can effectively resolve the dithering issue in low-demand areas.

Reinforcement Learning-Based Cooperative Optimal Output Regulation via Distributed Adaptive Internal Model.

In this article, a data-driven distributed control method is proposed to solve the cooperative optimal output regulation problem of leader-follower multiagent systems. Different from traditional studies on cooperative output regulation, a distributed adaptive internal model is originally developed, which includes a distributed internal model and a distributed observer to estimate the leader's dynamics. Without relying on the dynamics of multiagent systems, we have proposed two reinforcement learning algorithms, policy iteration and value iteration, to learn the optimal controller through online input and state data, and estimated values of the leader's state. By combining these methods, we have established a basis for connecting data-distributed control methods with adaptive dynamic programming approaches in general since these are the theoretical foundation from which they are built.

H∞ Codesign for Uncertain Nonlinear Control Systems Based on Policy Iteration Method.

In this article, the problem of H∞ codesign for nonlinear control systems with unmatched uncertainties and adjustable parameters is investigated. The main purpose is to solve the adjustable parameters and H∞ controller simultaneously so that better robust control performance can be achieved. By introducing a bounded function and defining a special cost function, the problem of solving the Hamilton-Jacobi-Isaacs equation is transformed into an optimization problem with nonlinear inequality constraints. Based on the sum of squares technique, a novel policy iteration algorithm is proposed to solve the problem of the H∞ codesign. Moreover, one modified algorithm for optimizing the robust performance index is given. The convergence and the performance improvement of new iteration policy algorithms are proved. Simulation results are presented to demonstrate the effectiveness of the proposed algorithms.

Reinforcement Learning Control of Robotic Knee With Human-in-the-Loop by Flexible Policy Iteration.

We are motivated by the real challenges presented in a human-robot system to develop new designs that are efficient at data level and with performance guarantees, such as stability and optimality at system level. Existing approximate/adaptive dynamic programming (ADP) results that consider system performance theoretically are not readily providing practically useful learning control algorithms for this problem, and reinforcement learning (RL) algorithms that address the issue of data efficiency usually do not have performance guarantees for the controlled system. This study fills these important voids by introducing innovative features to the policy iteration algorithm. We introduce flexible policy iteration (FPI), which can flexibly and organically integrate experience replay and supplemental values from prior experience into the RL controller. We show system-level performances, including convergence of the approximate value function, (sub)optimality of the solution, and stability of the system. We demonstrate the effectiveness of the FPI via realistic simulations of the human-robot system. It is noted that the problem we face in this study may be difficult to address by design methods based on classical control theory as it is nearly impossible to obtain a customized mathematical model of a human-robot system either online or offline. The results we have obtained also indicate the great potential of RL control to solving realistic and challenging problems with high-dimensional control inputs.

Convergence of Dynamic Programming on the Semidefinite Cone for Discrete-Time Infinite-Horizon LQR

The goal of this article is to investigate new and simple convergence analysis of dynamic programming for the linear–quadratic regulator problem of discrete-time linear time-invariant systems. In particular, bounds on errors are given in terms of both matrix inequalities and matrix norm. Under a mild assumption on the initial parameter, we prove that the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q$</tex-math></inline-formula> -value iteration exponentially converges to the optimal solution. Moreover, a global asymptotic convergence is also presented. These results are then extended to the policy iteration. We prove that in contrast to the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q$</tex-math></inline-formula> -value iteration, the policy iteration always converges exponentially fast. An example is given to illustrate the results.

Imposing Healthy Hip Motion Pattern and Range by Exoskeleton Control for Individualized Assistance

Powered exoskeletons are promising devices to improve the walking patterns of people with neurological impairments. Providing personalized external assistance though is challenging due to uncertainties and the time-varying nature of human-robot interaction. Recently, human-in-the-loop (HIL) optimization has been investigated for providing assistance to minimize energetic expenditure, usually quantified by metabolic cost. However, this full-body global effect evaluation may not directly reflect the local functions of the targeted joint(s). This makes it difficult to assess the direct effect when robotic assistance is provided. In addition, the HIL optimization method usually does not take into account local joint trajectories, a consideration that is important in imposing healthy joint movements and gait patterns for individuals with lower limb motor deficits. In this paper, we propose a model-free reinforcement learning (RL)-based control framework to achieve a normative range of motion and gait pattern of the hip joint during walking. Our RL-based control provides personalized assistance torque profile by heuristically manipulating three control parameters for hip flexion and extension, respectively, during walking. A least square policy iteration was devised to optimize a cost function associated with control efforts and hip joint trajectory errors by tuning the control parameters. To evaluate the performance of the design approach, a compression sleeve was used to constrain the hip joint of unimpaired human participants to simulate motor deficits. The proposed RL control successfully achieved the desired goal of enlarging the hip joint's range of motion in three participants walking on a treadmill.

H∞-Based Minimal Energy Adaptive Control With Preset Convergence Rate.

This work studies the H∞ -based minimal energy control with a preset convergence rate (PCR) problem for a class of disturbed linear time-invariant continuous-time systems with matched external disturbance. This problem aims to design an optimal controller so that the energy of the control input satisfies a predetermined requirement. Moreover, the closed-loop system asymptotic stability with PCR is ensured simultaneously. To deal with this problem, a modified game algebraic Riccati equation (MGARE) is proposed, which is different from the game algebraic Riccati equation in the traditional H∞ control problem due to the state cost being lost. Therefore, a unique positive-definite solution of the MGARE is theoretically analyzed with its existing conditions. In addition, based on this formulation, a novel approach is proposed to solve the actuator magnitude saturation problem with the system dynamics being exactly known. To relax the requirement of the knowledge of system dynamics, a model-free policy iteration approach is proposed to compute the solution of this problem. Finally, the effectiveness of the proposed approaches is verified through two simulation examples.

Optimal Latency-Oriented Coding and Scheduling in Parallel Queuing Systems

The evolution of 5G and Beyond networks has enabled new applications with stringent end-to-end latency requirements, but providing reliable low-latency service with high throughput over public wireless networks is still a significant challenge. One of the possible ways to solve this is to exploit path diversity, encoding the information flow over multiple streams across parallel links. The challenge presented by this approach is the design of joint coding and scheduling algorithms that adapt to the state of links to take full advantage of path diversity. In this paper, we address this problem for a synchronous traffic source that generates data blocks at regular time intervals (e.g., a video with constant frame rate) and needs to deliver each block within a predetermined deadline. We first develop a closed-form performance analysis in the simple case of two parallel servers without any buffering and single-packet blocks, and propose a model for the general problem based on a Markov Decision Process (MDP). We apply policy iteration to obtain the coding and scheduling policy that maximizes the fraction of source blocks delivered within the deadline: our simulations show the drawbacks of different commonly applied heuristic solutions, drawing general design insights on the optimal policy.

Continuous Action Reinforcement Learning From a Mixture of Interpretable Experts.

Reinforcement learning (RL) has demonstrated its ability to solve high dimensional tasks by leveraging non-linear function approximators. However, these successes are mostly achieved by 'black-box' policies in simulated domains. When deploying RL to the real world, several concerns regarding the use of a 'black-box' policy might be raised. In order to make the learned policies more transparent, we propose in this paper a policy iteration scheme that retains a complex function approximator for its internal value predictions but constrains the policy to have a concise, hierarchical, and human-readable structure, based on a mixture of interpretable experts. Each expert selects a primitive action according to a distance to a prototypical state. A key design decision to keep such experts interpretable is to select the prototypical states from trajectory data. The main technical contribution of the paper is to address the challenges introduced by this non-differentiable prototypical state selection procedure. Experimentally, we show that our proposed algorithm can learn compelling policies on continuous action deep RL benchmarks, matching the performance of neural network based policies, but returning policies that are more amenable to human inspection than neural network or linear-in-feature policies.

Autonomous obstacle avoidance control method based on safe adaptive reinforcement learning

Obstacle avoidance is an important issue in the motion planning of autonomous unmanned systems. Therefore, designing an effective avoidance control method is crucial. For further improving the decision-making process, this paper presents a novel autonomous obstacle avoidance control method based on reinforcement learning that generates a safe motion trajectory in an adaptive manner. First, the barrier function is utilized to design a smooth penalty function in the cost function, thereby transforming the avoidance problem into an unconstrained optimal control problem. Then, adaptive reinforcement learning is implemented by using an actor-critic neural network architecture and policy iteration, in which the critic network uses the state-following kernel function to approximate the cost function while the actor network provides an approximate optimal control policy. During this learning process, the simulated experience is obtained through state extrapolation such that the critic network can use experience replay for reliable local exploration. Finally, simulation experiments on simplified drone systems and a nonlinear numerical system are provided. The proposed method can generate a safe motion trajectory in real time with comparable performance.

Optimal Leader-Following Consensus Control of Multi-Agent Systems: A Neural Network Based Graphical Game Approach

In this article, the optimal leader-following consensus control problem of multi-agent systems is solved using a novel neural network-based (NN-based) integrated heuristic dynamic programming (IHDP) algorithm under a distributed learning framework. The optimal controller is designed to find a Nash equilibrium of the multi-agent differential graphical game. According to the principle of optimality, this problem can be transformed into solving a set of coupled Hamilton-Jacobi-Bellman (HJB) equations, which are difficult to solve directly because of the nonlinearity and strong coupling of the equations. The proposed algorithm combines policy iteration (PI) and value iteration (VI), and the Nash equilibrium of performance index among agents is achieved distributedly, i.e., all agents update their control inputs only use local information. The convergence of the algorithm is rigorously proved. Neural network (NN) approximation implementation is given for the unknown structures of local value functions of agents. The effectiveness of the algorithm is verified by numerical simulation.

ADP-Based Remote Secure Control for Networked Control Systems Under Unknown Nonlinear Attacks in Sensors and Actuators

In this article, we concentrate on a class of networked control systems (NCSs) under the unknown nonlinear attacks, which target the sensors and actuators with remote communication channels. To attenuate the attacks remotely, the remote secure control scheme based on the adaptive dynamic programming (ADP) is proposed. First, the dynamic process of NCSs with nonlinear sensors and actuators attacks is analyzed and an adaptive remote estimator is designed to obtain the state information of the NCSs from a long distance, which is the key to achieve the purpose of remote control. Moreover, to compensate for the unknown nonlinear attacks, the adaptive law of the remote estimator is presented based on recursive least square. Furthermore, we consider the influence of the estimation errors, the minimax control theory is applied to improve the design of the policy iteration algorithm, and the ADP-based remote secure control strategy is proposed to rely on the remote estimation information. Then, the neural networks are built to implement the online algorithm. Finally, results’ analysis illustrates on two networked power systems the effectiveness of the proposed method.

Adaptive dynamic event-triggered control for constrained modular reconfigurable robot

Compared with traditional robot, modular reconfigurable robot (MRR) has the advantages of strong environmental adaptability and flexible task completion. According to the optimal tracking control problem (OTCP) of MRR under some restricted conditions, this paper puts forward a constrained dynamic event-triggered control (DETC) for MRR system with disturbance through adaptive dynamic programming (ADP), which can minimize the information interaction quantity under the premise of system stability and expected control effect. In view of the uncertainty of model coupling part, the identification network is used to estimate the dynamics of MRR and the estimation error is proved to be uniformly ultimate bounded (UUB). The other three groups of critic, action and disturbance neural networks (NNs) are established by the approximation principle of ADP. The optimal control pair is obtained through policy iteration (PI) with DETC, and the triggering condition is designed based on the asymptotic stability of MRR system. At last, the strengths of the algorithm in this paper are validated through simulation experiments.

Intertemporal Pricing via Nonparametric Estimation: Integrating Reference Effects and Consumer Heterogeneity

Problem definition: We consider intertemporal pricing in the presence of reference effects and consumer heterogeneity. Our research question encompasses how to estimate heterogeneous consumer reference effects from data and how to efficiently compute the optimal pricing policy. Academic/practical relevance: Understanding reference effects is essential for designing pricing policies in modern retailing. Our work contributes to this area by incorporating consumer heterogeneity under arbitrary distributions. Methodology: We propose a mixed logit demand model that allows arbitrary joint distributions of valuations, responsiveness to prices, and responsiveness to reference prices among consumers. We use a nonparametric estimation method to learn consumer heterogeneity from transaction data. Further, we formulate the pricing optimization as an infinite horizon dynamic programming problem and solve it by applying a modified policy iteration algorithm. Results: Moreover, we investigate the structure of optimal pricing policies and prove the suboptimality of constant pricing policies even when all consumers are loss-averse according to the classical definition. Our numerical studies show that our estimation and optimization framework improves the expected revenue of retailers via accounting for heterogeneity. We validate our model using real data from JD.com, a large E-commerce retailer, and find empirical evidence of consumer heterogeneity. Managerial implications: In practice, ignoring consumer heterogeneity may lead to a significant loss of revenue. Furthermore, heterogeneous reference effect offers a strong motive for promotions and price fluctuations. History: This paper has been accepted as part of the 2020 MSOM Data Driven Research Challenge. Funding: This work was partially supported by the National Natural Science Foundation of China [Grant 71991462]. Supplemental Material: The e-companion is available at https://doi.org/10.1287/msom.2022.1134 .