Discounted Value Function Research Articles

Lifelong reinforcement learning tracking control of nonlinear strict-feedback systems using multilayer neural networks with constraints

This paper presents a novel safe integral reinforcement learning (IRL)-based optimal trajectory tracking scheme for nonlinear systems with uncertain dynamics that is subject to constraints. We leverage multilayer neural networks (MNNs) for actor-critic MNNs along with an NN identifier in the backstepping process for minimizing a discounted value function. A time-varying barrier Lyapunov function (TVBLF) is utilized for handling constraints and to provide safety assurances. Online weight update laws for the actor and critic MNNs are derived that are driven by Bellman error and control input error. We introduce an online lifelong learning (LL) method in the critic NN, utilizing the Bellman error in MNNs to address catastrophic forgetting. The method’s effectiveness is demonstrated through simulations on mobile robot multitask tracking. The paper concludes with a stability analysis of the closed-loop system.

Optimal trajectory tracking of uncertain nonlinear continuous-time strict-feedback systems with dynamic constraints

A novel method for optimal trajectory tracking of uncertain nonlinear systems by using adaptive dynamic programming (ADP)-based integral reinforcement learning (IRL) and neural networks (NNs) is proposed. The method utilises an actor-critic framework with optimal backstepping to minimise a discounted value function and uses a single-layer NN identifier for estimating the unknown dynamics. For safety assurance, a time-varying barrier Lyapunov function (TVBLF) is used in the control design to handle the constraints. A novel weight-tuning law by using the control input error, integral Bellman error, and the NN identifier is used for the actor-critic framework. A novel lifelong learning (LL)-based method for critic NN is utilised in an optimal framework by incorporating the Bellman error to mitigate catastrophic forgetting in multitasking scenarios. Stability analysis using Lyapunov stability theory is obtained for the overall closed-loop system. Simulations of leader-follower mobile robot formation control show a 25% reduction in the cost in multitasking scenarios.

Robust Average-Reward Reinforcement Learning

Robust Markov decision processes (MDPs) aim to find a policy that optimizes the worst-case performance over an uncertainty set of MDPs. Existing studies mostly have focused on the robust MDPs under the discounted reward criterion, leaving the ones under the average-reward criterion largely unexplored. In this paper, we develop the first comprehensive and systematic study of robust average-reward MDPs, where the goal is to optimize the long-term average performance under the worst case. Our contributions are four-folds: (1) we prove the uniform convergence of the robust discounted value function to the robust average-reward function as the discount factor γ goes to 1; (2) we derive the robust average-reward Bellman equation, characterize the structure of its solution set, and prove the equivalence between solving the robust Bellman equation and finding the optimal robust policy; (3) we design robust dynamic programming algorithms, and theoretically characterize their convergence to the optimal policy; and (4) we design two model-free algorithms unitizing the multi-level Monte-Carlo approach, and prove their asymptotic convergence

Continual online learning-based optimal tracking control of nonlinear strict-feedback systems: application to unmanned aerial vehicles

A novel optimal trajectory tracking scheme is introduced for nonlinear continuous-time systems in strict feedback form with uncertain dynamics by using neural networks (NNs). The method employs an actor-critic-based NN backstepping technique for minimizing a discounted value function along with an identifier to approximate unknown system dynamics that are expressed in augmented form. Novel online weight update laws for the actor and critic NNs are derived by using both the NN identifier and Hamilton-Jacobi-Bellman residual error. A new continual lifelong learning technique utilizing the Fisher Information Matrix via Hamilton-Jacobi-Bellman residual error is introduced to obtain the significance of weights in an online mode to overcome the issue of catastrophic forgetting for NNs, and closed-loop stability is analyzed and demonstrated. The effectiveness of the proposed method is shown in simulation by contrasting the proposed with a recent method from the literature on an underactuated unmanned aerial vehicle, covering both its translational and attitude dynamics.

Optimal Learning Output Tracking Control: A Model-Free Policy Optimization Method With Convergence Analysis.

Optimal learning output tracking control (OLOTC) in a model-free manner has received increasing attention in both the intelligent control and the reinforcement learning (RL) communities. Although the model-free tracking control has been achieved via off-policy learning and Q -learning, another popular RL idea of direct policy learning, with its easy-to-implement feature, is still rarely considered. To fill this gap, this article aims to develop a novel model-free policy optimization (PO) algorithm to achieve the OLOTC for unknown linear discrete-time (DT) systems. The iterative control policy is parameterized to directly improve the discounted value function of the augmented system via the gradient-based method. To implement this algorithm in a model-free manner, a model-free two-point policy gradient (PG) algorithm is designed to approximate the gradient of discounted value function by virtue of the sampled states and the reference trajectories. The global convergence of model-free PO algorithm to the optimal value function is demonstrated with the sufficient quantity of samples and proper conditions. Finally, numerical simulation results are provided to validate the effectiveness of the present method.

Markov decision processes under risk sensitivity: A discount vanishing approach

This paper considers Markov decision processes (MDPs) with risk-sensitivity. The aim is to explore the effects of state transience and non-communication on the optimal control of the system. A vanishing discount approach is investigated. The approximating system is the MDP evaluated by the usual discounted exponential utility. After being appropriately normalized, it is shown that the optimal discounted value functions converge to the optimal risk-sensitive averages as the discount factor goes to 1. These value functions are shown to depend on certain order structure of the state space. In this way it is also proved that the optimal policies for the discounted system converge to the optimal ones for the risk-sensitive system as the discount factor goes to 1. In proving these, an ordered partition of the state space is introduced, which is closely related to the characteristics of state communication, transience and absorption.

Robust Average-Reward Markov Decision Processes

In robust Markov decision processes (MDPs), the uncertainty in the transition kernel is addressed by finding a policy that optimizes the worst-case performance over an uncertainty set of MDPs. While much of the literature has focused on discounted MDPs, robust average-reward MDPs remain largely unexplored. In this paper, we focus on robust average-reward MDPs, where the goal is to find a policy that optimizes the worst-case average reward over an uncertainty set. We first take an approach that approximates average-reward MDPs using discounted MDPs. We prove that the robust discounted value function converges to the robust average-reward as the discount factor goes to 1, and moreover when it is large, any optimal policy of the robust discounted MDP is also an optimal policy of the robust average-reward. We further design a robust dynamic programming approach, and theoretically characterize its convergence to the optimum. Then, we investigate robust average-reward MDPs directly without using discounted MDPs as an intermediate step. We derive the robust Bellman equation for robust average-reward MDPs, prove that the optimal policy can be derived from its solution, and further design a robust relative value iteration algorithm that provably finds its solution, or equivalently, the optimal robust policy.

Space manipulator optimal impedance control using integral reinforcement learning

This paper examines the optimal impedance control problem for large-scale space manipulator operational tasks with unknown contact dynamics and partial measurements. More specifically, by quantifying the interaction performance using a discounted value function, the optimal impedance control problem is tactfully transformed into a linear quadratic tracking problem. By resorting to the historical inputs and outputs, an improved state reconstruction method is presented, which obviates the velocity measurement. Unlike the existing state reconstruction method for continuous-time systems, the estimation bias caused by probing noise is completely eliminated under the improved state reconstruction method. Based on this, a novel model-free value iteration integral reinforcement learning algorithm is developed to approximate optimal impedance parameters. Compared with the earlier integral reinforcement learning algorithms, the proposed algorithm not only averts any prior contact dynamics knowledge and full-state measurements, but also eliminates the heavy dependence on the specific initial stabilization control. In addition, the implementation and convergence of the proposed algorithm are discussed successively. Finally, numerical simulations verify the effectiveness of the theoretical results.

Homogenization for sub-riemannian Lagrangians

We consider Lagrangians that are defined only on the horizontal distribution of a sub-riemannian manifold. The associated Hamiltonian is neither strictly convex nor coercive. We prove a result on homogenization of the Hamilton–Jacobi equation following the approach in Contreras et al (2015 Calc. Var. Partial Differ. Equ. 52 237–52). To establish the result, we need to extend weak KAM and Aubry–Mather theories to the sub-riemannian setting. We obtain a Tonelli theorem to dispend of the Lagrangian dynamics tools used in standard weak KAM theory. In the way towards our main result we observe that, the long time convergence of the Lax-Oleinik semigroup, and the vanishing discount convergence of the discounted value function hold.

Robust tracking control with reinforcement learning for nonlinear‐constrained systems

AbstractThis article considers the robust tracking control problem of uncertain nonlinear systems with asymmetric input constraints. Initially, the tracking error dynamics and the desired trajectory dynamics are constructed as an augmented system. Then, with a discounted value function being introduced for the nominal augmented system, the original tracking control problem is transformed into a constrained optimal control problem. To solve the constrained optimal control problem, its related Hamilton–Jacobi–Bellman equation (HJBE) is developed. After that, a critic approximator is constructed to solve the HJBE in the reinforcement learning framework. To tune the parameters used in the critic approximator, a novel concurrent learning technique is introduced, which could remove the persistence of excitation condition. Moreover, the uniform ultimate boundedness of the tracking error and the parameters' estimation error of the critic approximator is assured via the Lyapunov's approach. Finally, a spring‐mass‐damper mechanical system and a robot manipulator system are given to validate the theoretical claims.

Multi-agent Q-Learning control of spacecraft formation flying reconfiguration trajectories

This paper presents a novel approach based on multi-agent reinforcement learning for spacecraft formation flying reconfiguration tracking problems. In this scheme, spacecrafts learn the control strategy via transfer learning. For this matter, a new generalized discounted value function is introduced for the tracking problems. Due to the digital nature of spacecraft computer systems, local optimal controllers are developed for the spacecrafts in discrete-time. The stability of the controller is proven. Two Q-learning algorithms are proposed, in each of which the optimal control solution is learned on-line without knowledge about the system dynamics. In the first algorithm, each agent learns the optimal control independently. In the second one, each agent shares the learned information with other agents. Next, the collision avoidance capability is provided. The effectiveness of the presented schemes is verified through simulations and compared with each other.

Event-Triggered ADP for Tracking Control of Partially Unknown Constrained Uncertain Systems.

An event-triggered adaptive dynamic programming (ADP) algorithm is developed in this article to solve the tracking control problem for partially unknown constrained uncertain systems. First, an augmented system is constructed, and the solution of the optimal tracking control problem of the uncertain system is transformed into an optimal regulation of the nominal augmented system with a discounted value function. The integral reinforcement learning is employed to avoid the requirement of augmented drift dynamics. Second, the event-triggered ADP is adopted for its implementation, where the learning of neural network weights not only relaxes the initial admissible control but also executes only when the predefined execution rule is violated. Third, the tracking error and the weight estimation error prove to be uniformly ultimately bounded, and the existence of a lower bound for the interexecution times is analyzed. Finally, simulation results demonstrate the effectiveness of the present event-triggered ADP method.

Adaptive dynamic programming‐based event‐triggered optimal tracking control

AbstractIn this article, an event‐triggered constrained optimal tracking control algorithm using integral reinforcement learning (IRL) is developed. First, the constrained optimal tracking control problem is transformed into an optimal regulation problem by employing an augmented system with a discounted value function. Then, IRL is introduced to solve the Hamilton‐Jacobi‐Bellman equation, where the drift dynamics and reference dynamics are not required. The learning of neural network weights is event‐triggered and there is no restriction on the initial control to be admissible. The involvement of event‐triggering mechanism alleviates the pressure of data transmission on the network to some extent, which is suitable for control systems with limited computational and communication resources. Moreover, the nonexistence of Zeno behavior and the stability of the impulsive system are proved, respectively. Finally, the application of the algorithm on a mass‐spring‐damper system verifies its effectiveness.

Average Cost Optimality in Partially Observed Lost-Sales Inventory Systems

We consider a partially observable inventory system in which the inventory level can only be observed when it reaches zero, the unmet demand is lost, and replenishment orders must be decided so as to minimize the long-run average cost. This problem has an infinite-dimensional state space and is therefore difficult to analyze. We prove the existence of an average-cost optimal policy, using the so-called vanishing discount factor approach. As a main methodological contribution, we provide a way to verify the key condition -- the uniform boundedness of the relative discounted value function -- for the partially observed system. To accomplish that we construct -- for the partially observed system -- a valid policy which in a certain sense copies the actions of another policy for the process with a different initial state. To our best knowledge, this paper is the first dealing with inventory models with partially observable inventory levels under the long-run average cost criterion.

Generalized value iteration for discounted optimal control with stability analysis

Abstract In this work, the generalized value iteration with a discount factor is developed for optimal control of discrete-time nonlinear systems, which is initialized with a positive definite value function rather than zero. The convergence analysis of the discounted value function sequence is provided. The condition for the discount factor is given to guarantee the stability of the controlled plants. With this operation, the iterative control policy that asymptotically stabilizes the closed-loop system can be determined. The introduction of a discount factor has eased some conditions that the system dynamics and the initialization of the generalized value iteration need to fulfill. It is not required that the initial control policy is stabilizing. In the iteration process, under some conditions, if the system is asymptotically stable at the current iteration, then it can be guaranteed that the iterative control policies after this current iteration step also are stabilizing. It is convenient and practical to evaluate the asymptotic stability of the closed-loop system using the iterative control policy. A numerical example with physical background is carried out to validate the present results.

H∞ Tracking Control for Linear Discrete-Time Systems: Model-Free Q-Learning Designs

In this letter, a novel model-free Q-learning based approach is developed to solve the H ∞ tracking problem for linear discrete-time systems. A new exponential discounted value function is introduced that includes the cost of the whole control input and tracking error. The tracking Bellman equation and the game algebraic Riccati equation (GARE) are derived. The solution to the GARE leads to the feedback and feedforward parts of the control input. A Q-learning algorithm is then developed to learn the solution of the GARE online without requiring any knowledge of the system dynamics. Convergence of the algorithm is analyzed, and it is also proved that probing noises in maintaining the persistence of excitation (PE) condition do not result in any bias. An example of the F-16 aircraft short period dynamics is developed to validate the proposed algorithm.

Model-free Optimal Tracking Control for an Aircraft Skin Inspection Robot with Constrained-input and Input Time-delay via Integral Reinforcement Learning

This paper presents a model-free optimal tracking control algorithm for an aircraft skin inspection robot with constrained-input and input time-delay. To tackle the input time-delay problem, the original system is transformed into a delay-free system with constrained-input and unknown input coupling term. In order to overcome the optimal control problem subject to constrained-input, a discounted value function is employed. In general, it is known that the HJB equation does not admit a classical smooth solution. Moreover, since the input coupling term of the delay-free system is unknown, a model-free integral reinforcement learning(IRL) algorithm which only requires the system sampling data generated by arbitrary different control inputs and external disturbances is proposed. The model-free IRL method is implemented on an actor-critic neural network (NN) structure. A system sampling data set is utilized to learn the value function and control policy. Finally, the simulation verifies the effectiveness of the proposed algorithm.

STOCHASTIC SETUP-COST INVENTORY MODEL WITH BACKORDERS AND QUASICONVEX COST FUNCTIONS

This paper studies a periodic-review single-commodity setup-cost inventory model with backorders and holding/backlog costs satisfying quasiconvexity assumptions. We show that the Markov decision process for this inventory model satisfies the assumptions that lead to the validity of optimality equations for discounted and average-cost problems and to the existence of optimal (s,S) policies. In particular, we prove the equicontinuity of the family of discounted value functions and the convergence of optimal discounted lower thresholds to the optimal average-cost lower threshold for some sequence of discount factors converging to 1. If an arbitrary nonnegative amount of inventory can be ordered, we establish stronger convergence properties: (i) the optimal discounted lower thresholds converge to optimal average-cost lower threshold; and (ii) the discounted relative value functions converge to average-cost relative value function. These convergence results previously were known only for subsequences of discount factors even for problems with convex holding/backlog costs. The results of this paper also hold for problems with fixed lead times.

The Vanishing Discount Approach in a class of Zero-Sum Finite Games with Risk-Sensitive Average Criterion

This work concerns zero-sum stochastic games with finite state space and compact action sets. The game is driven by two players and it is assumed that player 1 has a nonull and constant risk-sensitivity coefficient, so that a random cost is assessed via an exponential disutility function. The main objective of the paper is to show that as the discount factor increases to 1, an appropriate normalization of the risk-sensitive discounted value function converges to the risk-sensitive average value function. Besides standard continuity-compactness conditions, that result is obtained assuming that (i) the state space is communicating under the action of any pair of stationary deterministic policies and (ii) a minorization property holds at some state. Additionally, it is shown that, if as in the classical risk-neutral context, the risk-sensitive discounted value function is multiplied by 1 minus the discount factor, then the resulting normalization converges to the arithmetic mean of the risk-sensitive average value over the interval joining 0 and the risk-sensitivity coefficient.

Vanishing discount approximations in controlled Markov chains with risk-sensitive average criterion

Abstract This work concerns Markov decision chains on a finite state space. The decision-maker has a constant and nonnull risk sensitivity coefficient, and the performance of a control policy is measured by two different indices, namely, the discounted and average criteria. Motivated by well-known results for the risk-neutral case, the problem of approximating the optimal risk-sensitive average cost in terms of the optimal risk-sensitive discounted value functions is addressed. Under suitable communication assumptions, it is shown that, as the discount factor increases to 1, appropriate normalizations of the optimal discounted value functions converge to the optimal average cost, and to the functional part of the solution of the risk-sensitive average cost optimality equation.