Approximate/adaptive Dynamic Programming Research Articles

Reinforcement Learning Control of Robotic Knee With Human-in-the-Loop by Flexible Policy Iteration.

We are motivated by the real challenges presented in a human-robot system to develop new designs that are efficient at data level and with performance guarantees, such as stability and optimality at system level. Existing approximate/adaptive dynamic programming (ADP) results that consider system performance theoretically are not readily providing practically useful learning control algorithms for this problem, and reinforcement learning (RL) algorithms that address the issue of data efficiency usually do not have performance guarantees for the controlled system. This study fills these important voids by introducing innovative features to the policy iteration algorithm. We introduce flexible policy iteration (FPI), which can flexibly and organically integrate experience replay and supplemental values from prior experience into the RL controller. We show system-level performances, including convergence of the approximate value function, (sub)optimality of the solution, and stability of the system. We demonstrate the effectiveness of the FPI via realistic simulations of the human-robot system. It is noted that the problem we face in this study may be difficult to address by design methods based on classical control theory as it is nearly impossible to obtain a customized mathematical model of a human-robot system either online or offline. The results we have obtained also indicate the great potential of RL control to solving realistic and challenging problems with high-dimensional control inputs.

Online [formula omitted] control for continuous-time nonlinear large-scale systems via single echo state network

In this article, the H∞ optimal control for nonlinear large-scale interconnected systems is designed using novel adaptive/approximate dynamic programming (ADP) method and single echo state network (ESN). All subsystems of the large-scale systems are combined into an augmented system. Subsequently, the H∞ problem is transformed into a zero-sum game problem, where a nonlinear partial differential Hamilton-Jacobi-Isaacs (HJI) equation is required to be solved. However, the HJI equation has no analytically solution, so this paper develops a novel online ADP algorithm to approximatively obtain the index function and control policy simultaneously employing single ESN without requiring the initial admissible control. The stability of the closed-loop augmented system and the ESN output weights is analyzed using the Lyapunov theorem and guaranteed to be uniformly ultimate boundedness (UUB). A large-scale system example including three subsystems is simulated to show the feasibility of the suggested control scheme.

Adaptive Reinforcement Learning Strategy with Sliding Mode Control for Unknown and Disturbed Wheeled Inverted Pendulum

This paper develops a novel adaptive integral sliding-mode control (SMC) technique to improve the tracking performance of a wheeled inverted pendulum (WIP) system, which belongs to a class of continuous time systems with input disturbance and/or unknown parameters. The proposed algorithm is established based on an integrating between the advantage of online adaptive reinforcement learning control and the high robustness of integral sliding-mode control (SMC) law. The main objective is to find a general structure of integral sliding mode control law that can guarantee the system state reaching a sliding surface in finite time. An adaptive/approximate optimal control based on the approximate/adaptive dynamic programming (ADP) is responsible for the asymptotic stability of the closed loop system. Furthermore, the convergence possibility of proposed output feedback optimal control was determined without the convergence of additional state observer. Finally, the theoretical analysis and simulation results validate the performance of the proposed control structure.

Pareto optimal control of the mean-field stochastic systems by adaptive dynamic programming algorithm

The Pareto game for the model-free continuous-time stochastic system is studied through approximate/adaptive dynamic programming (ADP) in this paper. Firstly, the model-based online iterative algorithm is proposed, and it is proved that the control iterative sequence converges to the Pareto efficient solution, but the algorithm requires complete system parameters. Then, we derive the model-free iterative equation and develop the ADP algorithm to calculate the equation by collecting updated states and input information online. From the derivation of the ADP algorithm, the model-free iterative equation and the model-based iterative equation have the same solution, which means that the ADP algorithm can approximate the Pareto optimal solution. Next, the convergence analysis shows that the Pareto optimal strategy is uniquely determined by the ADP algorithm. Finally, two simulation examples confirm the feasibility of the ADP algorithm.

An improvement of single-network adaptive critic design for nonlinear systems with asymmetry constraints

Traditional approximate/adaptive dynamic programming (ADP) methods can handle a very special class of systems subject to symmetry constraints. In this study, I extend the exiting ADP to a broader class of nonlinear dynamic systems with asymmetry constraints. Firstly, I propose a novel nonquadratic cost function, based on which the developed optimal controller by solving Hamilton–Jacobi–Bellman equation can limit its value to arbitrarily prescribed bound. Then, to avoid “curse of dimensionality”, I approximately implement the addressed controller via single-network adaptive critic design. Fuzzy Hyperbolic Model is introduced to construct the single critic network by approximating optimal cost function, from which I further derive the optimal control law. The potential advantages are that the control structure is simple and the computational load is low. Lyapunov synthesis proves the ultimately uniformly bounded stability of closed-loop control system. Finally, numerical simulation results verify the efficiency and superiority of the proposed approach.

Output-feedback adaptive optimal control of interconnected systems based on robust adaptive dynamic programming

This paper studies the adaptive and optimal output-feedback problem for continuous-time uncertain systems with nonlinear dynamic uncertainties. Data-driven output-feedback control policies are developed by approximate/adaptive dynamic programming (ADP) based on both policy iteration and value iteration methods. The obtained adaptive and optimal output-feedback controllers differ from the existing literature on the ADP in that they are derived from sampled-data systems theory and are guaranteed to be robust to dynamic uncertainties. A small-gain condition is given under which the overall system is globally asymptotically stable at the origin. An application to power systems is given to test the effectiveness of the proposed approaches.

Revisiting approximate dynamic programming and its convergence.

Value iteration-based approximate/adaptive dynamic programming (ADP) as an approximate solution to infinite-horizon optimal control problems with deterministic dynamics and continuous state and action spaces is investigated. The learning iterations are decomposed into an outer loop and an inner loop. A relatively simple proof for the convergence of the outer-loop iterations to the optimal solution is provided using a novel idea with some new features. It presents an analogy between the value function during the iterations and the value function of a fixed-final-time optimal control problem. The inner loop is utilized to avoid the need for solving a set of nonlinear equations or a nonlinear optimization problem numerically, at each iteration of ADP for the policy update. Sufficient conditions for the uniqueness of the solution to the policy update equation and for the convergence of the inner-loop iterations to the solution are obtained. Afterwards, the results are formed as a learning algorithm for training a neurocontroller or creating a look-up table to be used for optimal control of nonlinear systems with different initial conditions. Finally, some of the features of the investigated method are numerically analyzed.

Learning Control of Dynamical Systems Based on Markov Decision Processes: Research Frontiers and Outlooks

摘要: 基于马氏决策过程(Markov decision process, MDP)的动态系统学习控制是近年来一个涉及机器学习、控制理论和运筹学等多个学科的交叉研究方向, 其主要目标是实现系统在模型复杂或者不确定等条件下基于数据驱动的多阶段优化控制. 本文对基于MDP的动态系统学习控制理论、算法与应用的发展前沿进行综述,重点讨论增强学习(Reinforcement learning, RL)与近似动态规划(Approximate dynamic programming, ADP)理论与方法的研究进展,其中包括时域差值学习理论、求解连续状态与行为空间MDP的值函数逼近方法、 直接策略搜索与近似策略迭代、自适应评价设计算法等,最后对相关研究领域的应用及发展趋势进行分析和探讨. 关键词: 学习控制 / Markov决策过程 / 增强学习 / 近似动态规划 / 机器学习 / 自适应控制

A boundedness result for the direct heuristic dynamic programming

Approximate/adaptive dynamic programming (ADP) has been studied extensively in recent years for its potential scalability to solve large state and control space problems, including those involving continuous states and continuous controls. The applicability of ADP algorithms, especially the adaptive critic designs has been demonstrated in several case studies. Direct heuristic dynamic programming (direct HDP) is one of the ADP algorithms inspired by the adaptive critic designs. It has been shown applicable to industrial scale, realistic and complex control problems. In this paper, we provide a uniformly ultimately boundedness (UUB) result for the direct HDP learning controller under mild and intuitive conditions. By using a Lyapunov approach we show that the estimation errors of the learning parameters or the weights in the action and critic networks remain UUB. This result provides a useful controller convergence guarantee for the first time for the direct HDP design.

Approximate Dynamic Programming for Optimal Stationary Control With Control-Dependent Noise

This brief studies the stochastic optimal control problem via reinforcement learning and approximate/adaptive dynamic programming (ADP). A policy iteration algorithm is derived in the presence of both additive and multiplicative noise using Itô calculus. The expectation of the approximated cost matrix is guaranteed to converge to the solution of some algebraic Riccati equation that gives rise to the optimal cost value. Moreover, the covariance of the approximated cost matrix can be reduced by increasing the length of time interval between two consecutive iterations. Finally, a numerical example is given to illustrate the efficiency of the proposed ADP methodology.

Adaptive dynamic programming for online solution of a zero-sum differential game

This paper will present an approximate/adaptive dynamic programming (ADP) algorithm, that uses the idea of integral reinforcement learning (IRL), to determine online the Nash equilibrium solution for the two-player zerosum differential game with linear dynamics and infinite horizon quadratic cost. The algorithm is built around an iterative method that has been developed in the control engineering community for solving the continuous-time game algebraic Riccati equation (CT-GARE), which underlies the game problem. We here show how the ADP techniques will enhance the capabilities of the offline method allowing an online solution without the requirement of complete knowledge of the system dynamics. The feasibility of the ADP scheme is demonstrated in simulation for a power system control application. The adaptation goal is the best control policy that will face in an optimal manner the highest load disturbance.

Adaptive Dynamic Programming: An Introduction

In this article, we introduce some recent research trends within the field of adaptive/approximate dynamic programming (ADP), including the variations on the structure of ADP schemes, the development of ADP algorithms and applications of ADP schemes. For ADP algorithms, the point of focus is that iterative algorithms of ADP can be sorted into two classes: one class is the iterative algorithm with initial stable policy; the other is the one without the requirement of initial stable policy. It is generally believed that the latter one has less computation at the cost of missing the guarantee of system stability during iteration process. In addition, many recent papers have provided convergence analysis associated with the algorithms developed. Furthermore, we point out some topics for future studies.

Issues on Stability of ADP Feedback Controllers for Dynamical Systems

This paper traces the development of neural-network (NN)-based feedback controllers that are derived from the principle of adaptive/approximate dynamic programming (ADP) and discusses their closed-loop stability. Different versions of NN structures in the literature, which embed mathematical mappings related to solutions of the ADP-formulated problems called "adaptive critics" or "action-critic" networks, are discussed. Distinction between the two classes of ADP applications is pointed out. Furthermore, papers in "model-free" development and model-based neurocontrollers are reviewed in terms of their contributions to stability issues. Recent literature suggests that work in ADP-based feedback controllers with assured stability is growing in diverse forms.