Iterative Dynamic Programming Research Articles

A study about control for rolling objects and beam system

The ball and beam system consists of a long beam which can be tilted by a servo motor together with a ball rolling back and forth on top of the beam. In this article, an equation of motion of the rolling objects and the beam are derived in consideration of characteristics of driving source. By considering the changes in the number of rolling object, trajectories for positioning control are calculated by iterative dynamic programming method. And the dynamic characteristics of the system controlled based on the trajectory for saving energy are analyzed theoretically and investigated experimentally.

Analysis of manipulator in consideration of impact absorption between link and projected object

In this article, equations of motion of a manipulator are derived in consideration of the characteristics of DC servomotors, and a performance criterion for saving energy is defined in consideration of energy consumption of the driving source. When the manipulator is operated in a vertical plane, the system is highly non-linear due to gravity, and an analytical solution cannot be found. By considering for catching the object that moves parabolic, trajectories for saving energy are calculated by iterative dynamic programming method. And, the dynamic characteristics of two-link system controlled based on the trajectory for saving energy are analyzed theoretically. When the object moves parabolic, measurement method for the force of collision between link and object is examined by experiment.

Economical control of indoor air quality in underground metro station using an iterative dynamic programming-based ventilation system

A set-point of ventilation control system plays an important role for efficient ventilation inside metro stations, since it affects level of indoor air pollutants and ventilation energy consumption concurrently. In this study, to maintain indoor air quality (IAQ) at a comfortable range with a lower ventilation energy consumption, the optimal set-points of the ventilation control system were determined. The concentration of air pollutants inside the station shows a periodic diurnal variation in accordance with the number of passengers and subway frequency. To consider the diurnal variation of IAQ, an iterative dynamic programming (IDP) that searches for a piecewise control policy by separating whole system duration into several stages was applied. The optimal set-points of the ventilation control system in underground D-subway station, Korea were determined at every 3, 2, and 1 h, respectively. Then, according to the set-point changes, the ventilation controller was adjusted to an appropriate ventilation fan speed, correlating to the amount of outdoor PM10 that flows into the station. The results showed that the ventilation control system with the IDP-based optimal set-points has a better economical ventilation performance than manual ventilation system, with a 4.6% decrease in energy consumption, maintaining a comfortable IAQ level inside the station.

Value iteration and adaptive dynamic programming for data-driven adaptive optimal control design

This paper presents a novel non-model-based, data-driven adaptive optimal controller design for linear continuous-time systems with completely unknown dynamics. Inspired by the stochastic approximation theory, a continuous-time version of the traditional value iteration (VI) algorithm is presented with rigorous convergence analysis. This VI method is crucial for developing new adaptive dynamic programming methods to solve the adaptive optimal control problem and the stochastic robust optimal control problem for linear continuous-time systems. Fundamentally different from existing results, the a priori knowledge of an initial admissible control policy is no longer required. The efficacy of the proposed methodology is illustrated by two examples and a brief comparative study between VI and earlier policy-iteration methods.

Iterative Adaptive Dynamic Programming for Solving Unknown Nonlinear Zero-Sum Game Based on Online Data.

H∞ control is a powerful method to solve the disturbance attenuation problems that occur in some control systems. The design of such controllers relies on solving the zero-sum game (ZSG). But in practical applications, the exact dynamics is mostly unknown. Identification of dynamics also produces errors that are detrimental to the control performance. To overcome this problem, an iterative adaptive dynamic programming algorithm is proposed in this paper to solve the continuous-time, unknown nonlinear ZSG with only online data. A model-free approach to the Hamilton-Jacobi-Isaacs equation is developed based on the policy iteration method. Control and disturbance policies and value are approximated by neural networks (NNs) under the critic-actor-disturber structure. The NN weights are solved by the least-squares method. According to the theoretical analysis, our algorithm is equivalent to a Gauss-Newton method solving an optimization problem, and it converges uniformly to the optimal solution. The online data can also be used repeatedly, which is highly efficient. Simulation results demonstrate its feasibility to solve the unknown nonlinear ZSG. When compared with other algorithms, it saves a significant amount of online measurement time.

A novel optimal tracking control scheme for a class of discrete-time nonlinear systems using generalised policy iteration adaptive dynamic programming algorithm

ABSTRACTIn this paper, a novel iterative adaptive dynamic programming (ADP) algorithm, called generalised policy iteration ADP algorithm, is developed to solve optimal tracking control problems for discrete-time nonlinear systems. The idea is to use two iteration procedures, including an i-iteration and a j-iteration, to obtain the iterative tracking control laws and the iterative value functions. By system transformation, we first convert the optimal tracking control problem into an optimal regulation problem. Then the generalised policy iteration ADP algorithm, which is a general idea of interacting policy and value iteration algorithms, is introduced to deal with the optimal regulation problem. The convergence and optimality properties of the generalised policy iteration algorithm are analysed. Three neural networks are used to implement the developed algorithm. Finally, simulation examples are given to illustrate the performance of the present algorithm.

ADP based optimal tracking control for a class of linear discrete-time system with multiple delays

In this paper, a new optimal tracking control scheme for a class of linear discrete-time systems with multiple delays is proposed using an iterative adaptive dynamic programming algorithm. First, the minimum delays equivalent system of the original system with multiple delays is derived based on the coordinate transformation technique, and then the corresponding Bellman equation and Q-function are presented, respectively. Then, a model-free optimal tracking controller is designed to minimize the augmented quadratic performance index based on the adaptive dynamic programming algorithm, which without requiring any knowledge of the system dynamics. Simulation results show the effectiveness of the proposed control scheme.

Analysis of the two-link manipulator in consideration of the horizontal motion about object

In this article, equations of motion of two-link-manipulator are derived in consideration of characteristics of driving source. By considering for horizontal motion about object, trajectories for saving energy are calculated by iterative dynamic programming method. And, the dynamic characteristics of two-link system controlled based on the trajectory for saving energy are analyzed theoretically. When the object moves parabolic, measurement method for the force of collision between link and object is examined by fundamental experiment.

Generalized Policy Iteration Adaptive Dynamic Programming for Discrete-Time Nonlinear Systems

This paper is concerned with a novel generalized policy iteration algorithm for solving optimal control problems for discrete-time nonlinear systems. The idea is to use an iterative adaptive dynamic programming algorithm to obtain iterative control laws which make the iterative value functions converge to the optimum. Initialized by an admissible control law, it is shown that the iterative value functions are monotonically nonincreasing and converge to the optimal solution of Hamilton-Jacobi-Bellman equation, under the assumption that a perfect function approximation is employed. The admissibility property is analyzed, which shows that any of the iterative control laws can stabilize the nonlinear system. Neural networks are utilized to implement the generalized policy iteration algorithm, by approximating the iterative value function and computing the iterative control law, respectively, to achieve approximate optimal control. Finally, numerical examples are presented to verify the effectiveness of the present generalized policy iteration algorithm.

A novel policy iteration based deterministic Q-learning for discrete-time nonlinear systems

In this chapter, a novel iterative Q-learning algorithm, called “policy iteration-based deterministic Q-learning algorithm,” is developed to solve the optimal control problems for discrete-time deterministic nonlinear systems. The idea is to use an iterative adaptive dynamic programming (ADP) technique to construct the iterative control law which optimizes the iterative Q function. When the optimal Q function is obtained, the optimal control law can be achieved by directly minimizing the optimal Q function, where the mathematical model of the system is not necessary. Convergence property is analyzed to show that the iterative Q function is monotonically nonincreasing and converges to the solution of the optimality equation. It is also proven that any of the iterative control laws is a stable control law. Neural networks are used to implement the policy iteration-based deterministic Q-learning algorithm, by approximating the iterative Q function and the iterative control law, respectively. Finally, two simulation examples are presented to illustrate the performance of the developed algorithm.

CAB-Align: A Flexible Protein Structure Alignment Method Based on the Residue-Residue Contact Area.

Proteins are flexible, and this flexibility has an essential functional role. Flexibility can be observed in loop regions, rearrangements between secondary structure elements, and conformational changes between entire domains. However, most protein structure alignment methods treat protein structures as rigid bodies. Thus, these methods fail to identify the equivalences of residue pairs in regions with flexibility. In this study, we considered that the evolutionary relationship between proteins corresponds directly to the residue–residue physical contacts rather than the three-dimensional (3D) coordinates of proteins. Thus, we developed a new protein structure alignment method, contact area-based alignment (CAB-align), which uses the residue–residue contact area to identify regions of similarity. The main purpose of CAB-align is to identify homologous relationships at the residue level between related protein structures. The CAB-align procedure comprises two main steps: First, a rigid-body alignment method based on local and global 3D structure superposition is employed to generate a sufficient number of initial alignments. Then, iterative dynamic programming is executed to find the optimal alignment. We evaluated the performance and advantages of CAB-align based on four main points: (1) agreement with the gold standard alignment, (2) alignment quality based on an evolutionary relationship without 3D coordinate superposition, (3) consistency of the multiple alignments, and (4) classification agreement with the gold standard classification. Comparisons of CAB-align with other state-of-the-art protein structure alignment methods (TM-align, FATCAT, and DaliLite) using our benchmark dataset showed that CAB-align performed robustly in obtaining high-quality alignments and generating consistent multiple alignments with high coverage and accuracy rates, and it performed extremely well when discriminating between homologous and nonhomologous pairs of proteins in both single and multi-domain comparisons. The CAB-align software is freely available to academic users as stand-alone software at http://www.pharm.kitasato-u.ac.jp/bmd/bmd/Publications.html.

Two-Step Method for Dynamic Optimization of Inequality State Constrained Systems Using Iterative Dynamic Programming

A two-step method for the solution of dynamic optimization problems with state inequality constraints using iterative dynamic programming (IDP) is presented. In the first step, a preliminary control policy along with approximate values for the start and end points of the state-constrained arc is obtained. The final solution is obtained in the second step by using the preliminary control policy as an input along with an analytical expression for the control during the constrained portion of the state trajectory. The use of flexible stage lengths in IDP helps in accurately locating the start and end points of the state-constrained arc. For the examples considered, the method results in better values of the performance index than those previously reported in the literature. It also results in a significant reduction in computation time as compared to the existing method for the solution of problems with a single active state constraint and a single control using IDP.

Multibattery Optimal Coordination Control for Home Energy Management Systems via Distributed Iterative Adaptive Dynamic Programming

In this paper, a novel distributed iterative adaptive dynamic programming (ADP) method is developed to solve the multibattery optimal coordination control problems for home energy management systems. According to system transformations, the multi-input optimal control problem is transformed into a single-input optimal control problem, where all the batteries are implemented at their worst performance. Next, based on the worst-performance optimal control law, an effective distributed iterative ADP algorithm is developed, where, in each iteration, only a single-input optimization problem is implemented. Convergence properties of the distributed iterative ADP algorithm are developed to show that the iterative performance index function converges to the optimum. Finally, numerical analysis is given to illustrate the performance of the developed algorithm.

Model-based estimation of optimal dissolved oxygen profile in Agrobacterium sp. fed-batch fermentation for improvement of curdlan production under nitrogen-limited condition

Dissolved oxygen tension (DOT) is an important factor influencing curdlan fermentative production by Agrobacterium sp. ATCC 31749. Therefore, optimization of dissolved oxygen profile is crucial to improve curdlan production. In this study, an unstructured mathematical models that consider nitrogen source as growth-limiting substrates were developed to estimate the kinetic parameters of cell growth, curdlan formation, and glucose consumption during Agrobacterium sp. ATCC 31749 fermentation at various DOTs (5–75%). The estimated kinetic parameters were used to establish models with Origin 8.0. All models of kinetic parameters were coupled with the kinetic model derived from Agrobacterium sp. ATCC 31749 fermentation to determine the effect of DOT on curdlan fermentation. The combined kinetic models satisfactorily fit the experimental data. The optimal dissolved oxygen profile was obtained using genetic algorithms coupled with iterative dynamic programming algorithms in MATLAB 10b. The optimal dissolved oxygen during curdlan production (20–120h) was approximately 52% DOT, and the maximum curdlan concentration of 48.5g/L was obtained at this DOT profile. To our knowledge, this study is the first to optimize DOT for curdlan fermentation under nitrogen-limited conditions. This report significantly contributes to the improvement of curdlan production on an industrial scale.

Nonlinear neuro-optimal tracking control via stable iterative Q-learning algorithm

This paper discusses a new policy iteration Q-learning algorithm to solve the infinite horizon optimal tracking problems for a class of discrete-time nonlinear systems. The idea is to use an iterative adaptive dynamic programming (ADP) technique to construct the iterative tracking control law which makes the system state track the desired state trajectory and simultaneously minimizes the iterative Q function. Via system transformation, the optimal tracking problem is transformed into an optimal regulation problem. The policy iteration Q-learning algorithm is then developed to obtain the optimal control law for the regulation system. Initialized by an arbitrary admissible control law, the convergence property is analyzed. It is shown that the iterative Q function is monotonically non-increasing and converges to the optimal Q function. It is proven that any of the iterative control laws can stabilize the transformed nonlinear system. Two neural networks are used to approximate the iterative Q function and compute the iterative control law, respectively, for facilitating the implementation of policy iteration Q-learning algorithm. Finally, two simulation examples are presented to illustrate the performance of the developed algorithm.

Optimal distributed synchronization control for continuous-time heterogeneous multi-agent differential graphical games

In this paper, a new optimal distributed synchronization control scheme for the consensus problem of heterogeneous multi-agent differential graphical games is developed by iterative adaptive dynamic programming (ADP). The main idea is to use iterative ADP technique to obtain the iterative control law which makes all the agents track a given dynamics and simultaneously makes the iterative value function reach the Nash equilibrium. In the developed heterogeneous multi-agent differential graphical games, the agent of each node is different from one another. The dynamics and performance index function for each node depend only on local neighborhood information. A cooperative policy iteration algorithm is presented to achieve the optimal distributed synchronization control law for the agent of each node, where the coupled Hamilton–Jacobi equations for optimal synchronization control of heterogeneous multi-agent differential games can be avoided. Convergence analysis is developed to show that the iterative value functions of heterogeneous multi-agent differential graphical games can converge to the Nash equilibrium. Two simulation examples are given to show the effectiveness of the developed optimal control scheme.

Analysis and constrained optimal impulsive control of a Holling-II type trophic system with two sources

In this paper, we study a multi-trophic prey–predator system composed by top- and intermediate-level consumers and two food sources. Biomass fluxes across trophic levels are modeled by means of Holling-II type functional responses and one of the sources is subject to periodic impulsive events consisting in biomass injections. Referring to this system, we analyze the local and global stability properties of periodic eradicated solutions and establish permanence properties for the populations in terms of relations between the model parameters and the intensity of the impulses. Secondly, we formulate an optimal control problem for this system as well as an Iterative Dynamic Programming (IDP) scheme for its solution. By means of the proposed algorithm we specify the intensity of the impulsive controls to be applied periodically in order to regulate the biomass levels along time. In our formulation, the intensity of the periodic impulsive events depends on the biomass of the upper-level consumer population, as we can often see in an agricultural framework wherein the quantity of human workforce can significantly affect the size of crop periodically collected.

Motion planning of autonomous mobile robots by iterative dynamic programming

We propose a new offline motion planning method for autonomous mobile robots. To minimize traveling time, a smooth path and a time-optimal velocity profile should be generated under kinematic and dynamic constraints. In this study, we develop an effective and practical method to generate a good solution with lower computation time. The initial path is obtained from a Voronoi diagram and spline function, and is improved by iteratively changing via-points. We apply a dynamic programming algorithm to change the via-points, and a Hermite interpolation to generate a smooth trajectory. Simulation results are presented to verify the performance of the proposed method.

Experimental analysis of the two-link manipulator in consideration of the relative motion between link and object

In this article, equations of motion of two-link manipulator are derived in consideration of characteristics of driving source. By considering for different heights of falling object, trajectories for saving energy are calculated by iterative dynamic programming method. And the dynamic characteristics of two-link system controlled based on the trajectory for saving energy are also analyzed theoretically and investigated experimentally.

Infinite horizon self-learning optimal control of nonaffine discrete-time nonlinear systems.

In this paper, a novel iterative adaptive dynamic programming (ADP)-based infinite horizon self-learning optimal control algorithm, called generalized policy iteration algorithm, is developed for nonaffine discrete-time (DT) nonlinear systems. Generalized policy iteration algorithm is a general idea of interacting policy and value iteration algorithms of ADP. The developed generalized policy iteration algorithm permits an arbitrary positive semidefinite function to initialize the algorithm, where two iteration indices are used for policy improvement and policy evaluation, respectively. It is the first time that the convergence, admissibility, and optimality properties of the generalized policy iteration algorithm for DT nonlinear systems are analyzed. Neural networks are used to implement the developed algorithm. Finally, numerical examples are presented to illustrate the performance of the developed algorithm.