Iterative Dynamic Programming Research Articles

An iterative -optimal control scheme for a class of discrete-time nonlinear systems with unfixed initial state

In this paper, a finite horizon iterative adaptive dynamic programming (ADP) algorithm is proposed to solve the optimal control problem for a class of discrete-time nonlinear systems with unfixed initial state. A new ϵ-optimal control algorithm based on the iterative ADP approach is proposed that makes the performance index function iteratively converge to the greatest lower bound of all performance indices within an error ϵ in finite time. The convergence analysis of the proposed ADP algorithm in terms of performance index function and control policy is conducted. The optimal number of control steps can also be obtained by the proposed ϵ-optimal control algorithm for the unfixed initial state. Neural networks are used to approximate the performance index function, and compute the optimal control policy, respectively, for facilitating the implementation of the ϵ-optimal control algorithm. Finally, a simulation example is given to show the effectiveness of the proposed method.

The Canadian minimum duration truck driver scheduling problem

In Canada transport companies must ensure that truck drivers can comply with Canadian Commercial Vehicle Drivers Hours of Service Regulations. Canadian regulations comprise the provisions found in US hours of service regulations as well as additional constraints on the maximum amount of driving and the minimum amount of off-duty time on each day. This paper presents a mixed integer programming formulation and an iterative dynamic programming approach for minimising the duration of truck driver schedules complying with Canadian hours of service regulations. Computational experiments show that schedule durations can be significantly reduced compared with a previously presented approach which only focuses on feasibility.

Optimization of Bioethanol Ethanol Production in Fed-batch Fermentation

This study focuses on improving the productivity of a fed-batch ethanol fermentation process by developing and implementing in real time, an optimum feeding policy. A constraint-based stoichiometric model is developed using the systems biology approach, which can quantitatively predict the cellular behavior of the E coli strain, KO11. However, the predictions from this model are accurate under low concentrations of substrate. In order to extend the usage of the model to higher substrate concentrations, we modified this model by introducing changes in glucose utilization rate to restrict the metabolic capacity of the cell. Using the proposed model, the fed-batch optimization problem becomes a constrained optimization problem, and a modified Iterative Dynamic Programming (IDP) algorithm was developed with an adaptive-stage updating methodology, which was applied to solve the global optimization problem. Experiments were carried out based on the optimal feed profile generated using the modified stoichiometric model and the Iterative Dynamic Programming algorithm. Approximately 90% of the theoretical ethanol yield was obtained; the fermentation consumed a total of 521.5 g of glucose and produced 237.5 g of ethanol.

Optimal control of polymer flooding based on mixed-integer iterative dynamic programming

Polymer flooding is one of the most important technologies for enhanced oil recovery. In this article, a mixed-integer optimal control model of distributed parameter systems (DPS) for the injection strategies is established, which involves the performance index as maximum of the profit, the governing equations as the fluid flow equations of polymer flooding and some inequalities constraints, such as polymer concentration and injection amount limitation. The control variables are the volume size, the injection concentration of each slug and the terminal flooding time. For the constant injection rate, the slug size is determined by the integer time stage length, and thus the integer variables are introduced in the DPS. To cope with the optimal control problem (OCP) of this DPS, a mixed-integer iterative dynamic programming incorporating a special truncation procedure to handle integer restrictions on stage lengths is proposed. First, the OCP with variable time stage lengths is transformed into a fixed time stage problem by introducing a normalised time variable. Then, the optimisation procedure is carried out at each stage and preceded backwards in a systematic way. Finally, the numerical results of an example illustrate the effectiveness of the proposed method.

Finite-horizon neuro-optimal tracking control for a class of discrete-time nonlinear systems using adaptive dynamic programming approach

In this paper, a finite-horizon neuro-optimal tracking control strategy for a class of discrete-time nonlinear systems is proposed. Through system transformation, the optimal tracking problem is converted into designing a finite-horizon optimal regulator for the tracking error dynamics. Then, with convergence analysis in terms of cost function and control law, the iterative adaptive dynamic programming (ADP) algorithm via heuristic dynamic programming (HDP) technique is introduced to obtain the finite-horizon optimal tracking controller which makes the cost function close to its optimal value within an ɛ - error bound. Three neural networks are used as parametric structures to implement the algorithm, which aims at approximating the cost function, the control law, and the error dynamics, respectively. Two simulation examples are included to complement the theoretical discussions.

Analysis of a manipulator in relation to collision between a link and an object

In this article, equations of motion of a manipulator are derived after consideration of the characteristics of the driving source. By considering a collision between a link and an object, and considering the active motion to absorb the kinetic energy of the object, the trajectories for saving energy are calculated by the iterative dynamic programming (IDP) method. The dynamic characteristics of manipulator control based on the trajectory for saving energy are also analyzed theoretically and investigated experimentally.

The neural paradigm for complex systems: new algorithms and applications

This special issue of Neural Computing and Applications includes 14 original articles, which are extended versions of selected papers from the Eighth International Symposium on Neural Networks (ISNN 2011). ISNN 2011 provided a high-level international forum for scientists, engineers, and educators to present the stateof-the-art of neural networks research and its applications in diverse fields. The symposium featured plenary speeches given by worldwide renowned scholars, regular sessions with broad coverage, and some special sessions focusing on interesting topics for the neural networks scientific community. Based on the recommendation of symposium organizers and reviewers, a number of authors were invited to resubmit an extended version of their conference papers for this special issue of Neural Computing and Applications. All these journal articles went through the same rigorous review procedure by at least three independent experts before being accepted for publication. This special issue focuses on new hot topics in the field of Neural Networks for Complex Systems including new algorithms and applications. The selected 14 articles can be indeed divided into two main groups. The first group consists of five theoretical papers dealing with some emerging and important issues regarding complex neural architectures (D. Gong et al. and Z. Wang et al.) and advanced adaptive dynamic programming schemes for optimal control (D. Wang et al., R. Song and H. Zhang, C. Song and J. Ye). The second group contains nine application papers focusing on various complex systems real-world applications and involving new computational intelligence approaches: they include power system management (T. Lan et al., D. Liu and T. Huang, Z. Liu et al., D. Qian et al.), time series prediction (H. He et al.), acoustics and artificial vision (X. Song et al. and J. Qu et al.), path-planning (C. Xiong et al.), and position tracking (H. Dong et al.). The special issue starts with the first group of papers and with the contribution by D. Gong et al., dealing with the synchronization problem in general complex networks: A new criterion is proposed on purpose resulting in a simpler and less computational demanding method for synchronization analysis, especially if compared to the approaches appeared in the literature so far. Then, in the work by Z. Wang et al., the global asymptotic stability problem for a class of recurrent neural networks with multi-time scale is studied. Some novel stability criteria, also consistent from a biological viewpoint, are proposed on the basis of linear matrix inequality technique for the concerned neural network, which sufficiently consider the inhibitory actions in the different memories. Some relevant contributions in the adaptive dynamic programming field also belong to this first group. D. Wang et al. propose a novel neural network-based iterative adaptive dynamic programming (ADP) algorithm for S. Squartini (&) Universita Politecnica delle Marche, Via Brecce Bianche 31, 60131 Ancona, Italy e-mail: s.squartini@univpm.it

A neural-network-based iterative GDHP approach for solving a class of nonlinear optimal control problems with control constraints

In this paper, a novel neural-network-based iterative adaptive dynamic programming (ADP) algorithm is proposed. It aims at solving the optimal control problem of a class of nonlinear discrete-time systems with control constraints. By introducing a generalized nonquadratic functional, the iterative ADP algorithm through globalized dual heuristic programming technique is developed to design optimal controller with convergence analysis. Three neural networks are constructed as parametric structures to facilitate the implementation of the iterative algorithm. They are used for approximating at each iteration the cost function, the optimal control law, and the controlled nonlinear discrete-time system, respectively. A simulation example is also provided to verify the effectiveness of the control scheme in solving the constrained optimal control problem.

Using IDP Algorithm for Solving OCP Governed by aElectric Power Generated

Iterative Dynamic method was introduced by Luss for solving OCP (Optimal Control Problems) in his book. In this article, by doing a deformation, the problems are converted into a measure one with some positive theoretical coefficient and by extending space and then applying discretization scheme, the optimal pair of trajectory and control are determined as a finite linear programming. We apply this new method for solving an OCP governed by a electric power generated with initial and boundary conditions and integral criterion ,and we will using Iterative dynamic method as iteration for Solving OCP model Index Terms—Iterative Dynamic Programming (IDP), optimal control (OCP), algorithm, simulation, electric power generated system.

An iterative adaptive dynamic programming method for solving a class of nonlinear zero-sum differential games

In this paper, a new iterative adaptive dynamic programming (ADP) method is proposed to solve a class of continuous-time nonlinear two-person zero-sum differential games. The idea is to use the ADP technique to obtain the optimal control pair iteratively which makes the performance index function reach the saddle point of the zero-sum differential games. If the saddle point does not exist, the mixed optimal control pair is obtained to make the performance index function reach the mixed optimum. Stability analysis of the nonlinear systems is presented and the convergence property of the performance index function is also proved. Two simulation examples are given to illustrate the performance of the proposed method.

Analysis of a manipulator in consideration of the relative motion between a tray and an object

In this article, equations of motion of a manipulator, whose mechanism has a tray installed with a passive revolute joint, are derived after consideration of the characteristics of the driving source. Considering the relative motion between the tray and the object, the trajectories of the velocity for saving energy are calculated by iterative dynamic programming. Also, the dynamic characteristics of manipulator control based on the trajectory for saving energy are analyzed theoretically and investigated experimentally.

Optimal control laws for time-delay systems with saturating actuators based on heuristic dynamic programming

In this paper a new iterative heuristic dynamic programming (HDP) algorithm is proposed to solve the optimal control problem for a class of nonlinear discrete time-delay systems with saturating actuators. Considering the saturation nonlinearity in the actuators, a nonquadratic performance index function is introduced. In the meantime, a state modification is used to deal with the obstacle induced by time delays. In the new iterative HDP algorithm the local and global optimization searching processes are developed to solve the optimal feedback control problem with convergence analysis. In the presented iterative HDP algorithm, two neural networks are used to facilitate the implementation of the iterative algorithm. Finally, two simulation examples are given to demonstrate the convergence and feasibility of the proposed optimal control scheme.

Time-Optimal Control of Dividing-Wall Distillation Columns

Time-optimal control of startup traditional distillation columns by iterative programming proposed by Woinaroschy for ideal [Ind. Eng. Chem. Res. 2008, 47, 4158] and nonideal mixtures [Ind. Eng. Chem. Res. 2009, 48, 3873] is extended to the case of dividing-wall distillation columns. The minimization of distillation startup time is performed by iterative dynamic programming employing randomly chosen candidates for admissible control. The control variables are the reflux ratio, the reboiler heat duty, and the side-draw flow rate. The dynamic distillation model proposed by the author in the previous papers is applied. Two illustrative case studies for the separation in a dividing-wall column with sieve trays and lateral downcomers are presented as follows: the separation of an ideal benzene−toluene−ethylbenzene ternary mixture and the separation of a nonideal methanol−ethanol−1-propanol mixture. In another case study, a conventional two-column system is presented in comparison to the dividing-wall column. As in the cases of traditional distillation columns, the startup time decrease and the corresponding reboiler energy savings are significant for each of the control variables.

IAlign: a method for the structural comparison of protein–protein interfaces

Protein-protein interactions play an essential role in many cellular processes. The rapid accumulation of protein-protein complex structures provides an unprecedented opportunity for comparative studies of protein-protein interactions. To facilitate such studies, it is necessary to develop an accurate and efficient computational algorithm for the comparison of protein-protein interaction modes. While there are many structural comparison approaches developed for individual proteins, very few methods are available for protein-protein complexes. We present a novel interface alignment method, iAlign, for the structural alignment of protein-protein interfaces. New scoring schemes for measuring interface similarity are introduced, and an iterative dynamic programming algorithm is implemented. We find that the similarity scores follow extreme value distributions. Using statistical models, we empirically estimate their statistical significance, which is in good agreement with manual classifications by human experts. Large-scale tests of iAlign were conducted on both artificial docking models and experimental structures. In a benchmark test on 1517 dimers, iAlign successfully detects biologically related, structurally similar protein-protein interfaces at a coverage percentage of 90% and an error per query of 0.05. When compared against previously published methods, iAlign is substantially more accurate and efficient. The iAlign software package is freely available at http://cssb.biology.gatech.edu/iAlign.

Investigation on Energetic Optimization Problems of Stochastic Thermodynamics with Iterative Dynamic Programming

The energetic optimization problem, e.g., searching for the optimal switch- ing protocol of certain system parameters to minimize the input work, has been extensively studied by stochastic thermodynamics. In current work, we study this problem numerically with iterative dynamic programming. The model systems under investigation are toy actuators consisting of spring-linked beads with loading force imposed on both ending beads. For the simplest case, i.e., a one-spring actuator driven by tuning the stiffness of the spring, we compare the optimal control protocol of the stiffness for both the overdamped and the underdamped situations, and discuss how inertial effects alter the irreversibility of the driven process and thus modify the optimal protocol. Then, we study the systems with multiple degrees of freedom by constructing oligomer actuators, in which the harmonic interaction between the two ending beads is tuned externally. With the same rated output work, actuators of different constructions demand different minimal input work, reflecting the influence of the internal degrees of freedom on the performance of the actuators.

An Optimal Control Scheme for a Class of Discrete-time Nonlinear Systems with Time Delays Using Adaptive Dynamic Programming

Abstract In this paper, an optimal control scheme for a class of nonlinear systems with time delays in both state and control variables with respect to a quadratic performance index function is proposed using a new iterative adaptive dynamic programming (ADP) algorithm. By introducing a delay matrix function, the explicit expression of the optimal control is obtained using the dynamic programming theory and the optimal control can iteratively be obtained using the adaptive critic technique. Convergence analysis is presented to prove that the performance index function can reach the optimum by the proposed method. Neural networks are used to approximate the performance index function, compute the optimal control policy, solve delay matrix function, and model the nonlinear system, respectively, for facilitating the implementation of the iterative ADP algorithm. Two examples are given to demonstrate the validity of the proposed optimal control scheme.

USING ORDER ADMISSION CONTROL TO MAXIMIZE REVENUE UNDER CAPACITY UTILIZATION REQUIREMENTS IN MTO B2B INDUSTRIES

An order admission model is developed to introduce revenue management into MTO B2B in- dustries. This solution is designed to resolve a common order management dilemma faced by manufacturers with severe seasonal demands. Considering the requirement that manufacturing industries be risk averse, the objective is to maximize the expected total revenue under a targeted capacity utilization constraint. The deterministic admission problem and its auxiliary model are presented and solved through an iterative dynamic programming algorithm. Through this algorithm, a reward-threshold order admission policy can be established at the beginning of the planning horizon, helping companies to achieve a balance between profitability and stability.

Neural-Network-Based Near-Optimal Control for a Class of Discrete-Time Affine Nonlinear Systems With Control Constraints

In this paper, the near-optimal control problem for a class of nonlinear discrete-time systems with control constraints is solved by iterative adaptive dynamic programming algorithm. First, a novel nonquadratic performance functional is introduced to overcome the control constraints, and then an iterative adaptive dynamic programming algorithm is developed to solve the optimal feedback control problem of the original constrained system with convergence analysis. In the present control scheme, there are three neural networks used as parametric structures for facilitating the implementation of the iterative algorithm. Two examples are given to demonstrate the convergence and feasibility of the proposed optimal control scheme.

Handling Inequality Constraints in Optimal Control by Problem Reformulation

Establishment of optimal control for systems, where constraints involve both control and state, is very difficult. In some problems the difficulty is reduced significantly by transforming the optimal control problem. For illustration, the optimal control of a nonisothermal fed-batch reactor with heat removal constraint is considered. Although there are only two control variables, the feed rate and the temperature, the heat removal rate constraint makes the optimal control problem very difficult. To parametrize the optimal control problem, the time interval is divided into P time stages of variable length, and piecewise constant control is used at each time stage. Establishment of the optimal control policy is very challenging owing to the low sensitivity, the heat removal constraint, and the need for a large number of time stages for adequate approximation. However, by reformulating the optimal control problem where heat generation, rather than temperature, is used as a control variable, we are able to get greater accuracy with a smaller number of time stages. The optimal control policy is established with iterative dynamic programming and checked with LJ-optimization procedure.

Time-Optimal Control of Startup Distillation of Nonideal Mixtures

In this work, the time-optimal control of startup distillation columns by iterative programming proposed by Woinaroschy [Ind. Eng. Chem. Res. 2008, 47, 4158.] is extended to the case of nonideal mixtures. The minimization of distillation startup time is performed by iterative dynamic programming employing randomly chosen candidates for admissible control. The control variables are the reflux ratio and/or the reboiler heat duty. The dynamic distillation model proposed by the author in the mentioned work is applied with a preliminary computation of suitable correlation relations for equilibrium constants. An illustrative application for separation of a propene-propane mixture at plant scale is presented. Because of the small relative volatility of this mixture and respective high reflux ratio, the decrease of the reboiler heat consumption corresponding to the minimization of the startup time is significant.