Iterative Dynamic Programming Research Articles

Solving multi‐objective dynamic optimization problems with fuzzy satisfying method

AbstractThis article proposes a novel algorithm integrating iterative dynamic programming and fuzzy aggregation to solve multi‐objective optimal control problems. First, the optimal control policies involving these objectives are sequentially determined. A payoff table is then established by applying each optimal policy in series to evaluate these multiple objectives. Considering the imprecise nature of decision‐maker's judgment, these multiple objectives are viewed as fuzzy variables. Simple monotonic increasing or decreasing membership functions are then defined for degrees of satisfaction for these linguistic objective functions. The optimal control policy is finally searched by maximizing the aggregated fuzzy decision values. The proposed method is rather easy to implement. Two chemical processes, Nylon 6 batch polymerization and Penicillin G fed‐batch fermentation, are used to demonstrate that the method has a significant potential to solve real industrial problems. Copyright © 2003 John Wiley & Sons, Ltd.

Efficient Differential Evolution algorithms for multimodal optimal control problems

Many methods for solving optimal control problems, whether direct or indirect, rely upon gradient information and therefore may converge to a local optimum. Global optimisation methods like Evolutionary algorithms, overcome this problem. In this work it is investigated how well novel and easy to understand Evolutionary algorithms, referred to as Differential Evolution (DE) algorithms, and claimed to be very efficient when they are applied to solve static optimisation problems, perform on solving multimodal optimal control problems. The results show that within the class of evolutionary methods, Differential Evolution algorithms are very robust, effective and highly efficient in solving the studied class of optimal control problems. Thus, they are able of mitigating the drawback of long computation times commonly associated with Evolutionary algorithms. Furthermore, in locating the global optimum these Evolutionary algorithms present some advantages over the Iterative Dynamic Programming (IDP) algorithm, which is an alternative global optimisation approach for solving optimal control problems. At present little knowledge is available to the selection of the algorithm parameters in the DE algorithm when they are applied to solve optimal control problems. Our study provides guidelines for this selection. In contrast to the IDP algorithm the DE algorithms have only a few algorithm parameters that are easily determined such that multimodal optimal control problems are solved effectively and efficiently.

Optimization of semicontinuous emulsion polymerization reactions by IDP procedure with variable time intervals

Optimization of a semicontinuous emulsion polymerization reaction is carried out using a mathematical model derived from first principles to represent the process. Iterative Dynamic Programming (IDP), being a straightforward procedure that allows the implementation of optimization constraints, is an attractive tool to optimize highly complex and nonlinear equation systems such as those that are used to represent the emulsion polymerization processes. In order to minimize the reaction time of a semicontinuous reaction, time intervals with varying lengths were used allowing the establishment of the optimal operation conditions where the final time is not specified. The optimization procedure is applied to two emulsion polymerization systems: (a) butyl acrylate/vinyl acetate copolymerization with controlled composition; (b) vinyl acetate homopolymerization with specified molecular weight. Results show that significant reductions of reaction time are possible and that their extent depends on the reactor's heat removal capacity and on the formulated constraints.

Demand subscription services - an iterative dynamic programming for the substation suffering from capacity shortage

The iterative dynamic programming (IDP) is presented in this paper to carry out the demand subscription services (DSS) for reducing the peak load of the substation, which is suffering from capacity shortage. The optimal or near optimal schedule of the interruptible load can be obtained by the enumerative search structure of the algorithm. Besides, the shortcoming of the enumerative approach, in which enormous memory is needed, can also be circumvented in light of the iterative procedure. Moreover, for avoiding the influence of the uncertainties of the practical load on the results of the IDP, the heuristic inference rules (HIR) are introduced in the paper. The approach cannot only adjust the schedule of the interruptible load to enhance the robustness for the uncertainties but also possess the ability of real-time operation. Three cases of demand subscription services are implemented through the proposed algorithms. The results show that the interruptible load scheduling can reduce the system load effectively and the load capacity reduced by the DSS follows close on the trajectory of the peak load.

重力下でＰＴＰ制御される２自由度Ｄ．Ｄ．マニピュレータの省エネルギー軌道

A performance criterion for saving energy is defined in consideration of energy consumption of driving source. It is necessary for saving energy of direct-drive manipulator to make the mechanism lighter. In this paper, equations of motion of a closed-type manipulator, whose mechanism can easily be made lighter, are derived in consideration of characteristics of the DC servomotors. When the manipulator is operated in a vertical plane, the system is highly non-linear due to gravity and an analytical solution can not be found. Then, a numerical approach is necessary. Considering the final condition about angular displacement and angular velocity, trajectories of velocity for saving energy are calculated by iterative dynamic programming. Initial searching region of dynamic programming is a shape which is surrounded by two sine-wave translated in parallel. This proposed searching region is used as a initial range of the iteration method, and the region is shifted along the axis of coordinate of angular displacement to minimize the energy consumption of the motor. The dynamic characteristics of manipulator controlled based on above mentioned trajectory are analyzed theoretically and investigated experimentally.

Optimal reaction conditions for the minimization of energy consumption and by-product formation in a poly(ethylene terephthalate) process

We determined the optimal reaction conditions to minimize the energy cost and the quantities of by-products for a poly(ethylene terephthalate) process by using the iterative dynamic programming (IDP) algorithm. Here, we employed a sequence of three reactor models: the semibatch transesterification reactor model, the semibatch prepolymerization reactor model, and the rotating-disc-type polycondensation reactor model. We selectively chose or developed the reactor models by incorporating experimentally verified kinetic models reported in the literature. We established the model for the entire reactor system by connecting the three reactor models in series and by resolving some joint problems arising when different types of reactor models were interconnected. On the basis of the simulation results of the reactor system, we scrutinized the cause and effect between the reaction conditions and the final quality of the polymer product. Here, we set up the optimization strategy by using IDP on the basis of the integrated reactor model, and the process variables with significant influence on the properties of polymer were selected as control variables with the help of a simulation study. With this method, we could refine the reaction conditions at the end of each iteration step by contracting the spectra of control regions, and the iteration process finally stopped when the profile of the optimal trajectory converged. We also took the constraints on the control variables into account to guarantee polymer quality and to suppress side reactions. Constituting six different strategies by setting weighting vectors differently, we examined the differences in optimal trajectories, the trend of optimality, and the quality of the final polymer product. For each of the strategies, we conducted the optimization to examine whether the number-average degree of polymerization approached the desired value. © 2002 Wiley Periodicals, Inc. J Appl Polym Sci 86: 993–1008, 2002

Closed-Loop Composition and Molecular Weight Control of a Copolymer Latex Using Near-Infrared Spectroscopy

A control algorithm is designed and implemented experimentally for the simultaneous closed-loop control of the composition and number-average molecular weight of a copolymer latex. The developed control algorithm is based on a predictive control strategy and uses an iterative dynamic programming (IDP) algorithm for process optimization and design of the optimum dynamic trajectories at each sampling time. The control of semibatch methyl methacrylate (MMA) and butyl acrylate (BuA) emulsion copolymerizations is used to illustrate the approach. Monomer compositions and polymer holdup are measured in-line and in situ with the help of near-infrared spectroscopy (NIRS), while average molecular weights are inferred with the help of a process model. The manipulated variables are the flow rates of three distinct feed streams, which contain known concentrations of both monomer species and of a chain-transfer agent. The results obtained when different process disturbances are introduced, such as feed failure and addition of unknown amounts of inhibitor to the reactor charge, show unequivocally that NIRS can be used successfully for the in-line and in situ simultaneous control of copolymer composition and average molecular weights in emulsion copolymerizations.

Dynamic optimization of non-linear emulsion copolymerization systems: Open-loop control of composition and molecular weight distribution

Time optimal monomer and chain-transfer agent feed profiles were computed and implemented experimentally for the simultaneous control of copolymer composition and molecular weight distribution in non-linear emulsion copolymerization systems. Iterative dynamic programming was used for the off-line calculation of the optimal feed policies. This approach can deal with constrained optimization of systems described by complex mathematical models, as those needed for the emulsion copolymerization kinetics, especially when the computation of the whole molecular weight distribution is included. The proposed approach was applied to the semibatch methylmethacrylate (MMA)/ n-butylacrylate ( n-BA) emulsion copolymerization, using n-dodecanethiol as chain-transfer agent, and allowed the production of copolymers with constant composition and with well-defined molecular weight distributions.

Time Optimal Control of a Binary Distillation Column

AbstractTime optimal control problems involving high dimensional systems are generally very difficult to solve. Instead of using suboptimal control based on reduced system, iterative dynamic programming is used directly to establish the time optimal control policy of a binary distillation column described by 11 ordinary differential equations. The resulting optimal control policy that is bang‐bang in nature was obtained very rapidly. The minimum time of 2236 seconds obtained here is substantially better than 4120 seconds obtained by a suboptimal control policy. Simulations show that the procedure may be used for on‐line optimal control of the distillation column, since the optimal control policy can be established very fast on a personal computer.

NEURAL NETWORK BASED ON-LINE RE-OPTIMISATION OF FED-BATCH PROCESSES USING ITERATIVE DYNAMIC PROGRAMMING FOR DISCRETE-TIME SYSTEMS

Optimisation of fed-batch processes can be described as a constrained nonlinear end-point dynamic optimisation problem. Although iterative dynamic programming (IDP) is feasible, it is usually very time-consuming and very difficult to apply to on-line optimisation because of solving the non-linear differential-algebraic equations of the process model in each iteration. The replacement of a rigorous mechanistic model by an equivalent neural network (NN) model takes the advantage of high speed processing, since simulation with a NN model involves only a few non-iterative algebraic calculations. To use IDP algorithm for NN model based on-line re-optimisation, a modified algorithm is proposed and is called as iterative dynamic programming for discrete-time system (IDP/DTS). The novel IDP/DTS algorithm can obtain a reduction of many times in computational time compared to the conventional IDP algorithm. In this paper, an effective optimisation and control scheme for on-line re-optimisation of fed-batch processes is proposed based on NN models and the novel IDP/DTS algorithm. The proposed scheme is illustrated using simulation studies of an ethanol fermentation process.

Further developments in the new approach to boundary condition iteration in optimal control

AbstractIn solving the boundary value problem resulting from the use of Pontryagin's maximum principle, a transformation matrix is used to relate the sensitivity of the final state to the initial state. This avoids the need to solve the (n × n) differential equation to give the transition matrix, and yields very rapid convergence to the optimum. To ensure convergence, iterative dynamic programming (IDP) is used for a number of passes to yield good starting conditions for this boundary condition iteration procedure. Clipping technique is used to handle constraints on control. Five optimal control problems are used to illustrate and to test the procedure.

Numerical solution of dynamic optimization problems with flexible inequality constraints by iterative dynamic programming

A solution strategy for optimizing the dynamic systems with flexible inequality constraints is proposed. To apply fuzzy inference in solution, the flexible portion in the problem is treated as fuzzy constraints. After functional values are bounded in a region, the objective function of this problem can also be fuzzified easily. When the problem is formulated as a fuzzy dynamic optimization problem, the iterative dynamic programming integrated with fuzzy inference is adopted to find the solution. Two examples are employed, demonstrating the facility of the proposed algorithm.

On solving optimal control problems with free initial condition using iterative dynamic programming

AbstractFor solving optimal control problems where the initial conditions of some of the state variables are not specified, a procedure based on iterative dynamic programming (IDP) is presented. In this procedure, the free initial conditions are taken to be additional control variables for the first time stage only; then the search for the optimal initial conditions and also the optimal control policy is carried out simultaneously using IDP. The procedure is straightforward, and as illustrated with two nonlinear optimal control problems, for each case the optimum performance index is readily obtained.

Dynamic optimization of semicontinuous emulsion copolymerization reactions: composition and molecular weight distribution

Iterative dynamic programming (IDP) is used to compute optimal monomer and chain transfer agent feed profiles to produce polymer with specified copolymer composition and molecular weight distribution. The approach used allows the implementation of constrained optimization procedures for systems described by complex mathematical models, as those needed for proper description of emulsion copolymerization reactors, especially when the computation of the whole molecular weight distribution is desired. The proposed approach is applied to the semicontinuous methyl methacrylate (MMA)/butyl acrylate (BuA) emulsion copolymerization in stirred tank reactors, using dodecanethiol as chain transfer agent (CTA). It is shown that this technique allows the effective computation of feed policies for the production of copolymers with constant composition and well-defined molecular weight distributions.

Bang‐Bang solution of nonlinear time‐optimal control problems using a semi‐exhaustivesearch

AbstractAt times, the objective is to seek a bang‐bang control policy for nonlinear time‐optimal control problems. The usefulness of iterative dynamic programming (IDP) has been shown in the literature for solving such problems. However, the convergence to the optimal solution has been obtained from about 50% of the guessed values near the optimum. In this paper, we present a semiexhaustive search method for seeking such solutions and a comparison is made with the IDP. The results show that the convergence can be obtained from a significantly higher number of guessed values chosen over a much wider region around the optimum.

Receding horizon iterative dynamic programming with discrete time models

This contribution proposes a modified version of the Iterative Dynamic Programming (IDP) method. Two main differences to the original method are introduced. The new algorithm deals with discrete-time input–output models compared to continuous-time state–space models described by a set of ODE/DAE used in the original method. The main purpose of these modifications is to reduce computational load of the original method, estimate the process models more easily, and to enable its use on-line in receding horizon predictive control framework.

Bounded-parameter Markov decision processes

In this paper, we introduce the notion of a bounded-parameter Markov decision process (BMDP) as a generalization of the familiar exact MDP. A bounded-parameter MDP is a set of exact MDPs specified by giving upper and lower bounds on transition probabilities and rewards (all the MDPs in the set share the same state and action space). BMDPs form an efficiently solvable special case of the already known class of MDPs with imprecise parameters (MDPIPs). Bounded-parameter MDPs can be used to represent variation or uncertainty concerning the parameters of sequential decision problems in cases where no prior probabilities on the parameter values are available. Bounded-parameter MDPs can also be used in aggregation schemes to represent the variation in the transition probabilities for different base states aggregated together in the same aggregate state. We introduce interval value functions as a natural extension of traditional value functions. An interval value function assigns a closed real interval to each state, representing the assertion that the value of that state falls within that interval. An interval value function can be used to bound the performance of a policy over the set of exact MDPs associated with a given bounded-parameter MDP. We describe an iterative dynamic programming algorithm called interval policy evaluation that computes an interval value function for a given BMDP and specified policy. Interval policy evaluation on a policy π computes the most restrictive interval value function that is sound, i.e., that bounds the value function for π in every exact MDP in the set defined by the bounded-parameter MDP. We define optimistic and pessimistic criteria for optimality, and provide a variant of value iteration (Bellman, 1957) that we call interval value iteration that computes policies for a BMDP that are optimal with respect to these criteria. We show that each algorithm we present converges to the desired values in a polynomial number of iterations given a fixed discount factor.

Suboptimal Robust Control of Fast Electro-Magnetic Actuators

Electro-magnetic actuators can provide engine valves with variable valve timing to best suit the prevalent engine speed and load. All electro-magnetic valve actuators, however, suffer at least to some degree from high seating velocities, which result in un desire able audible noise and early fatigue of the valve and valve seat. This problem can be solved by closed-loop control of the valve motion. The controller has to be robust, because of a wide range of pressures in the combustion chamber during valve opening. Additionally the required electrical energy has to be minimal. Therefore the proposed controller is based on variable structure control. The energy-optimal sliding-surfaces are computed by Iterative Dynamic Programming technique. The control signals are calculated by a first-order nonlinear observer.

Iterative dynamic programming

The query optimizer is one of the most important components of a database system. Most commercial query optimizers today are based on a dynamic-programming algorithm, as proposed in Selinger et al. [1979]. While this algorithm produces good optimization results (i.e, good plans), its high complexity can be prohibitive if complex queries need to be processed, new query execution techniques need to be integrated, or in certain programming environments (e.g., distributed database systems). In this paper, we present and thoroughly evaluate a new class of query optimization algorithms that are based on a principle that we calliterative dynamic programming, or IDP for short. IDP has several important advantages: First, IDP-algorithms produce the best plans of all known algorithms in situations in which dynamic programming is not viable because of its high complexity. Second, some IDP variants are adaptive and produce as good plans as dynamic programming if dynamic programming is viable and as good-as possible plans if dynamic programming turns out to be not viable. Three, all IDP-algorithms can very easily be integrated into an existing optimizer which is based on dynamic programming.

Dynamic modeling and optimal fed-batch feeding strategies for a two-phase partitioning bioreactor.

A dynamic model for the degradation of phenol in a two-phase partitioning bioreactor has been developed based on mechanistic balances around the bioreactor. The key process characteristics including substrate transfer between the organic and aqueous phases, substrate inhibition, oxygen limitation, and cell entrainment were incorporated into the model. The model predictions were validated against existing experimental data obtained for a 2-L bioreactor, and good correlation was observed for the time frames of the simulations, as well as for trends in cell and substrate concentrations. Optimal fed-batch, phenol feeding strategies were then developed based on two approaches: (1) maximization of phenol consumption in a fixed time interval and (2) consumption of a fixed amount of phenol in minimal time. The optimal feeding policies, determined using the Iterative Dynamic Programming algorithm, provided substantial improvements in the amount of phenol consumed when compared to a typical experimental heuristic approach. For example, 45.73 g of phenol was predicted to be consumed in 50 h (not including lag phase) using the optimal feeding profile compared to 10.26 g of phenol consumed in the simulated experimental approach. Oxygen limitation was predicted to be a recurring operational challenge in the partitioning bioreactor, and had a strong impact on the optimization results.