Neuro-dynamic Programming Approach Research Articles

Zero-sum differential game guidance law for missile interception engagement via neuro-dynamic programming

In this paper, for solving the nonlinear control problem of the missile intercepting the maneuvering target, a novel nonlinear zero-sum differential game guidance law is proposed via the neuro-dynamic programming approach. First, the continuous-time nonlinear differential game problem is transformed into solving the nonlinear Hamilton–Jacobi–Isaacs (HJI) equation. Then, a critic neural network is designed to solve the corresponding nonlinear HJI equation. An adaptive weight tuning law for the critic weights is proposed, where an additional term is added to ensure the stability of the closed-loop nonlinear system. Furthermore, the uniform ultimate boundedness of the closed-loop system and the critic NN weights estimation error are proved with the Lyapunov approach. Finally, some simulation results are presented to demonstrate the effectiveness of the proposed differential game guidance law for nonlinear interception.

Neuro-dynamic programming for optimal control of macroscopic fundamental diagram systems

The macroscopic fundamental diagram (MFD) can effectively reduce the spatial dimension involved in dynamic optimization of traffic performance for large-scale networks. Solving the Hamilton-Jacobi-Bellman (HJB) equation takes center stage in yielding solutions to the optimal control problem. At the core of solving the HJB equation is the value function that represents choosing a sequence of actions to optimize the system performance. However, this problem generally becomes intractable for possible discontinuities in the solution and the curse of dimensionality for systems with all but modest dimension. To address these challenges, a neural network is used to approximate the value function to obtain the optimal controls through policy iteration. Furthermore, a saturated operator is embedded in the neural network approximator to handle the difficulty caused by the control and state constraints. This policy iteration can be implemented as an iterative data-driven technique that integrates with the model-based optimal design based on real-time observations. Numerical experiments are conducted to show that the neuro-dynamic programming approach can achieve optimization goals while stabilizing the system by regulating the traffic state to the desired uncongested equilibrium.

An Online Energy Management Control for Hybrid Electric Vehicles Based on Neuro-Dynamic Programming

Hybrid electric vehicles are a compromise between traditional vehicles and pure electric vehicles and can be part of the solution to the energy shortage problem. Energy management strategies (EMSs) are highly related to energy utilization in HEVs’ fuel economy. In this research, we have employed a neuro-dynamic programming (NDP) method to simultaneously optimize fuel economy and battery state of charge (SOC). In this NDP method, the critic network is a multi-resolution wavelet neural network based on the Meyer wavelet function, and the action network is a conventional wavelet neural network based on the Morlet function. The weights and parameters of both networks are obtained by an algorithm of backpropagation type. The NDP-based EMS has been applied to a parallel HEV and compared with a previously reported NDP EMS and a stochastic dynamic programing-based method. Simulation results under ADVISOR2002 have shown that the proposed NDP approach achieves better performance than both the methods. These indicate that the proposed NDP EMS, and the CWNN and MRWNN, are effective in approximating a nonlinear system.

A neuro-dynamic programming approach to the optimal stand management problem

Some ideas of neuro-dynamic programming (NDP) are illustrated by considering the problem of optimally managing a forest stand under uncertainty. Because reasonable growth models require state information such as height (or age), basal area, and stand diameter, as well as an indicator variable for treatments that have been performed on the stand, they can easily lead to very large state spaces that include continuous variables. Realistic stand management policies include silvicultural options such as pre-commercial and commercial thinning as well as post-harvest treatments. We are interested in problems that are stochastic in their basic growth dynamics, in market prices, and in disturbances, ranging from insects to fire to hurricanes. NDP algorithms are appropriate for problems with large dimensions that may lack a simple model of dynamics and stochastic processes. This paper looks at applying these ideas in the context of a multispecies model. Results show that policies obtained using NDP are optimal within a 95% confidence interval or better. The set of states and controls incorporated into our NDP model allows us to develop optimal policies with a level of detail not typically seen in the forestry literature.

Neurodynamic Programming Approach for the PID Controller Adaptation

Abstract In this paper it is proposed a novel procedure for adaptive PID controller tuning applying Approximate Dynamic Programming. This procedure approximately solves the Bellman Equation for an optimum control design using an Actor-Critic scheme. An Artificial Neural Network is employed as a Critic to estimate the gradient of cost-to-go function. The PID Controller works as the Actor. This controller is commonly employed in a large number of industrial process. The method is applicable to optimal tuning in nonlinear control problems. Examples illustrate the effectiveness of the design scheme.

A Neuro-Dynamic Programming approach to synthesize optimal dispatching rules in logistics

AbstractA dynamic model of a logistic node of a transportation network is presented and a performance index comprising transportation and inventory costs is considered. The optimization problem is solved considering a class of policies called full capacity. The derivation of optimal dispatching rules by integer programming or dynamic programming is usually intractable in realistic scenarios. For this reason, a neuro-dynamic programming (NDP) approach is pursued. Several simulation results are discussed, in particular for what concerns the role of transportation costs in the characterization of the optimal policies.

Minimising total cost with regular and emergency outsourcing sources: a neuro-dynamic programming approach

This study models and explains the business situation of an organisation which has regular and emergency outsourcing sources and where decisions have to be made at the beginning of every period regarding how much to order from these sources, so as to balance between different cost components of current and future periods. Previous works in this area have tried to attack this problem with dynamic programming. In this project, neuro-dynamic programming has been applied, and the reasons for doing so have been clearly stated. This model not only derives policies in order to minimise the expected total discounted cost over a period of time, but also enables the system to learn to make such decisions, and to improve its actions by using reinforcement learning. The performance of the present work has been measured quantitatively and has been compared with the models stated in literature. This study will be very useful for the organisations where such business problems exist or are likely to exist. This study will also be of great use to researchers, who are keen to understand and model the given business situation with distribution independent demand based models.

THE APPLICATION OF CONTROL USING NEURO-DYNAMIC PROGRAMMING WITH A FEATURE MAP

For the general nonlinear processes control problem, the most rigorous approach is to use dynamic optimization. However, as the size of the problem grows, the Dynamic Programming (DP) approach is suffered from the burden of calculation which is called as ‘Curse of Dimensionality'. To overcome this problem, a cost-approximator can be used to obtain the optimal control input policy minimizing the value of cost, called the Neuro-Dynamic Programming (NDP) algorithm. In this study, the NDP algorithm was applied to pH neutralization process in both simulations and experiments. A systematic approach of NDP to a pH neutralization process has been proposed and the performance of the NDP approach has been evaluated through the comparison with PI control. Also, a few related issues such as selection of approximator and a feature map of the states are investigated.

Control of pH neutralization process using simulation based dynamic programming

The pH neutralization process has long been taken as a representative benchmark problem of nonlinear chemical process control due to its nonlinearity and time-varying nature. For general nonlinear processes, it is difficult to control with a linear model-based control method so nonlinear controls must be considered. Among the numerous approaches suggested, the most rigorous approach is the dynamic optimization. However, as the size of the problem grows, the dynamic programming approach suffers from the curse of dimensionality. In order to avoid this problem, the Neuro-Dynamic Programming (NDP) approach was proposed by Bertsekas and Tsitsiklis [1996]. The NDP approach is to utilize all the data collected to generate an approximation of optimal cost-to-go function which was used to find the optimal input movement in real time control. The approximation could be any type of function such as polynomials, neural networks, etc. In this study, an algorithm using NDP approach was applied to a pH neutralization process to investigate the feasibility of the NDP algorithm and to deepen the understanding of the basic characteristics of this algorithm. As the approximator, the neural network which requires training and the k-nearest neighbor method which requires querying instead of training are investigated. The approximator has to use data from the optimal control strategy. If the optimal control strategy is not readily available, a suboptimal control strategy can be used instead. However, the laborious Bellman iterations are necessary in this case. For pH neutralization process it is rather easy to devise an optimal control strategy. Thus, we used an optimal control strategy and did not perform the Bellman iteration. Also, the effects of constraints on control moves are studied. From the simulations, the NDP method outperforms the conventional PID control.