Neuro-dynamic Programming Research Articles

STOCHASTIC APPROXIMATE SCHEDULING BY NEURODYNAMIC LEARNING

The paper suggests a stochastic approximate solution to scheduling problems with unrelated parallel machines. The presented method is based on neurodynamic programming (reinforcement learning and feed-forward artificial neural networks). For various scheduling environments (static-dynamic, deterministic-stochastic) different variants of episodic Q-learning rules are proposed. A way to improve the avoidance of local minima is also discussed. Some investigations on the exploration strategy function approximation and parallelizing the solution are made. Finally a few experimental results are shown.

NEURO-DYNAMIC PROGRAMMING FOR THE EXPLORATION OF UNKNOWN GRAPHS

In this paper, the problem of exploring stochastic graphs is addressed. The definition of the entropy related to the a-priori unknown parameters (the lengths of the a-priori unknown links) leads to the formulation of the problem as a stochastic optimal control one. The application of exact Dynamic Programming suffers the so-called curse of dimensionality. To overcome this drawback, an approximate technique is proposed making use of Neuro-Dynamic Programming. Exploiting the concept of frontier, any approximate solution of the problem is shown to generate a “proper” policy.

AN RBF BASED NEURO-DYNAMIC APPROACH FOR THE CONTROL OF STOCHASTIC DYNAMIC SYSTEMS

This paper presents a neuro-dynamic programming methodology for the control of markov decision processes. The proposed method can be considered as a variant of the optimistic policy iteration, where radial basis function (RBF) networks are employed as a compact representation of the cost-to-go function and the Λ-LSPE is used for policy evaluation. We also emphasize the reformulation of the Bellman equation around the post-decision state in order to circumvent the calculation of the expectation. The proposed algorithm is applied to a retailer-inventory management problem.

Control of pH neutralization process using simulation based dynamic programming

The pH neutralization process has long been taken as a representative benchmark problem of nonlinear chemical process control due to its nonlinearity and time-varying nature. For general nonlinear processes, it is difficult to control with a linear model-based control method so nonlinear controls must be considered. Among the numerous approaches suggested, the most rigorous approach is the dynamic optimization. However, as the size of the problem grows, the dynamic programming approach suffers from the curse of dimensionality. In order to avoid this problem, the Neuro-Dynamic Programming (NDP) approach was proposed by Bertsekas and Tsitsiklis [1996]. The NDP approach is to utilize all the data collected to generate an approximation of optimal cost-to-go function which was used to find the optimal input movement in real time control. The approximation could be any type of function such as polynomials, neural networks, etc. In this study, an algorithm using NDP approach was applied to a pH neutralization process to investigate the feasibility of the NDP algorithm and to deepen the understanding of the basic characteristics of this algorithm. As the approximator, the neural network which requires training and the k-nearest neighbor method which requires querying instead of training are investigated. The approximator has to use data from the optimal control strategy. If the optimal control strategy is not readily available, a suboptimal control strategy can be used instead. However, the laborious Bellman iterations are necessary in this case. For pH neutralization process it is rather easy to devise an optimal control strategy. Thus, we used an optimal control strategy and did not perform the Bellman iteration. Also, the effects of constraints on control moves are studied. From the simulations, the NDP method outperforms the conventional PID control.

Fractionation in radiation treatment planning

Radiotherapy treatment is often delivered in a fractionated manner over a period of time. Emerging delivery devices are able to determine the actual dose that has been delivered at each stage facilitating the use of adaptive treatment plans that compensate for errors in delivery. We formulate a model of the day-to-day planning problem as a stochastic program and exhibit the gains that can be achieved by incorporating uncertainty about errors during treatment into the planning process. Due to size and time restrictions, the model becomes intractable for realistic instances. We show how heuristics and neuro-dynamic programming can be used to approximate the stochastic solution, and derive results from our models for realistic time periods. These results allow us to generate practical rules of thumb that can be immediately implemented in current planning technologies.

Valuation of American Options via Basis Functions

After a brief review of recent developments in the pricing and hedging of American options, this paper modifies the basis function approach to adaptive control and neuro-dynamic programming, and applies it to develop: 1) nonparametric pricing formulas for actively traded American options and 2) simulation-based optimization strategies for complex over-the-counter options, whose optimal stopping problems are prohibitively difficult to solve numerically by standard backward induction algorithms because of the curse of dimensionality. An important issue in this approach is the choice of basis functions, for which some guidelines and their underlying theory are provided.

Call admission control in cellular networks: A reinforcement learning solution

AbstractIn this paper, we address the call admission control (CAC) problem in a cellular network that handles several classes of traffic with different resource requirements. The problem is formulated as a semi‐Markov decision process (SMDP) problem. We use a real‐time reinforcement learning (RL) [neuro‐dynamic programming (NDP)] algorithm to construct a dynamic call admission control policy. We show that the policies obtained using our TQ‐CAC and NQ‐CAC algorithms, which are two different implementations of the RL algorithm, provide a good solution and are able to earn significantly higher revenues than classical solutions such as guard channel. A large number of experiments illustrates the robustness of our policies and shows how they improve quality of service (QoS) and reduce call‐blocking probabilities of handoff calls even with variable traffic conditions. Copyright © 2004 John Wiley & Sons, Ltd.

Optimization of a Fed-Batch Bioreactor Using Simulation-Based Approach

Abstract We use simulation-based approach to find the optimal feeding strategy for cloned invertase expression in Saccharomyces cerevisiae in a fed-batch bioreactor. The optimal strategy maximizes the productivity and minimizes the fermentation time. This procedure is motivated from Neuro Dynamic Programming (NDP) literature. wherein the optimal solution is parameterized in the form of a cost-to-go or profit-to-go functions. The proposed approach uses simulations from a heuristic feeding policy as a starting point to generate the profit-to-go vs state data. An artificial neural network is used to obtain profit-to-go as a function of system state. Iterations of Bellman equation are used to improve the profit function . The profit-to-go function thus obtained, is then implemented in an online controller, which essentially converts infinite horizon problem into an equivalent one-step-ahead problem.

Adaptation and Learning in Distributed Production Control

Distributed (agent-based) control architectures offer prospects of reduced complexity, high flexibility and a high robustness against disturbances in manufacturing. However, it has also turned out that distributed control architectures, usually banning all forms of hierarchy, cannot guarantee optimum performance and the system behaviour can be unpredictable. In the paper machine learning approaches such as neurodynamic programming and simulated annealing are described for managing changes and disturbances in manufacturing systems, and to decrease the computational costs of the scheduling process. The results demonstrate the applicability of the proposed solutions, which can contribute to significant improvements in system performance, keeping the known benefits of distributed control.

From Perturbation Analysis to Markov Decision Processes and Reinforcement Learning

The goals of perturbation analysis (PA), Markov decision processes (MDPs), and reinforcement learning (RL) are common: to make decisions to improve the system performance based on the information obtained by analyzing the current system behavior. In this paper, we study the relations among these closely related fields. We show that MDP solutions can be derived naturally from performance sensitivity analysis provided by PA. Performance potential plays an important role in both PA and MDPs; it also offers a clear intuitive interpretation for many results. Reinforcement learning, TD(λ), neuro-dynamic programming, etc., are efficient ways of estimating the performance potentials and related quantities based on sample paths. The sensitivity point of view of PA, MDP, and RL brings in some new insight to the area of learning and optimization. In particular, gradient-based optimization can be applied to parameterized systems with large state spaces, and gradient-based policy iteration can be applied to some nonstandard MDPs such as systems with correlated actions, etc. Potential-based on-line approaches and their advantages are also discussed.

Neuro dynamic programming algorithms for computing optimal control of production lines

Neuro dynamic programming algorithms for computing optimal control of production lines

Neuro-dynamic programming method for MPC

In this paper, we present how the approach of Neuro-Dynamic Programming(NDP) can be used to combat two important deficiencies of the conventional Model Predictive Control (MPC) formulation, the sometimes exorbitant on-line computational requirement and the inability to consider the evolution of uncertainty in the optimal control calculation. We use a simple Van de Vusse reactor to investigate the feasibility of the proposed approach and identify further research issues.

Comparing neuro-dynamic programming algorithms for the vehicle routing problem with stochastic demands

The paper considers a version of the vehicle routing problem where customers’ demands are uncertain. The focus is on dynamically routing a single vehicle to serve the demands of a known set of geographically dispersed customers during real-time operations. The goal consists of minimizing the expected distance traveled in order to serve all customers’ demands. Since actual demand is revealed upon arrival of the vehicle at the location of each customer, fully exploiting this feature requires a dynamic approach. This work studies the suitability of the emerging field of neuro-dynamic programming (NDP) in providing approximate solutions to this difficult stochastic combinatorial optimization problem. The paper compares the performance of two NDP algorithms: optimistic approximate policy iteration and a rollout policy. While the former improves the performance of a nearest-neighbor policy by 2.3%, the computational results indicate that the rollout policy generates higher quality solutions. The implication for the practitioner is that the rollout policy is a promising candidate for vehicle routing applications where a dynamic approach is required. Scope and purpose Recent years have seen a growing interest in the development of vehicle routing algorithms to cope with the uncertain and dynamic situations found in real-world applications (see the recent survey paper by Powell et al. [1]). As noted by Psaraftis [2], dramatic advances in information and communication technologies provide new possibilities and opportunities for vehicle routing research and applications. The enhanced capability of capturing the information that becomes available during real-time operations opens up new research directions. This informational availability provides the possibility of developing dynamic routing algorithms that take advantage of the information that is dynamically revealed during operations. Exploiting such information presents a significant challenge to the operations research/management science community. The single vehicle routing problem with stochastic demands [3] provides an example of a simple, yet very difficult to solve exactly, dynamic vehicle routing problem [2, p. 157] . The problem can be formulated as a stochastic shortest path problem [4] characterized by an enormous number of states. Neuro-dynamic programming [5,6] is a recent methodology that can be used to approximately solve very large and complex stochastic decision and control problems. In this spirit, this paper is meant to study the applicability of neuro-dynamic programming algorithms to the single-vehicle routing problem with stochastic demands.

Call admission control and routing in integrated services networks using neuro-dynamic programming

We consider the problem of call admission control (CAC) and routing in an integrated services network that handles several classes of calls of different value and with different resource requirements. The problem of maximizing the average value of admitted calls per unit time (or of revenue maximization) is naturally formulated as a dynamic programming problem, but is too complex to allow for an exact solution. We use methods of neuro-dynamic programming (NDP) [reinforcement learning (RL)], together with a decomposition approach, to construct dynamic (state-dependent) call admission control and routing policies. These policies are based on state-dependent link costs, and a simulation-based learning method is employed to tune the parameters that define these link costs. A broad set of experiments shows the robustness of our policy and compares its performance with a commonly used heuristic.

Missile defense and interceptor allocation by neuro-dynamic programming

This paper proposes a solution methodology for a missile defense problem involving the sequential allocation of defensive resources over a series of engagements. The problem is cast as a dynamic programming/Markovian decision problem, which is computationally intractable by exact methods because of its large number of states and its complex modeling issues. We employed a neuro-dynamic programming framework, whereby the cost-to-go function is approximated using neural network architectures that are trained on simulated data. We report on the performance obtained using several different training methods, and we compare this performance with the optimal approach.

An Approximation Algorithm for Optimal Stopping

We propose an algorithm for solving optimal stopping problems, and we provide theoretical results concerning convergence and approximation error. This algorithm can be viewed as a "neuro-dynamic programming" method for optimal stopping problems. The applicability of the algorithm is illustrated through a computational case study involving the pricing of a path-dependent financial derivative security that gives rise to an optimal stopping problem with a one-hundred-dimensional state space.

Neuro-Dynamic Programming [Book News & Reviews

Neuro-Dynamic Programming [Book News & Reviews