Approximate Dynamic Programming Method Research Articles

Pathwise Optimization for Merchant Energy Production

We study merchant energy production modeled as a compound switching and timing option. The resulting Markov decision process is intractable. State-of-the-art approximate dynamic programming methods applied to realistic instances of this model yield policies with large optimality gaps that are attributed to a weak upper (dual) bound on the optimal policy value. We extend path-wise optimization from stopping models to merchant energy production to investigate this issue. We apply principal component analysis and block coordinate descent in novel ways to respectively precondition and solve the ensuing ill conditioned and large scale linear program, which even a cutting-edge commercial solver is unable to handle directly. Compared to standard methods, our approach leads to substantially tighter dual bounds and smaller optimality gaps at the expense of considerably larger computational effort. Specifically, we provide numerical evidence for the near optimality of the operating policies based on least squares Monte Carlo and compute slightly better ones using our approach on a set of existing benchmark ethanol production instances. These findings suggest that both these policies are effective for the class of models we investigate. Our research has potential relevance for other commodity merchant operations settings.

Approximate Optimal Motion Planning to Avoid Unknown Moving Avoidance Regions

In this article, an infinite-horizon optimal regulation problem is considered for a control-affine nonlinear autonomous agent subject to input constraints in the presence of dynamic avoidance regions. A local model-based approximate dynamic programming method is implemented to approximate the value function in a local neighborhood of the agent. By performing local approximations, prior knowledge of the locations of avoidance regions is not required. To alleviate the a priori knowledge of the number of avoidance regions in the operating domain, an extension is provided that modifies the value function approximation. The developed feedback-based motion planning strategy guarantees uniformly ultimately bounded convergence of the approximated control policy to the optimal policy while also ensuring the agent remains outside avoidance regions. Simulations are included to demonstrate the preliminary development for a kinematic unicycle and generic nonlinear system. Results from three experiments are also presented to illustrate the performance of the developed method, where a quadcopter achieves approximate optimal regulation while avoiding three mobile obstacles. To demonstrate the developed method, known avoidance regions are used in the first experiment, unknown avoidance regions are used in the second experiment, and an unknown time-varying obstacle directed by a remote pilot is included in the third experiment.

Approximate dynamic programming for the military inventory routing problem

The United States Army can benefit from effectively utilizing cargo unmanned aerial vehicles (CUAVs) to perform resupply operations in combat environments to reduce the use of manned (ground and aerial) resupply that incurs risk to personnel. We formulate a Markov decision process (MDP) model of an inventory routing problem (IRP) with vehicle loss and direct delivery, which we label the military IRP (MILIRP). The objective of the MILIRP is to determine CUAV dispatching and routing policies for the resupply of geographically dispersed units operating in an austere, combat environment. The large size of the problem instance motivating this research renders dynamic programming algorithms inappropriate, so we utilize approximate dynamic programming (ADP) methods to attain improved policies (relative to a benchmark policy) via an approximate policy iteration algorithmic strategy utilizing least squares temporal differencing for policy evaluation. We examine a representative problem instance motivated by resupply operations experienced by the United States Army in Afghanistan both to demonstrate the applicability of our MDP model and to examine the efficacy of our proposed ADP solution methodology. A designed computational experiment enables the examination of selected problem features and algorithmic features vis-a-vis the quality of solutions attained by our ADP policies. Results indicate that a 4-crew, 8-CUAV unit is able to resupply 57% of the demand from an 800-person organization over a 3-month time horizon when using the ADP policy, a notable improvement over the 18% attained using a benchmark policy. Such results inform the development of procedures governing the design, development, and utilization of CUAV assets for the resupply of dispersed ground combat forces.

Influence of unmanned combat aerial vehicle agility on short-range aerial combat effectiveness

The flight agility of an unmanned combat aerial vehicle (UCAV) determines its ability to rapidly transition from one state to another. In this paper, the influence of flight agility on short-range aerial combat effectiveness is quantitatively investigated. This research is based on one-on-one three-dimensional aerial combat engagements. First, a 6-DOF mathematical model of the UCAV is established, and a nonlinear dynamic inverse controller is designed. The flight agility is calculated based on the 180∘ flight heading reverse maneuver, and the influence of the control law parameters is studied. To implement autonomous intelligent aerial combat engagements, a three-dimensional approximate dynamic programming method is proposed. The performance of the designed algorithm is validated through both Monte Carlo simulations with various initial conditions and engagements with another algorithm for comparison. After training, the aerial combat simulation framework is constructed by combining the point-mass-based guidance law and a nonlinear UCAV model. Combat engagements are conducted between UCAVs with different configurations. The quantitative results are evaluated through Monte Carlo simulations and correlation analysis.

A data-driven approximate dynamic programming approach based on association rule learning: Spacecraft autonomy as a case study

Dynamic programming (DP) and Markov Decision Process (MDP) offer powerful tools for formulating, modeling, and solving decision making problems under uncertainty. In real-world applications, the applicability of DP is limited by severe scalability issues. These issues can be addressed by Approximate Dynamic Programming (ADP) techniques. ADP methods are based on the assumption of having either a proper estimation of the underlying state transition probability distributions or a simulation mechanism with the capability of generating samples according to such probability distributions. In this paper, we present a data-driven ADP-based approach, which can offer an alternative in case such assumption cannot be guaranteed. In particular, when varying the set-up of the MDP state transition probability matrix, different policies can be calculated through exact DP or ADP methods. Such policies are then processed by an Apriori-based algorithm to find frequent association rules within them. A pruning procedure is used to select the most suitable association rules, and finally an Association Classifier infers the optimal policy in all the possible circumstances. We show a detailed application of the proposed approach for the calculation of a proper mission operations plan for spacecrafts with a high level of on-board autonomy.

Least squares policy iteration with instrumental variables vs. direct policy search: comparison against optimal benchmarks using energy storage

This article studies least-squares approximate policy iteration (API) methods with parametrized value-function approximation. We study several variations of the policy evaluation phase, namely, Bellman error minimization, Bellman error minimization with instrumental variables, projected Bellman error minimization, and projected Bellman error minimization with instrumental variables. For a general discrete-time stochastic control problem, Bellman error minimization policy evaluation using instrumental variables is equivalent to both variants of the projected Bellman error minimization. An alternative to these API methods is direct policy search based on knowledge gradient. The practical performance of these three approximate dynamic programming methods, (i) least squares API with Bellman error minimization, (ii) least squares API with Bellman error minimization with instrumental variables, and (iii) direct policy search, are investigated in the context of an application in energy storage operations management. We create a library of test problems using real-world data and apply value iteration to find their optimal policies. These optimal benchmarks are then used to compare the developed approximate dynamic programming policies. Our analysis indicates that least-squares API with instrumental variables Bellman error minimization prominently outperforms least-squares API with Bellman error minimization. However, these approaches underperform our direct policy search implementation.

Adaptive railway traffic control using approximate dynamic programming

This study presents an adaptive railway traffic controller for real-time operations based on approximate dynamic programming (ADP). By assessing requirements and opportunities, the controller aims to limit consecutive delays resulting from trains that entered a control area behind schedule by sequencing them at a critical location in a timely manner, thus representing the practical requirements of railway operations. This approach depends on an approximation to the value function of dynamic programming after optimisation from a specified state, which is estimated dynamically from operational experience using reinforcement learning techniques. By using this approximation, the ADP avoids extensive explicit evaluation of performance and so reduces the computational burden substantially. In this investigation, we explore formulations of the approximation function and variants of the learning techniques used to estimate it. Evaluation of the ADP methods in a stochastic simulation environment shows considerable improvements in consecutive delays by comparison with the current industry practice of First-Come-First-Served sequencing. We also found that estimates of parameters of the approximate value function are similar across a range of test scenarios with different mean train entry delays.

Finite-Horizon Optimal Tracking Guidance for Aircraft Based on Approximate Dynamic Programming

Referring to the optimal tracking guidance of aircraft, the conventional time based kinematics model is transformed into a downrange based model by independent variable replacement. The deviations of in-flight altitude and flight path angle are penalized and corrected to achieve high precision tracking of reference trajectory. The tracking problem is solved as a linear quadratic regulator applying small perturbation theory, and the approximate dynamic programming method is used to cope with the solving of finite-horizon optimization. An actor-critic structure is established to approximate the optimal tracking controller and minimum cost function. The least squares method and Adam optimization algorithm are adopted to learn the parameters of critic network and actor network, respectively. A boosting trajectory with maximum final velocity is generated by Gauss pseudospectral method for the validation of guidance strategy. The results show that the trained feedback control parameters can effectively resist random wind disturbance, correct the initial altitude and flight path angle deviations, and achieve the goal of following a given trajectory.

Resource allocation in congested queueing systems with time-varying demand: An application to airport operations

Motivated by the need to develop time-efficient methods for minimizing operational delays at severely congested airports, we consider a problem involving the distribution of a common resource between two sources of time-varying demand. We formulate this as a dynamic program in which the objective is based on second moments of stochastic queue lengths and show that, for sufficiently high volumes of demand, optimal values can be well-approximated by quadratic functions of the system state. We identify conditions which enable the strong performance of myopic policies and develop approaches to the design of heuristic policies by means of approximate dynamic programming (ADP) methods. Numerical experiments suggest that our ADP-based heuristics, which require very little computational effort, are able to improve substantially upon the performances of more naive decision-making policies, particularly if exogenous system parameters vary considerably as functions of time.

Can Deep Reinforcement Learning Improve Inventory Management? Performance and Implementation of Dual Sourcing-Mode Problems

(Forthcoming in Manufacturing & Service Operations Management) Problem definition: Is Deep Reinforcement Learning (DRL) effective at solving inventory problems? Academic/practical relevance: Given that DRL has successfully been applied in computer games and robotics, supply chain researchers and companies are interested in its potential in inventory management. We provide a rigorous performance evaluation of DRL in three classic and intractable inventory problems: lost sales, dual-sourcing and multi-echelon inventory management. Methodology: We model each inventory problem as a Markov Decision Process and apply and tune the Asynchronous Advantage Actor Critic (A3C) DRL algorithm for a variety of parameter settings. Results: We demonstrate that the A3C algorithm can match performance of state-of-the-art heuristics and other approximate dynamic programming methods. While the initial tuning was computationally- and time-demanding, only small changes to the tuning parameters were needed for the other studied problems. Managerial implications: Our study provides evidence that DRL can effectively solve inventory problems. This is especially promising when problem-dependent heuristics are lacking. Yet generating structural policy insight or designing specialized policies that are (ideally provably) near optimal remains desirable.

Adaptive Railway Traffic Control using Approximate Dynamic Programming

Abstract This study presents an adaptive railway traffic controller for real-time operations based on approximate dynamic programming (ADP). By assessing requirements and opportunities, the controller aims to limit consecutive delays resulting from trains that entered a control area behind schedule by sequencing them at a critical location in a timely manner, thus representing the practical requirements of railway operations. This approach depends on an approximation to the value function of dynamic programming after optimisation from a specified state, which is estimated dynamically from operational experience using reinforcement learning techniques. By using this approximation, the ADP avoids extensive explicit evaluation of performance and so reduces the computational burden substantially. In this investigation, we explore formulations of the approximation function and variants of the learning techniques used to estimate it. Evaluation of the ADP methods in a stochastic simulation environment shows considerable improvements in consecutive delays by comparison with the current industry practice of First-Come-First-Served sequencing. We also found that estimates of parameters of the approximate value function are similar across a range of test scenarios with different mean train entry delays.

Incremental Approximate Dynamic Programming for Nonlinear Adaptive Tracking Control with Partial Observability

Approximate dynamic programming is a class of reinforcement learning, which solves adaptive, optimal control problems and tackles the curse of dimensionality with function approximators. Within this category, linear approximate dynamic programming provides a model-free control method by systematically using a quadratic cost-to-go function. Although efficient, linear approximate dynamic programming methods are difficult to apply to nonlinear systems or time-varying systems. To overcome the above limitations, this paper proposes an adaptive nonlinear tracking control method based on incremental approximate dynamic programming, which combines the advantages of linear approximate dynamic programming and incremental nonlinear control techniques. This is a model-free method for unknown, nonlinear systems and time-varying references. The trait of separating the local model information from the cost function approximation makes this method an option for partially observable control problems. This paper, therefore, proposes two reference tracking controllers for different observability conditions: the direct measurement of the full state, and the partially observable tracking error. In each condition, two algorithms are developed for off-line learning and online learning, respectively. These algorithms are applied to attitude control of a spacecraft disturbed by internal liquid sloshing. The results demonstrate that the proposed algorithms accurately deal with the unknown, time-varying internal dynamics while retaining efficient control, even with only partial observability.

Output Feedback Q-Learning Control for the Discrete-Time Linear Quadratic Regulator Problem.

Approximate dynamic programming (ADP) and reinforcement learning (RL) have emerged as important tools in the design of optimal and adaptive control systems. Most of the existing RL and ADP methods make use of full-state feedback, a requirement that is often difficult to satisfy in practical applications. As a result, output feedback methods are more desirable as they relax this requirement. In this paper, we present a new output feedback-based Q-learning approach to solving the linear quadratic regulation (LQR) control problem for discrete-time systems. The proposed scheme is completely online in nature and works without requiring the system dynamics information. More specifically, a new representation of the LQR Q-function is developed in terms of the input-output data. Based on this new Q-function representation, output feedback LQR controllers are designed. We present two output feedback iterative Q-learning algorithms based on the policy iteration and the value iteration methods. This scheme has the advantage that it does not incur any excitation noise bias, and therefore, the need of using discounted cost functions is circumvented, which in turn ensures closed-loop stability. It is shown that the proposed algorithms converge to the solution of the LQR Riccati equation. A comprehensive simulation study is carried out, which illustrates the proposed scheme.

Horizontal combinations of online and offline approximate dynamic programming for stochastic dynamic vehicle routing

Stochastic and dynamic vehicle routing problems gain increasing attention in the research community. In these problems, routing plans are dynamically updated based on realizations of stochastic information. Due to the complexity of the corresponding Markov decision processes (MDPs), the calculation of optimal policies for these problems is usually not possible and researchers draw on heuristical methods of approximate dynamic programming (ADP). These methods use simulation to approximate the value of a state and decision in the MDP. The simulations are either conducted offline or online. Offline methods such as value function approximations (VFAs) generally neglect the full detail of the state space due to aggregation. Online methods such as rollout algorithms (RAs) are often not able to capture decision and transition space sufficiently due to runtime limitations. In this paper, we alleviate this tradeoff by combining two methods of ADP, an online RA and an offline VFA in two ways. In addition to the integration of the VFA as a base policy into the online RA to strengthen the RA’s simulations, we also limit the RA’s simulation horizon, estimating the remaining reward-to-go again via the VFA. For two stochastic dynamic routing problems from the literature, we show how this combination outperforms state-of-the-art solutions while simultaneously reducing the required time for online calculations.

Approximate Dynamic Programming for Large-scale Unit Commitment Problems

Unit commitment (UC) is crucial for reducing the generation cost of short-term power system scheduling. This study proposes a new approximate dynamic programming (ADP) method to solve large-scale UC considering ramp rate limits of thermal units. To solve the “curse of dimensionality,” a policy iteration algorithm using post-decision state variables is utilized for value function approximation. The post-decision state variables dramatically simplify the algorithm by avoiding the need to compute the value of all feasible states within Bellman’s equation. The proposed method is applied to systems ranging in size from 10 to 1000 units to optimize the generation scheduling of the units in a day. And the time horizon of the UC problem is split into 96 time periods. It is shown that ADP can converge to promising results in polynomial time complexity for large-scale UC.

An approximate dynamic programming method for the optimal control of Alkai-Surfactant-Polymer flooding

Since the complexity, coupling, distributed parameter, etc. of alkali-surfactant-polymer (ASP) flooding, common optimization methods cannot acquire the optimal solutions well. This paper brings an optimal control method for ASP flooding based on approximate dynamic programming (ADP). At first, take the net present value (NPV) as the performance index. Then the Actor-Critic algorithm based on gradient descent method is adopted to get the optimal injection strategy, in which Actor and Critic are used to approximate the control and value function, respectively. To improve the approximation performance, the linear approximation basis function based on system characteristic is constructed. Furthermore, to train and predict the control and value function in next step, a temporal difference (TD) learning algorithm is introduced to update the weight coefficients. Then, the control in ADP is generated according to the Gauss function and its weight is updated according to the sigmoid function of TD error, so that the optimal control can be searched. At last, the enhanced oil recovery problem of ASP flooding with four injection wells and nine production wells is solved by the proposed method to test the effect of proposed method.

An approximate dynamic programming method for the multi-period technician scheduling problem with experience-based service times and stochastic customers

In this paper, we study how an organization can recognize that individuals learn when assigning employees to tasks. By doing so, an organization can meet current demands and position the capabilities of their workforce for the yet unknown demands in future days. Specifically, we study a variant of the technician and task scheduling problem in which the tasks to be performed in the current day are known, but there is uncertainty regarding the tasks to be performed in subsequent days. To solve this problem, we present an Approximate Dynamic Programming-based approach that incorporates into daily assignment decisions estimates of the long-term benefits associated with experience accumulation. We benchmark this approach against an approach that only considers the impact of experience accumulation on just the next day's productivity and show that the ADP approach outperforms this one-step lookahead approach. Finally, based on the results from an extensive computational study we derive insights into how an organization can schedule their employees in a manner that enables meeting both near and long-term demands.

Optimal Dynamic Liquidation of an Inventory via Forward Markets

In commodity markets, a trader selling her inventory over a finite time horizon has access not only to the spot market but also to forward contracts. The trader has the choice to sell at the spot price, or to short a forward contract for later delivery, or a combination of both. While the commodity is hedged with the forward contract and carried in the inventory, the trader can dynamically adjust the maturity of the forward contract at each time step until the inventory is completely depleted. This paper investigates the optimal liquidation strategy for such a trader in the above framework, which can be considered a dynamic extension of the cash and carry arbitrage that may arise in contango markets. It is proved, independent of the underlying stochastic forward price model, that a ‘partial sale’ strategy involving the sale of the inventory across different contracts is not optimal. In addition, under a linearity assumption on the forward price differences, the optimally selected forward maturity is limited to a subset comprising the immediate (spot), next, or last time stage. The theoretical results imply that the optimal actions set is considerably smaller than the feasible set. An Approximate Dynamic Programming (ADP) policy is developed, and compared to the solution achieved by an exact grid-based technique and a Forward Dynamic Optimization (FDO) approach. The risk and return characteristics of ADP and FDO methods are contrasted. The theoretical propositions are verified numerically, and found to reduce computational times significantly.

Output-Feedback Control of Unknown Linear Discrete-Time Systems With Stochastic Measurement and Process Noise via Approximate Dynamic Programming.

This paper studies the optimal output-feedback control problem for unknown linear discrete-time systems with stochastic measurement and process noise. A dithered Bellman equation with the innovation covariance matrix is constructed via the expectation operator given in the form of a finite summation. On this basis, an output-feedback-based approximate dynamic programming method is developed, where the terms depending on the innovation covariance matrix are available with the aid of the innovation covariance matrix identified beforehand. Therefore, by iterating the Bellman equation, the resulting value function can converge to the optimal one in the presence of the aforementioned noise, and the nearly optimal control laws are delivered. To show the effectiveness and the advantages of the proposed approach, a simulation example and a velocity control experiment on a dc machine are employed.

An approximate dynamic programming approach to attended home delivery management

We propose a new method of controlling demand through delivery time slot pricing in attended home delivery management with a focus on developing an approach suitable for industry-scale implementation. To this end, we exploit a relatively simple yet effective way of approximating delivery costs by decomposing the overall delivery problem into a collection of smaller, area-specific problems. These cost estimations serve as inputs into an approximate dynamic programming method that provides estimates of the opportunity cost associated with having a customer from a specific area book delivery in a specific time slot. These estimates depend on the area and on the delivery time slot under consideration.Using real, large-scale industry data, we estimate a demand model including a multinomial logit model of customers’ delivery time slot choice, and show in simulation studies that we can improve profits by over two percent in all tested instances relative to using a fixed-price policy commonly encountered in e-commerce. These improvements are achieved despite making strong assumptions in estimating delivery cost. These assumptions allow us to reduce computational run-time to a level suitable for real-time decision making on delivery time slot feasibility and pricing. Our approach provides quantitative insight into the importance of incorporating expected future order displacement costs into opportunity cost estimations alongside marginal delivery costs.