Approximate Policy Iteration Research Articles

A Reinforcement Learning Approach to Autonomous Decision Making of Intelligent Vehicles on Highways

Autonomous decision making is a critical and difficult task for intelligent vehicles in dynamic transportation environments. In this paper, a reinforcement learning approach with value function approximation and feature learning is proposed for autonomous decision making of intelligent vehicles on highways. In the proposed approach, the sequential decision making problem for lane changing and overtaking is modeled as a Markov decision process with multiple goals, including safety, speediness, smoothness, etc. In order to learn optimized policies for autonomous decision-making, a multiobjective approximate policy iteration (MO-API) algorithm is presented. The features for value function approximation are learned in a data-driven way, where sparse kernel-based features or manifold-based features can be constructed based on data samples. Compared with previous RL algorithms such as multiobjective Q-learning, the MO-API approach uses data-driven feature representation for value and policy approximation so that better learning efficiency can be achieved. A highway simulation environment using a 14 degree-of-freedom vehicle dynamics model was established to generate training data and test the performance of different decision-making methods for intelligent vehicles on highways. The results illustrate the advantages of the proposed MO-API method under different traffic conditions. Furthermore, we also tested the learned decision policy on a real autonomous vehicle to implement overtaking decision and control under normal traffic on highways. The experimental results also demonstrate the effectiveness of the proposed method.

Feature-based aggregation and deep reinforcement learning: a survey and some new implementations

In this paper we discuss policy iteration methods for approximate solution of a finite-state discounted Markov decision problem, with a focus on feature-based aggregation methods and their connection with deep reinforcement learning schemes. We introduce features of the states of the original problem, and we formulate a smaller &#x201C aggregate &#x201D Markov decision problem, whose states relate to the features. We discuss properties and possible implementations of this type of aggregation, including a new approach to approximate policy iteration. In this approach the policy improvement operation combines feature-based aggregation with feature construction using deep neural networks or other calculations. We argue that the cost function of a policy may be approximated much more accurately by the nonlinear function of the features provided by aggregation, than by the linear function of the features provided by neural network-based reinforcement learning, thereby potentially leading to more effective policy improvement.

Approximate dynamic programming for the aeromedical evacuation dispatching problem: Value function approximation utilizing multiple level aggregation

Sequential resource allocation decision-making for the military medical evacuation of wartime casualties consists of identifying which available aeromedical evacuation (MEDEVAC) assets to dispatch in response to each casualty event. These sequential decisions are complicated due to uncertainty in casualty demand (i.e., severity, number, and location) and service times. In this research, we present a Markov decision process model solved using a hierarchical aggregation value function approximation scheme within an approximate policy iteration algorithmic framework. The model seeks to optimize this sequential resource allocation decision under uncertainty of how to best dispatch MEDEVAC assets to calls for service. The policies determined via our approximate dynamic programming (ADP) approach are compared to optimal military MEDEVAC dispatching policies for two small-scale problem instances and are compared to a closest-available MEDEVAC dispatching policy that is typically implemented in practice for a large-scale problem instance. Results indicate that our proposed approximation scheme provides high-quality, scalable dispatching policies that are more easily employed by military medical planners in the field. The identified ADP policies attain 99.8% and 99.5% optimal for the 6- and 12-zone problem instances investigated, as well as 9.6%, 9.2%, and 12.4% improvement over the closest-MEDEVAC policy for the 6-, 12-, and 34-zone problem instances investigated.

Spare Parts Inventory Management with Substitution-Dependent Reliability

Motivated to apply sustainable supply chain principles to air-pollution control systems, this paper presents a dynamic inventory-management approach where substitution is possible to maintain these systems’ equipment. An air-pollution control system’s subsequent reliability depends on the replacement equipment selected. The corresponding problem is formulated as a stochastic dynamic program. Because the state and action space are prohibitively large, the approximate policy iteration algorithm is adapted to generate high-quality solutions. Therefore, this work replaces the value function with an affine combination of nonlinear basis functions and shows that a relaxation of the policy improvement step requires the solving of a mixed integer linear program. This approach helps in the designing of an algorithm that improves the quality of the approximation by solving a convex optimization problem. To assess the quality of resulting solutions, a lower bound is developed by considering a relaxation of the problem. In addition, two classes of heuristics are proposed based on a rolling-horizon two-stage stochastic programming formulation of the problem and a standard base-stock ordering policy. The performance of proposed policies is tested on a variety of settings, and results show that the approximate dynamic programming policies are near-optimal in the settings of interest and significantly outperform available benchmarks. The following analysis reveals that the proposed inventory replenishment policies resemble a base-stock policy with occasional deviations, and assignment and substitution decisions are determined by balancing the reliability with ordering, holding, and shortage costs. The online supplement is available at https://doi.org/10.1287/ijoc.2017.0794 .

Approximate policy iteration for dynamic resource-constrained project scheduling

We study non-preemptive scheduling problems where heterogeneous projects stochastically arrive over time. The projects include precedence-constrained tasks that require multiple resources. Incomplete projects are held in queues. When a queue is full, an arriving project must be rejected. The goal is to choose which tasks to start in each time-slot to maximize the infinite-horizon discounted expected profit. We provide a weakly coupled Markov decision process (MDP) formulation and apply a simulation-based approximate policy iteration method. Extensive numerical results are presented.

Estimate and approximate policy iteration algorithm for discounted Markov decision models with bounded costs and Borel spaces

Estimate and approximate policy iteration algorithm for discounted Markov decision models with bounded costs and Borel spaces

Bellman residuals minimization using online support vector machines

In this paper we present and theoretically study an Approximate Policy Iteration (API) method called A P I − B R M 𝜖 using a very effective implementation of incremental Support Vector Regression (SVR) to approximate the value function able to generalize Reinforcement Learning (RL) problems with continuous (or large) state space. A P I − B R M 𝜖 is presented as a non-parametric regularization method based on an outcome of the Bellman Residual Minimization (BRM) able to minimize the variance of the problem. The proposed method can be cast as incremental and may be applied to the on-line agent interaction framework of RL. Being also based on SVR which are based on convex optimization, is able to find the global solution of the problem. A P I − B R M 𝜖 using SVR can be seen as a regularization problem using 𝜖−insensitive loss. Compared to standard squared loss also used in regularization, this allows to naturally build a sparse solution for the approximation function. We extensively analyze the statistical properties of A P I − B R M 𝜖 founding a bound which controls the performance loss of the algorithm under some assumptions on the kernel and assuming that the collected samples are not-i.i.d. following a β−mixing process. Some experimental evidence and performance for well known RL benchmarks are also presented.

Tracking control optimization scheme of continuous-time nonlinear system via online single network adaptive critic design method

In this paper, the optimal tracking control problem (OTCP) for a class of continuous-time nonlinear systems with infinite horizon cost is discussed. An online adaptive critic design method is proposed to learn the solution of OTCP by constructing an augmented system associated with a discounted performance function, which is composed of the tracking errors and reference trajectory dynamics. Only one neural network (NN) is used as critic module for approximating the performance function in the solution procedure, and thus the architecture is simpler than the typical action-critic structure, which needs more computational load from neural networks. Therefore, by the means of the approximate policy iteration, the tracking errors get converged to a region near zero and the parameters of critic module get converged to the optimal ones based on our proposed method. Both the convergence of the NN weights and the stability of the tracking error dynamics are guaranteed by the Lyapunov theory. Two simulation examples are proposed to verify the effectiveness of the proposed method.

Least Squares Policy Iteration with Instrumental Variables vs. Direct Policy Search: Comparison Against Optimal Benchmarks Using Energy Storage

This paper studies approximate policy iteration (API) methods which use least-squares Bellman error minimization for policy evaluation. We address several of its enhancements, namely, Bellman error minimization with instrumental variables, projected Bellman error minimization, and projected Bellman error minimization with instrumental variables. For a general discrete-time stochastic control problem, Bellman error minimization policy evaluation using instrumental variables is equivalent to both variants of projected Bellman error minimization. An alternative to these API methods is direct policy search based on knowledge gradient. The practical performance of these three approximate dynamic programming methods are investigated in the context of an application in energy storage, integrated with an intermittent wind energy supply to fully serve a stochastic time-varying electricity demand. We create a library of test problems using real-world data and apply value iteration to find their optimal policies. These benchmarks are then used to compare the developed policies. Our analysis indicates that API with instrumental variables Bellman error minimization prominently outperforms API with least-squares Bellman error minimization. However, these approaches underperform our direct policy search implementation.

Multi-agent reinforcement learning using ordinal action selection and approximate policy iteration

Multi-agent reinforcement learning (MARL) provides a useful and flexible framework for multi-agent coordination in uncertain dynamic environments. However, the generalization ability and scalability of algorithms to large problem sizes, already problematic in single-agent RL, is an even more formidable obstacle in MARL applications. In this paper, a new MARL method based on ordinal action selection and approximate policy iteration called OAPI (Ordinal Approximate Policy Iteration), is presented to address the scalability issue of MARL algorithms in common-interest Markov Games. In OAPI, an ordinal action selection and learning strategy is integrated with distributed approximate policy iteration not only to simplify the policy space and eliminate the conflicts in multi-agent coordination, but also to realize the approximation of near-optimal policies for Markov Games with large state spaces. Based on the simplified policy space using ordinal action selection, the OAPI algorithm implements distributed approximate policy iteration utilizing online least-squares policy iteration (LSPI). This resulted in multi-agent coordination with good convergence properties with reduced computational complexity. The simulation results of a coordinated multi-robot navigation task illustrate the feasibility and effectiveness of the proposed approach.

Simulation-based approximate policy iteration for dynamic patient scheduling for radiation therapy.

We study radiation therapy scheduling problem where dynamically and stochastically arriving patients of different types are scheduled to future days. Unlike similar models in the literature, we consider cancellation of treatments. We formulate this dynamic multi-appointment patient scheduling problem as a Markov Decision Process (MDP). Since the MDP is intractable due to large state and action spaces, we employ a simulation-based approximate dynamic programming (ADP) approach to approximately solve our model. In particular, we develop Least-square based approximate policy iteration for solving our model. The performance of the ADP approach is compared with that of a myopic heuristic decision rule.

Least squares approximate policy iteration for learning bid prices in choice-based revenue management

We consider the revenue management problem of capacity control under customer choice behavior. An exact solution of the underlying stochastic dynamic program is difficult because of the multi-dimensional state space and, thus, approximate dynamic programming (ADP) techniques are widely used. The key idea of ADP is to encode the multi-dimensional state space by a small number of basis functions, often leading to a parametric approximation of the dynamic program’s value function. In general, two classes of ADP techniques for learning value function approximations exist: mathematical programming and simulation. So far, the literature on capacity control largely focuses on the first class.In this paper, we develop a least squares approximate policy iteration (API) approach which belongs to the second class. Thereby, we suggest value function approximations that are linear in the parameters, and we estimate the parameters via linear least squares regression. Exploiting both exact and heuristic knowledge from the value function, we enforce structural constraints on the parameters to facilitate learning a good policy. We perform an extensive simulation study to investigate the performance of our approach. The results show that it is able to obtain competitive revenues compared to and often outperforms state-of-the-art capacity control methods in reasonable computational time. Depending on the scarcity of capacity and the point in time, revenue improvements of around 1% or more can be observed. Furthermore, the proposed approach contributes to simulation-based ADP, bringing forth research on numerically estimating piecewise linear value function approximations and their application in revenue management environments.

Provision of Regulation Service by Smart Buildings

Regulation service (RS) reserves, a critical type of bi-directional capacity reserves, are provided today by expensive and environmentally unfriendly centralized fossil fuel generators. This paper investigates provision of RS reserves by the demand side. We consider a smart building operator that is capable of modulating the aggregate consumption of the building loads via price signals in response to an unanticipated RS signal that an independent system operator broadcasts. We first model the RS signal and load behavior, and formulate the related stochastic dynamic programming (DP) problem. Then, in order to deal with the complexity of the DP problem resulting from the uncountably infinite allowable price set, we characterize certain key properties of the DP dynamics, solve the DP problem for a discretized price policy to observe the structure of the optimal policy and re-capture the continuous price policy in an analytic approximate policy iteration (API) algorithm using the above properties and structure. We finally provide numerical evidence that the novel API algorithm converges to a continuous dynamic price policy that outperforms optimal discretized price policies in both computational effort and average cost.

Approximate dynamic programming for the dispatch of military medical evacuation assets

Military medical planners must consider the dispatching of aerial military medical evacuation (MEDEVAC) assets when preparing for and executing major combat operations. The launch authority seeks to dispatch MEDEVAC assets such that prioritized battlefield casualties are transported quickly and efficiently to nearby medical treatment facilities. We formulate a Markov decision process (MDP) model to examine the MEDEVAC dispatching problem. The large size of the problem instance motivating this research suggests that conventional exact dynamic programming algorithms are inappropriate. As such, we employ approximate dynamic programming (ADP) techniques to obtain high quality dispatch policies relative to current practices. An approximate policy iteration algorithmic strategy is applied that utilizes least squares temporal differencing for policy evaluation. We construct a representative planning scenario based on contingency operations in northern Syria both to demonstrate the applicability of our MDP model and to examine the efficacy of our proposed ADP solution methodology. A designed computational experiment is conducted to determine how selected problem features and algorithmic features affect the quality of solutions attained by our ADP policies. Results indicate that the ADP policy outperforms the myopic policy (i.e., the default policy in practice) by up to nearly 31% with regard to a lifesaving performance metric, as compared for a baseline scenario. Moreover, the ADP policy provides decreased MEDEVAC response times and utilization rates. These results benefit military medical planners interested in the development and implementation of cogent MEDEVAC tactics, techniques, and procedures for application in combat situations with a high operations tempo.

Sell or store? An ADP approach to marketing renewable energy

In deregulated markets, electricity is usually traded in advance, and the advance commitments have a time lag of several periods. For example, in the German intraday market, the seller commits to providing electricity 45 min before the 15-min interval in which delivery has to be made. We consider the problem of a producer that generates energy from stochastic, renewable sources, such as solar or wind and uses a storage device with conversion losses. We model the problem as a Markov Decision Process and consider lagged commitments for the first time in the literature. The problem is solved using an innovative approximate dynamic programming approach. Its key elements are the analytical derivation of the optimal action based on the value function approximation and a new combination of approximate policy iteration with classical backward induction. The new approach is quite general with regard to the stochastic processes describing the energy production and price evolution. We demonstrate the application of our approach by considering a wind farm/storage combination. A numerical study using real-world data shows the applicability and performance of the new approach and investigates how the storage device's parameters influence profit.

Manifold-Based Reinforcement Learning via Locally Linear Reconstruction.

Feature representation is critical not only for pattern recognition tasks but also for reinforcement learning (RL) methods to solve learning control problems under uncertainties. In this paper, a manifold-based RL approach using the principle of locally linear reconstruction (LLR) is proposed for Markov decision processes with large or continuous state spaces. In the proposed approach, an LLR-based feature learning scheme is developed for value function approximation in RL, where a set of smooth feature vectors is generated by preserving the local approximation properties of neighboring points in the original state space. By using the proposed feature learning scheme, an LLR-based approximate policy iteration (API) algorithm is designed for learning control problems with large or continuous state spaces. The relationship between the value approximation error of a new data point and the estimated values of its nearest neighbors is analyzed. In order to compare different feature representation and learning approaches for RL, a comprehensive simulation and experimental study was conducted on three benchmark learning control problems. It is illustrated that under a wide range of parameter settings, the LLR-based API algorithm can obtain better learning control performance than the previous API methods with different feature representation schemes.

Regularized policy iteration with nonparametric function spaces

We study two regularization-based approximate policy iteration algorithms, namely REG-LSPI and REG-BRM, to solve reinforcement learning and planning problems in discounted Markov Decision Processes...

Sell or Store? An ADP Approach to Marketing Renewable Energy

In deregulated markets, electricity is usually traded in advance and the advance commitments have a time lag of several periods. For example, in the German intraday market, the seller commits to providing electricity 45 minutes before the 15-minute interval in which delivery has to be made. We consider the problem of a producer that generates energy from stochastic, renewable sources, such as solar or wind, and uses a storage device with conversion losses. We model the problem as a Markov Decision Process and consider lagged commitments for the first time in the literature. The problem is solved using an innovative approximate dynamic programming approach. Its key elements are the analytical derivation of the optimal action based on the value function approximation, and a new combination of approximate policy iteration with classical backward induction. The new approach is quite general with regard to the stochastic processes describing the energy production and price evolution. We demonstrate the application of our approach by considering a wind farm/storage combination.A numerical study using real world data shows the applicability and performance of the new approach and investigates how the storage device’s parameters influence profit.

Classification-Based Approximate Policy Iteration

Tackling large approximate dynamic programming or reinforcement learning problems requires methods that can exploit regularities of the problem in hand. Most current methods are geared towards exploiting the regularities of either the value function or the policy. We introduce a general classification-based approximate policy iteration (CAPI) framework that can exploit regularities of both. We establish theoretical guarantees for the sample complexity of CAPI-style algorithms, which allow the policy evaluation step to be performed by a wide variety of algorithms, and can handle nonparametric representations of policies. Our bounds on the estimation error of the performance loss are tighter than existing results.

Convergence Proof of Approximate Policy Iteration for Undiscounted Optimal Control of Discrete-Time Systems

Approximate policy iteration (API) is studied to solve undiscounted optimal control problems in this paper. A discrete-time system with the continuous-state space and the finite-action set is considered. As approximation technique is used for the continuous-state space, approximation errors exist in the calculation and disturb the convergence of the original policy iteration. In our research, we analyze and prove the convergence of API for undiscounted optimal control. We use an iterative method to implement approximate policy evaluation and demonstrate that the error between approximate and exact value functions is bounded. Then, with the finite-action set, the greedy policy in policy improvement is generated directly. Our main theorem proves that if a sufficiently accurate approximator is used, API converges to the optimal policy. For implementation, we introduce a fuzzy approximator and verify the performance on the puddle world problem.