Approximate Policy Iteration Algorithm Research Articles

DNN-Approximations of Control Laws for Nonlinear Systems with Polytopic Constraints

This work focuses on deep artificial feed-forward neural networks as parametric function approximators in optimal control of discrete-time nonlinear but affine-in-control systems subject to polytopic constraints on the continuous states and control inputs. The neural networks are either considered as approximators of optimal control laws or as approximators of the corresponding cost functions. In both cases, an approach is developed for determining the approximated optimal control inputs without violating the constraints. A simple approximate policy iteration algorithm exploiting both approaches is finally presented and illustrated for a numerical example.

An approximate dynamic programming approach for solving an air combat maneuvering problem

Within visual range air combat involves execution of highly complex and dynamic activities, requiring rapid, sequential decision-making to achieve success. Fighter pilots spend years perfecting tactics and maneuvers for these types of combat engagements, yet the ongoing emergence of unmanned, autonomous vehicle technologies elicits a natural question — can an autonomous unmanned combat aerial vehicle (AUCAV) be imbued with the necessary artificial intelligence to perform challenging air combat maneuvering tasks independently? We formulate and solve the air combat maneuvering problem (ACMP) to examine this important question, developing a Markov decision process (MDP) model to control a defending AUCAV seeking to destroy an attacking adversarial vehicle. The MDP model includes a 5-degree-of-freedom, point-mass aircraft state transition model to accurately represent both kinematics and energy while maneuvering. An approximate dynamic programming (ADP) approach is proposed wherein we develop and test an approximate policy iteration algorithm that implements value function approximation via neural network regression to attain high-quality maneuver policies for the AUCAV. A representative intercept scenario is specified for testing purposes wherein the AUCAV must engage and destroy an adversary aircraft attempting to penetrate the defended airspace. Several designed experiments are conducted to determine how aircraft velocity and adversary maneuvering tactics impact the efficacy of the proposed ADP solution approach and to enable efficient algorithm parameter tuning. ADP-generated policies are compared to two benchmark maneuver policies constructed from two reward shaping functions found in the ACMP literature, attaining improved mean probabilities of kill for 24 of 36 air combat situations considered.

Safe model‐based reinforcement learning for nonlinear optimal control with state and input constraints

AbstractSafety is a critical factor in reinforcement learning (RL) in chemical processes. In our previous work, we had proposed a new stability‐guaranteed RL for unconstrained nonlinear control‐affine systems. In the approximate policy iteration algorithm, a Lyapunov neural network (LNN) was updated while being restricted to the control Lyapunov function, and a policy was updated using a variation of Sontag's formula. In this study, we additionally consider state and input constraints by introducing a barrier function, and we extend the applicable type to general nonlinear systems. We augment the constraints into the objective function and use the LNN added with a Lyapunov barrier function to approximate the augmented value function. Sontag's formula input with this approximate function brings the states into its lower level set, thereby guaranteeing the constraints satisfaction and stability. We prove the practical asymptotic stability and forward invariance. The effectiveness is validated using four tank system simulations.

A reinforcement learning approach to the stochastic cutting stock problem

We propose a formulation of the stochastic cutting stock problem as a discounted infinite-horizon Markov decision process. At each decision epoch, given current inventory of items, an agent chooses in which patterns to cut objects in stock in anticipation of the unknown demand. An optimal solution corresponds to a policy that associates each state with a decision and minimizes the expected total cost. Since exact algorithms scale exponentially with the state-space dimension, we develop a heuristic solution approach based on reinforcement learning. We propose an approximate policy iteration algorithm in which we apply a linear model to approximate the action-value function of a policy. Policy evaluation is performed by solving the projected Bellman equation from a sample of state transitions, decisions and costs obtained by simulation. Due to the large decision space, policy improvement is performed via the cross-entropy method. Computational experiments are carried out with the use of realistic data to illustrate the application of the algorithm. Heuristic policies obtained with polynomial and Fourier basis functions are compared with myopic and random policies. Results indicate the possibility of obtaining policies capable of adequately controlling inventories with an average cost up to 80% lower than the cost obtained by a myopic policy.

Efficient Batch-Mode Reinforcement Learning Using Extreme Learning Machines

As a class of batch-mode reinforcement learning (RL) methods for Markov decision problems with large or continuous state spaces, approximate policy iteration (API) has received increasing attention in the past decades. One open problem in the design of API algorithms is how to construct the basis functions or features for value function approximation (VFA). In this paper, we propose a novel batch-mode RL approach with randomly projected features for VFA. The proposed approach can be viewed as an extension of extreme learning machines (ELMs) to RL problems so it can be called ELM-API. The ELMs have been popularly studied in supervised learning problems, but there is not much work on the extension of ELMs to learning control problems. The proposed approach has advantages over the previous API algorithms in that the features for VFA can be quickly generated without complex parameter selection and the performance will be adaptive to different sample sets in batch-mode RL. In particular, the ELM-API approach can realize fast and efficient feature reconstruction when training sample sets are relatively small. Comprehensive simulation studies on two benchmark learning control problems were carried out to test the performance of API algorithms with different feature construction methods. It is shown that the ELM-API algorithm can obtain comparable or better performance than the previous API approaches. To further show the effectiveness of ELM-API in real-world applications, the simulation results on a more challenging high-dimensional lane-changing decision problem in dynamic traffic environment are also reported, which show the capability of the ELM-API algorithm in learning satisfactory lane-changing policies with high data efficiency.

Surgical scheduling under uncertainty by approximate dynamic programming

Surgical scheduling consists of selecting surgeries to be performed within a day, while jointly assigning operating rooms, starting times and the required resources. Patients can be elective or emergency/urgent. The scheduling of surgeries in an operating theatre with common resources to emergency or urgent and elective cases is highly subject to uncertainties not only on the duration of an intervention but mainly on the arrival of emergency or urgent cases. At the beginning of the day we are given a candidate set of elective surgeries with and an expected duration and a time window the surgery must start, but the expected duration and the time window of an emergency or urgent case become known when the surgery arrives. The day is divided into decision stages. Due to the dynamic nature of the problem, at the beginning of each stage the planner can make decisions taking into account the new information available. Decisions can be to schedule arriving surgeries, and to reschedule or cancel surgeries not started yet. The objective is to minimize the total expected cost composed of terms related to refusing arriving surgeries, to canceling scheduled surgeries, and to starting surgeries out of their time window. We address the problem with an approximate dynamic programming approach embedding an integer programming formulation to support decision making. We propose a dynamic model and an approximate policy iteration algorithm making use of basis functions to capture the impact of decisions to the future stages. Computational experiments have shown with statistical significance that the proposed algorithm outperforms a lookahead reoptimization approach.

A Reinforcement Learning Approach to Autonomous Decision Making of Intelligent Vehicles on Highways

Autonomous decision making is a critical and difficult task for intelligent vehicles in dynamic transportation environments. In this paper, a reinforcement learning approach with value function approximation and feature learning is proposed for autonomous decision making of intelligent vehicles on highways. In the proposed approach, the sequential decision making problem for lane changing and overtaking is modeled as a Markov decision process with multiple goals, including safety, speediness, smoothness, etc. In order to learn optimized policies for autonomous decision-making, a multiobjective approximate policy iteration (MO-API) algorithm is presented. The features for value function approximation are learned in a data-driven way, where sparse kernel-based features or manifold-based features can be constructed based on data samples. Compared with previous RL algorithms such as multiobjective Q-learning, the MO-API approach uses data-driven feature representation for value and policy approximation so that better learning efficiency can be achieved. A highway simulation environment using a 14 degree-of-freedom vehicle dynamics model was established to generate training data and test the performance of different decision-making methods for intelligent vehicles on highways. The results illustrate the advantages of the proposed MO-API method under different traffic conditions. Furthermore, we also tested the learned decision policy on a real autonomous vehicle to implement overtaking decision and control under normal traffic on highways. The experimental results also demonstrate the effectiveness of the proposed method.

Spare Parts Inventory Management with Substitution-Dependent Reliability

Motivated to apply sustainable supply chain principles to air-pollution control systems, this paper presents a dynamic inventory-management approach where substitution is possible to maintain these systems’ equipment. An air-pollution control system’s subsequent reliability depends on the replacement equipment selected. The corresponding problem is formulated as a stochastic dynamic program. Because the state and action space are prohibitively large, the approximate policy iteration algorithm is adapted to generate high-quality solutions. Therefore, this work replaces the value function with an affine combination of nonlinear basis functions and shows that a relaxation of the policy improvement step requires the solving of a mixed integer linear program. This approach helps in the designing of an algorithm that improves the quality of the approximation by solving a convex optimization problem. To assess the quality of resulting solutions, a lower bound is developed by considering a relaxation of the problem. In addition, two classes of heuristics are proposed based on a rolling-horizon two-stage stochastic programming formulation of the problem and a standard base-stock ordering policy. The performance of proposed policies is tested on a variety of settings, and results show that the approximate dynamic programming policies are near-optimal in the settings of interest and significantly outperform available benchmarks. The following analysis reveals that the proposed inventory replenishment policies resemble a base-stock policy with occasional deviations, and assignment and substitution decisions are determined by balancing the reliability with ordering, holding, and shortage costs. The online supplement is available at https://doi.org/10.1287/ijoc.2017.0794 .

Estimate and approximate policy iteration algorithm for discounted Markov decision models with bounded costs and Borel spaces

Estimate and approximate policy iteration algorithm for discounted Markov decision models with bounded costs and Borel spaces

Provision of Regulation Service by Smart Buildings

Regulation service (RS) reserves, a critical type of bi-directional capacity reserves, are provided today by expensive and environmentally unfriendly centralized fossil fuel generators. This paper investigates provision of RS reserves by the demand side. We consider a smart building operator that is capable of modulating the aggregate consumption of the building loads via price signals in response to an unanticipated RS signal that an independent system operator broadcasts. We first model the RS signal and load behavior, and formulate the related stochastic dynamic programming (DP) problem. Then, in order to deal with the complexity of the DP problem resulting from the uncountably infinite allowable price set, we characterize certain key properties of the DP dynamics, solve the DP problem for a discretized price policy to observe the structure of the optimal policy and re-capture the continuous price policy in an analytic approximate policy iteration (API) algorithm using the above properties and structure. We finally provide numerical evidence that the novel API algorithm converges to a continuous dynamic price policy that outperforms optimal discretized price policies in both computational effort and average cost.

Manifold-Based Reinforcement Learning via Locally Linear Reconstruction.

Feature representation is critical not only for pattern recognition tasks but also for reinforcement learning (RL) methods to solve learning control problems under uncertainties. In this paper, a manifold-based RL approach using the principle of locally linear reconstruction (LLR) is proposed for Markov decision processes with large or continuous state spaces. In the proposed approach, an LLR-based feature learning scheme is developed for value function approximation in RL, where a set of smooth feature vectors is generated by preserving the local approximation properties of neighboring points in the original state space. By using the proposed feature learning scheme, an LLR-based approximate policy iteration (API) algorithm is designed for learning control problems with large or continuous state spaces. The relationship between the value approximation error of a new data point and the estimated values of its nearest neighbors is analyzed. In order to compare different feature representation and learning approaches for RL, a comprehensive simulation and experimental study was conducted on three benchmark learning control problems. It is illustrated that under a wide range of parameter settings, the LLR-based API algorithm can obtain better learning control performance than the previous API methods with different feature representation schemes.

Regularized policy iteration with nonparametric function spaces

We study two regularization-based approximate policy iteration algorithms, namely REG-LSPI and REG-BRM, to solve reinforcement learning and planning problems in discounted Markov Decision Processes...

A hierarchical path planning approach based on A⁎ and least-squares policy iteration for mobile robots

In this paper, we propose a novel hierarchical path planning approach for mobile robot navigation in complex environments. The proposed approach has a two-level structure. In the first level, the A⁎ algorithm based on grids is used to find a geometric path quickly and several path points are selected as subgoals for the next level. In the second level, an approximate policy iteration algorithm called least-squares policy iteration (LSPI) is used to learn a near-optimal local planning policy that can generate smooth trajectories under kinematic constraints of the robot. Using this near-optimal local planning policy, the mobile robot can find an optimized path by sequentially approaching the subgoals obtained in the first level. One advantage of the proposed approach is that the kinematic characteristics of the mobile robot can be incorporated into the LSPI-based path optimization procedure. The second advantage is that the LSPI-based local path optimizer uses an approximate policy iteration algorithm which has been proven to be data-efficient and stable. The training of the local path optimizer can use sample experiences collected randomly from any reasonable sampling distribution. Furthermore, the LSPI-based local path optimizer has the ability of dealing with uncertainties in the environment. For unknown obstacles, it just needs to replan the path in the second level rather than the whole planner. Simulations for path planning in various types of environments have been carried out and the results demonstrate the effectiveness of the proposed approach.

A simulation based approximate dynamic programming approach to multi-class, multi-resource surgical scheduling

This paper presents a model and solution methodology for scheduling patients in a multi-class, multi-resource surgical system. Specifically, given a master schedule that provides a cyclic breakdown of total OR availability into specific daily allocations to each surgical specialty, the model provides a scheduling policy for all surgeries that minimizes a combination of the lead time between patient request and surgery date, overtime in the operating room and congestion in the wards. To the best of our knowledge, this paper is the first to determine a surgical schedule based on making efficient use of both the operating rooms and the recovery beds. Such a problem can be formulated as Markov Decision Process model but the size of any realistic problem makes traditional solution methods intractable. We develop a version of the Least Squares Approximate Policy Iteration algorithm and test our model on data from a local hospital to demonstrate the success of the resulting policy.

Policy Iteration Based on Stochastic Factorization

When a transition probability matrix is represented as the product of two stochastic matrices, one can swap the factors of the multiplication to obtain another transition matrix that retains some fundamental characteristics of the original. Since the derived matrix can be much smaller than its precursor, this property can be exploited to create a compact version of a Markov decision process (MDP), and hence to reduce the computational cost of dynamic programming. Building on this idea, this paper presents an approximate policy iteration algorithm called policy iteration based on stochastic factorization, or PISF for short. In terms of computational complexity, PISF replaces standard policy iteration's cubic dependence on the size of the MDP with a function that grows only linearly with the number of states in the model. The proposed algorithm also enjoys nice theoretical properties: it always terminates after a finite number of iterations and returns a decision policy whose performance only depends on the quality of the stochastic factorization. In particular, if the approximation error in the factorization is sufficiently small, PISF computes the optimal value function of the MDP. The paper also discusses practical ways of factoring an MDP and illustrates the usefulness of the proposed algorithm with an application involving a large-scale decision problem of real economical interest.

A hierarchical reinforcement learning approach for optimal path tracking of wheeled mobile robots

Robust motion control is fundamental to autonomous mobile robots. In the past few years, reinforcement learning (RL) has attracted considerable attention in the feedback control of wheeled mobile robot. However, it is still difficult for RL to solve problems with large or continuous state spaces, which is common in robotics. To improve the generalization ability of RL, this paper presents a novel hierarchical RL approach for optimal path tracking of wheeled mobile robots. In the proposed approach, a graph Laplacian-based hierarchical approximate policy iteration (GHAPI) algorithm is developed, in which the basis functions are constructed automatically using the graph Laplacian operator. In GHAPI, the state space of an Markov decision process is divided into several subspaces and approximate policy iteration is carried out on each subspace. Then, a near-optimal path-tracking control strategy can be obtained by GHAPI combined with proportional-derivative (PD) control. The performance of the proposed approach is evaluated by using a P3-AT wheeled mobile robot. It is demonstrated that the GHAPI-based PD control can obtain better near-optimal control policies than previous approaches.

Approximate Policy Iteration for Markov Control Revisited

Q-Learning is based on value iteration and remains the most popular choice for solving Markov Decision Problems (MDPs) via reinforcement learning (RL), where the goal is to bypass the transition probabilities of the MDP. Approximate policy iteration (API) is another RL technique, not as widely used as Q-Learning, based on modified policy iteration. In this paper, we present and analyze an API algorithm for discounted reward based on (i) a classical temporal differences update for policy evaluation and (ii) simulation-based mean estimation for policy improvement. Further, we analyze for convergence API algorithms based on Q-factors for (i) discounted reward and (ii) for average reward MDPs. The average reward algorithm is based on relative value iteration; we also present results from some numerical experiments with it.

On Solving Event-Based Optimization With Average Reward Over Infinite Stages

Event-based optimization (EBO) provides a unified framework for problems in which decisions can be made only when certain events occur. Because the event sequence usually is not Markovian, the optimal policy could depend on the entire event history, which is hard to implement in practice. So most existing studies focus on memoryless policies, which make decisions only based on the current observable events. But it remains open how to find the optimal memoryless policies in general, leaving alone to solve the EBO optimally. In this technical note, we address these two important questions for infinite-stage EBOs with finite state and action spaces and make the following three major contributions. First, we extend our previous studies on finite-stage EBOs and convert infinite-stage EBOs to partially observable Markov decision processes (POMDPs). The belief process of this POMDP is called belief-event decision process (BEDP). Under certain well-known conditions, the optimal policies of BEDPs can be achieved within stationary Markov deterministic policies. Second, assuming optimal stationary policies exist, the performance difference and derivative formulas are developed. Potentials of memoryless event-based policies are shown to be piecewise linear functions, and thus can be efficiently estimated through sample paths. Third, a potential-based approximate policy iteration algorithm is developed to obtain near-optimal memoryless policies. The convergence and performance loss bound of this algorithm are analyzed.

Hierarchical Approximate Policy Iteration With Binary-Tree State Space Decomposition

In recent years, approximate policy iteration (API) has attracted increasing attention in reinforcement learning (RL), e.g., least-squares policy iteration (LSPI) and its kernelized version, the kernel-based LSPI algorithm. However, it remains difficult for API algorithms to obtain near-optimal policies for Markov decision processes (MDPs) with large or continuous state spaces. To address this problem, this paper presents a hierarchical API (HAPI) method with binary-tree state space decomposition for RL in a class of absorbing MDPs, which can be formulated as time-optimal learning control tasks. In the proposed method, after collecting samples adaptively in the state space of the original MDP, a learning-based decomposition strategy of sample sets was designed to implement the binary-tree state space decomposition process. Then, API algorithms were used on the sample subsets to approximate local optimal policies of sub-MDPs. The original MDP was decomposed into a binary-tree structure of absorbing sub-MDPs, constructed during the learning process, thus, local near-optimal policies were approximated by API algorithms with reduced complexity and higher precision. Furthermore, because of the improved quality of local policies, the combined global policy performed better than the near-optimal policy obtained by a single API algorithm in the original MDP. Three learning control problems, including path-tracking control of a real mobile robot, were studied to evaluate the performance of the HAPI method. With the same setting for basis function selection and sample collection, the proposed HAPI obtained better near-optimal policies than previous API methods such as LSPI and KLSPI.

A framework and a mean-field algorithm for the local control of spatial processes

The Markov Decision Process (MDP) framework is a tool for the efficient modelling and solving of sequential decision-making problems under uncertainty. However, it reaches its limits when state and action spaces are large, as can happen for spatially explicit decision problems. Factored MDPs and dedicated solution algorithms have been introduced to deal with large factored state spaces. But the case of large action spaces remains an issue. In this article, we define graph-based Markov Decision Processes (GMDPs), a particular Factored MDP framework which exploits the factorization of the state space and the action space of a decision problem. Both spaces are assumed to have the same dimension. Transition probabilities and rewards are factored according to a single graph structure, where nodes represent pairs of state/decision variables of the problem. The complexity of this representation grows only linearly with the size of the graph, whereas the complexity of exact resolution grows exponentially. We propose an approximate solution algorithm exploiting the structure of a GMDP and whose complexity only grows quadratically with the size of the graph and exponentially with the maximum number of neighbours of any node. This algorithm, referred to as MF-API, belongs to the family of Approximate Policy Iteration (API) algorithms. It relies on a mean-field approximation of the value function of a policy and on a search limited to the suboptimal set of local policies. We compare it, in terms of performance, with two state-of-the-art algorithms for Factored MDPs: SPUDD and Approximate Linear Programming (ALP). Our experiments show that SPUDD is not generally applicable to solving GMDPs, due to the size of the action space we want to tackle. On the other hand, ALP can be adapted to solve GMDPs. We show that ALP is faster than MF-API and provides solutions of similar quality for most problems. However, for some problems MF-API provides significantly better policies, and in all cases provides a better approximation of the value function of approximate policies. These promising results show that the GMDP model offers a convenient framework for modelling and solving a large range of spatial and structured planning problems, that can arise in many different domains where processes are managed over networks: natural resources, agriculture, computer networks, etc.