Near-optimal Policy Research Articles

Dynamic server assignment in an extended machine-repair model

This article considers an extension of the classic machine-repair problem. The machines, apart from receiving service from a single repairman, now also supply service themselves to queues of products. The extended model can be viewed as a two-layered queueing network, in which the queues of products in the first layer are generally correlated, due to the fact that the machines have to share the repairman’s capacity in the second layer. Of particular interest is the dynamic control problem of how the repairman should allocate his/her capacity to the machines at any point in time so that the long-term average (weighted) sum of the queue lengths of the first-layer queues is minimized. Since the optimal policy for the repairman cannot be found analytically due to the correlations in the queue lengths, a near-optimal policy is proposed. This is obtained by combining intuition and results from queueing theory with techniques from Markov decision theory. Specifically, the relative value functions for several policies for which the model can be decomposed in less complicated subsystems are studied, and the results are combined with the classic one-step policy improvement algorithm. The resulting policy is easy to apply, is scalable in the number of machines, and is shown to be highly accurate over a wide range of parameter settings.

Optimal Maintenance Policies for Automated Demand Response Devices

Demand-side participation is now widely recognized as being extremely critical for satisfying the growing electricity demand in the U.S. The primary mechanism for demand management in the U.S. is demand response (DR) programs that attempt to reduce or shift demand by giving incentives to participating customers via price discounts or rebate payments. Utilities that offer DR programs rely on automated DR devices (ADRs) to automate the response to DR signals. The ADRs are faulty, but the working state of the ADR is not directly observable; one can, however, attempt to infer it from the power consumption during DR events. The utility loses revenue when a malfunctioning ADR does not respond to a DR signal; however, sending a maintenance crew to check and reset the ADR also incurs costs. In this paper, we show that the problem of maintaining a pool of ADRs using a limited number of maintenance crews can be formulated as a restless bandit problem, and that one can compute a near-optimal policy for this problem using Whittle indices. We show that the Whittle indices can be efficiently computed using a variational Bayes procedure even when the load-shed magnitude is noisy, and when there is a random mismatch between the clocks at the utility and at the meter. The results of our numerical experiments suggest that the Whittle-index-based approximate policy is within 4% of the optimal solution for all reasonably low values of the signal-to-noise ratio in the meter readings.

Approximate planning and verification for large Markov decision processes

We focus on the planning and verification problems for very large probabilistic systems, such as Markov decision processes (MDPs), from a complexity point of view. More precisely, we deal with the problem of designing an efficient approximation method to compute a near-optimal policy for the planning problem in discounted MDPs and the satisfaction probabilities of interesting properties, like reachability or safety, over the Markov chain obtained by restricting the MDP to the near-optimal policy. In this paper, we present two different approaches. The first one is based on sparse sampling while the second uses a variant of the multiplicative weights update algorithm. The complexity of the first approximation method is independent of the size of the state space and uses only a probabilistic generator of the MDP. We give a complete analysis of this approach, for which the control parameter is mainly the targeted quality of the approximation. The second approach is more prospective and is different in the sense that the method can be controlled dynamically by observing its speed of convergence. Parts of this paper have already been presented in Lassaigne and Peyronnet (in Proceedings of the ACM Symposium on applied computing, SAC 2012, pp 1314---1319, ACM 2012), by the same authors.

Piecewise linear approximations for the static–dynamic uncertainty strategy in stochastic lot-sizing

In this paper, we develop a unified mixed integer linear modelling approach to compute near-optimal policy parameters for the non-stationary stochastic lot sizing problem under static–dynamic uncertainty strategy. The proposed approach applies to settings in which unmet demand is backordered or lost; and it can accommodate variants of the problem for which the quality of service is captured by means of backorder penalty costs, non-stockout probabilities, or fill rate constraints. This approach has a number of advantages with respect to existing methods in the literature: it enables seamless modelling of different variants of the stochastic lot sizing problem, some of which have been previously tackled via ad hoc solution methods and some others that have not yet been addressed in the literature; and it produces an accurate estimation of the expected total cost, expressed in terms of upper and lower bounds based on piecewise linearisation of the first order loss function. We illustrate the effectiveness and flexibility of the proposed approach by means of a computational study.

Interference-Constrained Optimal Power-Adaptive Amplify-and-Forward Relaying and Selection for Underlay Cognitive Radios

In an underlay cognitive radio (CR) system, a secondary user can transmit when the primary is transmitting but is subject to tight constraints on the interference it causes to the primary receiver. Amplify-and-forward (AF) relaying is an effective technique that significantly improves the performance of a CR by providing an alternate path for the secondary transmitter's signal to reach the secondary receiver. We present and analyze a novel optimal relay gain adaptation policy (ORGAP) in which the relay is interference aware and optimally adapts both its gain and transmit power as a function of its local channel gains. ORGAP minimizes the symbol error probability at the secondary receiver subject to constraints on the average relay transmit power and on the average interference caused to the primary. It is different from ad hoc AF relaying policies and serves as a new and fundamental theoretical benchmark for relaying in an underlay CR. We also develop a near-optimal and simpler relay gain adaptation policy that is easy to implement. An extension to a multirelay scenario with selection is also developed. Our extensive numerical results for single and multiple relay systems quantify the power savings achieved over several ad hoc policies for both MPSK and MQAM constellations.

An approximate dynamic programming approachto decision making in the presence of uncertainty for surfactant-polymer flooding

The least squares Monte Carlo method is a decision evaluation method that can capture the effect of uncertainty and the value of flexibility of a process. The method is a stochastic approximate dynamic programming approach to decision making. It is based on a forward simulation coupled with a recursive algorithm which produces the near-optimal policy. It relies on the Monte Carlo simulation to produce convergent results. This incurs a significant computational requirement when using this method to evaluate decisions for reservoir engineering problems because this requires running many reservoir simulations. The objective of this study was to enhance the performance of the least squares Monte Carlo method by improving the sampling method used to generate the technical uncertainties used in obtaining the production profiles. The probabilistic collocation method has been proven to be a robust and efficient uncertainty quantification method. By using the sampling methods of the probabilistic collocation method to approximate the sampling of the technical uncertainties, it is possible to significantly reduce the computational requirement of running the decision evaluation method. Thus, we introduce the least squares probabilistic collocation method. The decision evaluation considered a number of technical and economic uncertainties. Three reservoir case studies were used: a simple homogeneous model, the PUNQ-S3 model, and a modified portion of the SPE10 model. The results show that using the sampling techniques of the probabilistic collocation method produced relatively accurate responses compared with the original method. Different possible enhancements were discussed in order to practically adapt the least squares probabilistic collocation method to more realistic and complex reservoir models. Furthermore, it is desired to perform the method to evaluate high-dimensional decision scenarios for different chemical enhanced oil recovery processes using real reservoir data.

Scenario Reduction for Probabilistic Robot Path Planning in the Presence of Preferences on Hidden States

In practical robot motion planning, robots usually do not have a full model of their surrounding, and hence no complete and correct plan can be prepared in advance. In other words, in real-world scenarios a mobile robot operates in a partially known environment, meaning that there exists incomplete information about the true state of several ‘hidden’ variables, such as open/closed doors, which may represent potential blockages of the path of the robot. Consequently, a robot may have a probability distribution estimation of the states of the hidden variables and a preference over the possible values of such hidden variables. In this paper, to deal with the problem of choosing optimal policies for planning in off-line mode, a linear programming model is developed that incorporates the probability distribution of hidden variables. Furthermore, a heuristic method is proposed for planning in the presence of numerous hidden variables which relies on optimistic assumptive planning and produces near-optimal policies.

Optimal back-to-front airplane boarding

The problem of finding an optimal back-to-front airplane boarding policy is explored, using a mathematical model that is related to the 1+1 polynuclear growth model with concave boundary conditions and to causal sets in gravity. We study all airplane configurations and boarding group sizes. Optimal boarding policies for various airplane configurations are presented. Detailed calculations are provided along with simulations that support the main conclusions of the theory. We show that the effectiveness of back-to-front policies undergoes a phase transition when passing from lightly congested airplanes to heavily congested airplanes. The phase transition also affects the nature of the optimal or near-optimal policies. Under what we consider to be realistic conditions, optimal back-to-front policies lead to a modest 8-12% improvement in boarding time over random (no policy) boarding, using two boarding groups. Having more than two groups is not effective.

Approximation Algorithms for the Stochastic Lot-Sizing Problem with Order Lead Times

We develop new algorithmic approaches to compute provably near-optimal policies for multiperiod stochastic lot-sizing inventory models with positive lead times, general demand distributions, and dynamic forecast updates. The policies that are developed have worst-case performance guarantees of 3 and typically perform very close to optimal in extensive computational experiments. The newly proposed algorithms employ a novel randomized decision rule. We believe that these new algorithmic and performance analysis techniques could be used in designing provably near-optimal randomized algorithms for other stochastic inventory control models and more generally in other multistage stochastic control problems.

Decision Making Under Uncertainty: Applying the Least-Squares Monte Carlo Method in Surfactant-Flooding Implementation

Summary This study introduces a decision-making evaluation method for flexibility in surfactant flooding. The method aims to capture the effects of uncertainty in the time series for both technical and economic parameters and produce a near-optimal policy with respect to these uncertainties as they vary with time. The evaluation method used was the least-squares Monte Carlo (LSM) method, which is best-suited for evaluating flexibility in project implementation. The decision analyzed was that of finding the best time to start surfactant flooding during the lifetime of a field under uncertainty. The study was conducted on two reservoir models: a 3D homogeneous model and a 2D heterogeneous model. The technical uncertainties considered were the residual oil saturation (ROS) to the surfactant flood, surfactant adsorption, and reservoir heterogeneity, and the main economic uncertain parameters considered were oil price, surfactant cost, and water-injection and -production costs. The results show that the LSM method provides a decision-making tool that was able to capture the value of flexibility in surfactant-flooding implementation and provides some insight into the effect of uncertainty on decision making, which can help mitigate adverse circumstances should they arise or capture the upside potential if circumstances prove beneficial. The results found that the optimal policy obtained was reliable and that heterogeneity and different well-placement patterns affect the value of flexibility and optimal policy for different reservoir models. Furthermore, possible extensions to enhance the LSM method were discussed.

Service Provision Control in Federated Service Providing Systems

Different from traditional P2P systems, individuals nodes of a Federated Service Providing (FSP) system play a more active role by offering a variety of domain-specific services. The service provision control (SPC) problem is an important problem of the FSP system and will be tackled in this paper within a stochastic optimization framework through several steps. The first step focuses on using stochastic differential equations (SDEs) to model and analyze the dynamic evolution of the service demand. Driven by the SDE model, expected future performance of a FSP system is analytically evaluated in the second step. Step three utilizes the differential evolution (DE) algorithm to identify near-optimal service-providing policies for each node. The service subscription protocol is further proposed in step four to help every node adjust its local policy in accordance with the services provided by other nodes. The four steps together implement a complete solution of the SPC problem and will be called the SDE-based service-provision control (SSPC) mechanism in this paper. Experimental evaluation of the mechanism has been reported in the paper. The results show that our approach is effective in tackling the SPC problem and may be therefore suitable for many practical applications.

Replenishment policies for empty containers in an inland multi-depot system

This research focuses on the problem of positioning empty containers in a port area with multiple depots. Three options are considered: positioning from other overseas ports, inland positioning between depots and leasing. The policies for empty-container management are as follows: a coordinated (s, S) inventory policy for overseas positioning, a (ri, Ri) policy at each depot for inland positioning, and a simple leasing policy with zero lead-time and infinite capacity. For the inland positioning policy, four different methods are proposed to reposition empty containers between depots. The objective is to obtain the optimal policy in order to minimize the expected total cost consisting of: inventory holding, overseas positioning, inland positioning and leasing costs. A simulation-based genetic algorithm is developed to find near-optimal policies. Numerical examples are given to demonstrate the effectiveness of the proposed approach and examine the sensitivity of results with respect to system parameters.

Optimal Policy for Inventory Transfer Between Two Depots With Backlogging

This technical note considers the optimal control problem of transferring empty containers between two depots over a multiperiod planning horizon to minimize the total cost comprising inventory holding costs, empty container transfer costs, and demand backlog costs. The problem involves random supply and random demand. Formulating the problem as a stochastic dynamic program, we show that the value function is not convex so the traditional method of analysis cannot be applied. We present an alternative approach by focusing on the local properties of the value function such as the first and second derivatives on a region-wise basis. This enables us to establish the structural characteristics of the optimal policy, e.g., several monotonic switching curves divide the state space into seven control regions. Based on the established structural properties, we develop a simple near-optimal policy. We provide a numerical example to illustrate the analytical results.

Dynamic local interaction model. Modèle et algorithms

This article introduces DyLIM, a model derivated from DEC-POMDP. DyLIM is an interaction based approach, in which the agents are supposed to interact only sometimes, with some other agents (contrary to classical approaches which are based on the assumption of permanent and strong interactions). Moreover, DyLIM deals with evolving interactions (as opposed to predefined ones). We are then able to describe the multiagent problem as a set of individual ones (sometimes interdependent), which breaks the combinatorial complexity. In a second time, we describe two different algorithms able to compute a solution for DyLIM, and we give experimental results based on a set of standard benchmarks. We finally show how our approach computes near-optimal policies, while scaling up to many agents.

Induced states in a decision tree constructed by Q-learning

This paper develops a tree-construction method based on the framework of reinforcement learning (RL). The induction of a decision tree is regarded as a problem of RL, where the optimal policy should be found to obtain the maximal accumulated information gain. The proposed approach consists of two stages. At the first stage, the emulation/demonstration stage, sensory-action data in a mechatronic system or samples of training patterns are generated by an operator or stimulator. The records of these emulation data are aggregated into components of the state space represented by a decision tree. State aggregation for decartelization of a state space consists of two phases: split estimation and tree growing. In the split estimation phase, an inducer estimates long-term evaluations of splits at visited nodes. In the second phase, the inducer grows the tree by the predicted long-term evaluations, which is approximated by a neural network model. At the second stage, the learned behavior or classifier is shaped by the RL scheme with a discretized state space constructed by the decision tree derived from the previous stage. Unlike the conventional greedy procedure for constructing and pruning a the tree, the proposed method casts the sequential process of tree induction to policy iterations, where policies for node split are evaluated and improved repeatedly until an optimal or near-optimal policy is obtained. The splitting criterion regarded as an action policy is based on long-term evaluations of payoff instead of using immediate evaluations on impurity. A comparison with CART (classification and regression tree) and C4.5 on several benchmark datasets is presented. Furthermore, to show its applications for learning control, the proposed method is applied further to construct a so-called tree-based reinforcement learning method, where the mechanism works with a discrete state space derived from the proposed method. The results show the feasibility and high performance of the proposed system as a state partition by comparison with the renowned Adaptive Heuristic Critic (AHC) model.

Sum-rate optimal power policies for energy harvesting transmitters in an interference channel

This paper considers a two-user Gaussian interference channel with energy harvesting transmitters. Different than conventional battery powered wireless nodes, energy harvesting transmitters have to adapt transmission to availability of energy at a particular instant. In this setting, the optimal power allocation problem to maximize the sum throughput with a given deadline is formulated. The convergence of the proposed iterative coordinate descent method for the problem is proved and the short-term throughput maximizing offline power allocation policy is found. Examples for interference regions with known sum capacities are given with directional waterfilling interpretations. Next, stochastic data arrivals are addressed. Finally, online and/or distributed near-optimal policies are proposed. Performance of the proposed algorithms are demonstrated through simulations.

Intelligent Control of a Sensor-Actuator System via Kernelized Least-Squares Policy Iteration

In this paper a new framework, called Compressive Kernelized Reinforcement Learning (CKRL), for computing near-optimal policies in sequential decision making with uncertainty is proposed via incorporating the non-adaptive data-independent Random Projections and nonparametric Kernelized Least-squares Policy Iteration (KLSPI). Random Projections are a fast, non-adaptive dimensionality reduction framework in which high-dimensionality data is projected onto a random lower-dimension subspace via spherically random rotation and coordination sampling. KLSPI introduce kernel trick into the LSPI framework for Reinforcement Learning, often achieving faster convergence and providing automatic feature selection via various kernel sparsification approaches. In this approach, policies are computed in a low-dimensional subspace generated by projecting the high-dimensional features onto a set of random basis. We first show how Random Projections constitute an efficient sparsification technique and how our method often converges faster than regular LSPI, while at lower computational costs. Theoretical foundation underlying this approach is a fast approximation of Singular Value Decomposition (SVD). Finally, simulation results are exhibited on benchmark MDP domains, which confirm gains both in computation time and in performance in large feature spaces.

Computing Near-Optimal Policies in Generalized Joint Replenishment

We provide a practical methodology for solving the generalized joint replenishment (GJR) problem, based on a mathematical programming approach to approximate dynamic programming. We show how to automatically generate a value function approximation basis built upon piecewise-linear ridge functions by developing and exploiting a theoretical connection with the problem of finding optimal cyclic schedules. We provide a variant of the algorithm that is effective in practice, and we exploit the special structure of the GJR problem to provide a coherent, implementable framework.

Constraint-Based Local Search for Inventory Control Under Stochastic Demand and Lead Time

In this paper, we address the general multiperiod production/inventory problem with nonstationary stochastic demand and supplier lead time under service-level constraints. A replenishment cycle policy is modeled. We propose two hybrid algorithms that blend constraint programming and local search for computing near-optimal policy parameters. Both algorithms rely on a coordinate descent local search strategy; what differs is the way this strategy interacts with the constraint programming solver. These two heuristics are first, compared for small instances against an existing optimal solution method. Second, they are tested and compared with each other in terms of solution quality and run time on a set of larger instances that are intractable for the exact approach. Our numerical experiments show the effectiveness of our methods.

Near-Optimal Tracking Control of a Nonholonomic Mobile Robot with Uncertainties

A combined kinematic/torque control law is developed by using a backstepping design approach for a nonholonomic mobile robot with two driving wheels mounted on the same axis to track a reference trajectory. The auxiliary velocity control inputs are designed for the kinematic steering system to make the posture error asymptotically stable. Next, a computed-torque controller is designed such that the mobile robot's velocities converge on the given velocity inputs in an optimal manner by converting the tracking control problem into the regulation problem whereby the uncertainties in the dynamics of mobile robots are considered. The proposed online and forward-in-time policy iteration (PI) algorithm based on approximate dynamic programming (ADP) is used to solve the optimal control problem with unknown internal dynamics by using single neural networks (NNs) to approximate the cost function. Afterwards, the near-optimal control policy can be computed directly according to the cost function, which removes the action network appearing in the ordinary ADP method. The stability of the dynamical extension system is demonstrated using Lyapunov methods. The simulation results are provided to demonstrate the effectiveness of the proposed approach.