Large-scale Markov Decision Process Research Articles

Sketched Newton Value Iteration for Large-Scale Markov Decision Processes

Value Iteration (VI) is one of the most classic algorithms for solving Markov Decision Processes (MDPs), which lays the foundations for various more advanced reinforcement learning algorithms, such as Q-learning. VI may take a large number of iterations to converge as it is a first-order method. In this paper, we introduce the Newton Value Iteration (NVI) algorithm, which eliminates the impact of action space dimension compared to some previous second-order methods. Consequently, NVI can efficiently handle MDPs with large action spaces. Building upon NVI, we propose a novel approach called Sketched Newton Value Iteration (SNVI) to tackle MDPs with both large state and action spaces. SNVI not only inherits the stability and fast convergence advantages of second-order algorithms, but also significantly reduces computational complexity, making it highly scalable. Extensive experiments demonstrate the superiority of our algorithms over traditional VI and previously proposed second-order VI algorithms.

Energy-Efficient Joint Task Assignment and Migration in Data Centers: A Deep Reinforcement Learning Approach

Energy-efficient task scheduling in data centers is a critical issue and has drawn wide attention. However, the task execution times are mixed and hard to estimate in a real-world data center. It has been conspicuously neglected by existing work that scheduling decisions made at tasks’ arrival times are likely to cause energy waste or idle resources over time. To fill in such gaps, in this paper, we jointly consider assignment and migration for mixed duration tasks and devise a novel energy-efficient task scheduling algorithm. Task assignment can improve resource utilization, and migration is required when long-running tasks run in low-load servers. Specifically: 1) We formulate mixed duration task scheduling as a large-scale Markov Decision Process (MDP) problem; 2) To solve such a large-scale MDP problem, we design an efficient Deep Reinforcement Learning (DRL) algorithm to make assignment and migration decisions. To make the DRL algorithm more practical in real scenarios, multiple optimizations are proposed to achieve online training; 3) Experiments with real-world data have shown that our algorithm outperforms the existing baselines 14% on average in terms of energy consumption while keeping the same level of Quality of Service (QoS).

Matching Uncertain Renewable Supply with Electric Vehicle Charging Demand—A Bi-Level Event-Based Optimization Method

The matching between dynamic supply of renewable power generation and flexible charging demand of the Electric Vehicles (EVs) can not only increase the penetration of renewables but also reduce the load to the state electric power grid. The challenges herein are the curse of dimensionality, the multiple decision making stages involved, and the uncertainty of both the supply and demand sides. Event-Based Optimization (EBO) provides a new way to solve large-scale Markov decision process. Considering different spatial scales, we develop a bi-level EBO model in this paper which can both catch the changes on the macro and micro levels. By proper definition, the size of event space stays fixed with the scale of the problem, which shows good scalability in online optimization. Then a bi-level Q-learning method is developed to solve the problem iteratively. We demonstrate the performance of the method by numerical examples. Our method outperforms other methods both in performance and scalability.

Energy-Efficient Dynamic Virtual Machine Management in Data Centers

Efficient virtual machine (VM) management can dramatically reduce energy consumption in data centers. Existing VM management algorithms fall into two categories based on whether the VMs’ resource demands are assumed to be static or dynamic. The former category fails to maximize the resource utilization as they cannot adapt to the dynamic nature of VMs’ resource demands. Most approaches in the latter category are heuristic and lack theoretical performance guarantees. In this paper, we formulate the dynamic VM management as a large-scale Markov decision process (MDP) problem and derive an optimal solution. Our analysis of real-world data traces supports our choice of the modeling approach. However, solving the large-scale MDP problem suffers from the curse of dimensionality. Therefore, we further exploit the special structure of the problem and propose an approximate MDP-based dynamic VM management method, called MadVM. We prove the convergence of MadVM and analyze the bound of its approximation error. Moreover, we show that MadVM can be implemented in a distributed system with at most two times of the optimal migration cost. Extensive simulations based on two real-world workload traces show that MadVM achieves significant performance gains over two existing baseline approaches in power consumption, resource shortage, and the number of VM migrations. Specifically, the more intensely the resource demands fluctuate, the more MadVM outperforms.

Approximate Value Iteration for Risk-Aware Markov Decision Processes

We consider large-scale Markov decision processes (MDPs) with a time-consistent risk measure of variability in cost under the risk-aware MDP paradigm. Previous studies showed that risk-aware MDPs, based on a minimax approach to handling risk, can be solved using dynamic programming for small- to medium-sized problems. However, due to the “curse of dimensionality,” MDPs that model real-life problems are typically prohibitively large for such approaches. In this technical note, we employ an approximate dynamic programming approach and develop a family of simulation-based algorithms to approximately solve large-scale risk-aware MDPs with time-consistent risk measures. In parallel, we develop a unified convergence analysis technique to derive sample complexity bounds for this new family of algorithms.

Timing it Right: Balancing Inpatient Congestion Versus Readmission Risk at Discharge

When to discharge a patient plays an important role in hospital patient flow management as well as the quality of care and patient outcomes. In this work, we develop and implement a data-integrated decision support framework to aid hospitals in managing the delicate balance between readmission risk at discharge and ward congestion. We formulate a large-scale Markov Decision Process (MDP) that integrates a personalized readmission prediction model to dynamically prescribe both how many and which patients to discharge on each day. Due to patient heterogeneity and the fact that length-of-stay is not memoryless, the MDP suffers the curse of dimensionality. We leverage structural properties and analytical solutions for a special cost setting to transform the MDP into a univariate optimization; this leads to a novel, efficient dynamic heuristic. Further, for our decision framework to be implementable in practice, we build a unified prediction model that integrates several statistical methods and provides key inputs to the decision framework; existing off-the-shelf readmission prediction models alone could not adequately parametrize our decision support. Through extensive counterfactual analyses, we demonstrate the value of our discharge decision tool over our partner hospital's historical discharge behavior. We also obtain generalizable insights by applying the tool to a broad range of hospital types through a high-fidelity simulation. Lastly, we showcase an implementation of our tool at our partner hospital, to demonstrate broader applicability through our framework's plug-and-play design for integration with general hospital data-systems and workflows.

CoSchd: Coordinated Scheduling With Channel and Load Awareness for Alleviating Cellular Congestion

Although cellular networks can be provisioned according to the peak demand, they usually experience large fluctuations in both channel conditions and traffic load level. Scheduling with both channel and load awareness allows us to exploit the delay tolerance of data traffic to alleviate network congestion, and thus reduce the peak. However, solving the optimal scheduling problem leads to a large-scale Markov decision process (MDP) with extremely high complexity. In this paper, we propose a scalable and distributed approach to this problem, called Coordinated Scheduling (CoSchd). CoSchd decomposes the large-scale MDP problem into many individual MDP problems, each of which can be solved independently by each user under a limited amount of coordination signals from the base station (BS). We show that CoSchd is close to optimal when the number of users becomes large. Furthermore, we propose an approximation of CoSchd that iteratively updates the scheduling policy based on online measurements. Simulation results demonstrate that exploiting channel and load awareness with CoSchd can effectively alleviate cellular network congestion.

Learning the variance of the reward-to-go

In Markov decision processes (MDPs), the variance of the reward-to-go is a natural measure of uncertainty about the long term performance of a policy, and is important in domains such as finance, resource allocation, and process control. Currently however, there is no tractable procedure for calculating it in large scale MDPs. This is in contrast to the case of the expected reward-to-go, also known as the value function, for which effective simulation-based algorithms are known, and have been used successfully in various domains. In this paper we extend temporal difference (TD) learning algorithms to estimating the variance of the reward-to-go for a fixed policy. We propose variants of both TD(0) and LSTD(λ) with linear function approximation, prove their convergence, and demonstrate their utility in an option pricing problem. Our results show a dramatic improvement in terms of sample efficiency over standard Monte-Carlo methods, which are currently the state-of-the-art.

Efficient Algorithms for Budget-Constrained Markov Decision Processes

Discounted, discrete-time, discrete state-space, discrete action-space Markov decision processes (MDPs) form a classical topic in control, game theory, and learning, and as a result are widely applied, increasingly, in very large-scale applications. Many algorithms have been developed to solve large-scale MDPs. Algorithms based on value iteration are particularly popular, as they are more efficient than the generic linear programming approach, by an order of magnitude in the number of states of the MDP. Yet in the case of budget constrained MDPs, no more efficient algorithm than linear programming is known. The theoretically slower running times of linear programming may limit the scalability of constrained MDPs piratically; while, theoretically, it invites the question of whether the increase is somehow intrinsic. In this technical note we show that it is not, and provide two algorithms for budget-constrained MDPs that are as efficient as value iteration. Denoting the running time of value iteration by VI, and the magnitude of the input by U, for an MDP with m expected budget constraints our first algorithm runs in time O(poly(m, log U) · VI). Given a pre-specified degree of precision, η, for satisfying the budget constraints, our second algorithm runs in time O(log m · poly(log U) · (1/η 2 ) · VI), but may produce solutions that overutilize each of the m budgets by a multiplicative factor of 1 + η. In fact, one can substitute value iteration with any algorithm, possibly specially designed for a specific MDP, that solves the MDP quickly to achieve similar theoretical guarantees. Both algorithms restrict attention to constrained infinite-horizon MDPs under discounted costs.

Scalable approximate policies for Markov decision process models of hospital elective admissions.

To demonstrate the feasibility of using stochastic simulation methods for the solution of a large-scale Markov decision process model of on-line patient admissions scheduling. The problem of admissions scheduling is modeled as a Markov decision process in which the states represent numbers of patients using each of a number of resources. We investigate current state-of-the-art real time planning methods to compute solutions to this Markov decision process. Due to the complexity of the model, traditional model-based planners are limited in scalability since they require an explicit enumeration of the model dynamics. To overcome this challenge, we apply sample-based planners along with efficient simulation techniques that given an initial start state, generate an action on-demand while avoiding portions of the model that are irrelevant to the start state. We also propose a novel variant of a popular sample-based planner that is particularly well suited to the elective admissions problem. Results show that the stochastic simulation methods allow for the problem size to be scaled by a factor of almost 10 in the action space, and exponentially in the state space. We have demonstrated our approach on a problem with 81 actions, four specialities and four treatment patterns, and shown that we can generate solutions that are near-optimal in about 100s. Sample-based planners are a viable alternative to state-based planners for large Markov decision process models of elective admissions scheduling.

A Fast Algorithm for Large-Scale MDP-Based Systems in Smart Grid

In this study, we investigate the fast algorithms for the Large-Scale Markov Decision Process (LSMDP) problem in smart gird. Markov decision process is one of the efficient mathematical tools to solve the control and optimization problems in wireless smart grid systems. However, the complexity and the memory requirements exponentially increase when the number of system state grows in. Moreover, the limited computational ability and small size of memory on board constraint the application of wireless smart grid systems. As a result, it is impractical to implement those LSMDP-based approaches in such systems. Therefore, we propose the fast algorithm with low computational overhead and good performance in this study. We first derive the factored MDP representation, which substitutes LSMDP in a compact way. Based on the factored MDP, we propose the fast algorithm, which considerably reduces the size of state space and remains reasonable performance compared to the optimal solution.

On State Aggregation to Approximate Complex Value Functions in Large-Scale Markov Decision Processes

Many small electronic devices such as cell phones and wireless sensors have restrictive memory space, computing power, and battery. The pervasive applications of these devices in industry, military, and our daily lives require simple policies that are easy to implement, and can be executed in a reasonably short time. The Markov decision process (MDP) provides a general framework for these policy optimization problems with complexity constraint. In many cases, we can use a powerful computer to find the optimal (or a good) policy and the value function first, and then approximate by a simple one. The approximation usually depends on heuristics or experiences because the relationship between the complexity of a function and the approximation error is not clear in general. In this paper we assume the optimal value function is known (or a reasonably good estimate is available) and consider how to approximate a complex value function. Due to the broad application of state aggregation in the large-scale MDP, we focus on piecewise constant approximate value functions and use the number of aggregated states to measure the complexity of a value function. We quantify how the complexity of a value function affects the approximation error. When the optimal value function is known for sure we develop an algorithm that finds the best simple state aggregation within polynomial time. When we have estimates of the optimal value function, we apply ordinal optimization to find good simple state aggregations with high probability. The algorithms are demonstrated on a node activation policy optimization problem in wireless sensor network. We hope this work can shed some insight on how to find simple policies with good performances.

A Multi-Step Reinforcement Learning Algorithm

Reinforcement learning (RL) is a state or action value based machine learning method which approximately solves large-scale Markov Decision Process (MDP) or Semi-Markov Decision Process (SMDP). A multi-step RL algorithm called Sarsa(,k) is proposed, which is a compromised variation of Sarsa and Sarsa(). It is equivalent to Sarsa if k is 1 and is equivalent to Sarsa() if k is infinite. Sarsa(,k) adjust its performance by setting k value. Two forms of Sarsa(,k), forward view Sarsa(,k) and backward view Sarsa(,k), are constructed and proved equivalent in off-line updating.

Battery-Aware Adaptive Modulation Based on Large-Scale MDP

We treat the problem of designing the optimal transmission scheme that is adapted to the battery state, the channel and buffer conditions, and the incoming traffic rate. We assume that the battery states can be tracked at every time slot, so that the problem is formulated as a large-scale Markov decision process (MDP). An efficient sparse sampling method is employed to obtain a solution. Simulation results are provided to demonstrate that the proposed schemes can considerably increase the lifetime of the battery-powered wireless systems while satisfying QoS (quality of Service) constraints.

Algorithms for aggregated limiting average Markov decision problems

We consider discrete time Markov Decision Process (MDP) with finite state and action spaces under average reward optimality criterion. The decomposition theory, in Ross and Varadarajan [11], leads to a natural partition of the state space into strongly communicating classes and a set of states that are transient under all stationary strategies. Then, an optimal pure strategy can be obtained from an optimal strategy for some smaller aggregated MDP. This decomposition gives an efficient method for solving large-scale MDPs. In this paper, we consider deterministic MDPs and we construct a simple algorithm, based on graph theory, to determine an aggregated optimal policy. In the case of MDPs without cycles, we propose an algorithm for computing aggregated optimal strategies. In the general case, we propose some new improving algorithms for computing aggregated optimal strategies.

Simulation-based policy generation using large-scale Markov decision processes

This paper presents a new problem-solving approach, termed simulation-based policy generation (SPG), that is able to generate solutions to problems that may otherwise be computationally intractable. The SPG method uses a simulation of the original problem to create an approximating Markov decision process (MDP) model which is then solved via traditional MDP solution approaches. Since this approximating MDP is a fairly rich and robust sequential optimization model, solution policies can be created which represent an intelligent and structured search of the policy space. An important feature of the SPG approach is its adaptive nature, in that it uses the original simulation model to generate improved aggregation schemes, allowing the approach to be applied in situations where the underlying problem structure is largely unknown. In order to illustrate the performance of the SPG methodology, we apply it to a common but computationally complex problem of inventory control, and we briefly discuss its application to a large-scale telephone network routing problem.

Computation reduction for a large-scale Markov process model to analyse multistage production systems

In order to solve a large-scale Markov decision process which optimizes the control strategy of a multistaged production/inventory system, a computation reduction and memory management system was used. A ‘nuclear’ state space of relatively small size was found to substitute for the original state space. A circular comparison technique and a location index storage method were used to save computer memory used for storing the model

Suboptimal policy determination for large-scale Markov decision processes, Part 1: Description and bounds

This paper is the first of two papers that present and evaluate an approach for determining suboptimal policies for large-scale Markov decision processes (MDP). Part 1 is devoted to the determination of bounds that motivate the development and indicate the quality of the suboptimal design approach; Part 2 is concerned with the implementation and evaluation of the suboptimal design approach. The specific MDP considered is the infinite-horizon, expected total discounted cost MDP with finite state and action spaces. The approach can be described as follows. First, the original MDP is approximated by a specially structured MDP. The special structure suggests how to construct associated smaller, more computationally tractable MDP's. The suboptimal policy for the original MDP is then constructed from the solutions of these smaller MDP's. The key feature of this approach is that the state and action space cardinalities of the smaller MDP's are exponential reductions of the state and action space cardinalities of the original MDP.

Suboptimal policy determination for large-scale Markov decision processes, part 2: Implementation and numerical evaluation

We present an implementation of the procedure for determining a suboptimal policy for a large-scale Markov decision process (MDP) presented in Part 1. An operation count analysis illuminates the significant computational benefits of this procedure for determining an optimal policy relative to a procedure for determining a suboptimal policy based on state and action space aggregation. Results of a preliminary numerical study indicate that the quality of the suboptimal policy produced by the 3MDP approach shows promise.

Computation techniques for large scale undiscounted markov decision processes

AbstractIn this paper we consider computation techniques associated with the optimization of large scale Markov decision processes. Markov decision processes and the successive approximation procedure of White are described. Then a procedure for scaling continuous time and renewal processes so that they are amenable to the White procedure is discussed. The effect of the scale factor value on the convergence rate of the procedure and insights into proper scale factor selection are given.