Large Markov Decision Processes Research Articles

A new parallelized of hierarchical value iteration algorithm for discounted Markov decision processes

Markov Decision Process (MDP) is a popular mathematical framework for modeling stochastic sequential problems under uncertainty. These models appear in many applications, such as computer science, engineering, telecommunications, and finance, among others. One of the most challenging goals is to deal with complexity reduction in the case of large MDP. In this paper; we propose an optimal strategy deals with large MDP under discount reward. The proposed approach is based on an intelligent combination of a decomposition technique and an efficient parallel strategy. The global MDP is splitting into several 'sub-MDPs', subsequently, these MDPs are classified by level following the strongly connected components principle. A master-slave strategy base on Message Passing Interface (MPI) is proposed to solve the obtained problem. The performance of the proposed approach is shown in terms of scalability, cost, and execution speed.

Decomposition Methods for Solving Finite‐Horizon Large MDPs

Conventional algorithms for solving Markov decision processes (MDPs) become intractable for a large finite state and action spaces. Several studies have been devoted to this issue, but most of them only treat infinite‐horizon MDPs. This paper is one of the first works to deal with non‐stationary finite‐horizon MDPs by proposing a new decomposition approach, which consists in partitioning the problem into smaller restricted finite‐horizon MDPs, each restricted MDP is solved independently, in a specific order, using the proposed hierarchical backward induction (HBI) algorithm based on the backward induction (BI) algorithm. Next, the sub‐local solutions are combined to obtain a global solution. An example of racetrack problems shows the performance of the proposal decomposition technique.

Leveraging experience in lazy search

Lazy graph search algorithms are efficient at solving motion planning problems where edge evaluation is the computational bottleneck. These algorithms work by lazily computing the shortest potentially feasible path, evaluating edges along that path, and repeating until a feasible path is found. The order in which edges are selected is critical to minimizing the total number of edge evaluations: a good edge selector chooses edges that are not only likely to be invalid, but also eliminates future paths from consideration. We wish to learn such a selector by leveraging prior experience. We formulate this problem as a Markov Decision Process (MDP) on the state of the search problem. While solving this large MDP is generally intractable, we show that we can compute oracular selectors that can solve the MDP during training. With access to such oracles, we use imitation learning to find effective policies. If new search problems are sufficiently similar to problems solved during training, the learned policy will choose a good edge evaluation ordering and solve the motion planning problem quickly. We evaluate our algorithms on a wide range of 2D and 7D problems and show that the learned selector outperforms baseline commonly used heuristics. We further provide a novel theoretical analysis of lazy search in a Bayesian framework as well as regret guarantees on our imitation learning based approach to motion planning.

UAV-Aided Information and Energy Transmissions for Cognitive and Sustainable 5G Networks

To develop sustainable fifth generation (5G) wireless networks and utilize the unused spectrum, this paper focuses on cognitive radio (CR) based wireless information and energy transmissions from an unmanned aerial vehicle (UAV) to multiple low-power ground terminals (GTs). By practically considering the location-dependent air-to-ground (A2G) channel states and the non-linear energy harvesting (EH), we propose a dynamic fly-hover-transmit scheme, where the UAV successively flies between GTs, and hovers close to each GT for efficient wireless energy transfer (WET) or wireless information transfer (WIT) when the primary user (PU) is idle. By causally and optimally determining the UAV’s mobility and transmit power for each selected transmission mode (WIT, WET, or being silent), we formulate the UAV’s sum-throughput maximization over all GTs as a constrained Markov decision process (MDP) problem with battery energy constraints at all GTs and the UAV. Due to the infinitely large MDP system state space, this problem is difficult to solve. We then decompose this problem into two subproblems, by first deciding the UAV’s transmission mode and power above a given GT, and then optimizing the UAV movement policy over multiple GTs. In the first subproblem, we propose an approximate to the complicated MDP value function of low complexity in closed-form, and then analytically derive the threshold-based suboptimal transmission policies. In the second subproblem, we optimally solve a simple-but-fundamental two-GT case, and then extend the general location-dependent GT weight design to an efficient suboptimal UAV movement policy. Simulation results show the significantly improved system performance under the proposed suboptimal policies over various benchmarks in dynamic networks.

A deep learning approach for the dynamic dispatching of unreliable machines in re-entrant production systems

This research combines deep neural network (DNN) and Markov decision processes (MDP) for the dynamic dispatching of re-entrant production systems. In re-entrant production systems, jobs enter the same workstation multiple times and dynamic dispatching oftentimes aims to dynamically assign different priorities to various job groups to minimise weighted cycle time or maximise throughput. MDP is an effective tool for dynamic production control, but it suffers from two major challenges in dynamic control problems. First, the curse of dimensionality limits the computational performance of solving large MDP problems. Second, a different model should be built and solved after system configuration is changed. DNN is used to overcome both challenges by learning directly from optimal dispatching policies generated by MDP. Results suggest that a properly trained DNN model can instantly generate near-optimal dynamic control policies for large problems. The quality of the DNN solution is compared with the optimal dynamic control policies through the standard K-fold cross-validation test and discrete event simulation. On average, the performance of the DNN policy is within 2% of optimal in both tests. The proposed artificial intelligence algorithm illustrates the potential of machine learning methods in manufacturing applications.

Optimal policy for structure maintenance: A deep reinforcement learning framework

The cost-effective management of aged infrastructure is an issue of worldwide concern. Markov decision process (MDP) models have been used in developing structural maintenance policies. Recent advances in the artificial intelligence (AI) community have shown that deep reinforcement learning (DRL) has the potential to solve large MDP optimization tasks. This paper proposes a novel automated DRL framework to obtain an optimized structural maintenance policy. The DRL framework contains a decision maker (AI agent) and the structure that needs to be maintained (AI task environment). The agent outputs maintenance policies and chooses maintenance actions, and the task environment determines the state transition of the structure and returns rewards to the agent under given maintenance actions. The advantages of the DRL framework include: (1) a deep neural network (DNN) is employed to learn the state-action Q value (defined as the predicted discounted expectation of the return for consequences under a given state-action pair), either based on simulations or historical data, and the policy is then obtained from the Q value; (2) optimization of the learning process is sample-based so that it can learn directly from real historical data collected from multiple bridges (i.e., big data from a large number of bridges); and (3) a general framework is used for different structure maintenance tasks with minimal changes to the neural network architecture. Case studies for a simple bridge deck with seven components and a long-span cable-stayed bridge with 263 components are performed to demonstrate the proposed procedure. The results show that the DRL is efficient at finding the optimal policy for maintenance tasks for both simple and complex structures.

Migration Modeling and Learning Algorithms for Containers in Fog Computing

Fog Computing (FC) is a flexible architecture to support distributed domain-specific applications with cloud-like quality of service. However, current FC still lacks the mobility support mechanism when facing many mobile users with diversified application quality requirements. Such mobility support mechanism can be critical such as in the industrial internet where human, products, and devices are moveable. To fill in such gaps, in this paper we propose novel container migration algorithms and architecture to support mobility tasks with various application requirements. Our algorithms are realized from three aspects: 1) We consider mobile application tasks can be hosted in a container of a corresponding fog node that can be migrated, taking the communication delay and computational power consumption into consideration; 2) We further model such container migration strategy as multiple dimensional Markov Decision Process (MDP) spaces. To effectively reduce the large MDP spaces, efficient deep reinforcement learning algorithms are devised to achieve fast decision-making and 3) We implement the model and algorithms as a container migration prototype system and test its feasibility and performance. Extensive experiments show that our strategy outperforms the existing baseline approaches 2.9, 48.5 and 58.4 percent on average in terms of delay, power consumption, and migration cost, respectively.

Offline reinforcement learning with task hierarchies

In this work, we build upon the observation that offline reinforcement learning (RL) is synergistic with task hierarchies that decompose large Markov decision processes (MDPs). Task hierarchies can allow more efficient sample collection from large MDPs, while offline algorithms can learn better policies than the so-called “recursively optimal” or even hierarchically optimal policies learned by standard hierarchical RL algorithms. To enable this synergy, we study sample collection strategies for offline RL that are consistent with a provided task hierarchy while still providing good exploration of the state-action space. We show that naive extensions of uniform random sampling do not work well in this case and design a strategy that has provably good convergence properties. We also augment the initial set of samples using additional information from the task hierarchy, such as state abstraction. We use the augmented set of samples to learn a policy offline. Given a capable offline RL algorithm, this policy is then guaranteed to have a value greater than or equal to the value of the hierarchically optimal policy. We evaluate our approach on several domains and show that samples generated using a task hierarchy with a suitable strategy allow significantly more sample-efficient convergence than standard offline RL. Further, our approach also shows more sample-efficient convergence to policies with value greater than or equal to hierarchically optimal policies found through an online hierarchical RL approach.

Accelerated decomposition techniques for large discounted Markov decision processes

Many hierarchical techniques to solve large Markov decision processes (MDPs) are based on the partition of the state space into strongly connected components (SCCs) that can be classified into some levels. In each level, smaller problems named restricted MDPs are solved, and then these partial solutions are combined to obtain the global solution. In this paper, we first propose a novel algorithm, which is a variant of Tarjan’s algorithm that simultaneously finds the SCCs and their belonging levels. Second, a new definition of the restricted MDPs is presented to ameliorate some hierarchical solutions in discounted MDPs using value iteration (VI) algorithm based on a list of state-action successors. Finally, a robotic motion-planning example and the experiment results are presented to illustrate the benefit of the proposed decomposition algorithms.

Markov Decision Process-Based Distributed Conflict Resolution for Drone Air Traffic Management

Ensuring safety and providing timely conflict alerts to small unmanned aircraft, commonly known as drones, is important to their integration into civil airspace. This paper proposes a short-term conflict avoidance algorithm for an automated low-altitude, small unmanned aircraft traffic management system. The goal is to balance between aircraft safety and efficiency subject to uncertainty in the environment, aircraft, and pilot response. The proposed algorithm generates advisories for each aircraft to follow, and is based on decomposing a large multiagent Markov decision process and fusing their solutions. As a result, the method scales well and resolves conflicts efficiently. Further, this paper presents a massively distributed architecture used to implement the conflict avoidance algorithm on a practical scale in which simultaneous conflicts can be solved in parallel in tens of milliseconds. Our controller significantly outperforms two baseline algorithms based on uncoordinated and closest-threat heuristics. In terms of conflict probability, our simulations demonstrate that our method is an order of magnitude better than the latter and about 10% better than the former.

A Modified Policy Iteration Algorithm for Discounted Reward Markov Decision Processes

The running time of the classical algorithms of the Markov Decision Process (MDP) typically grows linearly with the state space size, which makes them frequently intractable. This paper presents a Modified Policy Iteration algorithm to compute an optimal policy for large Markov decision processes in the discounted reward criteria and under infinite horizon. The idea of this algorithm is based on the topology of the problem; moreover, an Open Multi-Processing (Open-MP) programming model is applied to attain efficient parallel performance in solving the Modified algorithm. General Terms Theoretical Informatics, Parallelizing ...

Sleeping experts and bandits approach to constrained Markov decision processes

This communique presents simple simulation-based algorithms for obtaining an approximately optimal policy in a given finite set in large finite constrained Markov decision processes. The algorithms are adapted from playing strategies for “sleeping experts and bandits” problem and their computational complexities are independent of state and action space sizes if the given policy set is relatively small. We establish convergence of their expected performances to the value of an optimal policy and convergence rates, and also almost-sure convergence to an optimal policy with an exponential rate for the algorithm adapted within the context of sleeping experts.

Online Planning for Large Markov Decision Processes with Hierarchical Decomposition

Markov decision processes (MDPs) provide a rich framework for planning under uncertainty. However, exactly solving a large MDP is usually intractable due to the “curse of dimensionality”— the state space grows exponentially with the number of state variables. Online algorithms tackle this problem by avoiding computing a policy for the entire state space. On the other hand, since online algorithm has to find a near-optimal action online in almost real time, the computation time is often very limited. In the context of reinforcement learning, MAXQ is a value function decomposition method that exploits the underlying structure of the original MDP and decomposes it into a combination of smaller subproblems arranged over a task hierarchy. In this article, we present MAXQ-OP—a novel online planning algorithm for large MDPs that utilizes MAXQ hierarchical decomposition in online settings. Compared to traditional online planning algorithms, MAXQ-OP is able to reach much more deeper states in the search tree with relatively less computation time by exploiting MAXQ hierarchical decomposition online. We empirically evaluate our algorithm in the standard Taxi domain—a common benchmark for MDPs—to show the effectiveness of our approach. We have also conducted a long-term case study in a highly complex simulated soccer domain and developed a team named WrightEagle that has won five world champions and five runners-up in the recent 10 years of RoboCup Soccer Simulation 2D annual competitions. The results in the RoboCup domain confirm the scalability of MAXQ-OP to very large domains.

A Modified Value Iteration Algorithm for Discounted Markov Decision Processes

As many real applications need a large amount of states, the classical methods are intractable for solving large Markov Decision Processes. The decomposition technique basing on the topology of each state in the associated graph and the parallelization technique are very useful methods to cope with this problem. In this paper, the authors propose a Modified Value Iteration algorithm, adding the parallelism technique. They test their implementation on artificial data using an Open MP that offers a significant speed-up.

Approximate planning and verification for large Markov decision processes

We focus on the planning and verification problems for very large probabilistic systems, such as Markov decision processes (MDPs), from a complexity point of view. More precisely, we deal with the problem of designing an efficient approximation method to compute a near-optimal policy for the planning problem in discounted MDPs and the satisfaction probabilities of interesting properties, like reachability or safety, over the Markov chain obtained by restricting the MDP to the near-optimal policy. In this paper, we present two different approaches. The first one is based on sparse sampling while the second uses a variant of the multiplicative weights update algorithm. The complexity of the first approximation method is independent of the size of the state space and uses only a probabilistic generator of the MDP. We give a complete analysis of this approach, for which the control parameter is mainly the targeted quality of the approximation. The second approach is more prospective and is different in the sense that the method can be controlled dynamically by observing its speed of convergence. Parts of this paper have already been presented in Lassaigne and Peyronnet (in Proceedings of the ACM Symposium on applied computing, SAC 2012, pp 1314---1319, ACM 2012), by the same authors.

Scalable approximate policies for Markov decision process models of hospital elective admissions.

To demonstrate the feasibility of using stochastic simulation methods for the solution of a large-scale Markov decision process model of on-line patient admissions scheduling. The problem of admissions scheduling is modeled as a Markov decision process in which the states represent numbers of patients using each of a number of resources. We investigate current state-of-the-art real time planning methods to compute solutions to this Markov decision process. Due to the complexity of the model, traditional model-based planners are limited in scalability since they require an explicit enumeration of the model dynamics. To overcome this challenge, we apply sample-based planners along with efficient simulation techniques that given an initial start state, generate an action on-demand while avoiding portions of the model that are irrelevant to the start state. We also propose a novel variant of a popular sample-based planner that is particularly well suited to the elective admissions problem. Results show that the stochastic simulation methods allow for the problem size to be scaled by a factor of almost 10 in the action space, and exponentially in the state space. We have demonstrated our approach on a problem with 81 actions, four specialities and four treatment patterns, and shown that we can generate solutions that are near-optimal in about 100s. Sample-based planners are a viable alternative to state-based planners for large Markov decision process models of elective admissions scheduling.

SOLVING THE SAILING PROBLEM WITH A NEW PRIORITIZED VALUE ITERATION

In this paper we tackle the sailing strategies problem, a stochastic shortest-path Markov decision process. The problem of solving large Markov decision processes accurately and quickly is challenging. Because the computational effort incurred is considerable, current research focuses on finding superior acceleration techniques. For instance, the convergence properties of current solution methods depend, to a great extent, on the order of backup operations. On one hand, algorithms such as topological sorting are able to find good orderings, but their overhead is usually high. On the other hand, shortest path methods, such as Dijkstra's algorithm, which is based on priority queues, have been applied successfully to the solution of deterministic shortest-path Markov decision processes. Here, we propose improved value iteration algorithms based on Dijkstra's algorithm for solving shortest path Markov decision processes. The experimental results on a stochastic shortest-path problem show the feasibility of our approach.

Linear Dynamic Programs for Resource Management

Sustainable resource management in many domains presents large continuous stochastic optimization problems, which can often be modeled as Markov decision processes (MDPs). To solve such large MDPs, we identify and leverage linearity in state and action sets that is common in resource management. In particular, we introduce linear dynamic programs (LDPs) that generalize resource management problems and partially observable MDPs (POMDPs). We show that the LDP framework makes it possible to adapt point-based methods--the state of the art in solving POMDPs--to solving LDPs. The experimental results demonstrate the efficiency of this approach in managing the water level of a river reservoir. Finally, we discuss the relationship with dual dynamic programming, a method used to optimize hydroelectric systems.

New prioritized value iteration for Markov decision processes

The problem of solving large Markov decision processes accurately and quickly is challenging. Since the computational effort incurred is considerable, current research focuses on finding superior acceleration techniques. For instance, the convergence properties of current solution methods depend, to a great extent, on the order of backup operations. On one hand, algorithms such as topological sorting are able to find good orderings but their overhead is usually high. On the other hand, shortest path methods, such as Dijkstra's algorithm which is based on priority queues, have been applied successfully to the solution of deterministic shortest-path Markov decision processes. Here, we propose an improved value iteration algorithm based on Dijkstra's algorithm for solving shortest path Markov decision processes. The experimental results on a stochastic shortest-path problem show the feasibility of our approach.

Using Bisimulation for Policy Transfer in MDPs

Knowledge transfer has been suggested as a useful approach for solving large Markov Decision Processes. The main idea is to compute a decision-making policy in one environment and use it in a different environment, provided the two are ”close enough”. In this paper, we use bisimulation-style metrics (Ferns et al., 2004) to guide knowledge transfer. We propose algorithms that decide what actions to transfer from the policy computed on a small MDP task to a large task, given the bisimulation distance between states in the two tasks. We demonstrate the inherent ”pessimism” of bisimulation metrics and present variants of this metric aimed to overcome this pessimism, leading to improved action transfer. We also show that using this approach for transferring temporally extended actions (Sutton et al., 1999) is more successful than using it exclusively with primitive actions. We present theoretical guarantees on the quality of the transferred policy, as well as promising empirical results.