Markov Decision Process Algorithm Research Articles

A Q-learning algorithm for Markov decision processes with continuous state spaces

We propose an online algorithm for solving a class of continuous-state Markov decision processes. The algorithm combines classical Q-learning with an asynchronous averaging procedure, which allows Q-function estimates at sampled state–action pairs to be adaptively updated based on observations collected along a single sample trajectory. These estimates are then used to iteratively construct an interpolation-based function approximator of the Q-function. We prove the convergence of the algorithm and provide numerical results to illustrate its performance.

Efficient Computation of Optimal Thresholds in Cloud Auto-scaling Systems

We consider a horizontal and dynamic auto-scaling technique in a cloud system where virtual machines hosted on a physical node are turned on and off to minimise energy consumption while meeting performance requirements. Finding cloud management policies that adapt the system to the load is not straightforward, and we consider here that virtual machines are turned on and off depending on queue load thresholds. We want to compute the optimal threshold values that minimize consumption costs and penalty costs (when performance requirements are not met). To solve this problem, we propose several optimisation methods, based on two different mathematical approaches. The first one is based on queueing theory and uses local search heuristics coupled with the stationary distributions of Markov chains. The second approach tackles the problem using Markov Decision Process (MDP) in which we assume that the policy is of a special multi-threshold type called hysteresis. We improve the heuristics of the former approach with the aggregation of Markov chains and queues approximation techniques. We assess the benefit of threshold-aware algorithms for solving MDPs. Then we carry out theoretical analyzes of the two approaches. We also compare them numerically and we show that all of the presented MDP algorithms strongly outperform the local search heuristics. Finally, we propose a cost model for a real scenario of a cloud system to apply our optimisation algorithms and to show their practical relevance. The major scientific contribution of the article is a set of fast (almost in real time) load-based threshold computation methods that can be used by a cloud provider to optimize its financial costs.

An immediate-return reinforcement learning for the atypical Markov decision processes.

The atypical Markov decision processes (MDPs) are decision-making for maximizing the immediate returns in only one state transition. Many complex dynamic problems can be regarded as the atypical MDPs, e.g., football trajectory control, approximations of the compound Poincaré maps, and parameter identification. However, existing deep reinforcement learning (RL) algorithms are designed to maximize long-term returns, causing a waste of computing resources when applied in the atypical MDPs. These existing algorithms are also limited by the estimation error of the value function, leading to a poor policy. To solve such limitations, this paper proposes an immediate-return algorithm for the atypical MDPs with continuous action space by designing an unbiased and low variance target Q-value and a simplified network framework. Then, two examples of atypical MDPs considering the uncertainty are presented to illustrate the performance of the proposed algorithm, i.e., passing the football to a moving player and chipping the football over the human wall. Compared with the existing deep RL algorithms, such as deep deterministic policy gradient and proximal policy optimization, the proposed algorithm shows significant advantages in learning efficiency, the effective rate of control, and computing resource usage.

Comprehensive Automated Driving Maneuvers under a Non-Signalized Intersection Adopting Deep Reinforcement Learning

Automated driving systems have become a potential approach to mitigating collisions, emissions, and human errors in mixed-traffic environments. This study proposes the use of a deep reinforcement learning method to verify the effects of comprehensive automated vehicle movements at a non-signalized intersection according to training policy and measures of effectiveness. This method integrates multilayer perceptron and partially observable Markov decision process algorithms to generate a proper decision-making algorithm for automated vehicles. This study also evaluates the efficiency of proximal policy optimization hyperparameters for the performance of the training process. Firstly, we set initial parameters and create simulation scenarios. Secondly, the SUMO simulator executes and exports observations. Thirdly, the Flow tool transfers these observations into the states of reinforcement learning agents. Next, the multilayer perceptron algorithm trains the input data and updates policies to generate the proper actions. Finally, this training checks the termination and iteration process. These proposed experiments not only increase the speeds of vehicles but also decrease the emissions at a higher market penetration rate and a lower traffic volume. We demonstrate that the fully autonomous condition increased the average speed 1.49 times compared to the entirely human-driven experiment.

An exponential lower bound for Zadeh’s pivot rule

The question whether the Simplex Algorithm admits an efficient pivot rule remains one of the most important open questions in discrete optimization. While many natural, deterministic pivot rules are known to yield exponential running times, the random-facet rule was shown to have a subexponential running time. For a long time, Zadeh’s rule remained the most prominent candidate for the first deterministic pivot rule with subexponential running time. We present a lower bound construction that shows that Zadeh’s rule is in fact exponential in the worst case. Our construction is based on a close relation to the Strategy Improvement Algorithm for Parity Games and the Policy Iteration Algorithm for Markov Decision Processes, and we also obtain exponential lower bounds for Zadeh’s rule in these contexts.

Structure Learning-Based Task Decomposition for Reinforcement Learning in Non-stationary Environments

Reinforcement learning (RL) agents empowered by deep neural networks have been considered a feasible solution to automate control functions in a cyber-physical system. In this work, we consider an RL-based agent and address the issue of learning via continual interaction with a time-varying dynamic system modeled as a non-stationary Markov decision process (MDP). We view such a non-stationary MDP as a time series of conventional MDPs that can be parameterized by hidden variables. To infer the hidden parameters, we present a task decomposition method that exploits CycleGAN-based structure learning. This method enables the separation of time-variant tasks from a non-stationary MDP, establishing the task decomposition embedding specific to time-varying information. To mitigate the adverse effect due to inherent noises of task embedding, we also leverage continual learning on sequential tasks by adapting the orthogonal gradient descent scheme with a sliding window. Through various experiments, we demonstrate that our approach renders the RL agent adaptable to time-varying dynamic environment conditions, outperforming other methods including state-of-the-art non-stationary MDP algorithms.

Numerical Simulation of Time-Optimal Path Planning for Autonomous Underwater Vehicles Using a Markov Decision Process Method

Many path planning algorithms developed for land or air based autonomous vehicles no longer apply under the water. A time-optimal path planning method for autonomous underwater vehicles (AUVs), based on a Markov decision process (MDP) algorithm, is proposed for the marine environment. Its performance is examined for different oceanic conditions, including complex coastal bathymetry and time-varying ocean currents, revealing advantages compared to the A* algorithm, a traditional path planning method. The ocean current is predicted using a regional ocean model and then provided to the MDP algorithm as a priori. A computation-efficient and feature-resolved spatial resolution are determined through a series of sensitivity experiments. The simulations demonstrate the importance to incorporate ocean currents in the path planning of AUVs in the real ocean. The MDP algorithm remains robust even if the ocean current is complex.

A Hybrid PAC Reinforcement Learning Algorithm for Human-Robot Interaction.

This paper offers a new hybrid probably approximately correct (PAC) reinforcement learning (RL) algorithm for Markov decision processes (MDPs) that intelligently maintains favorable features of both model-based and model-free methodologies. The designed algorithm, referred to as the Dyna-Delayed Q-learning (DDQ) algorithm, combines model-free Delayed Q-learning and model-based R-max algorithms while outperforming both in most cases. The paper includes a PAC analysis of the DDQ algorithm and a derivation of its sample complexity. Numerical results are provided to support the claim regarding the new algorithm’s sample efficiency compared to its parents as well as the best known PAC model-free and model-based algorithms in application. A real-world experimental implementation of DDQ in the context of pediatric motor rehabilitation facilitated by infant-robot interaction highlights the potential benefits of the reported method.

Temporal concatenation for Markov decision processes

We propose and analyze a temporal concatenation heuristic for solving large-scale finite-horizon Markov decision processes (MDP), which divides the MDP into smaller sub-problems along the time horizon and generates an overall solution by simply concatenating the optimal solutions from these sub-problems. As a “black box” architecture, temporal concatenation works with a wide range of existing MDP algorithms. Our main results characterize the regret of temporal concatenation compared to the optimal solution. We provide upper bounds for general MDP instances, as well as a family of MDP instances in which the upper bounds are shown to be tight. Together, our results demonstrate temporal concatenation's potential of substantial speed-up at the expense of some performance degradation.

A numerical study of Markov decision process algorithms for multi-component replacement problems

We present a unified modeling framework for Time-Based Maintenance (TBM) and Condition-Based Maintenance (CBM) for optimization of replacements in multi-component systems. The considered system has a K-out-of-N reliability structure, and components deteriorate according to a multivariate gamma process with Lévy copula dependence. The TBM and CBM models are formulated as Markov Decision Processes (MDPs), and optimal policies are found using dynamic programming. Solving the CBM model requires that the continuous deterioration process is discretized. We therefore investigate the discretization level required for obtaining a near-optimal policy. Our results indicate that a coarser discretization level than previously suggested in the literature is adequate, indicating that dynamic programming is a feasible approach for optimization in multi-component systems. We further demonstrate this through empirical results for the size limit of the MDP models when solved with an optimized implementation of modified policy iteration. The TBM model can generally be solved with more components than the CBM model, since the former has a sparser state transition structure. In the special case of independent component deterioration, transition probabilities can be calculated efficiently at runtime. This reduces the memory requirements substantially. For this case, we also achieved a tenfold speedup when using ten processors in a parallel implementation of algorithm. Altogether, our results show that the computational requirements for systems with independent component deterioration increase at a slower rate than for systems with stochastic dependence.

Focused Topological Value Iteration

Topological value iteration (TVI) is an effective algorithm for solving Markov decision processes (MDPs) optimally, which 1) divides an MDP into strongly-connected components, and 2) solves these components sequentially. Yet, TVI’s usefulness tends to degrade if an MDP has large components, because the cost of the division process isn’t offset by gains during solution. This paper presents a new algorithm to solve MDPs optimally, focused topological value iteration (FTVI). FTVI addresses TVI’s limitations by restricting its attention to connected components that are relevant for solving the MDP. Specifically, FTVI uses a small amount of heuristic search to eliminate provably sub-optimal actions; this pruning allows FTVI to find smaller connected components, thus running faster. We demonstrate that our new algorithm outperforms TVI by an order of magnitude, averaged across several domains. Surprisingly, FTVI also significantly outperforms popular ‘heuristically-informed’ MDP algorithms such as LAO*, LRTDP, and BRTDP in many domains, sometimes by as much as two orders of magnitude. Finally, we characterize the type of domains where FTVI excels — suggesting a way to an informed choice of solver.

Algorithms and conditional lower bounds for planning problems

We consider planning problems for graphs, Markov Decision Processes (MDPs), and games on graphs in an explicit state space. While graphs represent the most basic planning model, MDPs represent interaction with nature and games on graphs represent interaction with an adversarial environment. We consider two planning problems with k different target sets: (a) the coverage problem asks whether there is a plan for each individual target set; and (b) the sequential target reachability problem asks whether the targets can be reached in a given sequence. For the coverage problem, we present a linear-time algorithm for graphs, and quadratic conditional lower bound for MDPs and games on graphs. For the sequential target problem, we present a linear-time algorithm for graphs, a sub-quadratic algorithm for MDPs, and a quadratic conditional lower bound for games on graphs. Our results with conditional lower bounds, based on the boolean matrix multiplication (BMM) conjecture and strong exponential time hypothesis (SETH), establish (i) model-separation results showing that for the coverage problem MDPs and games on graphs are harder than graphs, and for the sequential reachability problem games on graphs are harder than MDPs and graphs; and (ii) problem-separation results showing that for MDPs the coverage problem is harder than the sequential target problem.

Secure indoor traffic modeling in long‐term evolution femtocell networks using efficient Markov decision process

AbstractMobile users spent most of their time in an indoor environment, therefore it is crucial to develop indoor traffic models for next‐generation networks. The significance of this paper is related to the predominant femtocell technology as it is anticipated to be a very crucial part of the modern mobile networks such as 5G and beyond networks. The Femtocell network can satisfy the increased bandwidth demand for video and data traffic with improved network capacity and security. Indoor data traffic exhibits different characteristics that need to be dealt with distinct criteria of the Poisson process and existing classifying techniques. Indoor traffic models, proposed in the literature do not consider long‐range dependent characteristics for traffic modeling. Moreover, these traffic models have the ability to capture large flows only. In this paper, we have provided an efficient quality of service‐based classifier for downlink transmission in long‐term evolution (LTE)‐femtocell network. Our proposed algorithm models the indoor traffic by Poisson Lomax Burst Process. Modeled traffic streams, after prioritization, are classified using Markov Decision Process (MDP) classifier. Furthermore, a detailed comparison of our proposed MDP algorithm with the state‐of‐the‐art algorithms in the literature has also been presented. The network performance parameters have been assessed in terms of throughput, delay, and jitter. Simulation results depict a significant improvement in throughput in contrast with Round Robin (RR), which is approximately 8%, 6%, 6%, and 7% for voice, video, BE, and BG services, respectively, while modified least weighted delay first (MLWDF) gives least performance among them. MDP also provides an improvement in delay and average jitter of four different traffic flows and mixed traffic users when compared with RR and MLWDF algorithms.

Multilevel Task Offloading and Resource Optimization of Edge Computing Networks Considering UAV Relay and Green Energy

Unmanned aerial vehicle (UAV)-assisted relay mobile edge computing (MEC) network is a prominent concept, where network deployment is flexible and network coverage is wide. In scenarios such as emergency communications and low-cost coverage, optimization of offloading methods and resource utilization are important ways to improve system effectiveness due to limited terminal and UAV energy and hardware equipment. A multilevel edge computing network resource optimization model on the basis of UAV fusion that provides relay forwarding and offload services is established by considering the initial energy state of the UAV, the green energy charging function, and the reliability of computing offload. With normalized system utility function maximization as the goal, a Markov decision process algorithm meets the needs of the practical application scene and provides a flexible and effective unloading mode. This algorithm is adopted to solve the optimal offloading mode and the optimal resource utilization scheme. Simulations verify the effectiveness and reliability of the proposed multilevel offloading model. The proposed model can optimize system resource allocation and effectively improve the utility function and user experience of computing offloading systems.

Simulation-Based Algorithms for Markov Decision Processes: Monte Carlo Tree Search from AlphaGo to AlphaZero

AlphaGo and its successors AlphaGo Zero and AlphaZero made international headlines with their incredible successes in game playing, which have been touted as further evidence of the immense potential of artificial intelligence, and in particular, machine learning. AlphaGo defeated the reigning human world champion Go player Lee Sedol 4 games to 1, in March 2016 in Seoul, Korea, an achievement that surpassed previous computer game-playing program milestones by IBM’s Deep Blue in chess and by IBM’s Watson in the U.S. TV game show Jeopardy. AlphaGo then followed this up by defeating the world’s number one Go player Ke Jie 3-0 at the Future of Go Summit in Wuzhen, China in May 2017. Then, in December 2017, AlphaZero stunned the chess world by dominating the top computer chess program Stockfish (which has a far higher rating than any human) in a 100-game match by winning 28 games and losing none (72 draws) after training from scratch for just four hours! The deep neural networks of AlphaGo, AlphaZero, and all their incarnations are trained using a technique called Monte Carlo tree search (MCTS), whose roots can be traced back to an adaptive multistage sampling (AMS) simulation-based algorithm for Markov decision processes (MDPs) published in Operations Research back in 2005 [Chang, HS, MC Fu, J Hu and SI Marcus (2005). An adaptive sampling algorithm for solving Markov decision processes. Operations Research, 53, 126–139.] (and introduced even earlier in 2002). After reviewing the history and background of AlphaGo through AlphaZero, the origins of MCTS are traced back to simulation-based algorithms for MDPs, and its role in training the neural networks that essentially carry out the value/policy function approximation used in approximate dynamic programming, reinforcement learning, and neuro-dynamic programming is discussed, including some recently proposed enhancements building on statistical ranking & selection research in the operations research simulation community.

Energy Efficient and Delay Aware 5G Multi-Tier Network

Multi-tier heterogeneous Networks (HetNets) with dense deployment of small cells in 5G networks are expected to effectively meet the ever increasing data traffic demands and offer improved coverage in indoor environments. However, HetNets are raising major concerns to mobile network operators such as complex distributed control plane management, handover management issue, increases latency and increased energy expenditures. Sleep mode implementation in multi-tier 5G networks has proven to be a very good approach for reducing energy expenditures. In this paper, a Markov Decision Process (MDP)-based algorithm is proposed to switch between three different power consumption modes of a base station (BS) for improving the energy efficiency and reducing latency in 5G networks. The MDP-based approach intelligently switches between the states of the BS based on the offered traffic while maintaining a prescribed minimum channel rate per user. Simulation results show that the proposed MDP algorithm together with the three-state BSs results in a significant gain in terms of energy efficiency and latency.

A Universal Empirical Dynamic Programming Algorithm for Continuous State MDPs

We propose universal randomized function approximation-based empirical value learning (EVL) algorithms for Markov decision processes. The “empirical” nature comes from each iteration being done empirically from samples available from simulations of the next state. This makes the Bellman operator a random operator. A parametric and a nonparametric method for function approximation using a parametric function space and a reproducing kernel Hilbert space respectively are then combined with EVL. Both function spaces have the universal function approximation property. Basis functions are picked randomly. Convergence analysis is performed using a random operator framework with techniques from the theory of stochastic dominance. Finite time sample complexity bounds are derived for both universal approximate dynamic programming algorithms. Numerical experiments support the versatility and computational tractability of this approach.

Underwater chemical plume tracing based on partially observable Markov decision process

Chemical plume tracing based on autonomous underwater vehicle uses chemical as a guidance to navigate and search in the unknown environments. To solve the key issue of tracing and locating the source, this article proposes a path-planning strategy based on partially observable Markov decision process algorithm and artificial potential field algorithm. The partially observable Markov decision process algorithm is used to construct a source likelihood map and update it in real time with environmental information from the sensors on autonomous underwater vehicle in search area. The artificial potential field algorithm uses the source likelihood map for accurately planning tracing path and guiding the autonomous underwater vehicle to track along the path until the source is detected. This article carries out simulation experiments on the proposed algorithm. The experimental results show that the algorithms have good performance, which is suitable for chemical plume tracing via autonomous underwater vehicle. Compared with the bionic method, the simulation results show that the proposed method has higher success rate and better stability than the bionic method.

Energy efficient and delay aware ternary-state transceivers for aerial base stations

In recent times, Aerial Base Stations(AeBSs) are being investigated to provide wireless coverage to terrestrial radio terminals. There are many advantages of using aerial platforms to provide wireless coverage, including larger coverage in remote areas and better line-of-sight conditions, etc. Energy is a scarce resource for the AeBSs, hence the wise management of energy is quite beneficial for the aerial network. In this context, we study the means of reducing the total energy consumption by designing and implementing an energy efficient AeBSs as presented in this paper. Implementing the sleep mode in the Base Stations (BSs) has been proven to be a very good approach for improving the energy efficiency and we propose a novel strategy for further improving energy efficiency by considering ternary state transceivers for AeBSs. Using the three state model, we propose a Markov Decision Process (MDP) based algorithm, which intelligently switches among three states of the transceivers based on the offered traffic meanwhile maintaining a prescribed minimum channel rate per user. We define a reward function for the MDP, which helps us to get an optimal policy for selecting a particular mode for the transceivers of the AeBS. Considering an AeBS with transceivers whose states are changeable, we perform simulations to analyse the performance of the algorithm. Our results show that, compared with the always active model, around 40% gain in the energy efficiency is achieved by using our proposed MDP algorithm together with the three-state transceivers model. We also show the energy-delay tradeoff in order to design an efficient AeBS.

PORCA: Modeling and Planning for Autonomous Driving Among Many Pedestrians

This letter presents a planning system for autonomous driving among many pedestrians. A key ingredient of our approach is Pedestrian Optimal Reciprocal Collision Avoidance, a pedestrian motion prediction model that accounts for both a pedestrian's global navigation intention and local interactions with the vehicle and other pedestrians. Unfortunately, the autonomous vehicle does not know the pedestrians’ intentions a priori and requires a planning algorithm that hedges against the uncertainty in pedestrian intentions. Our planning system combines a Partially Observable Markov Decision Process algorithm with the pedestrian motion model and runs in real time. Experiments show that it enables a robot scooter to drive safely, efficiently, and smoothly in a crowd with a density of nearly one person per square meter.