Original Markov Decision Processes Research Articles

Relaxed Equilibria for Time-Inconsistent Markov Decision Processes

This paper considers an infinite-horizon Markov decision process (MDP) that allows for general nonexponential discount functions in both discrete and continuous time. Because of the inherent time inconsistency, we look for a randomized equilibrium policy (i.e., relaxed equilibrium) in an intrapersonal game between an agent’s current and future selves. When we modify the MDP by entropy regularization, a relaxed equilibrium is shown to exist by a nontrivial entropy estimate. As the degree of regularization diminishes, the entropy-regularized MDPs approximate the original MDP, which gives the general existence of a relaxed equilibrium in the limit by weak convergence arguments. As opposed to prior studies that consider only deterministic policies, our existence of an equilibrium does not require any convexity (or concavity) of the controlled transition probabilities and reward function. Interestingly, this benefit of considering randomized policies is unique to the time-inconsistent case.

Managing Resources for Shared Micromobility: Approximate Optimality in Large-Scale Systems

We consider the problem of managing resources in shared micromobility systems (bike sharing and scooter sharing). An important task in managing such systems is periodic repositioning/recharging/sourcing of units to avoid stockouts or excess inventory at nodes with unbalanced flows. We consider a discrete-time model; each period begins with an initial inventory at each node in the network, and then, customers (demand) materialize at the nodes. Each customer picks up a unit at the origin node and drops it off at a randomly sampled destination node with an origin-specific probability distribution. We model the above network inventory management problem as an infinite horizon discrete-time discounted Markov decision process (MDP) and prove the asymptotic optimality of a novel mean-field approximation to the original MDP as the number of stations becomes large. To compute an approximately optimal policy for the mean-field dynamics, we provide an algorithm with a running time that is logarithmic in the desired optimality gap. Lastly, we compare the performance of our mean field-based policy with state-of-the-art heuristics via numerical experiments, including experiments using Austin scooter-sharing data. This paper was accepted by Jeannette Song, operations management. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.02023 .

Sample Efficient Deep Reinforcement Learning With Online State Abstraction and Causal Transformer Model Prediction.

Deep reinforcement learning (RL) typically requires a tremendous number of training samples, which are not practical in many applications. State abstraction and world models are two promising approaches for improving sample efficiency in deep RL. However, both state abstraction and world models may degrade the learning performance. In this article, we propose an abstracted model-based policy learning (AMPL) algorithm, which improves the sample efficiency of deep RL. In AMPL, a novel state abstraction method via multistep bisimulation is first developed to learn task-related latent state spaces. Hence, the original Markov decision processes (MDPs) are compressed into abstracted MDPs. Then, a causal transformer model predictor (CTMP) is designed to approximate the abstracted MDPs and generate long-horizon simulated trajectories with a smaller multistep prediction error. Policies are efficiently learned through these trajectories within the abstracted MDPs via a modified multistep soft actor-critic algorithm with a λ -target. Moreover, theoretical analysis shows that the AMPL algorithm can improve sample efficiency during the training process. On Atari games and the DeepMind Control (DMControl) suite, AMPL surpasses current state-of-the-art deep RL algorithms in terms of sample efficiency. Furthermore, DMControl tasks with moving noises are conducted, and the results demonstrate that AMPL is robust to task-irrelevant observational distractors and significantly outperforms the existing approaches.

Trajectory Synthesis for the Coordinated Inspection of a Spacecraft with Safety Guarantees

As outer space becomes increasingly congested, there exists a growing need for auxiliary spacecraft to perform support missions for existing satellites with guarantees for safety and mission success. We focus on a multispacecraft inspection mission, wherein a team of “deputy” spacecraft inspect a passive “chief” spacecraft by traveling to a set of inspection points while satisfying a set of safety constraints, namely, that they avoid aligning themselves with the sun, that they avoid colliding with one another, and that they avoid colliding with the chief. We model the deputy dynamics using the Clohessy–Wiltshire–Hill equations, and subsequently discretize the environment by exploiting elliptical natural motion trajectories. Using this finite state space, we construct a Markov decision process (MDP) model of the environment and determine the optimal sequence of inspection points for each deputy to visit by solving a vehicle routing problem. To ensure that the deputies satisfy the safety constraints, we form the product MDP of the original MDP and a nondeterministic Büchi automaton that encodes the sensing task and safety constraints. Using this product MDP, we propose a pair of decentralized algorithms that each seeks to minimize the weighted combination of the time and fuel required to safely complete the mission. The first is an offline algorithm that synthesizes a safe trajectory for each deputy that requires no communication at runtime, while the second is an online algorithm that enforces safety at runtime by leveraging communication between the deputies. We provide numerical examples demonstrating the efficacy of both proposed algorithms.

Dynamic Clustering and Resource Allocation Using Deep Reinforcement Learning for Smart-Duplex Networks

Ultra dense networks (UDNs) with smart-duplex (SD), which allows the base stations (BSs) to flexibly switch between the half-duplex (HD) and full-duplex (FD), are expected to support high-density transmissions. However, to centrally handle a large network is costly, while distributed processing may suffer from the severe performance loss due to the complicated intercell interferences in the UDNs. This article aims to balance the system performance and clustering cost of the SD UDNs by dividing all small cells into several clusters. A Markov decision process (MDP) problem is formulated to maximize the average weighted sum of network throughput and clustering cost for all clusters. To approximately solve this problem, we first adopt an affinity propagation method to determine the number of clusters and the center of each cluster. Then, by treating small cells as agents, the original MDP problem is proved to be equivalent to a multiagent MDP to maximize the average reward of all small cells. Next, a multiagent deep reinforcement learning (DRL) is proposed to jointly implement the dynamic clustering for noncenter small cells, resource allocation, and duplex mode selection. Simulation results show that SD has prominent advantages over both the HD and FD in UDNs, and the proposed multiagent DRL outperforms other clustering schemes under the considered scenarios.

Data-Driven Coordinated Charging for Electric Vehicles With Continuous Charging Rates: A Deep Policy Gradient Approach

In this article, we consider a parking lot that manages the charging processes of its parked electric vehicles (EVs). Upon arrival, each EV requests a certain amount of energy. This request should be fulfilled before the EV’s departure. It is of critical importance to coordinate the EVs’ charging rates to smooth out the load profile of the parking lot because inappropriate charging rates can lead to sharp spikes and fluctuations on the load profile, imposing negative effects on the power grid. Meanwhile, empirical studies show that many parking lots exhibit statistical patterns on EV dynamics. For example, the bulk of EVs arrives during rush hours. Therefore, in this article, we incorporate such patterns into charging rate coordination. Although the statistical patterns can be summarized from historical data, they are difficult to be analytically modeled. As a result, we adopt a model-free deep reinforcement learning approach. We also take the latest continuous charging rate control technology into consideration. The decision variables are thus continuous and a policy gradient algorithm is needed to perform reinforcement learning. Technically, we first formulate the problem as a Markov decision process (MDP) with unknown state transition probabilities. To further derive a deep policy gradient algorithm, the challenge lies in the inconsistent and state-dependent action space of the MDP model, due to the constraint to satisfy EVs’ energy demands before their scheduled departure. To tackle the challenge, we design a customized model for neural network training by extending the action space to be consistent and state independent, and revise the reward function to penalize the neural network output if it is beyond the action space of the original MDP model. With this customized model, we then develop a deep policy gradient algorithm based on the proximal policy gradient framework. Numerical results show that our algorithm outperforms the benchmarks.

Data-Driven Radar Selection and Power Allocation Method for Target Tracking in Multiple Radar System

In this article, a data-driven multiple radar system (MRS) resource allocation method for target tracking is developed based on deep reinforcement learning. The goal is to achieve the given tracking accuracy requirement with minimum long-short term power consumption by radar selection and MRS power allocation. In theory, thanks to the existence of the given tracking accuracy requirement, this problem can be modeled as a constrained Markov decision process (MDP). For this problem, a constrained deep reinforcement learning (DRL) is introduced based on deep deterministic policy gradient. Specifically, by relaxing the original constrained MDP problem to unconstrained MDP problem with Lagrangian relaxation procedure, the tracking accuracy requirement is introduced into the derivation of policy gradient of actor network in deep deterministic policy gradient, which makes the tracking performance with the resource allocation policy learned by DRL able to meet the given tracking requirements. Meanwhile, considering the limited radar and power resource of MRS, three output layers of the actor network is redesigned for determining the actions of radar selection and power allocation. In this way, the assignment and transmit power of each radar of MRS can be given in real time at each tracking interval. Simulation results have shown the effectiveness of the proposed method.

Solving K-MDPs

Markov Decision Processes (MDPs) are employed to model sequential decision-making problems under uncertainty. Traditionally, algorithms to solve MDPs have focused on solving large state or action spaces. With increasing applications of MDPs to human-operated domains such as conservation of biodiversity and health, developing easy-to-interpret solutions is of paramount importance to increase uptake of MDP policies. Here, we define the problem of solving K-MDPs, i.e., given an original MDP and a constraint on the number of states (K), generate a reduced state space MDP that minimizes the difference between the original optimal MDP value function and the reduced optimal K-MDP value function. Building on existing non-transitive and transitive approximate state abstraction functions, we propose a family of three algorithms based on binary search with sub-optimality bounded polynomially in a precision parameter: ϕQ*εK-MDP-ILP, ϕQ*dK-MDP and ϕa*dK-MDP. We compare these algorithms to a greedy algorithm (ϕQ*ε Greedy K-MDP) and clustering approach (k-means++ K-MDP). On randomly generated MDPs and two computational sustainability MDPs, ϕa*dK-MDP outperformed all algorithms when it could find a feasible solution. While numerous state abstraction problems have been proposed in the literature, this is the first time that the general problem of solving K-MDPs is suggested. We hope that our work will generate future research aiming at increasing the interpretability of MDP policies in human-operated domains.

Optimal Energy Cooperation Policy in Fusion Center-Based Sustainable Wireless Sensor Networks

In this paper, in order to maximize a long-term average system transmission rate, we propose an optimal energy cooperation policy for the sustainable wireless sensor networks where each pair of nodes can harvest energy and transfer energy in the pair. We formulate this optimization problem into a Markov decision process (MDP). We define a node pairing metric-relative energy distance ratio (REDR)-to design a node pairing algorithm which can decouple the original MDP into a multiple low dimension MDP. Then, value iteration and relative value iteration algorithms are brought to find optimal energy cooperation policies over finite and infinite horizons for each pair of nodes respectively. Simulation results show that the system transmission rate of the proposed policy with REDR based pairing algorithm outperforms the non-cooperation policies as well as the policies with straightforward pairing algorithms.

Age of Information Aware Radio Resource Management in Vehicular Networks: A Proactive Deep Reinforcement Learning Perspective

In this paper, we investigate the problem of age of information (AoI)-aware radio resource management for expected long-term performance optimization in a Manhattan grid vehicle-to-vehicle network. With the observation of global network state at each scheduling slot, the roadside unit (RSU) allocates the frequency bands and schedules packet transmissions for all vehicle user equipment-pairs (VUE-pairs). We model the stochastic decision-making procedure as a discrete-time single-agent Markov decision process (MDP). The technical challenges in solving the optimal control policy originate from high spatial mobility and temporally varying traffic information arrivals of the VUE-pairs. To make the problem solving tractable, we first decompose the original MDP into a series of per-VUE-pair MDPs. Then we propose a proactive algorithm based on long short-term memory and deep reinforcement learning techniques to address the partial observability and the curse of high dimensionality in local network state space faced by each VUE-pair. With the proposed algorithm, the RSU makes the optimal frequency band allocation and packet scheduling decision at each scheduling slot in a decentralized way in accordance with the partial observations of the global network state at the VUE-pairs. Numerical experiments validate the theoretical analysis and demonstrate the significant performance improvements from the proposed algorithm.

Easy Affine Markov Decision Processes

Individuals, firms, and governments often face the challenge of making optimal decisions in a dynamic setting amidst a changing and uncertain environment. Although Markov decision processes (MDPs) provide a powerful modeling framework for such problems, solving an MDP is generally difficult. In “Easy Affine Markov Decision Processes,” Ning and Sobel characterize decomposable affine MDPs, which can be solved easily. An MDP in this class has continuous multidimensional controlled states and actions, and Markov-modulated uncontrolled states. With affine immediate reward and transition dynamics and affine constraints on actions, these MDPs have a linear value function and a linear optimal policy. The linear coefficients can be computed easily and exactly by solving a small auxiliary MDP, which frees the solution of the original MDP from the curse of dimensionality and errors caused by discretizing the continuous state and action spaces. The applicability of decomposable affine MDPs is illustrated in fishery management, capacity portfolio management, and commodity procurement.

Dynamic scheduling for wireless multicast in massive MIMO HetNet

Consider a massive multiple input and multiple output (MIMO) heterogeneous network (HetNet) scenario where the macro base station is coupled with low-power nodes (LPNs) to provide wireless multicast transmission to user equipments (UEs) within a single macro cell. The Markov decision process (MDP) model is adopted to model the dynamic scheduling problem introduced by the mobility of UEs. With asymptotic conclusion for the massive MIMO multicast transmission, we propose a MDP-based off-line scheduling problem with determined system parameters and derive a linear programming (LP) model to solve the problem. As for the system with undetermined parameters, we adopt Q-learning algorithm to solve the optimal policy. To derive the low-complexity algorithm, the state aggregation method is utilized to build an series of sub-problems to estimate the original complex MDP problem. Finally we use the MDP Toolbox in MATLAB to run simulations to test the proposed algorithm. The performance of the proposed algorithm is shown and analyzed in details.

Countable state Markov decision processes with unbounded jump rates and discounted cost: optimality equation and approximations

This paper considers Markov decision processes (MDPs) with unbounded rates, as a function of state. We are especially interested in studying structural properties of optimal policies and the value function. A common method to derive such properties is by value iteration applied to the uniformised MDP. However, due to the unboundedness of the rates, uniformisation is not possible, and so value iteration cannot be applied in the way we need. To circumvent this, one can perturb the MDP. Then we need two results for the perturbed sequence of MDPs: 1. there exists a unique solution to the discounted cost optimality equation for each perturbation as well as for the original MDP; 2. if the perturbed sequence of MDPs converges in a suitable manner then the associated optimal policies and the value function should converge as well. We can model both the MDP and perturbed MDPs as a collection of parametrised Markov processes. Then both of the results above are essentially implied by certain continuity properties of the process as a function of the parameter. In this paper we deduce tight verifiable conditions that imply the necessary continuity properties. The most important of these conditions are drift conditions that are strongly related to nonexplosiveness.

Countable state Markov decision processes with unbounded jump rates and discounted cost: optimality equation and approximations

This paper considers Markov decision processes (MDPs) with unbounded rates, as a function of state. We are especially interested in studying structural properties of optimal policies and the value function. A common method to derive such properties is by value iteration applied to the uniformised MDP. However, due to the unboundedness of the rates, uniformisation is not possible, and so value iteration cannot be applied in the way we need. To circumvent this, one can perturb the MDP. Then we need two results for the perturbed sequence of MDPs: 1. there exists a unique solution to the discounted cost optimality equation for each perturbation as well as for the original MDP; 2. if the perturbed sequence of MDPs converges in a suitable manner then the associated optimal policies and the value function should converge as well. We can model both the MDP and perturbed MDPs as a collection of parametrised Markov processes. Then both of the results above are essentially implied by certain continuity properties of the process as a function of the parameter. In this paper we deduce tight verifiable conditions that imply the necessary continuity properties. The most important of these conditions are drift conditions that are strongly related to nonexplosiveness.

Online Planning for Large Markov Decision Processes with Hierarchical Decomposition

Markov decision processes (MDPs) provide a rich framework for planning under uncertainty. However, exactly solving a large MDP is usually intractable due to the “curse of dimensionality”— the state space grows exponentially with the number of state variables. Online algorithms tackle this problem by avoiding computing a policy for the entire state space. On the other hand, since online algorithm has to find a near-optimal action online in almost real time, the computation time is often very limited. In the context of reinforcement learning, MAXQ is a value function decomposition method that exploits the underlying structure of the original MDP and decomposes it into a combination of smaller subproblems arranged over a task hierarchy. In this article, we present MAXQ-OP—a novel online planning algorithm for large MDPs that utilizes MAXQ hierarchical decomposition in online settings. Compared to traditional online planning algorithms, MAXQ-OP is able to reach much more deeper states in the search tree with relatively less computation time by exploiting MAXQ hierarchical decomposition online. We empirically evaluate our algorithm in the standard Taxi domain—a common benchmark for MDPs—to show the effectiveness of our approach. We have also conducted a long-term case study in a highly complex simulated soccer domain and developed a team named WrightEagle that has won five world champions and five runners-up in the recent 10 years of RoboCup Soccer Simulation 2D annual competitions. The results in the RoboCup domain confirm the scalability of MAXQ-OP to very large domains.

Stochastic eco-routing in a signalized traffic network

In this paper, an eco-routing algorithm is developed for vehicles in a signalized traffic network. The proposed method incorporates a microscopic vehicle emission model into a Markov decision process (MDP). Instead of using GPS-based vehicle trajectory data, which are used by many existing eco-routing algorithm, high resolution traffic data including vehicle arrival and signal status information are used as primary inputs. The proposed method can work with any microscopic vehicle model that uses vehicle trajectories as inputs and gives related emission rates as outputs. Furthermore, a constrained eco-routing problem is proposed to deal with the situation where multiple costs present. This is done by transferring the original MDP based formulation to a linear programming formulation. Besides the primary cost, additional costs are considered as constraints. Two numerical examples are given using the field data obtained from City of Pasadena, California, USA. The eco-routing algorithm for single objective is compared against the traditional shortest path algorithm, Dijkstra’s algorithm. Average reductions of CO emission around 20% are observed.

Reachability-based model reduction for Markov decision process

This paper presents how to improve model reduction for Markov decision process (MDP), a technique that generates equivalent MDPs that can be smaller than the original MDP. In order to improve the current state-of-the-art, we take advantage of the information about the initial state of the environment. Given this initial state information, we perform a reachability analysis and then employ model reduction techniques to the reachable space of the original problem. Further, we also eliminate redundancies in the original MDP in order to speed up the model reduction phase. We also contribute by empirically comparing our technique against state-of-the-art model reduction techniques and MDP solvers that do not perform model reduction. The results show that our approach dominates the current model reduction algorithms and outperforms general MDP solvers in dense problems, i.e., problems in which actions have many probabilistic outcomes.

Stochastic Eco-routing in a Signalized Traffic Network

In this paper, an eco-routing algorithm is developed for vehicles in a signalized traffic network. The proposed method incorporates a microscopic vehicle emission model into a Markov decision process (MDP). Instead of using GPS-based vehicle trajectory data, which are used by many existing eco-routing algorithm, high resolution traffic data including vehicle arrival and signal status information are used as primary inputs. The proposed method can work with any microscopic vehicle model that uses vehicle trajectories as inputs and gives related emission rates as outputs. Furthermore, a constrained eco-routing problem is proposed to deal with the situation where multiple costs present. This is done by transferring the original MDP based formulation to a linear programming formulation. Besides the primary cost, additional costs are considered as constraints. Two numerical examples are given using the field data obtained from City of Pasadena, California, USA. The eco-routing algorithm for single objective is compared against the traditional shortest path algorithm, Dijkstra's algorithm. Average reductions of CO emission around 20% are observed.

Dynamic Request Routing for Online Video-on-Demand Service: A Markov Decision Process Approach

We investigate the request routing problem in the CDN-based Video-on-Demand system. We model the system as a controlled queueing system including a dispatcher and several edge servers. The system is formulated as a Markov decision process (MDP). Since the MDP formulation suffers from the so-called “the curse of dimensionality” problem, we then develop a greedy heuristic algorithm, which is simple and can be implemented online, to approximately solve the MDP model. However, we do not know how far it deviates from the optimal solution. To address this problem, we further aggregate the state space of the original MDP model and use the bounded-parameter MDP (BMDP) to reformulate the system. This allows us to obtain a suboptimal solution with a known performance bound. The effectiveness of two approaches is evaluated in a simulation study.

EFFICIENT ABSTRACTION SELECTION IN REINFORCEMENT LEARNING

This article addresses reinforcement learning problems based on factored Markov decision processes (MDPs) in which the agent must choose among a set of candidate abstractions, each build up from a different combination of state components. We present and evaluate a new approach that can perform effective abstraction selection that is more resource-efficient and/or more general than existing approaches. The core of the approach is to make selection of an abstraction part of the learning agent's decision-making process by augmenting the agent's action space with internal actions that select the abstraction it uses. We prove that under certain conditions this approach results in a derived MDP whose solution yields both the optimal abstraction for the original MDP and the optimal policy under that abstraction. We examine our approach in three domains of increasing complexity: contextual bandit problems, episodic MDPs, and general MDPs with context-specific structure. © 2013 Wiley Periodicals, Inc.