Infinite-horizon Markov Decision Process Research Articles

Markov decision process design: A framework for integrating strategic and operational decisions

We consider the problem of optimally designing a system for repeated use under uncertainty. We develop a modeling framework that integrates the design and operational phases, which are represented by a mixed-integer program and discounted-cost infinite-horizon Markov decision processes, respectively. We seek to simultaneously minimize the design costs and the subsequent expected operational costs. This problem setting arises naturally in several application areas, as we illustrate through examples. We derive a bilevel mixed-integer linear programming formulation for the problem and perform a computational study to demonstrate that realistic instances can be solved numerically.

Cost-aware defense for parallel server systems against reliability and security failures

Parallel server systems in transportation, manufacturing, and computing heavily rely on dynamic routing using connected cyber components for computation and communication. Yet, these components remain vulnerable to random malfunctions and malicious attacks, motivating the need for resilient dynamic routing that is both traffic-stabilizing and cost-efficient. In this paper, we consider a parallel server system with dynamic routing subject to reliability and stability failures. For the reliability setting, we consider an infinite-horizon Markov decision process where the system operator strategically activates a protection mechanism upon each job arrival based on traffic state observations. We prove that an optimal deterministic threshold protecting policy exists based on the dynamic programming recursion of the Hamilton–Jacobi–Bellman equation. For the security setting, we extend the model to an infinite-horizon stochastic game where the attacker strategically manipulates routing assignments. We show that both players follow a threshold strategy at every Markov perfect equilibrium. For both failure settings, we also analyze the stability of the traffic queues. Finally, we develop approximate dynamic programming algorithms to compute the optimal/equilibrium policies and present numerical examples/experiments for validation and illustration.

Admission control for a capacitated supply system with real-time replenishment information

Control towers can provide real-time information on logistic processes to support decision making. The question however, is how to make use of it and how much it may save. We consider this issue for a company supplying expensive spare parts and which has limited production capacity. Besides deciding on base stock levels, it can accept or reject customers. The real-time status information is captured by a k-Erlang distributed replenishment lead time. First we model the problem with patient customers as an infinite-horizon Markov decision process and minimize the total expected discounted cost. We prove that the optimal policy can be characterized using two thresholds: a base work storage level that determines when ordering takes place and an acceptance work storage level that determines when demand of customers should be accepted. In a numerical study, we show that using real-time status information on the replenishment item and adopting admission control can lead to significant cost savings. The cost savings are highest when the optimal admission threshold is a work storage level with a replenishment item halfway in process. This finding is different from the literature, where it is stated that the cost increase of ignoring real-time information is negligible under either the lost sales or the backordering case. Next we study the problem where customers are of limited patience. We find that the optimal admission policy is not always of threshold type. This is different from the literature which assumes an exponential production lead time.

Dynamic Project Expediting: A Stochastic Shortest-Path Approach

We deal with the problem of managing a project or a complex operational process by controlling the execution pace of the activities it comprises. We consider a setting in which these activities are clearly defined, are subject to precedence constraints, and progress randomly. We formulate a discrete-time, infinite-horizon Markov decision process in which the manager reviews progress in each period and decides which activities to expedite to balance expediting costs with delay costs. We derive structural properties for this dynamic project expediting problem. These enable us then to devise exact solution methods that we show to reduce computational burden significantly. We illustrate how our method generalizes and can be used to tackle a wide range of so-called stochastic shortest-path problems that are characterized by an intuitive property and can capture other applications, including medical decision-making and disease-modeling problems. Moreover, we also deal with the state identification issue for our problem, which is a challenging task in and of itself, owing to precedence constraints. We complement our analytical results with numerical experiments, demonstrating that both our solution and state identification methods significantly outperform extant methods for a supply chain example and for various randomly generated instances. This paper was accepted by Chung Piaw Teo, optimization. Funding: R. Mogre acknowledges support from the U.S.-UK Fulbright Commission and the Lloyd’s Tercentenary Research Foundation through the Fulbright-Lloyd’s Scholar Award, which allowed him to spend an extended period of time at Massachusetts Institute of Technology. Supplemental Material: The data files are available at https://doi.org/10.1287/mnsc.2023.4876 .

Age-Dependent Distributed MAC for Ultra-Dense Wireless Networks

We consider an ultra-dense wireless network with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$</tex-math> </inline-formula> channels and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M = N$</tex-math> </inline-formula> devices. Messages with fresh information are generated at each device according to a random process and need to be transmitted to an access point. The value of a message decreases as it ages, so each device searches for an idle channel to transmit the message as soon as it can. However, each channel probing is associated with a fixed cost (energy), so a device needs to adapt its probing rate based on the “age” of the message. At each device, the design of the optimal probing strategy can be formulated as an infinite horizon Markov Decision Process (MDP) where the devices compete with each other to find idle channels. While it is natural to view the system as a Bayesian game, it is often intractable to analyze such a system. Thus, we use the Mean Field Game (MFG) approach to analyze the system in a large-system regime, where the number of devices is very large, to understand the structure of the problem and to find efficient probing strategies. We present an analysis based on the MFG perspective. We begin by characterizing the space of valid policies and use this to show the existence of a Mean Field Nash Equilibrium (MFNE) in a constrained set for any general increasing cost functions with diminishing rewards. Further we provide an algorithm for computing the equilibrium for any given device, and the corresponding age-dependent channel probing policy.

Markov Decision Processes with Time-Varying Geometric Discounting

Canonical models of Markov decision processes (MDPs) usually consider geometric discounting based on a constant discount factor. While this standard modeling approach has led to many elegant results, some recent studies indicate the necessity of modeling time-varying discounting in certain applications. This paper studies a model of infinite-horizon MDPs with time-varying discount factors. We take a game-theoretic perspective – whereby each time step is treated as an independent decision maker with their own (fixed) discount factor – and we study the subgame perfect equilibrium (SPE) of the resulting game as well as the related algorithmic problems. We present a constructive proof of the existence of an SPE and demonstrate the EXPTIME-hardness of computing an SPE. We also turn to the approximate notion of epsilon-SPE and show that an epsilon-SPE exists under milder assumptions. An algorithm is presented to compute an epsilon-SPE, of which an upper bound of the time complexity, as a function of the convergence property of the time-varying discount factor, is provided.

Multi-provider NFV network service delegation via average reward reinforcement learning

In multi-provider 5G/6G networks, service delegation enables administrative domains to federate in provisioning NFV network services. Admission control in selecting the appropriate domain for service deployment, without prior knowledge of service requests’ statistical distributions, is fundamental to maximize average profit. This paper analyzes a general federation contract model for service delegation in various ways. First, under the assumption of known system dynamics, we obtain the theoretically optimal performance bound by formulating the admission control problem as an infinite-horizon Markov decision process (MDP) and solving it through dynamic programming, which is used as a benchmark to evaluate practical solutions. Second, we apply Reinforcement Learning (RL) to practically tackle the problem when the arrival and departure rates are not known. For the first time in this context, we analyze the performance of the widely used Q-Learning algorithm, and prove as it maximizes the discounted rewards, it is not an efficient solution due to its sensitivity to the discount factor. Then, we propose the average reward reinforcement learning approach (named “R-Learning”) to find the policy that directly maximizes the average profit. Finally, we evaluate different solutions through extensive simulations and experimentally using the 5Growth management and orchestration platform. Results confirm that the proposed R-Learning solution always outperforms Q-Learning and the greedy policies. Furthermore, while there is at most a 9% optimality gap in the ideal simulation environment, it competes with the MDP solution in the experimental assessment.

BATCH POLICY LEARNING IN AVERAGE REWARD MARKOV DECISION PROCESSES.

We consider the batch (off-line) policy learning problem in the infinite horizon Markov Decision Process. Motivated by mobile health applications, we focus on learning a policy that maximizes the long-term average reward. We propose a doubly robust estimator for the average reward and show that it achieves semiparametric efficiency. Further we develop an optimization algorithm to compute the optimal policy in a parameterized stochastic policy class. The performance of the estimated policy is measured by the difference between the optimal average reward in the policy class and the average reward of the estimated policy and we establish a finite-sample regret guarantee. The performance of the method is illustrated by simulation studies and an analysis of a mobile health study promoting physical activity.

Integrated optimisation of pricing, manufacturing, and procurement decisions of a make-to-stock system operating in a fluctuating environment

ABSTRACT Manufacturers experience random environmental fluctuations that influence their supply and demand processes directly. To cope with these environmental fluctuations, they typically utilise operational hedging strategies in terms of pricing, manufacturing, and procurement decisions. We focus on this challenging problem by proposing an analytical model. Specifically, we study an integrated problem of procurement, manufacturing, and pricing strategies for a continuous-review make-to-stock system operating in a randomly fluctuating environment with exponentially distributed processing times. The environmental changes are driven by a continuous-time discrete state-space Markov chain, and they directly affect the system's procurement price, raw material flow rate, and price-sensitive demand rate. We formulate the system as an infinite-horizon Markov decision process with a long-run average profit criterion and show that the optimal procurement and manufacturing strategies are of state-dependent threshold policies. Besides that, we provide several analytical results on the optimal pricing strategies. We introduce a linear programming formulation to numerically obtain the system's optimal decisions. We, particularly, investigate how production rate, holding cost, procurement price and demand variabilities, customers' price sensitivity, and interaction between supply and demand processes affect the system's performance measures through an extensive numerical study. Furthermore, our numerical results demonstrate the potential benefits of using dynamic pricing compared to that of static pricing. In particular, the profit enhancement being achieved with dynamic pricing can reach up to 15%, depending on the problem parameters.

Scheduling Schemes for Age Optimization in IoT Systems With Limited Retransmission Times

Age of Information (AoI) is a recently introduced metric to capture data freshness. In this article, we consider a multiuser single-destination Internet of Things (IoT) system with periodic state updating, and investigate the scheduling methods of minimizing the long-term average AoI with limited retransmission times. In view of the retransmission mode, the AoI optimization problems for retransmission with and without feedback are analyzed. For the retransmission without feedback, we formulate the expected decision loss (EDL) function of the AoI optimization problem with the finite retransmission times and propose a loss-greedy policy by minimizing the EDL at each step. For the retransmission with feedback, a potential optimal solution is to construct the AoI optimization problem as an infinite-horizon Markov decision process (MDP) and solve the corresponding Bellman optimal equations. However, this optimal solution is prone to suffer from the curse of dimensionality and is hard to implement. To address this issue, we decouple the infinite-horizon MDP into the finite-state MDP in each single frame, and then propose a low-complexity slot-based max-weight (SBMW) policy to minimize the long-term average AoI. Numerical results show that, compared with an ALOHA-like baseline policy, the proposed Loss-Greedy policy can achieve up to 48% reduction of the average AoI for the retransmission without feedback, while the SBMW policy can reduce the average AoI by 57% in the retransmission with feedback. The average performance gain of the proposed policies over the state-of-the-art policies is at least 10%.

Parameterized MDPs and Reinforcement Learning Problems-A Maximum Entropy Principle-Based Framework.

We present a framework to address a class of sequential decision-making problems. Our framework features learning the optimal control policy with robustness to noisy data, determining the unknown state and action parameters, and performing sensitivity analysis with respect to problem parameters. We consider two broad categories of sequential decision-making problems modeled as infinite horizon Markov decision processes (MDPs) with (and without) an absorbing state. The central idea underlying our framework is to quantify exploration in terms of the Shannon entropy of the trajectories under the MDP and determine the stochastic policy that maximizes it while guaranteeing a low value of the expected cost along a trajectory. This resulting policy enhances the quality of exploration early on in the learning process, and consequently allows faster convergence rates and robust solutions even in the presence of noisy data as demonstrated in our comparisons to popular algorithms, such as Q -learning, Double Q -learning, and entropy regularized Soft Q -learning. The framework extends to the class of parameterized MDP and RL problems, where states and actions are parameter dependent, and the objective is to determine the optimal parameters along with the corresponding optimal policy. Here, the associated cost function can possibly be nonconvex with multiple poor local minima. Simulation results applied to a 5G small cell network problem demonstrate the successful determination of communication routes and the small cell locations. We also obtain sensitivity measures to problem parameters and robustness to noisy environment data.

CVaR Optimization for MDPs

We study the problem of Conditional Value-at-Risk (CVaR) optimization for a finite-state Markov Decision Process (MDP) with total discounted costs and the reduction of this problem to a stochastic game with perfect information. The CVaR optimization problem for finite and infinite-horizon MDPs can be reformulated as a zero-sum stochastic game with a compact state space. This game has the following property: while the second player has perfect information including the knowledge of the decision chosen by the first player at the current time instance, the first player does not directly observe the augmented component of the state and does not know current and past decisions chosen by the second player. Using methods of convex analysis, we show optimal policies exist for this game and an optimal policy of the first player optimizes CVaR of the total discounted costs. In addition to proving existence of optimal policies, we provide algorithms for their computation.

Decomposition Methods for Solving Finite-Horizon Large MDPs

Conventional algorithms for solving Markov decision processes (MDPs) become intractable for a large finite state and action spaces. Several studies have been devoted to this issue, but most of them only treat infinite-horizon MDPs. This paper is one of the first works to deal with non-stationary finite-horizon MDPs by proposing a new decomposition approach, which consists in partitioning the problem into smaller restricted finite-horizon MDPs, each restricted MDP is solved independently, in a specific order, using the proposed hierarchical backward induction (HBI) algorithm based on the backward induction (BI) algorithm. Next, the sub-local solutions are combined to obtain a global solution. An example of racetrack problems shows the performance of the proposal decomposition technique.

Robust Markov Decision Processes with Data-Driven, Distance-Based Ambiguity Sets

We consider finite- and infinite-horizon Markov decision processes (MDPs) with unknown state-transition probabilities. They are assumed to belong to certain ambiguity sets, and the goal is to maximize the worst-case expected total discounted reward over all probabilities from these sets. Specifically, the ambiguity set for any state-action-stage triplet is a ball---it includes all probability mass functions (pmfs) within a certain distance from the empirical pmf constructed using historical, independent observations of state transitions. We prove that optimal values in the resulting robust MDPs (RMDPs) converge to the optimal value of the true MDP if the radii of the ambiguity balls approach zero as the sample-size diverges to infinity. In addition, robust optimal policies for sufficiently large sample-sizes are optimal to the true MDP. These results rely on a sufficient condition that links convergence of pmfs with respect to the distance function with their componentwise convergence in an appropriate space. Further, for finite sample-sizes, the optimal value of the RMDP provides a lower bound on the value of the robust optimal policy in the true MDP, with a high probability. A certain concentration inequality is sufficient for this out-of-sample performance guarantee. Several well-known distances satisfy these conditions. Numerical experiments suggest that one can choose from several distance functions to build computationally tractable RMDPs that exhibit good out-of-sample performance, and balance conservativeness with probabilistic guarantees.

Intelligent Sensing and Computing in Wireless Sensor Networks for Multiple Target Tracking

With sixth generation (6G) communication technologies, target sensing can be finished in milliseconds. The mobile tracking-oriented Internet of Things (MTT-IoT) as a kind of emerging application network can detect sensor nodes and track targets within their sensing ranges cooperatively. Nevertheless, huge data processing and low latency demands put tremendous pressure on the conventional architecture where sensing data is executed in the remote cloud and the short transmission distance of 6G channels presents new challenges into the design of network topology. To cope with the above difficulties, this paper proposes a new resource allocation scheme to perform delicate node scheduling and accurate tracking in multitarget tracking mobile networks. The dynamic tracking problem is formulated as an infinite horizon Markov Decision Process (MDP), where the state space that considers energy consumption, system responding delay, and target important degree is extended. A model-free reinforcement learning is applied to obtain satisfied tracking actions by frequent iterations, in which smart agents interact with the complicated environment directly. The performance of each episode is evaluated by the action-value function in search of the optimal reward. Simulation results demonstrate that the proposed scheme shows excellent tracking performance in terms of energy cost and tracking delay.

Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning

Demystifying the Curse of Horizon in Offline Reinforcement Learning in Order to Break It Offline reinforcement learning (RL), where we evaluate and learn new policies using existing off-policy data, is crucial in applications where experimentation is challenging and simulation unreliable, such as medicine. It is also notoriously difficult because the similarity (density ratio) between observed trajectories and those generated by any new policy diminishes exponentially as the horizon grows, known as the curse of horizon, which severely limits the application of offline RL whenever horizons are moderately long or even infinite. In “Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning,” Kallus and Uehara set out to understand these limits and when they can be broken. They precisely characterize the curse by deriving the semiparametric efficiency lower bounds for the policy-value estimation problem in different models. On the one hand, this shows why the curse necessarily plagues standard estimators: they work even in non-Markov models and therefore must be limited by the corresponding bound. On the other hand, greater efficiency is possible in certain Markovian models, and they give the first estimator achieving these much lower efficiency bounds in infinite-horizon Markov decision processes.

Deep Reinforcement Learning for Adaptive Network Slicing in 5G for Intelligent Vehicular Systems and Smart Cities

Intelligent vehicular systems and smart city applications are the fastest growing Internet-of-Things (IoT) implementations at a compound annual growth rate of 30%. In view of the recent advances in IoT devices and the emerging new breed of IoT applications driven by artificial intelligence (AI), the fog radio access network (F-RAN) has been recently introduced for the fifth-generation (5G) wireless communications to overcome the latency limitations of cloud-RAN (C-RAN). We consider the network slicing problem of allocating the limited resources at the network edge (fog nodes) to vehicular and smart city users with heterogeneous latency and computing demands in dynamic environments. We develop a network slicing model based on a cluster of fog nodes (FNs) coordinated with an edge controller (EC) to efficiently utilize the limited resources at the network edge. For each service request in a cluster, the EC decides which FN to execute the task, i.e., locally serve the request at the edge, or to reject the task and refer it to the cloud. We formulate the problem as infinite-horizon Markov decision process (MDP) and propose a deep reinforcement learning (DRL) solution to adaptively learn the optimal slicing policy. The performance of the proposed DRL-based slicing method is evaluated by comparing it with other slicing approaches in dynamic environments and for different scenarios of design objectives. Comprehensive simulation results corroborate that the proposed DRL-based EC quickly learns the optimal policy through interaction with the environment, which enables adaptive and automated network slicing for efficient resource allocation in dynamic vehicular and smart city environments.

A reinforcement learning approach to the stochastic cutting stock problem

We propose a formulation of the stochastic cutting stock problem as a discounted infinite-horizon Markov decision process. At each decision epoch, given current inventory of items, an agent chooses in which patterns to cut objects in stock in anticipation of the unknown demand. An optimal solution corresponds to a policy that associates each state with a decision and minimizes the expected total cost. Since exact algorithms scale exponentially with the state-space dimension, we develop a heuristic solution approach based on reinforcement learning. We propose an approximate policy iteration algorithm in which we apply a linear model to approximate the action-value function of a policy. Policy evaluation is performed by solving the projected Bellman equation from a sample of state transitions, decisions and costs obtained by simulation. Due to the large decision space, policy improvement is performed via the cross-entropy method. Computational experiments are carried out with the use of realistic data to illustrate the application of the algorithm. Heuristic policies obtained with polynomial and Fourier basis functions are compared with myopic and random policies. Results indicate the possibility of obtaining policies capable of adequately controlling inventories with an average cost up to 80% lower than the cost obtained by a myopic policy.

Distributed Optimization for Mobile Robots under Mobile Edge Computing Environment

Driven by the development of the Internet industry, mobile robots (MRs) technology has become increasingly mature and widely used in all walks of life. Since MRs are densely distributed in the network system, how to establish a reliable communication architecture to achieve good cooperation and resource sharing between MRs has become a research hotspot. In this respect, mobile edge computing (MEC) technology and millimeter wave (mmW) technology can provide powerful support. This paper proposes a mmW communication network architecture for distributed MRs in MEC environment. The mmW base station provides reliable communication services for MRs under the coverage of information cloud (IC). We design a joint resource and power allocation strategy aimed at minimizing network energy consumption. First, we use the Lyapunov optimization technique to transform the original infinite horizon Markov decision process (MDP) problem. Then, a semidistributed algorithm is introduced to solve the distributed optimization problem in the mmW network. By improving the autonomous decision-making ability of the mmW base station, the signaling overheads caused by information interaction are reduced, and information leakage is effectively avoided. Finally, the global optimal solution is obtained. Simulation results demonstrate the superiority of the proposed strategy.

Identifying proactive ICU patient admission, transfer and diversion policies in a public-private hospital network

Management of hospital beds is a high-impact issue for two-tier healthcare systems, due principally to their critical importance and high costs. Bed capacity in the public sector is generally insufficient to provide immediate care to all critical patients and thus a significant proportion of public expenditure is assigned to the diversion of patients for treatment in the private sector. We formulate and approximately solve a discounted infinite-horizon Markov Decision Process (MDP) that seeks to identify cost-effective policies for transferring ICU patients between hospitals or diverting them to private clinics. The solution approach employs an affine architecture for approximating the value function of the MDP model and solves the equivalent linear programming model using column generation. The approach can handle a high level of dimensionality, enabling it to consider the arriving patients’ many different diagnostic groups and their corresponding lengths of stay. The decisions generated through this approach often differ from the intuitive ones produced in a typical day-by-day decision process, that does not consider the impact of the current day’s decisions on the future performance of the system. In particular, the resulting policies will in many cases proactively transfer patients to a different public facility or divert them to a private one even though the hospital they first arrived at had beds available. The performance of the proposed method was evaluated by simulating a case study based on data from a hospital network in Santiago, Chile, producing savings of almost 49% due mostly to reduced usage of private services.