Infinite-horizon Markov Decision Process Research Articles

Approximate Dynamic Programming for Military Medical Evacuation Dispatching Policies

Military medical planners must consider how aerial medical evacuation (MEDEVAC) assets will be dispatched when preparing for and supporting high-intensity combat operations. The dispatching authority seeks to dispatch MEDEVAC assets to prioritized requests for service, such that battlefield casualties are effectively and efficiently transported to nearby medical-treatment facilities. We formulate and solve a discounted, infinite-horizon Markov decision process (MDP) model of the MEDEVAC dispatching problem. Because the high dimensionality and uncountable state space of our MDP model renders classical dynamic programming solution methods intractable, we instead apply approximate dynamic programming (ADP) solution methods to produce high-quality dispatching policies relative to the currently practiced closest-available dispatching policy. We develop, test, and compare two distinct ADP solution techniques, both of which utilize an approximate policy iteration (API) algorithmic framework. The first algorithm uses least-squares temporal differences (LSTD) learning for policy evaluation, whereas the second algorithm uses neural network (NN) learning. We construct a notional, yet representative planning scenario based on high-intensity combat operations in southern Azerbaijan to demonstrate the applicability of our MDP model and to compare the efficacies of our proposed ADP solution techniques. We generate 30 problem instances via a designed experiment to examine how selected problem features and algorithmic features affect the quality of solutions attained by our ADP policies. Results show that the respective policies determined by the NN-API and LSTD-API algorithms significantly outperform the closest-available benchmark policies in 27 (90%) and 24 (80%) of the problem instances examined. Moreover, the NN-API policies significantly outperform the LSTD-API policies in each of the problem instances examined. Compared with the closest-available policy for the baseline problem instance, the NN-API policy decreases the average response time of important urgent (i.e., life-threatening) requests by 39 minutes. These research models, methodologies, and results inform the implementation and modification of current and future MEDEVAC tactics, techniques, and procedures, as well as the design and purchase of future aerial MEDEVAC assets.

Approximate dynamic programming for the military aeromedical evacuation dispatching, preemption-rerouting, and redeployment problem

Military medical planners must consider how aeromedical evacuation (MEDEVAC) assets will be utilized when preparing for and supporting combat operations. This research examines the MEDEVAC dispatching, preemption-rerouting, and redeployment (DPR) problem. The intent of this research is to determine high-quality DPR policies that improve the performance of United States Army MEDEVAC systems and ultimately increase the combat casualty survivability rate. A discounted, infinite-horizon Markov decision process (MDP) model of the MEDEVAC DPR problem is formulated and solved via an approximate dynamic programming (ADP) strategy that utilizes a support vector regression value function approximation scheme within an approximate policy iteration algorithmic framework. The objective is to maximize the expected total discounted reward attained by the system. The applicability of the MDP model is examined via a notional, representative planning scenario based on high-intensity combat operations to defend Azerbaijan against a notional aggressor. Computational experimentation is performed to determine how selected problem features and algorithmic features affect the quality of solutions attained by the ADP-generated DPR policies and to assess the efficacy of the proposed solution methodology. The results from the computational experiments indicate the ADP-generated policies significantly outperform the two benchmark policies considered. Moreover, the results reveal that the average service time of high-precedence, time-sensitive requests decreases when an ADP policy is adopted during high-intensity conflicts. As the rate at which requests enter the MEDEVAC system increases, the performance gap between the ADP policy and the first benchmark policy (i.e., the currently practiced, closest-available dispatching policy) increases substantially. Conversely, as the rate at which requests enter the system decreases, the ADP performance improvement over both benchmark policies decreases, indicating the ADP policy provides little-to-no benefit over a myopic approach (e.g., as utilized in the benchmark policies) when the intensity of a conflict is low. Ultimately, this research informs the development and implementation of future tactics, techniques, and procedures for military MEDEVAC operations.

Dynamic Admission Quota Control With Controllable and Uncontrollable Demands and Random Service Time

This article considers an admission quota control which determines the maximum daily admission (known as the quota) of controllable demand that shares capacity with uncontrollable demand. General arrival processes and general discrete stochastic service time are assumed. An infinite-horizon Markov decision process model is proposed to maximize the expected total discounted net reward comprising admission revenues and overcapacity costs. Through formal proofs, the optimal quota control which makes decisions before knowing demand information is found to be equivalent to the optimal classical admission control which makes decisions after knowing such information when the service time is deterministic or no more than two periods. This equivalence is shown numerically when the service time is no more than four periods. Furthermore, structural properties of the optimal quota control and bounds of the optimal quota are established. Finally, several heuristic policies are proposed and compared by numerical experiments.

Scheduling for Mobile Edge Computing With Random User Arrivals—An Approximate MDP and Reinforcement Learning Approach

In this paper, we investigate the scheduling design of a mobile edge computing (MEC) system, where active mobile devices with computation tasks randomly appear in a cell. Every task can be computed at either the mobile device or the MEC server. We jointly optimize the task offloading decision, uplink transmission device selection and power allocation by formulating the problem as an infinite-horizon Markov decision process (MDP). Compared with most of the existing literature, this is the first attempt to address the transmission and computation optimization with random device arrivals in an infinite time horizon to our best knowledge. Due to the uncertainty in the device number and location, the conventional approximate MDP approaches addressing the curse of dimensionality cannot be applied. An alternative and suitable low-complexity solution framework is proposed in this work. We first introduce a baseline scheduling policy, whose value function can be derived analytically with the statistics of random mobile device arrivals. Then, one-step policy iteration is adopted to obtain a sub-optimal scheduling policy whose performance can be bounded analytically. The complexity of deriving the sub-optimal policy is reduced dramatically compared with conventional solutions of MDP by eliminating the complicated value iteration. To address a more general scenario where the statistics of random mobile device arrivals are unknown, a novel and efficient algorithm integrating reinforcement learning and stochastic gradient descent (SGD) is proposed to improve the system performance in an online manner. Simulation results show that the gain of the sub-optimal policy over various benchmarks is significant.

A novel decomposition-based method for solving general-product structure assemble-to-order systems

Assemble-to-order (ATO) strategies are common to many industries. Despite their popularity, ATO systems remain challenging to analyze. We consider a general-product structure ATO problem modeled as an infinite horizon Markov decision process. As the optimal policy of such a system is computationally intractable, we develop a heuristic policy that is based on a decomposition of the original system, into a series of two-component ATO subsystems. We show that our decomposition heuristic policy (DHP) possesses many properties similar to those encountered in special-product structure ATO systems. Extensive numerical experiments show that the DHP is very efficient. In particular, we show that the DHP requires less than 10−5 the time required to obtain the optimal policy, with an average percentage cost gap less than 4% for systems with up to 5 components and 6 products. We also show that the DHP outperforms the state aggregation heuristic of Nadar et al. (2018), in terms of cost and computational effort. We further develop an information relaxation-based lower bound on the performance of the optimal policy. We show that such a bound is very efficient with an average percentage gap not exceeding 0.5% for systems with up to 5 components and 6 products. Using this lower bound, we further show that the average suboptimality gap of the DHP is within 9% for two special-product structure ATO systems, with up to 9 components and 10 products. Using a sophisticated computing platform, we believe the DHP can handle systems with a large number of components and products.

A Dynamic Reliability-Aware Service Placement for Network Function Virtualization (NFV)

Network softwarization is one of the major paradigm shifts in the next generation of networks. It enables programmable and flexible management and deployment of the network. Network function virtualization (NFV) is referred to the deployment of software functions running on commodity servers instead of traditional hardware-based middle-boxes. It is an example of network softwarization. In NFV, a service is defined as a chain of software functions named service chain function (SFC). The process of allocating the resources of servers to the services, called service placement, is the most challenging mission in NFV. Dynamic nature of the service arrivals and departures as well as meeting the service level agreement make the service placement problem even more challenging. In this paper, we propose a model for dynamic reliability-aware service placement based on the simultaneous allocation of the main and backup servers. Then, we formulate the dynamic reliability-aware service placement as an infinite horizon Markov decision process (MDP), which aims to minimize the placement cost and maximize the number of admitted services. In the proposed MDP, the number of active services in the network is considered to be the state of the system, and the state of the idle resources is estimated based on it. Also, the number of possible admitted services is considered as the action of the presented MDP. To evaluate each possible action in the proposed MDP, we use a sub-optimal method based on the Viterbi algorithm named Viterbi-based Reliable Static Service Placement (VRSSP) algorithm. We determine the optimal policy based on value iteration method using an algorithm named VRSSP-based Value Iteration (VVI) algorithm. Eventually, through the extensive simulations, the superiority of the proposed model for dynamic reliability-aware service placement compared to the static solutions is inferred.

Renewal Monte Carlo: Renewal Theory-Based Reinforcement Learning

An online reinforcement learning algorithm called renewal Monte Carlo (RMC) is presented. RMC works for infinite horizon Markov decision processes with a designated start state. RMC is a Monte Carlo algorithm that retains the key advantages of Monte Carlo—viz., simplicity, ease of implementation, and low bias—while circumventing the main drawbacks of Monte Carlo—viz., high variance and delayed updates. Given a parameterized policy $\pi _\theta$ , the algorithm consists of three parts: estimating the expected discounted reward $\mathsf {R}_{\theta }$ and the expected discounted time $\mathsf {T}_\theta$ over a regenerative cycle; estimating the derivatives $\nabla _\theta \mathsf {R}_\theta$ and $\nabla _\theta \mathsf {T}_\theta$ ; and updating the policy parameters using stochastic approximation to find the roots of $\mathsf {R}_\theta \nabla _\theta \mathsf {T}_\theta - \mathsf {T}_\theta \nabla _\theta \mathsf {R}_\theta$ . It is shown that under mild technical conditions, RMC converges to a locally optimal policy. It is also shown that RMC works for postdecision state models as well. An approximate version of RMC is proposed where a regenerative cycle is defined as successive visits to a prespecified “renewal set”. It is shown that if the value function of the system is locally Lipschitz on the renewal set, then RMC converges to an approximate locally optimal policy. Three numerical experiments are presented to illustrate RMC and compare it with other state-of-the-art reinforcement learning algorithms.

Optimal maintenance policies for multi-level preventive maintenance with complex effects

We consider the problem of optimally maintaining a periodically inspected system with multi-level preventive maintenance whose effects are complex. At each inspection, the maintenance decision concerns whether a preventive maintenance action is needed and which level should be selected if preventive maintenance is desired. The objective is to minimize the total expected discounted cost including inspection and maintenance costs. We formulate an infinite-horizon Markov decision process model and establish sufficient conditions to ensure the existence of an optimal monotone control-limit type policy with respect to the system’s deterioration level and age. We also numerically explore the structure of the optimal policy with respect to two additional system states, the level of the last maintenance action and the time since the last maintenance action. Real-world pavement deterioration data is used in our computational experiments, and the results show that the optimal policy is typically of monotone control-limit type.

Path Planning for Minimizing the Expected Cost Until Success

Consider a general path planning problem of a robot on a graph with edge costs, where each node has a Boolean value of success or failure (with respect to some task) with a given probability. The objective is to plan a path for the robot on the graph that minimizes the expected cost until success. In this paper, it is our goal to bring a foundational understanding to this problem. We start by showing how this problem can be optimally solved by formulating it as an infinite-horizon Markov decision process, but with an exponential space complexity. We then formally prove its NP-hardness. To address the space complexity, we then propose a path planner, using a game-theoretic framework, that asymptotically gets arbitrarily close to the optimal solution. Moreover, we also propose two fast and nonmyopic path planners. To show the performance of our framework, we do extensive simulations for two scenarios: a rover on Mars searching for an object for scientific studies, and a robot looking for a connected spot to a remote station (with real data from downtown San Francisco). Our numerical results show a considerable performance improvement over existing state-of-the-art approaches.

Controlled Markov Processes With Safety State Constraints

This paper considers a Markov decision process (MDP) model with safety state constraints, which specify polytopic invariance constraints on the state probability distribution ( pd ) for all time epochs. Typically, in the MDP framework, safety is addressed indirectly by penalizing failure states through the reward function. However, such an approach does not allow imposing hard constraints on the state pd , which could be an issue for practical applications where the chance of failure must be limited to prescribed bounds. In this paper, we explicitly separate state constraints from the reward function. We provide analysis and synthesis methods to impose generalized safety constraints at all time epochs, unlike current constrained MDP approaches where such constraints can only be imposed on the stationary distributions. We show that, contrary to the unconstrained MDP policies, optimal safe MDP policies depend on the initial state pd . We present novel algorithms for both finite- and infinite-horizon MDPs to synthesize feasible decision-making policies that satisfy safety constraints for all time epochs and ensure that the performance is above a computable lower bound. Linear programming implementations of the proposed algorithms are developed, which are formulated by using the duality theory of convex optimization. A swarm control simulation example is also provided to demonstrate the use of proposed algorithms.

An approximate dynamic programming approach for sequential pig marketing decisions at herd level

One of the most important operations in the production of growing/finishing pigs is the marketing of pigs for slaughter. While pork production can be managed at different levels (animal, pen, section, or herd), it is beneficial to consider the herd level when determining the optimal marketing policy due to inter-dependencies, such as those created by fixed transportation costs and cross-level constraints.In this paper, we consider sequential marketing decisions at herd level. A high-dimensional infinite-horizon Markov decision process (MDP) is formulated which, due to the curse of dimensionality, cannot be solved using standard MDP optimization techniques. Instead, approximate dynamic programming (ADP) is applied to solve the model and find the best marketing policy at herd level. Under the total expected discounted reward criterion, the proposed ADP approach is first compared with a standard solution algorithm for solving an MDP at pen level to show the accuracy of the solution procedure. Next, numerical experiments at herd level are given to confirm how the marketing policy adapts itself to varying costs (e.g., transportation cost) and cross-level constraints. Finally, a sensitivity analysis for some parameters in the model is conducted and the marketing policy found by ADP is compared with other well-known marketing polices, often applied at herd level.

Purchasing, production, and sales strategies for a production system with limited capacity, fluctuating sales and purchasing prices

In many industries, the revenue and cost structures of manufacturers are directly affected by the volatility of purchasing and sales prices in the markets. We analyze the purchasing, production, and sales policies for a continuous-review discrete material flow production/inventory system with fluctuating and correlated purchasing and sales prices, exponentially distributed raw material and demand inter-arrival times, and processing time. The sales and purchasing prices are driven by the random environmental changes that evolve according to a discrete state space continuous-time Markov process. We model the system as an infinite-horizon Markov decision process under the average reward criterion and prove that the optimal purchasing, production, and sales strategies are state-dependent threshold policies. We propose a linear programming formulation to compute the optimal threshold levels. We examine the effects of the sales price variation, purchasing price variation, correlation between sales and purchasing prices, customer arrival rate and limited inventory capacities on the system performance measures, through a range of numerical experiments. We also examine under which circumstances the use of the optimal policy notably improves the system profit compared to the use of the buy low and sell high naive policy. We show that using the optimal purchasing, production, and sales policies allow manufacturers to improve their profits when the purchasing and sales prices fluctuate.

Scheduling Elective Surgeries with Markov Decision Process and Approximate Dynamic Programming

This paper deals with the dynamic advance scheduling of elective surgeries with multiple sources of uncertainties taken into consideration. A waiting list is established to facilitate the management of elective patients from different specialties. Each patient in the waiting list is assigned a dynamic priority which is dependent on the relative importance of specialty, urgency level, and actual waiting time. At the end of each week, the number and type of elective surgeries to be performed in the following week should be properly determined to minimize an integrated cost function, including the costs incurred by performing and delaying surgeries as well as the penalties for overuse of operating rooms and shortage of recovery beds. The studied problem is formulated as an infinite-horizon Markov decision process (MDP) model. Considering that conventional dynamic programming algorithms cannot efficiently solve MDP models for real-sized problems, we develop an approximate dynamic programming (ADP) approach that combines recursive least-squares temporal difference learning and mixed integer programming. Results of numerical experiments validate the efficiency and accuracy of the proposed ADP approach and indicate that this approach can be employed by hospital managers in the future to efficiently solve real-sized surgery scheduling problems.

Examining military medical evacuation dispatching policies utilizing a Markov decision process model of a controlled queueing system

Military medical planners must develop dispatching policies that dictate how aerial medical evacuation (MEDEVAC) units are utilized during major combat operations. The objective of this research is to determine how to optimally dispatch MEDEVAC units in response to 9-line MEDEVAC requests to maximize MEDEVAC system performance. A discounted, infinite horizon Markov decision process (MDP) model is developed to examine the MEDEVAC dispatching problem. The MDP model allows the dispatching authority to accept, reject, or queue incoming requests based on a request’s classification (i.e., zone and precedence level) and the state of the MEDEVAC system. A representative planning scenario based on contingency operations in southern Afghanistan is utilized to investigate the differences between the optimal dispatching policy and three practitioner-friendly myopic policies. Two computational experiments are conducted to examine the impact of selected MEDEVAC problem features on the optimal policy and the system performance measure. Several excursions are examined to identify how the 9-line MEDEVAC request arrival rate and the MEDEVAC flight speeds impact the optimal dispatching policy. Results indicate that dispatching MEDEVAC units considering the precedence level of requests and the locations of busy MEDEVAC units increases the performance of the MEDEVAC system. These results inform the development and implementation of MEDEVAC tactics, techniques, and procedures by military medical planners. Moreover, an analysis of solution approaches for the MEDEVAC dispatching problem reveals that the policy iteration algorithm substantially outperforms the linear programming algorithms executed by CPLEX 12.6 with regard to computational effort. This result supports the claim that policy iteration remains the superlative solution algorithm for exactly solving computationally tractable Markov decision problems.

Joint spectrum access and transmission power management for energy harvesting cognitive radio sensor networks

In this paper, we investigate the joint optimisation of spectrum access and the transmission power for an energy harvesting cognitive sensor node, which operates in time-slotted fashion with causal knowledge of channel conditions along with the energy harvesting states. Allowing for the sensing imperfection, we formulate this joint optimisation problem as an infinite-horizon discretetime Markov decision process (MDP), in which the cognitive sensor aims at maximising the long-term expected throughput. An optimal policy which specifies the spectrum access decision as well as the power level is proposed. It is indicated that the optimal long-term expected throughput is nondecreasing with the battery available energy. Moreover, we introduce a low-complexity policy and prove that the optimal low-complexity policy has a threshold structure with respect to the battery available energy. An efficient algorithm for deriving the optimal low-complexity policy is introduced. Finally, numerical results are presented to confirm the superiority of proposed policies.

Optimal control of a continuous-time W-configuration assemble-to-order system

We analyze a W-configuration assemble-to-order system with random lead times, random arrival of demand, and lost sales, in continuous time. Specifically, we assume exponentially distributed production and demand inter-arrival times. We formulate the problem as an infinite-horizon Markov decision process. We deviate from the standard approach by first characterizing a region (the recurrent region) of the state space where all properties of the cost function hold. We then characterize the optimal policy within this region. In particular, we show that within the recurrent region components are always produced. We also characterize the optimal component allocation policy which specifies whether an arriving product demand should be fulfilled. Our analysis reveals that the optimal allocation policy is counter-intuitive. For instance, even when one product dominates the other, in terms of lost sale cost and lost sale cost rate (i.e., demand rate times the lost sale cost), its demand may not have absolute priority over the other product's demand. We also show that the structure of the optimal policy remains the same for systems with batch production, Erlang distributed production times, and non-unitary product demand. Finally, we propose efficient heuristics that can be either used as an approximation to the optimal policy or can be used as a starting policy for the common algorithms that are used to obtain the optimal policy in an effort to reduce their computational time.

Timely Wireless Flows With General Traffic Patterns: Capacity Region and Scheduling Algorithms

Most existing wireless networking solutions are best-effort and do not provide any delay guarantee required by important applications, such as mobile multimedia conferencing and real-time control of cyber-physical systems. Recently, Hou and Kumar provided a novel framework for analyzing and designing delay-guaranteed wireless networking solutions. While inspiring, their idle-time-based analysis applies only to flows with a special traffic pattern called the frame-synchronized setting . The problem remains largely open for general traffic patterns . This paper addresses this challenge by proposing a general framework that characterizes and achieves the complete delay-constrained capacity region with general traffic patterns in single-hop downlink access-point wireless networks. We first show that the timely wireless flow problem is fundamentally an infinite-horizon Markov decision process (MDP). Then, we judiciously combine different simplification methods to prove that the timely capacity region can be characterized by a finite-size convex polygon. This for the first time allows us to characterize the timely capacity region of wireless flows with general traffic patterns. We then design three scheduling policies to optimize network utility and/or support feasible timely throughput vectors for general traffic patterns. The first policy achieves the optimal network utility and supports any feasible timely throughput vector but suffers from the curse of dimensionality. The second and third policies are inspired by our MDP framework and are of much lower complexity. Simulation results show that both achieve near-optimal performance and outperform other existing alternatives.

Linear programming formulation for non-stationary, finite-horizon Markov decision process models

Linear programming (LP) formulations are often employed to solve stationary, infinite-horizon Markov decision process (MDP) models. We present an LP approach to solving non-stationary, finite-horizon MDP models that can potentially overcome the computational challenges of standard MDP solution procedures. Specifically, we establish the existence of an LP formulation for risk-neutral MDP models whose states and transition probabilities are temporally heterogeneous. This formulation can be recast as an approximate linear programming formulation with significantly fewer decision variables.

Population-level intervention and information collection in dynamic healthcare policy

We develop a general framework for optimal health policy design in a dynamic setting. We consider a hypothetical medical intervention for a cohort of patients where one parameter varies across cohorts with imperfectly observable linear dynamics. We seek to identify the optimal time to change the current health intervention policy and the optimal time to collect decision-relevant information. We formulate this problem as a discrete-time, infinite-horizon Markov decision process and we establish structural properties in terms of first and second-order monotonicity. We demonstrate that it is generally optimal to delay information acquisition until an effect on decisions is sufficiently likely. We apply this framework to the evaluation of hepatitis C virus (HCV) screening in the general population determining which birth cohorts to screen for HCV and when to collect information about HCV prevalence.

Optimal Stochastic Power Allocation and Relay Selection for Energy Harvesting Systems

In this letter, we develop a stochastic joint relay selection and transmit power allocation scheme for a two-hop amplify-and-forward multi-relay network, where relays harvest renewable energies from the surrounding environment. We study the optimal policy for relay selection and power allocation under unknown statistics of the channel fading and energy arrival processes. In particular, we formulate a stochastic optimization problem as an infinite-horizon Markov decision process. Then we develop a low-complexity online algorithm that does not require the statistics of the channel fading and energy harvesting processes to be known. Simulation results reveal that the developed algorithm performs close to the offline scheme proposed in the literature.