Continuous-time Markov Decision Processes Research Articles

Continuous-Time Mean Field Markov Decision Models

We consider a finite number of N statistically equal agents, each moving on a finite set of states according to a continuous-time Markov Decision Process (MDP). Transition intensities of the agents and generated rewards depend not only on the state and action of the agent itself, but also on the states of the other agents as well as the chosen action. Interactions like this are typical for a wide range of models in e.g. biology, epidemics, finance, social science and queueing systems among others. The aim is to maximize the expected discounted reward of the system, i.e. the agents have to cooperate as a team. Computationally this is a difficult task when N is large. Thus, we consider the limit for N→∞.\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$N\\rightarrow \\infty .$$\\end{document} In contrast to other papers we treat this problem from an MDP perspective. This has the advantage that we need less regularity assumptions in order to construct asymptotically optimal strategies than using viscosity solutions of HJB equations. The convergence rate is 1/N\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$1/\\sqrt{N}$$\\end{document}. We show how to apply our results using two examples: a machine replacement problem and a problem from epidemics. We also show that optimal feedback policies from the limiting problem are not necessarily asymptotically optimal.

Enhancing military medical evacuation dispatching with armed escort management: comparing model-based reinforcement learning approaches

The military medical evacuation (MEDEVAC) dispatching problem involves determining optimal policies for evacuating combat casualties to maximize patient survivability during military operations. This study explores a variation of the MEDEVAC dispatching problem, focusing on controlling armed escorts using a Markov decision process (MDP) model and model-based reinforcement learning (RL) approaches. A discounted, continuous-time MDP model over an infinite horizon is developed to maximize the expected total discounted reward of the system. Two model-based RL solution approaches are proposed: one utilizing semi-gradient descent Q-learning and another employing semi-gradient descent SARSA. A computational example, set in western and central Africa during contingency operations, assesses the performance of the RL-generated policies against the myopic policy, which military medical planners currently employ. Solution quality is derived from expected response time, a crucial determinant of life-saving potential in MEDEVAC operations. The research also explores sensitivity analysis and excursion scenarios to evaluate the RL-generated policies further. By explicitly controlling armed escort assets, dispatching authorities can better manage the location and allocation of these resources throughout combat operations. The findings of this study have the potential to inform military medical planning, operations, and tactics, ultimately leading to improved MEDEVAC system performance and higher patient survivability rates.

Viscosity solutions approach to finite-horizon continuous-time Markov decision process

This paper investigates the optimal control problems for the finite-horizon continuous-time Markov decision processes with delay-dependent control policies. We develop compactification methods in decision processes and show that the existence of optimal policies. Subsequently, through the dynamic programming principle of the delay-dependent control policies, the differential-difference Hamilton-Jacobi-Bellman (HJB) equation in the setting of discrete space is established. Under certain conditions, we give the comparison principle and further prove that the value function is the unique viscosity solution to this HJB equation. Based on this, we show that among the class of delay-dependent control policies, there is an optimal one which is Markovian.

Condition-Based Production for Stochastically Deteriorating Systems: Optimal Policies and Learning

Problem definition: Production systems deteriorate stochastically due to use and may eventually break down, resulting in high maintenance costs at scheduled maintenance moments. This deterioration behavior is affected by the system’s production rate. Although producing at a higher rate generates more revenue, the system may also deteriorate faster. Production should thus be controlled dynamically to tradeoff deterioration and revenue accumulation in between maintenance moments. We study systems for which the relation between production and deterioration is known and the same for each system and systems for which this relation differs from system to system and needs to be learned on-the-fly. The decision problem is to find the optimal production policy given planned maintenance moments (operational) and the optimal interval length between such maintenance moments (tactical). Methodology/results: For systems with a known production-deterioration relation, we cast the operational decision problem as a continuous time Markov decision process and prove that the optimal policy has intuitive monotonic properties. We also present sufficient conditions for the optimality of bang-bang policies, and we partially characterize the structure of the optimal interval length, thereby enabling efficient joint optimization of the operational and tactical decision problem. For systems that exhibit variability in their production-deterioration relations, we propose a Bayesian procedure to learn the unknown deterioration rate under any production policy. Numerical studies indicate that on average across a wide range of settings (i) condition-based production increases profits by 50% compared with static production, (ii) integrating condition-based production and maintenance decisions increases profits by 21% compared with the state-of-the-art sequential approach, and (iii) our Bayesian approach performs close, especially in the bang-bang regime, to an Oracle policy that knows each system’s production-deterioration relation. Managerial implications: Production should be adjusted dynamically based on real-time condition monitoring and the tactical maintenance planning should anticipate and integrate these operational decisions. Our proposed framework assists managers to do so optimally. Funding: This work was supported by the Nederlandse Organisatie voor Wetenschappelijk Onderzoek [Grant 439.17.708]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/msom.2022.0473 .

Managing inventories of reusable containers for food take-away at a restaurant

Single-use packaging in the food services sector accounts for a substantial amount of waste, leading some restaurants to offer customers reusable containers for take-away or delivery orders. Once used, customers may return these containers to the same restaurant or another restaurant in the reusable container network or they may not return them at all. The restaurant hence faces both uncertain demand and returns for reusable containers and needs to decide on the number of containers to stock to serve its customers. We formulate this problem by modeling it as a continuous-time Markov Decision Process. Through a numerical study, we investigate the effect that different balances of demand and return intensities and their coupling have on the average total cost for the restaurant. We find that greater demand and return coupling reduces average costs, but the effects are most beneficial when the overall demand and returns of the restaurant are balanced. The restaurant can reduce costs by optimizing the supplier visit frequency in addition to the inventory level of clean containers after the supplier visit. The supplier’s choice of the level of the visit cost is important as smaller scale restaurants may be penalized by a larger supplier visit cost, dissuading them from participating in reusable container systems.

Managing priorities on on-demand service platforms with waiting time differentiation

Inspired by recent applications, this article considers the problem of an on-demand service platform (e.g., Uber, Lyft, Deliveroo) that differentiates the service of customers according to their willingness to wait and to pay. Upon arrival, customers can be admitted to the system or rejected and, at any time, service providers can be allocated to admitted customers or reserved for future customers. We formulate this problem as a continuous-time Markov decision process, which is used to find the optimal policy. Exploiting properties of the value function, we first show that the optimal policy is described by state-dependent admission thresholds and a strict priority rule with respect to service provider allocation. In a numerical study, we then compare the performance of the optimal policy with simpler policies. We find that the platform, the customers, and the service providers can benefit from a policy that controls customer admission and service provider reservation. Finally, some evidence of the robustness of the policy is provided in two extensions.

Reinforcement Learning for Omega-Regular Specifications on Continuous-Time MDP

Continuous-time Markov decision processes (CTMDPs) are canonical models to express sequential decision-making under dense-time and stochastic environments. When the stochastic evolution of the environment is only available via sampling, model-free reinforcement learning (RL) is the algorithm-of-choice to compute optimal decision sequence. RL, on the other hand, requires the learning objective to be encoded as scalar reward signals. Since doing such translations manually is both tedious and error-prone, a number of techniques have been proposed to translate high-level objectives (expressed in logic or automata formalism) to scalar rewards for discrete-time Markov decision processes. Unfortunately, no automatic translation exists for CTMDPs. We consider CTMDP environments against the learning objectives expressed as omega-regular languages. Omega-regular languages generalize regular languages to infinite-horizon specifications and can express properties given in popular linear-time logic LTL. To accommodate the dense-time nature of CTMDPs, we consider two different semantics of omega-regular objectives: 1) satisfaction semantics where the goal of the learner is to maximize the probability of spending positive time in the good states, and 2) expectation semantics where the goal of the learner is to optimize the long-run expected average time spent in the ''good states'' of the automaton. We present an approach enabling correct translation to scalar reward signals that can be readily used by off-the-shelf RL algorithms for CTMDPs. We demonstrate the effectiveness of the proposed algorithms by evaluating it on some popular CTMDP benchmarks with omega-regular objectives.

Managing production-inventory-maintenance systems with condition monitoring

Providing spare parts and maintenance services to customers has become an important revenue stream for equipment manufacturers. This paper studies a Production-Inventory-Maintenance (PIM) system where a manufacturer serves an installed base of machines. Each machine contains the same critical component, whose degradation behaviour is described by a Markov process and captured via condition monitoring. When the state of a component reaches a prespecified replacement threshold, the machine is considered to have failed and incurs downtime penalties until being repaired. There are a few repair servers and queueing dynamics follows the Machine Interference Problem (MIP). Component deterioration and server congestion provide advance demand information, which helps plan the production of spare parts in a make-to-stock routine. The unsatisfied requests for spare parts are fulfilled by emergency replenishments with lost-sales penalties. We formulate the production-control problem as a continuous-time Markov decision process, characterize the structure of the optimal policy, and develop an effective heuristic policy. Through numerical experiments, we reveal a number of notable differences between managing PIM systems and operating MIP systems alone: (1) Although identifying more degradation states has no value to MIP systems, PIM systems benefit substantially from such a refined state identification. (2) While MIP systems prefer more repair servers and higher repair operation efficiency, greater repair capacity may worsen the performance of PIM systems. (3) When the product of repair service rate and server number is fixed, a single server is always preferred in MIP systems. However, PIM systems could perform better with multiple less-efficient servers.

Continuous-Time Markov Decision Processes Under the Risk-Sensitive First Passage Discounted Cost Criterion

Continuous-Time Markov Decision Processes Under the Risk-Sensitive First Passage Discounted Cost Criterion

Solving the joint military medical evacuation problem via a random forest approximate dynamic programming approach

Members of the armed forces rely on having an effective and efficient medical evacuation (MEDEVAC) process for evacuating casualties from the battlefield to medical treatment facilities during combat operations. Determining which MEDEVAC units to task to respond to incoming requests for service, known as the MEDEVAC dispatching problem, is a natural sequential decision making problem and is the focus of this paper. Unlike previous research in this area, this paper incorporates triage classification errors and the possibility of having blood transfusion kits on board select MEDEVAC units. A discounted, infinite-horizon continuous-time Markov decision process model is formulated to examine the MEDEVAC dispatching problem and to compare generated dispatching policies to the currently practiced United States military policy. We utilize an approximate dynamic programming (ADP) technique that leverages a random forest value function approximation within an approximate policy iteration algorithmic framework to develop high-quality policies for both small-scale and large-scale problem instances. A representative planning scenario involving joint combat operations in South Korea is developed and utilized to investigate the differences between the various policies. Results from the analysis indicate that applying ADP techniques can improve current practices by as much as 29% with regard to a life-saving performance metric. This research is of particular interest to the military medical community and can inform the procedures of future military MEDEVAC operations.

A note on the existence of optimal stationary policies for average Markov decision processes with countable states

In many practical stochastic dynamic optimization problems with countable states, the optimal policy possesses certain structural properties. For example, the (s,S) policy in inventory control, the well-known cμ-rule and the recently discovered c/μ-rule (Xia et al. (2022)) in scheduling of queues. A presumption of such results is that an optimal stationary policy exists. There are many research works regarding to the existence of optimal stationary policies of Markov decision processes with countable state spaces (see, e.g., Bertsekas (2012); Hernández-Lerma and Lasserre (1996); Puterman (1994); Sennott (1999)). However, these conditions are usually not easy to verify in such optimization problems. In this paper, we study the optimization of long-run average of continuous-time Markov decision processes with countable state spaces. We provide an intuitive approach to prove the existence of an optimal stationary policy. The approach is simply based on compactness of the policy space, with a special designed metric, and the continuity of the long-run average in the space. Our method is capable to handle cost functions unbounded from both above and below, which makes a complementary contribution to the literature work where the cost function is unbounded from only one side. Examples are provided to illustrate the application of our main results.

Optimizing the First Response to Sepsis: An Electronic Health Record-Based Markov Decision Process Model

Sepsis is considered a medical emergency where delays in initial treatment are associated with increased morbidity and mortality, yet there is no gold standard for identifying sepsis onset and thus treatment timing. We leverage electronic health record (EHR) data with clinical expertise to develop a continuous-time Markov decision process (MDP) optimal stopping model that identifies the optimal first intervention action (anti-infective, fluid, or wait). To study the impact of initial treatment of patients at risk for developing sepsis, we define the delayed treatment population who received delayed treatment upon admission or during hospitalization and serves as an approximation of the natural history of sepsis. We apply the optimal first treatment policy to sample patient visits from the nondelayed treatment population. This analysis indicates the average risk of death could be reduced by approximately 2.2%, the average time until treatment could be reduced by 106 minutes, and the average severity of the treatment state could be reduced by 15.5% compared with the treatment they received in the hospital. We study the properties of the optimal policy to define an easily interpretable initial treatment heuristic that considers a patient’s organ dysfunction, location, and septic shock status. This generalizable framework can inform personalized treatment of patients at risk for sepsis. History: This paper has been accepted for the Decision Analysis Special Issue on Emerging Topics in Health Decision Analysis. Funding: This material is based upon work supported by the National Science Foundation [Grant 1522107 (North Carolina State University), 1522106 (Mayo Clinic), and 1833538 (Drexel University)].

Scheduling servers in a two-stage queue with abandonments and costs

AbstractWe consider the assignment of servers to two phases of service in a two-stage tandem queueing system when customers can abandon from each stage of service. New jobs arrive at both stations. Jobs arriving at station 1 may go through both phases of service and jobs arriving at station 2 may go through only one phase of service. Stage-dependent holding and lump-sum abandonment costs are incurred. Continuous-time Markov decision process formulations are developed that minimize discounted expected and long-run average costs. Because uniformization is not possible, we use the continuous-time framework and sample path arguments to analyze control policies. Our main results are conditions under which priority rules are optimal for the single-server model. We then propose and evaluate threshold policies for allocating one or more servers between the two stages in a numerical study. These policies prioritize a phase of service before “switching” to the other phase when total congestion exceeds a certain number. Results provide insight into how to adjust the switching rule to significantly reduce costs for specific input parameters as well as more general multi-server situations when neither preemption or abandonments are allowed during service and service and abandonment times are not exponential.

Nonzero-Sum Risk-Sensitive Continuous-Time Stochastic Games with Ergodic Costs

We study nonzero-sum stochastic games for continuous time Markov decision processes on a denumerable state space with risk-sensitive ergodic cost criterion. Transition rates and cost rates are allowed to be unbounded. Under a Lyapunov type stability assumption, we show that the corresponding system of coupled HJB equations admits a solution which leads to the existence of a Nash equilibrium in stationary strategies. We establish this using an approach involving principal eigenvalues associated with the HJB equations. Furthermore, exploiting appropriate stochastic representation of principal eigenfunctions, we completely characterize Nash equilibria in the space of stationary Markov strategies.

Gradual-Impulsive Control for Continuous-Time Markov Decision Processes with Total Undiscounted Costs and Constraints: Linear Programming Approach via a Reduction Method

We consider the constrained optimal control problem for the gradual-impulsive CTMDP model with the performance criteria being the expected total undiscounted costs (from the running cost and the cost from each time an impulse being applied). The discounted model is covered as a special case. We justify fully a reduction method, and close an open issue in the previous literature. The reduction method induces an equivalent but simpler standard CTMDP model with gradual control only, based on which, we establish effectively, under rather natural conditions, a linear programming approach for solving the concerned constrained optimal control problem.

Risk-sensitive discounted cost criterion for continuous-time Markov decision processes on a general state space

In this paper, we consider risk-sensitive discounted control problem for continuous-time jump Markov processes taking values in general state space. The transition rates of underlying continuous-time jump Markov processes and the cost rates are allowed to be unbounded. Under certain Lyapunov condition, we establish the existence and uniqueness of the solution to the Hamilton–Jacobi–Bellman equation. Also, we prove the existence of optimal risk-sensitive control in the class of Markov control and completely characterized the optimal control.

Asymptotic Optimality and Rates of Convergence of Quantized Stationary Policies in Continuous‐Time Markov Decision Processes

This paper is concerned with the asymptotic optimality of quantized stationary policies for continuous‐time Markov decision processes (CTMDPs) in Polish spaces with state‐dependent discount factors, where the transition rates and reward rates are allowed to be unbounded. Using the dynamic programming approach, we first establish the discounted optimal equation and the existence of its solutions. Then, we obtain the existence of optimal deterministic stationary policies under suitable conditions by more concise proofs. Furthermore, we discretize and incentivize the action space and construct a sequence of quantizer policies, which is the approximation of the optimal stationary policies of the CTMDPs, and get the approximation result and the rates of convergence on the expected discounted rewards of the quantized stationary policies. Also, we give an iteration algorithm on the approximate optimal policies. Finally, we give an example to illustrate the asymptotic optimality.

Optimal speed profile of a DVFS processor under soft deadlines

We consider a Dynamic Voltage and Frequency Scaling (DVFS) processor executing jobs with obsolescence deadlines: A job becomes obsolete and is removed from the system if it is not completed before its deadline. The objective is to design a dynamic speed policy for the processor that minimizes its average energy consumption plus an obsolescence cost per deadline miss. Under Poisson arrivals and exponentially distributed deadlines and job sizes, we show that this problem can be modeled as a continuous time Markov decision process (MDP) with unbounded state space and unbounded rates. While this MDP admits a continuous time optimality equation for its average cost, the standard uniformization approach is not applicable. Inspired by the scaling method introduced by Blok and Spieksma, we first define a family of truncated MDPs and we then show that the optimal speed profiles are increasing in the number of jobs in the system and are uniformly bounded by a constant. Finally, we show that these properties are inherited from the original (infinite) system. The proposed upper bound on the optimal speed profile is tight and is used to develop an extremely simple policy that accurately approximates the optimal average cost in heavy traffic conditions.

Sampling and Remote Estimation for the Ornstein-Uhlenbeck Process Through Queues: Age of Information and Beyond

Recently, a connection between the age of information and remote estimation error was found in a sampling problem of Wiener processes: If the sampler has no knowledge of the signal being sampled, the optimal sampling strategy is to minimize the age of information; however, by exploiting causal knowledge of the signal values, it is possible to achieve a smaller estimation error. In this paper, we generalize the previous study by investigating a problem of sampling a stationary Gauss-Markov process named the Ornstein-Uhlenbeck (OU) process, where we aim to find useful insights for solving the problems of sampling more general signals. The optimal sampling problem is formulated as a constrained continuous-time Markov decision process (MDP) with an uncountable state space. We provide an exact solution to this MDP: The optimal sampling policy is a threshold policy on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">instantaneous estimation error</i> and the threshold is found. Further, if the sampler has no knowledge of the OU process, the optimal sampling problem reduces to an MDP for minimizing a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">nonlinear</i> age of information metric. The age-optimal sampling policy is a threshold policy on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">expected estimation error</i> and the threshold is found. In both problems, the optimal sampling policies can be computed by low-complexity algorithms (e.g., bisection search and Newton’s method), and the curse of dimensionality is circumvented. These results hold for (i) general service time distributions of the queueing server and (ii) sampling problems both with and without a sampling rate constraint. Numerical results are provided to compare different sampling policies.

Risk-sensitive average continuous-time Markov decision processes with unbounded transition and cost rates

AbstractThis paper considers risk-sensitive average optimization for denumerable continuous-time Markov decision processes (CTMDPs), in which the transition and cost rates are allowed to be unbounded, and the policies can be randomized history dependent. We first derive the multiplicative dynamic programming principle and some new facts for risk-sensitive finite-horizon CTMDPs. Then, we establish the existence and uniqueness of a solution to the risk-sensitive average optimality equation (RS-AOE) through the results for risk-sensitive finite-horizon CTMDPs developed here, and also prove the existence of an optimal stationary policy via the RS-AOE. Furthermore, for the case of finite actions available at each state, we construct a sequence of models of finite-state CTMDPs with optimal stationary policies which can be obtained by a policy iteration algorithm in a finite number of iterations, and prove that an average optimal policy for the case of infinitely countable states can be approximated by those of the finite-state models. Finally, we illustrate the conditions and the iteration algorithm with an example.