Continuous-time Markov Decision Processes Research Articles

The risk probability criterion for discounted continuous-time Markov decision processes

In this paper, we consider the risk probability minimization problem for infinite discounted continuous-time Markov decision processes (CTMDPs) with unbounded transition rates. First, we introduce a class of policies depending on histories with the additional reward levels. Then, we construct the corresponding probability spaces, and also establish the non-explosion of the state process. Secondly, under suitable conditions we prove that the value function is a solution to the optimality equation for the probability criterion by an iteration technique, and obtain a value iteration algorithm to compute (at least approximate) the value function. Furthermore, under an additional condition we establish the uniqueness of the solution to the optimality equation and the existence of an optimal policy. Finally, we illustrate our results with two examples. The first one is used to verify our conditions for CTMDPs with unbounded transition rates, the second one for the numerical calculation of the value function and an optimal policy.

Constrained total undiscounted continuous-time Markov decision processes

The present paper considers the constrained optimal control problem with total undiscounted criteria for a continuous-time Markov decision process (CTMDP) in Borel state and action spaces. Under the standard compactness and continuity conditions, we show the existence of an optimal stationary policy out of the class of general nonstationary ones. In the process, we justify the reduction of the CTMDP model to a discrete-time Markov decision process (DTMDP) model based on the studies of the undiscounted occupancy and occupation measures. We allow that the controlled process is not necessarily absorbing, and the transition rates are not necessarily separated from zero, and can be arbitrarily unbounded; these features count for the main technical difficulties in studying undiscounted CTMDP models.

Precisely deciding CSL formulas through approximate model checking for CTMCs

The model checking problem of continuous-time Markov chains with respect to continuous-time stochastic logic was introduced and shown to be decidable by Aziz et al. [1,2] in 1996. Unfortunately, their proof is only constructive, but highly unpractical. Later in 2000, an efficient approximate algorithm was proposed by Baier et al. [3,5] for a sublogic with binary until. In this paper, we apply transcendental number theory and classical linear algebra to bridge the gap between the precise but unpractical algorithm, and the imprecise but efficient approximate algorithm. We prove that the approximate algorithm in [3,5] can be used as an off-the-shell tool for a precise model checking algorithm for binary until formulas. Further, we discuss extensions of our results to nested until and continuous-time Markov decision processes.

A Real-Time Access Control of Patient Service in the Outpatient Clinic

With the increasing demand of patients for limited healthcare services, e.g., expert physicians in general hospitals and the long on-site waiting time of patients, a real-time admission control (AC) policy was developed considering both distinction and fairness among heterogeneous patients. The AC policy chooses the customer for the next service from the waiting area to minimize the expected total disutility according to the current system state. A continuous-time Markov decision process in a finite horizon was developed, and a myopic optimal policy was derived using a switching curve to discriminate patient admission based on patient types and waiting time. The effect of each parameter on the switching curve is analyzed, and managerial insights are discussed. In the numerical experiment, an empirical case in the Pediatric Department of Peking University Third Hospital is conducted using simulation models. Three commonly used scheduling policies and the myopic policy (MP) are tested in both Markovian settings and a real-world scenario. Simulation results confirm the effectiveness of the MP, because the total disutility of patients is significantly alleviated. Furthermore, the service experience of patients is expected to improve with the application of the AC policy.

Mean-variance optimality for semi-Markov decision processes under first passage criteria

This paper deals with a first passage mean-variance problem for semi-Markov decision processes in Borel spaces. The goal is to minimize the variance of a total discounted reward up to the system's first entry to some target set, where the optimization is over a class of policies with a prescribed expected first passage reward. The reward rates are assumed to be possibly unbounded, while the discount factor may vary with states of the system and controls. We first develop some suitable conditions for the existence of first passage mean-variance optimal policies and provide a policy improvement algorithm for computing an optimal policy. Then, two examples are included to illustrate our results. At last, we show how the results here are reduced to the cases of discrete-time Markov decision processes and continuous-time Markov decision processes.

Engage Others or Leave it to the Source? On Optimal Message Replication in DTNs Under Imperfect Cooperation

The message replication strategy, namely the way message copies are generated and diffused (a.k.a. sprayed) in the network, is a fundamental, yet not thoroughly explored, component of all multi-copy message forwarding schemes in opportunistic networks. Whereas almost all related protocols rely to some extent on the assistance of intermediate nodes for relaying message copies towards their destination, the generation of new message copies may either involve them or be carried out exclusively by the message source. This paper first formulates and solves analytical models for the performance of the two most popular message spraying strategies under imperfect node cooperation and homogeneous exponentially distributed pairwise node inter-contact times. Numerical results suggest that as the node cooperation decreases source replication consistently outperforms binary replication, i.e., the optimal variant of intermediate node-assisted replication under nominal conditions of full node cooperation. The analytical conclusions are also verified for more realistic node mobility patterns through trace-driven experimentation. We then formulate the ideal selection of replication mode as a finite-horizon Continuous Time Markov Decision Process with restricted decision epochs and solve it for the optimal spraying policy. The optimal policy coincides with the binary-(source-) spraying in the presence of few (resp. many) misbehaving nodes. To better approximate it at intermediate levels of misbehavior intensity, we introduce and analyze a simple static spraying policy permitting source-destination space-time paths of up to three hops. Our work deepens the understanding of a core operation embedded in a broad range of multi-copy DTN forwarding protocols. At the same time, it advocates a more thorough approach to the design and evaluation of DTN forwarding that accounts for beyond-nominal conditions.

Average cost criterion induced by the regular utility function for continuous-time Markov decision processes

In this paper we study the average cost criterion induced by the regular utility function (U-average cost criterion) for continuous-time Markov decision processes. This criterion is a generalization of the risk-sensitive average cost and expected average cost criteria. We first introduce an auxiliary risk-sensitive first passage optimization problem and obtain the properties of the corresponding optimal value function under the slight conditions. Then we show that the pair of the optimal value functions of the risk-sensitive average cost criterion and the risk-sensitive first passage criterion is a solution to the optimality equation of the risk-sensitive average cost criterion allowing the risk-sensitivity parameter to take any nonzero value. Moreover, we have that the optimal value function of the risk-sensitive average cost criterion is continuous with respect to the risk-sensitivity parameter. Finally, we give the connections between the U-average cost criterion and the average cost criteria induced by the identity function and the exponential utility function, and prove the existence of a U-average optimal deterministic stationary policy in the class of all randomized Markov policies.

Data-driven adaptive dynamic programming for continuous-time fully cooperative games with partially constrained inputs

In this paper, the fully cooperative game with partially constrained inputs in the continuous-time Markov decision process environment is investigated using a novel data-driven adaptive dynamic programming method. First, the model-based policy iteration algorithm with one iteration loop is proposed, where the knowledge of system dynamics is required. Then, it is proved that the iteration sequences of value functions and control policies can converge to the optimal ones. In order to relax the exact knowledge of the system dynamics, a model-free iterative equation is derived based on the model-based algorithm and the integral reinforcement learning. Furthermore, a data-driven adaptive dynamic programming is developed to solve the model-free equation using generated system data. From the theoretical analysis, we prove that this model-free iterative equation is equivalent to the model-based iterative equations, which means that the data-driven algorithm can approach the optimal value function and control policies. For the implementation purpose, three neural networks are constructed to approximate the solution of the model-free iteration equation using the off-policy learning scheme after the available system data is collected in the online measurement phase. Finally, two examples are provided to demonstrate the effectiveness of the proposed scheme.

Continuous-Time Markov Decision Processes with Exponential Utility

In this paper, we consider a continuous-time Markov decision process (CTMDP) in Borel spaces, where the certainty equivalent with respect to the exponential utility of the total undiscounted cost is to be minimized. The cost rate is nonnegative. We establish the optimality equation. Under the compactness-continuity condition, we show the existence of a deterministic stationary optimal policy. We reduce the risk-sensitive CTMDP problem to an equivalent risk-sensitive discrete-time Markov decision process, which is with the same state and action spaces as the original CTMDP. In particular, the value iteration algorithm for the CTMDP problem follows from this reduction. We do not need impose any condition on the growth of the transition and cost rate in the state, and the controlled process could be explosive.

Optimality of Mixed Policies for Average Continuous-Time Markov Decision Processes with Constraints

This article concerns the average criteria for continuous-time Markov decision processes with N constraints. We show the following; (a) every extreme point of the space of performance vectors corresponding to the set of stable measures is generated by a deterministic stationary policy; and (b) there exists a mixed optimal policy, where the mixture is over no more than N + 1 deterministic stationary policies.

State-Policy Dynamics in Evolutionary Games

Standard evolutionary game theory framework is a useful tool to study large interacting systems and to understand the strategic behavior of individuals in such complex systems. Adding an individual state to model local feature of each player in this context allows one to study a wider range of problems in various application areas as networking, biology, etc. In this paper, we introduce such an extension of evolutionary game framework and particularly, we focus on the dynamical aspects of this system. Precisely, we study the coupled dynamics of the policies and the individual states inside a population of interacting individuals. We first define a general model by coupling replicator dynamics and continuous-time Markov decision processes, and we then consider a particular case of a two policies and two states evolutionary game. We first obtain a system of combined dynamics, and we show that the rest points of this system are equilibria profiles of our evolutionary game with individual state dynamics. Second, by assuming two different timescales between states and policies dynamics, we can compute explicitly the equilibria. Then, by transforming our evolutionary game with individual states into a standard evolutionary game, we obtain an equilibrium profile which is equivalent, in terms of occupation measures and expected fitness to the previous one. All our results are illustrated with numerical analysis.

Verifiable conditions for average optimality of continuous-time Markov decision processes

In this paper we provide another set of verifiable conditions for the average optimality of continuous-time Markov decision processes (CTMDP) in Polish spaces with unbounded transition rates. Under the new conditions which are imposed on the primitive data of the model of the CTMDP and thus easy to verify, we also establish the existence of an average optimal stationary policy. Finally, we propose two examples to illustrate the newness of the conditions.

A Continuous-Time Markov Decision Process-Based Method With Application in a Pursuit-Evasion Example

This paper presents a novel method—continuous-time Markov decision process (CTMDP)—to address the uncertainties in pursuit-evasion problem. The primary difference between the CTMDP and the Markov decision process (MDP) is that the former takes into account the influence of the transition time between the states. The policy iteration method-based potential performance for solving the CTMDP and its convergence are also presented. The results obtained by MDP-based method demonstrate that it is a special case of CTMDP-based method involving the identity transition rate matrix. To compare the methods, a well-known pursuit-evasion problem, involving two identical cars, is solved as a benchmark. The CTMDP-based method can provide a discretization solution that is close to the analytical solution obtained by the differential game method. Besides, it shows strong robustness against changes in the transition probability, as compared with the traditional MDP-based method. To the best of our knowledge, this is the first attempt to validate the influence of the transition time between the states in such a pursuit-evasion scenario, or in a similar application, solved by an MDP-related model. The CTMDP-based method offers a new approach to solving the pursuit-evasion problem and can be extended to similar optimization applications.

Optimal decisions for continuous time Markov decision processes over finite planning horizons

The computation of ϵ-optimal policies for continuous time Markov decision processes (CTMDPs) over finite time intervals is a sophisticated problem because the optimal policy may change at arbitrary times. Numerical algorithms based on time discretization or uniformization have been proposed for the computation of optimal policies. The uniformization based algorithm has shown to be more reliable and often also more efficient but is currently only available for processes where the gain or reward does not depend on the decision taken in a state. In this paper, we present two new uniformization based algorithms for computing ϵ-optimal policies for CTMDPs with decision dependent rewards over a finite time horizon. Due to a new and tighter upper bound the newly proposed algorithms cannot only be applied for decision dependent rewards, they also outperform the available approach for rewards that do not depend on the decision. In particular for models where the policy only rarely changes, optimal policies can be computed much faster.

Dynamic control of a tandem system with abandonments

The goal of this paper is to provide a model that is an extension of classic scheduling problems for a tandem queueing system by including customer impatience. In such scenarios, the server(s) must balance abandonments from each phase of service with the need to prioritize higher reward customers. This presents an interesting challenge since the trade-off between the cost of abandonments and revenue maximization is not at all clear. As a result of customer abandonments, traditional solution techniques are not available. In particular, uniformization is not possible since the transition rates are unbounded. We do our analysis in continuous time, using the continuous-time Markov decision process framework to discern simple relationships of the value functions depending on the starting state. We then use sample path arguments to analyze the structure of the optimal policy. We provide conditions under which an optimal policy is nonidling and never splits servers (except to avoid idling). We then consider the single-server model as a proxy for the decision problem when there are more customers at each station than there are servers. In this case, we provide conditions under which it is optimal to prioritize each station.

Continuous-time Markov decision processes with risk-sensitive finite-horizon cost criterion

This paper studies continuous-time Markov decision processes with a denumerable state space, a Borel action space, bounded cost rates and possibly unbounded transition rates under the risk-sensitive finite-horizon cost criterion. We give the suitable optimality conditions and establish the Feynman–Kac formula, via which the existence and uniqueness of the solution to the optimality equation and the existence of an optimal deterministic Markov policy are obtained. Moreover, employing a technique of the finite approximation and the optimality equation, we present an iteration method to compute approximately the optimal value and an optimal policy, and also give the corresponding error estimations. Finally, a controlled birth and death system is used to illustrate the main results.

Finite approximation for finite-horizon continuous-time Markov decision processes

In this paper we study the continuous-time Markov decision processes with a denumerable state space, a Borel action space, and unbounded transition and cost rates. The optimality criterion to be considered is the finite-horizon expected total cost criterion. Under the suitable conditions, we propose a finite approximation for the approximate computations of an optimal policy and the value function, and obtain the corresponding error estimations. Furthermore, our main results are illustrated with a controlled birth and death system.

Computable approximations for continuous-time Markov decision processes on Borel spaces based on empirical measures

In this paper, we propose an approach for approximating the value function and an ϵ-optimal policy of continuous-time Markov decision processes with Borel state and action spaces, with possibly unbounded cost and transition rates, under the total expected discounted cost optimality criterion. Under adequate assumptions, which in particular include that the transition rate has a density function with respect to a reference measure, together with piecewise Lipschitz continuity of the elements of the control model, we approximate the original controlled process by a model with finite state and action spaces. The approximation error is related to the 1-Wasserstein distance between suitably defined probability measures and approximating measures with finite support. We also study the case when the reference measure is approximated with empirical distributions and we show that convergence of the approximations takes place at an exponential rate in probability.

Continuous-time Markov decision processes under the risk-sensitive average cost criterion

This paper studies continuous-time Markov decision processes under the risk-sensitive average cost criterion. The state space is a finite set, the action space is a Borel space, the cost and transition rates are bounded, and the risk-sensitivity coefficient can take any positive real number. Under the mild conditions, we develop a new approach to establish the existence of a solution to the risk-sensitive average cost optimality equation and obtain the existence of an optimal deterministic stationary policy.

Constrained Continuous-Time Markov Decision Processes on the Finite Horizon

This paper studies the constrained (nonhomogeneous) continuous-time Markov decision processes on the finite horizon. The performance criterion to be optimized is the expected total reward on the finite horizon, while N constraints are imposed on similar expected costs. Introducing the appropriate notion of the occupation measures for the concerned optimal control problem, we establish the following under some suitable conditions: (a) the class of Markov policies is sufficient; (b) every extreme point of the space of performance vectors is generated by a deterministic Markov policy; and (c) there exists an optimal Markov policy, which is a mixture of no more than $$N+1$$N+1 deterministic Markov policies.