Continuous-time Controlled Markov Chains Research Articles

Discounted Continuous-Time Controlled Markov Chains: Convergence of Control Models

We are interested in continuous-time, denumerable state controlled Markov chains (CMCs), with compact Borel action sets, and possibly unbounded transition and reward rates, under the discounted reward optimality criterion. For such CMCs, we propose a definition of a sequence of control models {ℳ n } converging to a given control model ℳ, which ensures that the discount optimal reward and policies of ℳ n converge to those of ℳ. As an application, we propose a finite-state and finite-action truncation technique of the original control model ℳ, which is illustrated by approximating numerically the optimal reward and policy of a controlled population system with catastrophes. We study the corresponding convergence rates.

Discounted Continuous-Time Controlled Markov Chains: Convergence of Control Models

We are interested in continuous-time, denumerable state controlled Markov chains (CMCs), with compact Borel action sets, and possibly unbounded transition and reward rates, under the discounted reward optimality criterion. For such CMCs, we propose a definition of a sequence of control models {ℳn} converging to a given control model ℳ, which ensures that the discount optimal reward and policies of ℳn converge to those of ℳ. As an application, we propose a finite-state and finite-action truncation technique of the original control model ℳ, which is illustrated by approximating numerically the optimal reward and policy of a controlled population system with catastrophes. We study the corresponding convergence rates.

Uniform ergodicity of continuous-time controlled Markov chains: A survey and new results

We make a review of several variants of ergodicity for continuous-time Markov chains on a countable state space. These include strong ergodicity, ergodicity in weighted-norm spaces, exponential and subexponential ergodicity. We also study uniform exponential ergodicity for continuous-time controlled Markov chains, as a tool to deal with average reward and related optimality criteria. A discussion on the corresponding ergodicity properties is made, and an application to a controlled population system is shown.

Continuous-time controlled Markov chains with safety upper bound

This study introduces the notion of safety for the controlled Markov chains in the continuous-time horizon. The concept is a non-trivial extension of safety control for stochastic systems modelled as discrete-time Markov decision processes, where the safety means that the probability distributions of the system states will not visit the given forbidden set at any time. In this paper study a unit-interval-valued vector that serves as an upper bound on the state probability distribution vector characterises the forbidden set. A probability distribution is then called safe if it does not exceed the upper bound. Under mild conditions the author derives two results: (i) the necessary and sufficient conditions that guarantee the all-time safety of the probability distributions if the starting distribution is safe, and (ii) the characterisation of the supreme set of safe initial probability vectors that remain safe as time passes. In particular, study the paper identifies an upper bound on time and shows that if a distribution is always safe before that time, the distribution is safe at all times. Numerical examples are provided to illustrate the two results.

The vanishing discount approach to constrained continuous-time controlled Markov chains

We consider a denumerable state continuous-time controlled Markov chain (CMC) with possibly unbounded transition and reward rates. We deal with constrained optimality; that is, we want to maximize a discounted reward (an average reward) criterion subject to a constraint on a discounted cost (an average cost). We give conditions ensuring that the average constrained optimal reward and policies can be obtained as the limit, as the discount rate vanishes, of the corresponding discounted constrained optimal reward and policies. This extends to average constrained CMCs the standard results on the vanishing discount approach for the average unconstrained case. We also present an example showing that the vanishing discount results for constrained problems might not hold.

Approximating Ergodic Average Reward Continuous-Time Controlled Markov Chains

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> We study the approximation of an ergodic average reward continuous-time denumerable state Markov decision process (MDP) by means of a sequence of MDPs. Our results include the convergence of the corresponding optimal policies and the optimal gains. For a controlled upwardly skip-free process, we show some computational results to illustrate the convergence theorems. </para>

Variance minimization and the overtaking optimality approach to continuous-time controlled Markov chains

This paper deals with denumerable-state continuous-time controlled Markov chains with possibly unbounded transition and reward rates. It concerns optimality criteria that improve the usual expected average reward criterion. First, we show the existence of average reward optimal policies with minimal average variance. Then we compare the variance minimization criterion with overtaking optimality. We present an example showing that they are opposite criteria, and therefore we cannot optimize them simultaneously. This leads to a multiobjective problem for which we identify the set of Pareto optimal policies (also known as nondominated policies).

Ergodic Control of Continuous-Time Markov Chains with Pathwise Constraints

This paper deals with unichain ergodic continuous-time controlled Markov chains (CMCs) with a denumerable state space and possibly unbounded reward rates as well as unbounded transition rates. The problem we are concerned with is to find control policies that maximize a sample-path average reward over the family of admissible policies for which a certain pathwise average cost is below a given value with probability one. To study this problem, first, we give conditions for the existence of sample-path average optimal policies. Then we analyze constrained average reward CMCs with “expected” constraints, and, finally, we apply these results to our original control problem with pathwise constraints. Examples on the control of a queueing system and an epidemic process illustrate the feasibility of our approach.

Blackwell Optimality in the Class of Markov Policies for Continuous-Time Controlled Markov Chains

This paper deals with Blackwell optimality for continuous-time controlled Markov chains with compact Borel action space, and possibly unbounded reward (or cost) rates and unbounded transition rates. We prove the existence of a deterministic stationary policy which is Blackwell optimal in the class of all admissible (nonstationary) Markov policies, thus extending previous results that analyzed Blackwell optimality in the class of stationary policies. We compare our assumptions to the corresponding ones for discrete-time Markov controlled processes.

Bias Optimality for Continuous-Time Controlled Markov Chains

In this paper we give conditions for the existence of bias optimal policies in a class of continuous-time controlled Markov chains with unbounded reward and transition rates. Several characterizations of bias optimality are proposed. We also introduce new sets of conditions ensuring uniform exponential ergodicity of continuous-time controlled Markov chains.

The Laurent series, sensitive discount and Blackwell optimality for continuous-time controlled Markov chains

This paper gives conditions for the convergence of the Laurent series expansion for a class of continuous-time controlled Markov chains with possibly unbounded reward (or cost) rates and unbounded transition rates. That series is then used to study several optimization criteria, including n-discount optimality (for n=−1,0,1,...), Blackwell optimality, and the maximization of a certain vector criterion that in particular gives gain and bias optimality.

Continuous-Time Controlled Markov Chains with Discounted Rewards

This paper studies denumerable state continuous-time controlled Markov chains with the discounted reward criterion and a Borel action space. The reward and transition rates are unbounded, and the reward rates are allowed to take positive or negative values. First, we present new conditions for a nonhomogeneous Q(t)-process to be regular. Then, using these conditions, we give a new set of mild hypotheses that ensure the existence of ∈-optimal (∈≥0) stationary policies. We also present a ‘martingale characterization’ of an optimal stationary policy. Our results are illustrated with controlled birth and death processes.

Drift and monotonicity conditions for continuous-time controlled markov chains with an average criterion

We give conditions for the existence of average optimal policies for continuous-time controlled Markov chains with a denumerable state-space and Borel action sets. The transition rates are allowed to be unbounded, and the reward/cost rates may have neither upper nor lower bounds. In the spirit of the drift and monotonicity conditions for continuous-time Markov processes, we propose a new set of conditions on the controlled process' primitive data under which the existence of optimal (deterministic) stationary policies in the class of randomized Markov policies is proved using the extended generator approach instead of Kolmogorov's forward equation used in the previous literature, and under which the convergence of a policy iteration method is also shown. Moreover, we use a controlled queueing system to show that all of our conditions are satisfied, whereas those in the previous literature fail to hold.

Continuous-time controlled Markov chains

This paper concerns studies on continuous-time controlled Markov chains, that is, continuous-time Markov decision processes with a denumerable state space, with respect to the discounted cost criterion. The cost and transition rates are allowed to be unbounded and the action set is a Borel space. We first study control problems in the class of deterministic stationary policies and give very weak conditions under which the existence of $\varepsilon$-optimal ($\varepsilon\geq 0)$ policies is proved using the construction of a minimum Q-process. Then we further consider control problems in the class of randomized Markov policies for (1) regular and (2) nonregular Q-processes. To study case (1), first we present a new necessary and sufficient condition for a nonhomogeneous Q-process to be regular. This regularity condition, together with the extended generatorof a nonhomogeneous Markov process, is used to prove the existence of $\varepsilon$-optimal stationary policies. Our results for case (1) are illustrated by a Schlogl model with a controlled diffusion. For case (2), we obtain a similar result using Kolmogorov's forward equation for the minimum Q-process and we also present an example in which our assumptions are satisfied, but those used in the previous literature fail to hold.