Denumerable State Research Articles

Optimal Control of Probability on a Target Set for Continuous-Time Markov Chains

In this paper, a stochastic optimal control problem is considered for a continuous-time Markov chain taking values in a denumerable state space over a fixed finite horizon. The optimality criterion is the probability that the process remains in a target set before and at a certain time. The optimal value is a superadditive capacity of target sets. Under some minor assumptions for the controlled Markov process, we establish the dynamic programming principle, based on which we prove that the value function is a classical solution of the Hamilton-Jacobi-Bellman equation on a discrete lattice space. We then prove that there exists an optimal deterministic Markov control under the compactness assumption of control domain. We further prove that the value function is the unique solution of the HJB equation. We also consider the case starting from the outside of target set and give the corresponding results. Finally, we apply our results to two examples.

Risk-Sensitive Average Optimality for Discrete-Time Markov Decision Processes

In this paper we study the risk-sensitive average optimality for discrete-time Markov decision processes with denumerable states and unbounded costs. We derive the multiplicative Poisson equation under the suitable ergodicity conditions via an approximation method. Moreover, we prove the existence of a unique solution to the risk-sensitive average cost optimality equation and give an equivalent characterization of the set of all optimal stationary policies. Finally, we present the policy iteration algorithm and show its convergence.

Nonzero-Sum Risk-Sensitive Continuous-Time Stochastic Games with Ergodic Costs

We study nonzero-sum stochastic games for continuous time Markov decision processes on a denumerable state space with risk-sensitive ergodic cost criterion. Transition rates and cost rates are allowed to be unbounded. Under a Lyapunov type stability assumption, we show that the corresponding system of coupled HJB equations admits a solution which leads to the existence of a Nash equilibrium in stationary strategies. We establish this using an approach involving principal eigenvalues associated with the HJB equations. Furthermore, exploiting appropriate stochastic representation of principal eigenfunctions, we completely characterize Nash equilibria in the space of stationary Markov strategies.

Continuous-time zero-sum games with probability criterion

In this article, we investigate a zero-sum stochastic game for continuous-time Markov chain with denumerable state space and unbounded transition rates, under the probability criterion. Under suitable assumptions, we show the existence of value of the game and also characterize it as the unique solution of a pair of Shapley equations. We also establish the existence of a randomized stationary saddle point equilibrium.

A Discounted Approach in Communicating Average Markov Decision Chains Under Risk-Aversion

This work concerns with discrete-time Markov decision processes on a denumerable state space. Assuming that the decision maker is risk-averse with constant risk-sensitivity coefficient, the performance of a control policy is measured by an average criterion associated with a non-negative and bounded cost function. Under conditions ensuring that the optimal average cost is constant, but not necessarily determined via the average cost optimality equation, it is shown that a discounted criterion can be used to approximate the optimal average index.

Sensitivity of the stationary distributions of denumerable Markov chains

In this note we present new perturbation bounds for the stationary distribution of Markov chains with denumerable state space. Bounds are obtained using the Dobrushin’s coefficient δ(P) and the ergodic coefficient α(P). Bounds are shown to be relevant when α(P)<δ(P). In addition we present bounds only involving the coefficient α(P).

Analysis of non-reversible Markov chains via similarity orbits

AbstractIn this paper we develop an in-depth analysis of non-reversible Markov chains on denumerable state space from a similarity orbit perspective. In particular, we study the class of Markov chains whose transition kernel is in the similarity orbit of a normal transition kernel, such as that of birth–death chains or reversible Markov chains. We start by identifying a set of sufficient conditions for a Markov chain to belong to the similarity orbit of a birth–death chain. As by-products, we obtain a spectral representation in terms of non-self-adjoint resolutions of identity in the sense of Dunford [21] and offer a detailed analysis on the convergence rate, separation cutoff and L2-cutoff of this class of non-reversible Markov chains. We also look into the problem of estimating the integral functionals from discrete observations for this class. In the last part of this paper we investigate a particular similarity orbit of reversible Markov kernels, which we call the pure birth orbit, and analyse various possibly non-reversible variants of classical birth–death processes in this orbit.

Constrained Expected Average Stochastic Games for Continuous-Time Jump Processes

In this paper we study the nonzero-sum constrained stochastic games for continuous-time jump processes with denumerable states and possibly unbounded transition rates. The optimality criterion under consideration is the expected average payoff criterion and the payoff functions of the players are allowed to be unbounded. Under the reasonable conditions, we introduce an approximating sequence of the auxiliary game models and show the existence of stationary constrained Nash equilibria for these approximating game models via employing the average occupation measures and constructing a suitable multifunction. Moreover, we obtain that any limit point of the stationary constrained Nash equilibria for the approximating sequence of the game models is a constrained Nash equilibrium for the original game model. Furthermore, we use a controlled birth and death system to illustrate our main results.

Skip-free Markov chains

The aim of this paper is to develop a general theory for the class of skip-free Markov chains on denumerable state space. This encompasses their potential theory via an explicit characterization of their potential kernel expressed in terms of family of fundamental excessive functions, which are defined by means of the theory of Martin boundary. We also describe their fluctuation theory generalizing the celebrated fluctuations identities that were obtained by using the Wiener-Hopf factorization for the specific skip-free random walks. We proceed by resorting to the concept of similarity to identify the class of skip-free Markov chains whose transition operator has only real and simple eigenvalues. We manage to find a set of sufficient and easy-to-check conditions on the one-step transition probability for a Markov chain to belong to this class. We also study several properties of this class including their spectral expansions given in terms of Riesz basis, derive a necessary and sufficient condition for this class to exhibit a separation cutoff, and give a tighter bound on its convergence rate to stationarity than existing results.

Characterization of the Optimal Risk-Sensitive Average Cost in Denumerable Markov Decision Chains

This work is concerned with Markov decision chains on a denumerable state space. The controller has a positive risk-sensitivity coefficient, and the performance of a control policy is measured by a risk-sensitive average cost criterion. Besides standard continuity-compactness conditions, it is assumed that the state process is communicating under any stationary policy, and that the simultaneous Doeblin condition holds. In this context, it is shown that if the cost function is bounded from below, and the superior limit average index is finite at some point, then (i) the optimal superior and inferior limit average value functions coincide and are constant, (ii) the optimal average cost is characterized via an extended version of the Collatz-Wielandt formula in the theory of positive matrices, and (iii) an optimality inequality is established, from which a stationary optimal policy is obtained. Moreover, an explicit example is given to show that, even if the cost function is bounded, the strict inequality may occur in the optimality relation.

Risk-Sensitive Average Equilibria for Discrete-Time Stochastic Games

In this paper, we study the risk-sensitive average payoff criterion for the nonzero-sum discrete-time stochastic games with a denumerable state space. The risk-sensitivity coefficient can take positive values and negative values. Under the suitable conditions, we show the existence of a solution to the coupled equations by a technique of the discounted approximation, and obtain the existence of a stationary Nash equilibrium. Moreover, we present some verifiable sufficient conditions imposed on the primitive data of the model for the verification of our assumption and use an example to illustrate that our conditions are weaker than those in the existing literature.

Local Poisson Equations Associated with Discrete-Time Markov Control Processes

This paper provides a characterization of the optimal average cost function, when the long-run (risk-sensitive) average cost criterion is used. The Markov control model has a denumerable state space with finite set of actions, and the characterization presented is given in terms of a system of local Poisson equations, which gives as a by-product the existence of an optimal stationary policy.

HITTING TIME DISTRIBUTIONS FOR BIRTH–DEATH PROCESSES WITH BILATERAL ABSORBING BOUNDARIES

For the birth–death process on a finite state space with bilateral boundaries, we give a simpler derivation of the hitting time distributions by h-transform and φ-transform. These transforms can then be used to construct a quick derivation of the hitting time distributions of the minimal birth–death process on a denumerable state space with exit/regular boundaries.

Continuous-Time Constrained Stochastic Games under the Discounted Cost Criteria

In this paper, we consider the continuous-time nonzero-sum constrained stochastic games with the discounted cost criteria. The state space is denumerable and the action space of each player is a general Polish space, while the transition rates and cost functions are allowed to be unbounded from below and from above. The strategies for each player may be history-dependent and randomized. Models with these features seemingly have not been handled in the previous literature. By constructing a sequence of continuous-time finite-state game models to approximate the original denumerable-state game model, we prove the existence of constrained Nash equilibria for the constrained games with denumerable states.

Continuous-time Markov decision processes with risk-sensitive finite-horizon cost criterion

This paper studies continuous-time Markov decision processes with a denumerable state space, a Borel action space, bounded cost rates and possibly unbounded transition rates under the risk-sensitive finite-horizon cost criterion. We give the suitable optimality conditions and establish the Feynman–Kac formula, via which the existence and uniqueness of the solution to the optimality equation and the existence of an optimal deterministic Markov policy are obtained. Moreover, employing a technique of the finite approximation and the optimality equation, we present an iteration method to compute approximately the optimal value and an optimal policy, and also give the corresponding error estimations. Finally, a controlled birth and death system is used to illustrate the main results.

Finite approximation for finite-horizon continuous-time Markov decision processes

In this paper we study the continuous-time Markov decision processes with a denumerable state space, a Borel action space, and unbounded transition and cost rates. The optimality criterion to be considered is the finite-horizon expected total cost criterion. Under the suitable conditions, we propose a finite approximation for the approximate computations of an optimal policy and the value function, and obtain the corresponding error estimations. Furthermore, our main results are illustrated with a controlled birth and death system.

Phase Transitions for Controlled Markov Chains on Infinite Graphs

This paper is concerned with some long-run average cost problems for controlled Markov chains with a denumerable state space. The criterion to be optimized contains both reward and penalty functions. As a trade-off between reward and penalty, we observe certain phase transition phenomena. Our results also provide a stochastic optimal control interpretation for phase transitions of discrete homopolymers with finite attracting potentials.

Equivalent Markov processes under gauge group.

We have studied Markov processes on denumerable state space and continuous time. We found that all these processes are connected via gauge transformations. We have used this result before as a method to resolve equations, included the case in a previous work in which the sample space is time-dependent [Phys. Rev. E 90, 022125 (2014)]. We found a general solution through dilation of the state space, although the prior probability distribution of the states defined in this new space takes smaller values with respect to that in the initial problem. The gauge (local) group of dilations modifies the distribution on the dilated space to restore the original process. In this work, we show how the Markov process in general could be linked via gauge (local) transformations, and we present some illustrative examples for this result.

Risk-Sensitive Ergodic Control of Continuous Time Markov Processes With Denumerable State Space

In this article, we study risk-sensitive control problem with controlled continuous time Markov chain state dynamics. Using multiplicative dynamic programming principle along with the atomic structure of the state dynamics, we prove the existence and a characterization of optimal risk-sensitive control under geometric ergodicity of the state dynamics along with a smallness condition on the running cost.

First passage Markov decision processes with constraints and varying discount factors

This paper focuses on the constrained optimality problem (COP) of first passage discrete-time Markov decision processes (DTMDPs) in denumerable state and compact Borel action spaces with multi-constraints, state-dependent discount factors, and possibly unbounded costs. By means of the properties of a so-called occupation measure of a policy, we show that the constrained optimality problem is equivalent to an (infinite-dimensional) linear programming on the set of occupation measures with some constraints, and thus prove the existence of an optimal policy under suitable conditions. Furthermore, using the equivalence between the constrained optimality problem and the linear programming, we obtain an exact form of an optimal policy for the case of finite states and actions. Finally, as an example, a controlled queueing system is given to illustrate our results.