Unbounded Transition Research Articles

Risk-sensitive zero-sum games for continuous-time jump processes with unbounded rates and Borel spaces

This paper studies two-person zero-sum risk-sensitive stochastic games for continuous time jump processes with the following features: (1) the payoff and transition rates are unbounded and time-dependent, (2) the state and action spaces are general Borel sets, (3) the discount factors can vary in time. Differing from those with the risk-sensitive parameter as a differential variable in previous works, we introduce a new type of Shapley equation (NSE) with the time as a differential variable. Under some technical assumptions, we derive a representation of a solution to the NSE, which in turn is used to establish the existence of a solution to the NSE by iterations and approximation. With the help of the NSE, we prove the existence of the value of the game and a saddle-point equilibrium depending on the time, which explicitly shows the nonstationarity of the risk-sensitive discounted equilibria. We illustrate our results by two examples with unbounded transition and payoff rates.

On the quasi-ergodicity of absorbing Markov chains with unbounded transition densities, including random logistic maps with escape – CORRIGENDUM

On the quasi-ergodicity of absorbing Markov chains with unbounded transition densities, including random logistic maps with escape – CORRIGENDUM

Discounted cost exponential semi-Markov decision processes with unbounded transition rates: a service rate control problem with impatient customers

Discounted cost exponential semi-Markov decision processes with unbounded transition rates: a service rate control problem with impatient customers

On the quasi-ergodicity of absorbing Markov chains with unbounded transition densities, including random logistic maps with escape

Abstract In this paper, we consider absorbing Markov chains $X_n$ admitting a quasi-stationary measure $\mu $ on M where the transition kernel ${\mathcal P}$ admits an eigenfunction $0\leq \eta \in L^1(M,\mu )$ . We find conditions on the transition densities of ${\mathcal P}$ with respect to $\mu $ which ensure that $\eta (x) \mu (\mathrm {d} x)$ is a quasi-ergodic measure for $X_n$ and that the Yaglom limit converges to the quasi-stationary measure $\mu $ -almost surely. We apply this result to the random logistic map $X_{n+1} = \omega _n X_n (1-X_n)$ absorbed at ${\mathbb R} \setminus [0,1],$ where $\omega _n$ is an independent and identically distributed sequence of random variables uniformly distributed in $[a,b],$ for $1\leq a <4$ and $b>4.$

Risk-sensitive average continuous-time Markov decision processes with unbounded transition and cost rates

AbstractThis paper considers risk-sensitive average optimization for denumerable continuous-time Markov decision processes (CTMDPs), in which the transition and cost rates are allowed to be unbounded, and the policies can be randomized history dependent. We first derive the multiplicative dynamic programming principle and some new facts for risk-sensitive finite-horizon CTMDPs. Then, we establish the existence and uniqueness of a solution to the risk-sensitive average optimality equation (RS-AOE) through the results for risk-sensitive finite-horizon CTMDPs developed here, and also prove the existence of an optimal stationary policy via the RS-AOE. Furthermore, for the case of finite actions available at each state, we construct a sequence of models of finite-state CTMDPs with optimal stationary policies which can be obtained by a policy iteration algorithm in a finite number of iterations, and prove that an average optimal policy for the case of infinitely countable states can be approximated by those of the finite-state models. Finally, we illustrate the conditions and the iteration algorithm with an example.

Continuous-time zero-sum games with probability criterion

In this article, we investigate a zero-sum stochastic game for continuous-time Markov chain with denumerable state space and unbounded transition rates, under the probability criterion. Under suitable assumptions, we show the existence of value of the game and also characterize it as the unique solution of a pair of Shapley equations. We also establish the existence of a randomized stationary saddle point equilibrium.

Average stochastic games for continuous-time jump processes

We consider nonzero-sum games for continuous-time jump processes with unbounded transition rates under expected average payoff criterion. The state and action spaces are Borel spaces and reward rates are unbounded. We introduce an approximating sequence of stochastic game models with extended state space, for which the uniform exponential ergodicity is obtained. Moreover, we prove the existence of a stationary almost Markov Nash equilibrium by introducing auxiliary static game models. Finally, a cash flow model is employed to illustrate the results.

A useful technique for piecewise deterministic Markov decision processes

This note presents a technique that is useful for the study of piecewise deterministic Markov decision processes (PDMDPs) with general policies and unbounded transition intensities. This technique produces an auxiliary PDMDP from the original one. The auxiliary PDMDP possesses certain desired properties, which may not be possessed by the original PDMDP. We apply this technique to risk-sensitive PDMDPs with total cost criteria, and comment on its connection with the uniformization technique.

Solving Poisson’s equation for birth–death chains: Structure, instability, and accurate approximation

Poisson’s equation plays a fundamental role as a tool for performance evaluation and optimization of Markov chains. For continuous-time birth–death chains with possibly unbounded transition and cost rates as addressed herein, when analytical solutions are unavailable its numerical solution can in theory be obtained by a simple forward recurrence. Yet, this may suffer from numerical instability, which can hide the structure of exact solutions. This paper presents three main contributions: (1) it establishes a structural result (convexity of the relative cost function) under mild conditions on transition and cost rates, which is relevant for proving structural properties of optimal policies in Markov decision models; (2) it elucidates the root cause, extent and prevalence of instability in numerical solutions by standard forward recurrence; and (3) it presents a novel forward–backward recurrence scheme to compute accurate numerical solutions. The results are applied to the accurate evaluation of the bias and the asymptotic variance, and are illustrated in an example.

Continuous-time zero-sum stochastic game with stopping and control

We consider a zero-sum stochastic game for continuous-time Markov chain with countable state space and unbounded transition and pay-off rates. The additional feature of the game is that the controllers together with taking actions are also allowed to stop the process. Under suitable hypothesis we show that the game has a value and it is the unique solution of certain dynamic programming inequalities with bilateral constraints. In the process we also prescribe a saddle point equilibrium.

Multiconstrained Finite-Horizon Piecewise Deterministic Markov Decision Processes with Unbounded Transition Rates

This paper studies a multiconstrained problem for piecewise deterministic Markov decision processes (PDMDPs) with unbounded cost and transition rates. The goal is to minimize one type of expected finite-horizon cost over history-dependent policies while keeping some other types of expected finite-horizon costs lower than some tolerable bounds. Using the Dynkin formula for the PDMDPs, we obtain an equivalent characterization of occupancy measures and express the expected finite-horizon costs in terms of occupancy measures. Under suitable assumptions, the existence of constrained-optimal policies is shown, the linear programming formulation and its dual program for the constrained problem are derived, and the strong duality between the two programs is established. An example is provided to demonstrate our results.

Risk Probability Minimization Problems for Continuous-Time Markov Decision Processes on Finite Horizon

This article deals with a risk probability minimization problem for finite horizon continuous-time Markov decision processes with unbounded transition rates and history-dependent policies. Only using the assumption of nonexplosion of the controlled state process as well as the finiteness of actions available at each state, we not only establish the existence and uniqueness of a solution to the corresponding optimality equation, but also prove the existence of a risk probability optimal policy. Finally, we give two examples to illustrate our results: one example shows that the value iteration algorithm is provided for computing both the value function and an optimal risk probability policy, and the other shows the differences between the conditions in this article and those in the previous literature.

Constrained Expected Average Stochastic Games for Continuous-Time Jump Processes

In this paper we study the nonzero-sum constrained stochastic games for continuous-time jump processes with denumerable states and possibly unbounded transition rates. The optimality criterion under consideration is the expected average payoff criterion and the payoff functions of the players are allowed to be unbounded. Under the reasonable conditions, we introduce an approximating sequence of the auxiliary game models and show the existence of stationary constrained Nash equilibria for these approximating game models via employing the average occupation measures and constructing a suitable multifunction. Moreover, we obtain that any limit point of the stationary constrained Nash equilibria for the approximating sequence of the game models is a constrained Nash equilibrium for the original game model. Furthermore, we use a controlled birth and death system to illustrate our main results.

Stochastic stability of non‐linear impulsive semi‐Markov jump systems

In this study, the problem of stochastic stability of non-linear impulsive semi-Markov jump systems is investigated. Using the method of stochastic Lyapunov functions, the authors develop some sufficient conditions about stochastic stability for a class of non-linear impulsive semi-Markov jump systems with unbounded transition rates. Particularly, the obtained results generalise and complement some published literatures. Finally, some examples are given to show the effectiveness and advantages of the proposed results.

Finite horizon risk-sensitive continuous-time Markov decision processes with unbounded transition and cost rates

We consider a risk-sensitive continuous-time Markov decision process over a finite time duration. Under the conditions that can be satisfied by unbounded transition and cost rates, we show the existence of an optimal policy, and the existence and uniqueness of the solution to the optimality equation out of a class of possibly unbounded functions, to which the Feynman–Kac formula was also justified to hold.

Risk-Sensitive Discounted Continuous-Time Markov Decision Processes with Unbounded Rates

This paper attempts to study the risk-sensitive discounted continuous-time Markov decision processes with unbounded transition and cost rates. Different from the case of bounded transition/cost rat...

Nonzero-sum risk-sensitive finite-horizon continuous-time stochastic games

This paper concerns the nonzero-sum games for continuous-time jump processes with unbounded transition rates under the risk-sensitive finite-horizon cost criterion. The state space is a countable set and the costs are allowed to be unbounded in the game model. Under the suitable optimality conditions, by introducing an appropriate topology for the set of all randomized Markov multi-strategies and employing the risk-sensitive finite-horizon optimality equations of the players, we prove the existence of a randomized Markov Nash equilibrium in the class of all randomized history-dependent multi-strategies.

Finite-horizon piecewise deterministic Markov decision processes with unbounded transition rates

ABSTRACTThis paper is concerned with the problem of minimizing the expected finite-horizon cost for piecewise deterministic Markov decision processes. The transition rates may be unbounded, and the cost functions are allowed to be unbounded from above and from below. The optimality is over the general history-dependent policies, where the control is continuously acting in time. The infinitesimal approach is employed to establish the associated Hamilton-Jacobi-Bellman equation, via which the existence of optimal policies is proved. An example is provided to verify all the assumptions proposed.

Nonlocal heat equations: Regularizing effect, decay estimates and Nash inequalities

We study the short and large time behaviour of solutions of nonlocal heat equations of the form $\partial_tu+\mathcal{L} u = 0$. Here $\mathcal{L}$ is an integral operator given by a symmetric nonnegative kernel of Lévy type, that includes bounded and unbounded transition probability densities. We characterize when a regularizing effect occurs for small times and obtain $L^q$-$L^p$ decay estimates, $1≤ q < p < ∞$ when the time is large. These properties turn out to depend only on the behaviour of the kernel at the origin or at infinity, respectively, without need of any information at the other end. An equivalence between the decay and a restricted Nash inequality is shown. Finally we deal with the decay of nonlinear nonlocal equations of porous medium type $\partial_tu+\mathcal{L}Φ(u) = 0$.

Uniformization: Basics, extensions and applications

Uniformization, also referred to as randomization, is a well-known performance evaluation technique to model and analyse continuous-time Markov chains via an easier to performance measures via iteration of the one-step transition matrix of the discrete-time Markov chain. The number of iterations has a Poisson distribution with rate dominating the maximum exit rate from the states of the continuous-time Markov chain. This paper contains an expository presentation of uniformization techniques to increase awareness and to provide a formal and intuitive justification of several exact and approximate extensions, including: •exact uniformization for reward models,•exact uniformization for time-inhomogeneous rates,•a numerical comparison with simple time-discretization,•approximate uniformization for unbounded transition rates, and•exact uniformization for continuous state variables for non-exponential networks.Furthermore, several of these results are numerically illustrated for a processor sharing web server tandem model of practical interest.