Continuous-time Finite-state Markov Research Articles

Poisson approximations for Markov-driven point processes

An asymptotically finite bound is derived for the total variation distance between the distribution of N( t) and the Poisson distribution with mean E N( t) when N is a simple point process whose interpoint times are exponential with means determined by an ergodic, finite-state Markov chain and when it is a Cox process with a stationary, irreducible, finite-state continuous-time Markov chain for intensity.

Continuous-time markov chains and compound poisson process with circulant intensity matrices

A classical result states that for a finite-state homogeneous continuous-time Markov chain with finite state space and intensity matrix Q=(qk) the matrix of transition probabilities is given by . In contrast to this infinite series expression, we derive an explicit finite sum formula for this transition matrix for situations where Q is a circulant matrix. Our result is motivated by the compound Poisson process (with discrete random i.i.d. variable Yj and a Poisson counting process It is shown that the stochastic process Xt = Dt mod n is a Markov process on E with a circulant intensity matrix Q and we apply the previous results to calculate, e.g., the distribution and the expectation of Xt The process provides a stochastic model for,e.g., channel assignment in telecommunication, bus occupancies, box packing etc.

Minimax control of switching systems under sampling

We consider a general class of systems subject to two types of uncertainty: A continuous deterministic uncertainty that affects the system dynamics, and a discrete stochastic uncertainty that leads to jumps in the system structure at random times, with the latter described by a continuous-time finite state Markov chain. When only sampled values of the system state is available to the controller, along with perfect measurements on the state of the Markov chain, we obtain a characterization of minimax controllers, which involves the solutions of two finite sets of coupled PDEs, and a finite dimensional compensator. For the linear-quadratic case, a complete characterization is given in terms of coupled generalized Riccati equations, which also provides the solution to a particular H ∞ optimal control problem with randomly switching system structure and sampled state measurements.

An Introduction to Stochastic Modeling

Stochastic Modeling. Probability Review. The Major Discrete Distributions. @rtant Continuous Distributions. Some Elementary Exercises. Useful Functions, Integrals, and Sums. Conditional Probability and Conditional Expectation: The Discrete Case. The Dice Game Craps. Random Sums. Conditioning on a Continuous Random Variable. Markov Chains: Introduction: Definitions. Transition Probability Matrices of a Markov Chain. Some Markov Chain Models. First Step Analysis. Some Special Markov Chains. Functionals of Random Walks and Success Runs. Another Look at First Step Analysis. The Long Run Behavior of Markov Chains: Regular Transition Probability Matrices. Examples. The Classification of States. The Basic Limit Theorem of Markov Chains. Reducible Markov Chains. Sequential Decisions and Markov Chains. Poisson Processes: The Poisson Distribution and the Poisson Processes. The Law of Rare Events. Distributions Associated with the Poisson Process. The Uniform Distribution and Poisson Processes. Spatial Poisson Processes. Compound and Marked Poisson Processes. Continuous Time Markov Chains: Pure Birth Processes. Ptire Death Processes. Birth and Death Processes. The Limiting Behavior of Birth and Death Processes. Birth and Death Processes with Absorbing States. Finite State Continuous Time Markov Chains. Set Valued Processes. Renewal Phenomena: Definition of a Renewal Process and Related Concepts. Some Examples of Renewal Processes. The Poisson Process Viewed as a Renewal Process. The Asymptotic ]3ehavior as Renewal Process.

Numerical computation of response time distributions using stochastic reward nets

We consider the numerical computation of response time distributions for closed product form queueing networks using thetagged customer approach. We map this problem on to the computation of the time to absorption distribution of a finite-state continuous time Markov chain. The construction and solution of these Markov chains is carried out using a variation of stochastic Petri nets called stochastic reward nets (SRNs). We examine the effects of changing the service discipline and the service time distribution at a queueing center on the response time distribution. A multiserver queueing network example is also presented. While the tagged customer approach for computing the response time distribution is not new, this paper presents a new approach for computing the response time distributions using SRNs.

Approximation and decomposition of singularly perturbed stochastic hybrid systems

This paper considers a singularly perturbed stochastic hybrid system whose state equations are governed by a stochastic switching process, which is modelled as a nearly completely decomposable continuous time finite state Markov chain. Approximate models for the slow mode subsystem are derived over the intervals of rapid switching and their accuracy (the mean-squared errors between the approximate models and the original systems) is quantified. Aggregate models for the subsystem are obtained over a longer period. The almost surely exponential stability of the subsystem is studied. The behaviour of the fast subsystem which depends on the relative size of the perturbation parameters is analysed. The decomposition of the overall system and the switching process together into slow and fast subsystems is investigated. The results derived in the paper are shown to hold when the switching process is stationary and irreducible, and each group of strongly interacting states is irreducible and time reversible. Finally a simple example is detailed to illustrate the aforementioned techniques

On the convergence of jacobi and gauss-seidel iteration for steady-state probabilities of finite-state continuous-time markov chains

We consider two nonsingular versions of the problem described in the title. For one of these versions, we show by example that neither Jacobi nor Gauss-Seidel iteration is guaranteed to converge; for the other version, we outline a proof that both methods are guaranteed to converge.

Exact filters for the estimation of the number of transitions of finite-state continuous-time Markov processes

The problem of estimating the number of transitions of finite-state continuous-time Markov processes observed by a noisy sensor is considered. A finite-dimensional exact filter is derived, and using the EM algorithm (an extension of the Baum-Welch algorithm for the discrete-time case), an application is made to the problem of estimating the unknown transition matrix of a finite-state continuous-time Markov process.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Finite-time implications of relaxation times for stochastically monotone processes

For a continuous-time finite state Markov process with stationary distribution π, it is well-known thatP i (X t =j)-π j isO(e -λt ) ast→∞, for a certain λ. For a stochastically monotone process for which the reversed process is also stochastically monotone, one can obtain bounds valid for allt. Precisely,\(\sum\limits_i {\pi _j \mathop {\max |}\limits_j P_i } (X_t \leqq j) - \pi [0,j]| \leqq 2(\lambda t + 2)\) exp(-λt). The proof exploits duality for stochastically monotone processes.

First-passage times with PFr densities

It is shown that if a finite-state continuous-time Markov process can be uniformized such that the embedded Markov chain has a TPr (totally positive of order r) transition matrix, then the first-passage time from state 0 to any other state has a PFr (Polya frequency of order r) density. As a consequence, results of Keilson (1971), Esary, Marshall and Proschan (1973), Ghosh and Ebrahimi (1982) and Derman, Ross and Schechner (1983) are strengthened. It is also shown that some cumulative damage shock models, with an underlying compound Poisson process and ‘damages' which are not necessarily non-negative, are associated with wear processes having PFr first-passage times to any threshold. First-passage times with completely monotone densities are also discussed.

First-passage times with PFr densities

It is shown that if a finite-state continuous-time Markov process can be uniformized such that the embedded Markov chain has a TPr (totally positive of order r) transition matrix, then the first-passage time from state 0 to any other state has a PF r (Polya frequency of order r) density. As a consequence, results of Keilson (1971), Esary, Marshall and Proschan (1973), Ghosh and Ebrahimi (1982) and Derman, Ross and Schechner (1983) are strengthened. It is also shown that some cumulative damage shock models, with an underlying compound Poisson process and ‘damages' which are not necessarily non-negative, are associated with wear processes having PF r first-passage times to any threshold. First-passage times with completely monotone densities are also discussed.

Asymptotic Properties of a Finite State Continuous Time Markov Decision Process

We consider a continuous time Markov decision process with a finite state space. There is a specified terminal reward, but the reward or cost rate is always zero. The maximum expected final gain can then be obtained by means of the exponential of a certain sublinear operator on $R^n$ This representation allows us to describe the asymptotic properties of the reward vector. We prove that the expected reward always tends to a limit as the time parameter $t \to \infty $. If we assume that it is allowed to stop the process in any state, then we can construct an almost optimal stationary control. Finally, we characterize the case where the asymptotic gain is independent of the initial state.

A singular perturbation approach to modeling and control of Markov chains

Finite state continuous time Markov processes with weak interactions are modeled as singularly perturbed systems. Aggregate states are obtained using a grouping algorithm. Two-time scale expansions simplify cost equations and lead to decentralized optimization algorithms.

Markov Decision Processes with a New Optimality Criterion: Continuous Time

Standard finite state and action continuous time Markov decision processes with discounting are studied using a new optimality criterion called moment optimality. A policy is moment optimal if it lexicographically maximizes the sequence of signed moments of total discounted return with a positive (negative) sign if the moment is odd (even). It is shown constructively that a stationary policy is moment optimal among the class of piecewise constant policies by examining the negative of the Laplace transform of the total return random variable and its Taylor series expansion.

Nondiscounted Continuous Time Markovian Decision Process with Countable State Space

Previous article Nondiscounted Continuous Time Markovian Decision Process with Countable State SpacePrasadarao KakumanuPrasadarao Kakumanuhttps://doi.org/10.1137/0310016PDFBibTexSections ToolsAdd to favoritesExport CitationTrack CitationsEmail SectionsAbout[1] Kai Lai Chung, Markov chains with stationary transition probabilities, Second edition. Die Grundlehren der mathematischen Wissenschaften, Band 104, Springer-Verlag New York, Inc., New York, 1967xi+301 MR0217872 0146.38401 Google Scholar[2] Cyrus Derman, Denumerable state Markovian decision processes—average cost criterion, Ann. Math. Statist., 37 (1966), 1545–1553 MR0207138 0144.43102 CrossrefISIGoogle Scholar[3] Ronald A. Howard, Dynamic programming and Markov processes, The Technology Press of M.I.T., Cambridge, Mass., 1960viii+136 MR0118514 0091.16001 Google Scholar[4] P. Kakumanu, Continuous time Markov decision models with applications to optimization problems, Tech. Rep., 63, Cornell University, Ithaca, N.Y., 1969 Google Scholar[5] Prasadarao Kakumanu, Continuously discounted Markov decision model with countable state and action space, Ann. Math. Statist., 42 (1971), 919–926 MR0282651 0234.93027 CrossrefISIGoogle Scholar[6] Bruce L. Miller, Finite state continuous time Markov decision processes with an infinite planning horizon, J. Math. Anal. Appl., 22 (1968), 552–569 10.1016/0022-247X(68)90194-7 MR0228251 0157.50301 CrossrefISIGoogle Scholar[7] Sheldon M. Ross, Non-discounted denumerable Markovian decision models, Ann. Math. Statist., 39 (1968), 412–423 MR0226933 0157.50504 CrossrefISIGoogle Scholar[8] Howard M. Taylor, III, Markovian sequential replacement processes, Ann. Math. Statist., 36 (1965), 1677–1694 MR0189839 0139.37802 CrossrefISIGoogle Scholar[9] D. V. Widder, The Laplace Transform, Princeton University Press, Princeton, N.J., 1946 Google Scholar Previous article FiguresRelatedReferencesCited byDetails Ergodic BSDEs Driven by Markov ChainsSamuel N. Cohen and Ying Hu28 October 2013 | SIAM Journal on Control and Optimization, Vol. 51, No. 5AbstractPDF (350 KB)Linear Programming and Constrained Average Optimality for General Continuous-Time Markov Decision Processes in History-Dependent PoliciesXianping Guo, Yonghui Huang, and Xinyuan Song3 January 2012 | SIAM Journal on Control and Optimization, Vol. 50, No. 1AbstractPDF (300 KB)Optimal Control of Ergodic Continuous-Time Markov Chains with Average Sample-Path RewardsXianping Guo and Xi-Ren Cao26 July 2006 | SIAM Journal on Control and Optimization, Vol. 44, No. 1AbstractPDF (207 KB)On Homogeneous Markov Models with Continuous Time and Finite or Countable State SpaceA. A. Yushkevic and E. A. Fainberg17 July 2006 | Theory of Probability & Its Applications, Vol. 24, No. 1AbstractPDF (653 KB)Controlled Markov Models with Countable State Space and Continuous TimeA. A. Yushkevich17 July 2006 | Theory of Probability & Its Applications, Vol. 22, No. 2AbstractPDF (1900 KB) Volume 10, Issue 1| 1972SIAM Journal on Control History Submitted:26 May 1970Published online:18 July 2006 InformationCopyright © 1972 Society for Industrial and Applied MathematicsPDF Download Article & Publication DataArticle DOI:10.1137/0310016Article page range:pp. 210-220ISSN (print):0036-1402Publisher:Society for Industrial and Applied Mathematics

Optimal Timing of Control Messages for a Two-State Markov Process

The purpose of this paper is to extend the results of the analysis [9] on the optimal timing of messages.' The model and theorems [9] are applicable to a situation involving the of a single (or aggregated) decision-making unit's performance terms of a single (or aggregated) goal. This goal need not be constant over time. The model used [9] is a continuous-time finite horizon model whose formulation and optimization involved the use of a finite-state continuous-time Markov process and tools from continuous-time optimal theory. In the present paper, attention is restricted to a two-state process. This restriction decreases the generality of the results return for a more specific model and more specific theorems. A two-state process is typically chosen for analytical treatment because of expositional convenience or practical applicability. The two states of the process are defined by the usual general descriptions: in control or consistent with goal(s) and out of control or inconsistent with goal(s). This type of process has been analyzed using mathematical programming, quality-control, and Markov chain techniques.2

Finite state continuous time Markov decision processes with an infinite planning horizon

Finite state continuous time Markov decision processes with an infinite planning horizon

Finite State Continuous Time Markov Decision Processes with a Finite Planning Horizon

The system we consider may be in one of n states at any point in time and its probability law is a Markov process which depends on the policy (control) chosen. The return to the system over a given planning horizon is the integral (over that horizon) of a return rate which depends on both the policy and the sample path of the process. Our objective is to find a policy which maximizes the expected return over the given planning horizon. A necessary and sufficient condition for optimality is obtained, and a constructive proof is given that there is a piecewise constant policy which is optimal. A bound on the number of switches (points where the piecewise constant policy jumps) is obtained for the case where there are two states.

Contraction Mappings in the Theory Underlying Dynamic Programming

Contraction Mappings in the Theory Underlying Dynamic Programming