Parameters Of Markov Chain Research Articles

Truncated taylor series approximation to the state transition matrix of a continuous parameter finite Markov chain

A readily computable bound is established for the error in approximating the transition matrix for a continuous parameter Markov chain.

Approximations for the repairman problem with two repair facilities, II: Spares

The model considered here consists of n operating units which are subject to stochastic failure according to an exponential failure time distribution. These operating units are backed up by mn spare units. Failures can be of two types. With probability p (q) a failure is of type 1(2) and is sent to repair facility 1(2) for repair. Repair facility 1(2) operates as a -server queue with exponential repair times having parameter μ 1 (μ 2). The number of units waiting for or undergoing repair at each of the two facilities is a continuous parameter Markov chain with finite state space. This paper derives limit theorems for the stationary distribution of this Markov chain as n becomes large under the assumption that and mn grow linearly with n. These limit theorems give very useful approximations, in terms of the seven parameters characterizing the model, to a distribution that would be difficult to calculate in practice.

Approximations for the repairman problem with two repair facilities, II: Spares

The model considered here consists of n operating units which are subject to stochastic failure according to an exponential failure time distribution. These operating units are backed up by mn spare units. Failures can be of two types. With probability p (q) a failure is of type 1(2) and is sent to repair facility 1(2) for repair. Repair facility 1(2) operates as a -server queue with exponential repair times having parameter μ1 (μ2). The number of units waiting for or undergoing repair at each of the two facilities is a continuous parameter Markov chain with finite state space. This paper derives limit theorems for the stationary distribution of this Markov chain as n becomes large under the assumption that and mn grow linearly with n. These limit theorems give very useful approximations, in terms of the seven parameters characterizing the model, to a distribution that would be difficult to calculate in practice.

Some theorems on the transient covariance of Markov chains

Several authors have considered the covariance structure of continuous parameter Markov chains. Most of this work has dealt with particular process ses, notably Morse (1955) who analysed the simple M/M/1 queue and Bene-(1961) who considered a telephone trunking model. Furthermore, the results obtained apply only when the process has attained its limiting (stationary) distribution. A recent paper by Reynolds (1968) gave some general results for finite chains, still assuming stationarity. This note generalises the results obtained therein, and considers the covariance structure during the transient period prior to attaining the stationary distribution where this exists. In the case where no such distribution exists, the results are valid throughout the whole lifetime of the process.

Some theorems on the transient covariance of Markov chains

Several authors have considered the covariance structure of continuous parameter Markov chains. Most of this work has dealt with particular process ses, notably Morse (1955) who analysed the simple M/M/1 queue and Bene-(1961) who considered a telephone trunking model. Furthermore, the results obtained apply only when the process has attained its limiting (stationary) distribution. A recent paper by Reynolds (1968) gave some general results for finite chains, still assuming stationarity. This note generalises the results obtained therein, and considers the covariance structure during the transient period prior to attaining the stationary distribution where this exists. In the case where no such distribution exists, the results are valid throughout the whole lifetime of the process.

Representation of terpolymer kinetics on triangular coordinates

AbstractThe ideal steady‐state kinetics of polymer formation from two or more monomeric species can be derived from basic rate equations by using the theory of discrete parameter Markov chains. Advantages are: (1) steady‐state assumptions are not required; and (2) a particularly lucid graphical visulization is possible for the case where there are three monomeric species.

State spaces for Markov chains

If p ( t , i , j ) p(t,i,j) is the transition probability (from i to j in time t) of a continuous parameter Markov chain, with p ( 0 + , i , i ) = 1 p(0 + ,i,i) = 1 , entrance and exit spaces for p are defined. If L [ L ∗ ] L[{L^ \ast }] is an entrance [exit] space, the function p ( ⋅ , ⋅ , j ) [ p ( ⋅ , i , ⋅ ) / h ( ⋅ ) ] p( \cdot , \cdot ,j)[p( \cdot ,i, \cdot )/h( \cdot )] has a continuous extension to ( 0 , ∞ ) × L [ ( 0 , ∞ ) × L ∗ (0,\infty ) \times L[(0,\infty ) \times {L^ \ast } , for a certain norming function h on L ∗ {L^ \ast } ]. It is shown that there is always a space which is both an entrance and exit space. On this space one can define right continuous strong Markov processes, for the parameter interval [0, b], with the given transition function as conditioned by specification of the sample function limits at 0 and b.

The Martin boundary and completion of Markov chains

In this paper we treat a time-symmetrical Martin boundary theory for continuous parameter Markov chains. This is done by reversing the time sense of a Markov chainX t in such a way as to obtain a dual Markov chain $$\tilde X_t $$ , and considering the two chains together. Various relations between the Martin exit boundaries $$B_0^* $$ and $$\tilde B_0^* $$ of these processes are studied. The exit boundary $$B_0^* $$ of $$\tilde X_t $$ , is in a sense an entrance boundary forX t and vice versa. After a natural identification of certain points in $$B_0^* $$ and $$\tilde B_0^* $$ one can topologizeI ∪ $$B_0^* $$ ∪ $$\tilde B_0^* $$ in such a way thatboth X t and $$\tilde X_t $$ have standard modifications in this space which are right continuous, have left limits, and are strongly Markov.

A Stable Limit Theorem for Markov Chains

In [2] Doeblin obtained a central limit for discrete parameter Markov chains with discrete state space. In obtaining this theorem the principal tool is the Doeblin dissection of the sequence of partial sums of a functional of a Markov chain into a random sum of independent, identically distributed random variables. Indeed, given this dissection, the remainder of Doeblin's proof is, in essence, a proof of a random central limit theorem. Although Doeblin does not state a stable limit theorem for Markov chains, at the end of the paper he observes that the dissection should also be of use in obtaining such theorems, and comments on a possible method of obtaining theorems of this nature. In this paper three stable limit theorems for Markov chains together with the appropriate solidarity theorems are obtained depending on whether the index $\alpha$ of the limiting distribution is $ 1$. The principal tools used in obtaining these theorems are the Doeblin dissection, a well known result concerning the rate of growth of the sequence of norming coefficients of a random variable in the domain of attraction of a stable law, and a random stable limit theorem [8], [10].

A Note on Sequences of Continuous Parameter Markov Chains

A Note on Sequences of Continuous Parameter Markov Chains

Approximate Construction of Confidence Intervals for Parameters of Markov Chains

Approximate Construction of Confidence Intervals for Parameters of Markov Chains

Die Struktur der Ausscheideordnungen in der Invaliditätsversicherung im Lichte der modernen Mengen- und Wahrscheinlichkeitstheorie

The present treatise deals with the structure and functions of the multiple decrement tables in disability insurance on the basis of modern set and probability theories. In the first part, the author shows the set relations for the life table, the service table and the open disablement table. In this respect, two simple relations (see formula 1,39) are indicated for the two independent annual probabilities of exit in sense of the probability theory (probability of disablement). After elucidation of the pertinent theory, the process of decrement out of a closed service table is described as a continuous parameter Markov chain with non-homogeneous transition probability and finitely many states. The discussion of the homogeneous case leads further to an analysis of the mechanism of decrement due to two independent causes. Finally, the two distributions of the probabilities of remaining in the state or of leaving it are deduced.

Restricted transition probabilities and their applications to some problems in the dynamics of biological populations.

In this paper a theory of a class of restricted transition probabilities is developed and applied to a problem in the dynamics of biological populations under the assumption that the underlying stochastic process is a continuous time parameter Markov chain with stationary transition probabilities. The paper is divided into three parts. Part one contains sufficient background from the theory of Markov processes to define restricted transition probabilities in a rigorous manner. In addition, some basic concepts in the theory of stochastic processes are interpreted from the biological point of view. Part two is concerned with the problem of finding representations for restricted transition probabilities. Finally, in part three the theory of restricted transition probabilities is applied to the problem of finding and analyzing some properties of the distribution function of the maximum size attained by the population in a finite time interval for a rather wide class of Markov processes. Some other applications of restricted transition probabilities to other problems in the dynamics of biological populations are also suggested. These applications will be discussed more fully in a companion paper.

Potentials associated with recurrent Markov renewal processes

In [1] and more recently in [2], Chapters III and VII, Spitzer constructs potentials for a particular class of recurrent Markov chains (M.C.'s), namely, the class of recurrent random walks on the n-dimensional lattice of integers ( n = 1 or 2). In [3], [4], and [5], Kemeny and Snell construct a potential theory for arbitrary recurrent M.C.'s. Orey [6] has characterized potential kernels for recurrent M.C.'s. The potentials and kernels studied in these papers have properties which are analogous to those defined for transient M.C.'s. This paper extends the results of Kemeny and Snell to Markov renewal processes (MRP's). Section 1 contains a brief discussion of MRP's in general, a definition of the class of MRP's to be studied in this paper, and a summary of some needed relationships proved in other papers [7–10]. Section 2 contains some preliminary lemmas. In Section 3, a potential of functions is defined for the semi-Markov process (S-MP) associated with the MRP. The concept of a left-normal MRP is introduced and its relationship with a certain potential kernel is discussed. Section 4 contains applications of these results to the important case of a continuous parameter Markov chain (c.p.M.C.). Results are also obtained by means of “duality” for the potential of σ-finite signed measures defined for a c.p.M.C. In Section 5, a potential of functions is defined for an associated three-dimensional Markov process. Section 6 contains several miscellaneous results which have applications to previous sections. The MRP's considered are assumed to be irreducible, recurrent, nonlattice, and to satisfy hypothesis A (see below) unless otherwise specified. The nonlattice assumption causes no loss in generality, the lattice case (into which Kemeny and Snell's work falls) being handled by restricting t to take on only such values as are multiples of the span.

A Dynamic Programming Algorithm for Embedded Markov Chains when the Planning Horizon is at Infinity

This paper presents an algorithm for the solution of dynamic programming problems requiring the determination of optimal policies for the control of a special class of stochastic processes when the time horizon of the planning period is at infinity. These processes can be mathematically described as discrete time parameter Markov chains with a finite number of states which have been “embedded” in continuous time in the sense that the time between transitions is a random variable whose probability distribution depends only on the states between which the transition takes place. Such processes are called Markov-renewal processes. The Markov processes considered by R. A. Howard in [1] are really two special cases of this somewhat wider class of stochastic processes. In these two special cases, the algorithm of this paper is identical with Howard's. In fact, with only slight modification, Howard's algorithm can be extended to this wider class of stochastic processes.

Functions of a Markov Process That are Markovian

In this paper we are primarily concerned with discrete time parameter Markov processes {X(n)}, n = 0, 1, 2, • • • , with stationary transition mechanism. The processes {Y(n)} = {f(X(n))} generated by a given manyone function / and the processes {X(n)} with a fixed stationary transition mechanism are constructed. The processes {Y(n)} are in a one-to-one correspondence with the possible initial distributions of {X(n)\. The object of the paper is to determine conditions under which {Y(n)} is Markovian, whatever the initial distribution of {X(n)}. Necessary and sufficient conditions for the new processes {Y(n) to be Markovian are obtained under the assumption that the family of measures corresponding to the fixed transition mechanism (of {X(n)}) is dominated [4]. The conditions are expressed, of course, in terms of the function /(•) and the transition mechanism. Generally the processes {Y(n)} do not have stationary transition mechanism. The conditions simplify in the case of a continuous time parameter Markov chain. Some of the discussions may at times have the flavor of those used in considering the concept of sufficiency [4]. Some aspects of the problem discussed in the paper are touched on in [2],