Ergodic Classes Research Articles

Average cost Markov decision processes under the hypothesis of Doeblin

Average cost Markov decision processes (MDPs) with compact state and action spaces and bounded lower semicontinuous cost functions are considered. Kurano [7] has treated the general case in which several ergodic classes and a transient set are permitted for the Markov process induced by any randomized stationary policy under the hypothesis of Doeblin and showed the existence of a minimum pair of state and policy. This paper considers the same case as that discussed in Kurano [7] and proves some new results which give the existence theorem of an optimal stationary policy under some reasonable conditions.

Exact formula for sensitivity analysis of Markov chains

Given a family of Markov chains whose transition matrices depend on a parameter vector, we given an exact formula for the gradient of the equilibrium distribution with respect to that parameter even in the case of multiple ergodic classes and transient states. This formula generalizes previous results in the ergodic case.

Nonergodicity of local, length-conserving Monte Carlo algorithms for the self-avoiding walk

It is proved that every dynamic Monte Carlo algorithm for the self-avoiding walk based on a finite repertoire of local,N-conserving elementary moves is nonergodic (hereN is the number of bonds in the walk). Indeed, for largeN, each ergodic class forms an exponentially small fraction of the whole space. This invalidates (at least in principle) the use of the Verdier-Stockmayer algorithm and its generalizations for high-precision Monte Carlo studies of the self-avoiding walk.

Relaxed Markov processes

The concept of relaxing a Markov process is introduced; this is the creation of additional transitions between ergodic classes of the process in such a way as to conserve the existing equilibrium distribution within ergodic classes. The ‘open' version of a ‘closed' model of migration, polymerisation etc. often has this character. As further examples, generalized versions of Jackson networks and networks with clustering nodes are given.

Relaxed Markov processes

The concept of relaxing a Markov process is introduced; this is the creation of additional transitions between ergodic classes of the process in such a way as to conserve the existing equilibrium distribution within ergodic classes. The ‘open' version of a ‘closed' model of migration, polymerisation etc. often has this character. As further examples, generalized versions of Jackson networks and networks with clustering nodes are given.

Constrained Markov Decision Chains

We consider finite state and action discrete time parameter Markov decision chains. The objective is to provide an algorithm for finding a policy that minimizes the long-run expected average cost when there are linear side conditions on the limit points of the expected state-action frequencies. This problem has been solved previously only for the case where every deterministic stationary policy has at most one ergodic class. This note removes that restriction by applying the Dantzig-Wolfe decomposition principle.

On quasi-invariant measures in uniquely ergodic systems

Let X be a compact metrizable space, ~ (X) its G-algebra of Borel sets, and let T be a homeomorphism of X. We denote for a Borel measure # on X by T/2 the measure that is obtained by setting TI~(A)=I~(T -1 A), A e ~ . There exists always a Borel probability measure # such that T/2=# [10, p. 76]. If there exists only one such invariant measure then T is called uniquely ergodic. A measure # is called quasi-invariant for T if T# is equivalent to/2. In this paper we show that every uniquely ergodic homeomorphism of a compact metrizable space whose invariant measure is non-atomic possesses more than countably many non-atomic quasi-invariant ergodic measure classes. The problem of describing such measure classes has arisen in the theory of unitary representations of locally compact groups [12, p. 650]. In fact, we shall prove a more precise result. To state this result we need more terminology. Let (Y, ~ ) be a Borel structure and let ~ be an equivalence relation in Y. It can happen that the G-algebra ~ contains sets A i, i eN , such that

Existence of a Stationary Control for a Markov Chain Maximizing the Average Reward

The problem of optimal control of a discrete time stationary Markov chain with complete state information has been considered by many authors. The case with finitely many states and controls has been thoroughly investigated. Chains with infinitely many states or controls have also been considered with various assumptions concerning the reward function. In this paper the existence of a control maximizing the average reward is established for Markov chains with a finite number of states and an arbitrary compact set of possible actions in each state. It is assumed that there is only one ergodic class and no transient states in the chain for every control. The method of proof uses methods from convex programming, and is analogous to the linear programming approach used by Wolfe and Danzig.

A Note on Memoryless Rules for Controlling Sequential Control Processes

A Note on Memoryless Rules for Controlling Sequential Control Processes

Singular functions associated with Markov chains

We suppose {xi , i-= 1, 2, ... to be a chain with a finite number of states, 0, 1, * , M1, and consider the random variable X = Zf=1 xiMand its associated distribution function F(x) =Prob {X<x}. We write F(A)=Prob {XEA} =fAdF(x). F(A) is a completely additive probability measure on the Borel field of sets in [0, 1] generated by sets of the form { F(x) <a }. Harris [1] has shown that under very general conditions on the stationarity of the chain that F(x) is a purely singular function and that 0(t)t0-+-O0 where q(t) is the Fourier-Stieltjes transform ?k(t) =feitxdF(x). Wiener and Wintner [2 ] used the connection between the Lipschitz condition satisfied by F(x) and the behavior of +(t) to show that there are purely singular f unctions F(x) for which X (t)t = O (t-a) for all a < 1/2. Salem [3] showed the connection between the Hausdorff measure of the set E on which F(A) is concentrated and the behavior of +(t) for large t. Although in our case k(t)tiO, the Lipschitz condition and the Hausdorff dimension of E still play a role. Namely, when the xi form a stationary Markov chain, with a single ergodic class, they are the entropy, in the sense of Shannon [4], of the sequence {xi} considered as the sequence of states of a symbol-generating source. The dimensional number :(E) of a set EC [0, 1] is defined as follows: If / ,_maxi I i|, where {Ili is a set of intervals, and ECUIi, we say Cl,u = UIi is a covering of E of norm y. We let