Ergodic Classes Research Articles

Efficient learning for decomposing and optimizing random networks

In this study, we consider the problem of node ranking in a random network. A Markov chain is defined for the network, and its transition probability matrix is unknown but can be learned by sampling random interactions among nodes. Our objective is to decompose the Markov chain into several ergodic classes and select the best node in each ergodic class. We propose a dynamic sampling procedure, which gives a probability guarantee on correct decomposition and maximizes a weighted probability of correct selection of the best node in each ergodic class. Numerical experiment results demonstrate the efficiency of the proposed sampling procedure.

Analysis of Markov Influence Graphs

There is a growing interest in the analysis of networks found in the World Wide Web and in social networks. A common feature of these networks is that the finite-state Markov chain modeling the influence relation between nodes typically has several (nearly) ergodic classes. This paper introduces a new decomposition algorithm for Markov chains that allows to split the graph of the Markov chain into subgraphs such that the connectivity of the chain—measured by the Kemeny constant—is maximally decreased. In other words, we show how the structure dormant in a nearly decomposable chain can be brought to light. We present applications to influence ranking in social networks, decomposition of a social network into subnetworks, and cluster analysis.

The Jump Start Power Method: A New Approach for Computing the Ergodic Projector of a Finite Markov Chain

This article presents a new numerical method for approximately computing the ergodic projector of a finite Markov chain. Our approach requires neither structural information on the chain, such as, the identification of ergodic classes, transient states, or qualitative information, such as whether the chain is nearly decomposable or not. The theoretical deduction of the new method is corroborated by an extensive numerical study.

Convergence analysis for pure stationary strategies in repeated potential games: Nash, Lyapunov and correlated equilibria

In game theory the interaction among players obligates each player to develop a belief about the possible strategies of the other players, to choose a best-reply given those beliefs, and to look for an adjustment of the best-reply and the beliefs using a learning mechanism until they reach an equilibrium point. Usually, the behavior of an individual cost-function, when such best-reply strategies are applied, turns out to be non-monotonic and concluding that such strategies lead to some equilibrium point is a non-trivial task. Even in repeated games the convergence to a stationary equilibrium is not always guaranteed. The best-reply strategies analyzed in this paper represent the most frequent type of behavior applied in practice in problems of bounded rationality of agents considered within the Artificial Intelligence research area. They are naturally related with the, so-called, fixed-local-optimal actions or, in other words, with one step-ahead optimization algorithms widely used in the modern Intelligent Systems theory.This paper shows that for an ergodic class of finite controllable Markov games the best-reply strategies lead necessarily to a Lyapunov/Nash equilibrium point. One of the most interesting properties of this approach is that an expedient (or absolutely expedient) behavior of an ergodic system (repeated game) can be represented by a Lyapunov-like function non-decreasing in time. We present a method for constructing a Lyapunov-like function: the Lyapunov-like function replaces the recursive mechanism with the elements of the ergodic system that model how players are likely to behave in one-shot games. To show our statement, we first propose a non-converging state-value function that fluctuates (increases and decreases) between states of the Markov game. Then, we prove that it is possible to represent that function in a recursive format using a one-step-ahead fixed-local-optimal strategy. As a result, we prove that a Lyapunov-like function can be built using the previous recursive expression for the Markov game, i.e., the resulting Lyapunov-like function is a monotonic function which can only decrease (or remain the same) over time, whatever the initial distribution of probabilities. As a result, a new concept called Lyapunov games is suggested for a class of repeated games. Lyapunov games allow to conclude during the game whether the applied strategy provides the convergence to an equilibrium point (or not). The time for constructing a Potential (Lyapunov-like) function is exponential. Our algorithm tractably computes the Nash, Lyapunov and the correlated equilibria: a Lyapunov equilibrium is a Nash equilibrium, as well it is also a correlated equilibrium. Validity of the proposed method is successfully demonstrated both theoretically and practically by a simulated experiment related to the Duel game.

Efficient Algorithm for Computing the Ergodic Projector of Markov Multi-chains

Abstract This paper extends the Markov uni-chain series expansion theory to Markov multi-chains, i.e., to Markov chains having multiple ergodic classes and possible transient states. The introduced series expansion approximation (SEA) provides a controllable approximation for Markov multi-chain ergodic projectors which may be a useful tool in large-scale network analysis. As we will illustrate by means of numerical examples, the new algorithm is faster than the power algorithm for large networks.

On Approximations and Ergodicity Classes in Random Chains

We study the limiting behavior of a random dynamic system driven by a stochastic chain. Our interest is in the chains that are not necessarily ergodic but are decomposable into ergodic classes. To investigate the conditions under which the ergodic classes of a model can be identified, we introduce and study an <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$\ell_{1}$</tex> </formula> -approximation and infinite flow graph of the model. We show that the <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$\ell_{1}$</tex></formula> -approximations of random chains preserve certain limiting behavior. Using the <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex Notation="TeX">$\ell_{1}$</tex></formula> -approximations, we show how the connectivity of the infinite flow graph is related to the structure of the ergodic groups of the model. Our main result of this paper provides conditions under which the ergodicity groups of the model can be identified by considering the connected components in the infinite flow graph. We provide two applications of our main result to random networks, namely broadcast over time-varying networks and networks with random link failure.

Canonical simplicial gravity

A general canonical formalism for discrete systems is developed, which can handle varying phase space dimensions and constraints. The central ingredient is Hamilton's principal function that generates canonical time evolution and ensures that the canonical formalism reproduces the dynamics of the covariant formulation following directly from the action. We apply this formalism to simplicial gravity and (Euclidean) Regge calculus, in particular. A discrete forward/backward evolution is realized by gluing/removing single simplices step by step to/from a bulk triangulation and amounts to Pachner moves in the triangulated hypersurfaces. As a result, the hypersurfaces evolve in a discrete ‘multi-fingered’ time through the full Regge solution. Pachner moves are an elementary and ergodic class of homeomorphisms and generically change the number of variables, but can be implemented as canonical transformations on naturally extended phase spaces. Some moves introduce a priori free data that, however, may become fixed a posteriori by constraints arising in subsequent moves. The end result is a general and fully consistent formulation of canonical Regge calculus, thereby removing a longstanding obstacle in connecting covariant simplicial gravity models to canonical frameworks. The presented scheme is, therefore, interesting in view of many approaches to quantum gravity, but may also prove useful for numerical implementations.

The Geometric Markov Renewal Processes with Application to Finance

We introduce the geometric Markov renewal processes as a model for a security market and study this processes in a series scheme. We consider its approximations in the form of averaged, merged and double averaged geometric Markov renewal processes. Weak convergence analysis and rates of convergence of ergodic geometric Markov renewal processes are presented. Martingale properties, infinitesimal operators of geometric Markov renewal processes are presented and a Markov renewal equation for expectation is derived. As an application, we consider the case of two ergodic classes. Moreover, we consider a generalized binomial model for a security market induced by a position dependent random map as a special case of a geometric Markov renewal process.

Determinants and Longest Cycles of Graphs

We consider the Hamiltonian cycle problem on a given graph $G$. With such a graph we can associate a family $\mathcal{F}$ of probability transition matrices of Markov chains whose entries represent the probabilities of traversing corresponding arcs of the graph. When the underlying graph is Hamiltonian, we show the transition probability matrix induced by a Hamiltonian cycle maximizes—over $\mathcal{F}$—the determinant of a matrix that is a rank-one correction of the generator matrix of a Markov chain. In the case when the graph does not possess a Hamiltonian cycle, the above maximization yields a transition matrix of a chain with a longest simple cycle (in $G$) comprising that chain's unique ergodic class. These problems also have analogous eigenvalue interpretations.

NP-Hardness of checking the unichain condition in average cost MDPs

The unichain condition requires that every policy in an MDP result in a single ergodic class, and guarantees that the optimal average cost is independent of the initial state. We show that checking whether the unichain condition fails to hold is an NP-complete problem. We conclude with a brief discussion of the merits of the more general weak accessibility condition.

Two-time-scale Hybrid Filters: Near Optimality

This work develops a filtering scheme for hybrid systems. The process dictating the configuration or regimes is a continuous-time Markov chain with a finite state space. Exploiting hierarchical structure of the underlying system, the states of the Markov chain are divided into a number of groups so that it jumps rapidly within each group and slowly among different groups. Focusing on reduction of computational complexity, the filtering scheme includes the following steps: (1) partition the state space of the Markov chain into subspaces, (2) derive a limit system in which the states are averaged out with respect to the invariant distributions of the Markov chain, (3) use the limit system to design quadratic variation test statistics, and (4) use the test statistics to identify which ergodic class the aggregated process belongs to and to construct near-optimal filter. For demonstration, a numerical example is presented.

Perturbation analysis for denumerable Markov chains with application to queueing models

We study the parametric perturbation of Markov chains with denumerable state spaces. We consider both regular and singular perturbations. By the latter we mean that transition probabilities of a Markov chain, with several ergodic classes, are perturbed such that (rare) transitions among the different ergodic classes of the unperturbed chain are allowed. Singularly perturbed Markov chains have been studied in the literature under more restrictive assumptions such as strong recurrence ergodicity or Doeblin conditions. We relax these conditions so that our results can be applied to queueing models (where the conditions mentioned above typically fail to hold). Assumingν-geometric ergodicity, we are able to explicitly express the steady-state distribution of the perturbed Markov chain as a Taylor series in the perturbation parameter. We apply our results to quasi-birth-and-death processes and queueing models.

Perturbation analysis for denumerable Markov chains with application to queueing models

We study the parametric perturbation of Markov chains with denumerable state spaces. We consider both regular and singular perturbations. By the latter we mean that transition probabilities of a Markov chain, with several ergodic classes, are perturbed such that (rare) transitions among the different ergodic classes of the unperturbed chain are allowed. Singularly perturbed Markov chains have been studied in the literature under more restrictive assumptions such as strong recurrence ergodicity or Doeblin conditions. We relax these conditions so that our results can be applied to queueing models (where the conditions mentioned above typically fail to hold). Assuming ν-geometric ergodicity, we are able to explicitly express the steady-state distribution of the perturbed Markov chain as a Taylor series in the perturbation parameter. We apply our results to quasi-birth-and-death processes and queueing models.

Exponential bounds for discrete-time singularly perturbed Markov chains

This paper develops exponential type upper bounds for scaled occupation measures of singularly perturbed Markov chains in discrete time. By considering two-time scale in the Markov chains, asymptotic analysis is carried out. The cases of the fast changing transition probability matrix is irreducible and that are divisible into l ergodic classes are examined first; the upper bounds of a sequence of scaled occupation measures are derived. Then extensions to Markov chains involving transient states and/or nonhomogeneous transition probabilities are dealt with. The results enable us to further our understanding of the underlying Markov chains and related dynamic systems, which is essential for solving many control and optimization problems.

Markov Games with Several Ergodic Classes

We consider Markov games of the general form characterized by the property that, for all stationary strategies of players, the set of game states is partitioned into several ergodic sets and a transient set, which may vary depending on the strategies of players. As a criterion, we choose the mean payoff of the first player per unit time. It is proved that the general Markov game with a finite set of states and decisions of both players has a value, and both players have e-optimal stationary strategies. The correctness of this statement is demonstrated on the well-known Blackwell's example (“Big Match”).

On One Property of a Regular Markov Chain

We prove that if a certain row of the transition probability matrix of a regular Markov chain is subtracted from the other rows of this matrix and then this row and the corresponding column are deleted, then the spectral radius of the matrix thus obtained is less than 1. We use this property of a regular Markov chain for the construction of an iterative process for the solution of the Howard system of equations, which appears in the course of investigation of controlled Markov chains with single ergodic class and, possibly, transient states.

On a new method of Markov chain reduction

The practical usefulness of Markov models and Markovian decision process has been severely limited due to their extremely large dimension. Thus, a reduced model without sacrificing significant accuracy can be very interesting. The homogeneous finite Markov chain's long-run behaviour is given by the persistent states, obtained after the decomposition in classes of connected states. In this paper we expound a new reduction method for ergodic classes formed by such persistent states. An ergodic class has a steady-state independent of the initial distribution. This class constitutes an irreducible finite ergodic Markov chain, which evolves independently after the capture of the event. The reduction is made according to the significance of steady-state probabilities. For being treatable by this method, the ergodic chain must have the Two-Time-Scale property. The presented reduction method is an approximate method. We begin with an arrangement of irreducible Markov chain states, in decreasing order of their ste...

Mean-Variance Tradeoffs in an Undiscounted MDP

A stationary policy and an initial state in an MDP (Markov decision process) induce a stationary probability distribution of the reward. The problem analyzed here is generating the Pareto optima in the sense of high mean and low variance of the stationary distribution. In the unichain case, Pareto optima can be computed either with policy improvement or with a linear program having the same number of variables and one more constraint than the formulation for gain-rate optimization. The same linear program suffices in the multichain case if the ergodic class is an element of choice.

Asymptotic behavior of the transition probability of a homogeneous Markov process with a finite number of states in a series model

The asymptotic behavior of the transition probabilities Pij(n)(t/ɛn) for a homogeneous stochastically continuous Markov process with a finite set of states E where E can be represented in the form E=E0 U E1 U ... U Er, E0 is the set of unessential states, Ek, k=1, ..., r are ergodic classes.

The BFACF algorithm and knotted polygons

The BFACF algorithm applied to polygons involves sampling on a Markov chain whose state space is the set of all polygons. In three dimensions, for the simple cubic lattice. The authors prove that the ergodic classes of this Markov chain are the knot classes of the polygons.