Ergodic Homogeneous Markov Chain Research Articles

A sequential direct hybrid algorithm to compute stationary distribution of continuous-time Markov chain

In modeling a wide variety of problems of the real world using Markov processes, we usually get large sparse Markov chains. In the case of non-Markovian process, the behavior of a stochastic phenomenon or system can be approximated by continuous-time Markov chains (CTMC) using Phase-type distributions. However, this substantially increases the number of states, which requires a large amount of computation time and operational memory resources. Even the most recent computer hardware is not enough to provide a solution in almost real time, unless a specific algorithm is designed.In this paper, a new hybrid algorithm for the computation of a stationary distribution of a large ergodic homogeneous Markov chains with a finite state space and continuous time is suggested. This algorithm would solve the aforementioned challenges. Depending on model size and sparsity, it significantly reduces a computational time when compared with other recent methods, such as the SparseLU solver from Eigen library (version 3.9.9) and the MUMPS solver (MUltifrontal Massively Parallel sparse direct Solver, version 5.4.0). It is important to note that the algorithm works well independently on the structure of the generator matrix of a continuous-time Markov chain. For the study of the developed hybrid algorithm, the generator matrix is constructed randomly using a special algorithm, based on combining two methods: the slightly modified Grassmann, Taskar and Heyman (GTH) algorithm and the Gauss elimination method. The new idea is to remove the majority of states in a near-optimal order by the GTH method and complete computations for the remaining states by the Gauss method. The robust approach to identify a switching position from one algorithm to another one has been proposed and tested.

On improved bounds and conditions for the convergence of Markov chains

We continue the work of improving the rate of convergence of ergodic homogeneous Markov chains. The setting is more general than in previous papers: we are able to get rid of the assumption about a common dominating measure and consider the case of inhomogeneous Markov chains as well as more general state spaces. We give examples where the new bound for the rate of convergence is the same as (resp. better than) the classical Markov–Dobrushin inequality.

Quasi-metrics on graphs

In this paper we consider some questions of the theory and practice of mean first passage time quasi-metric, a generalized metric structure closely related to ergodic homogeneous Markov chains. In particular, we consider the structure and properties of mean first passage time quasimetrics based on several graph models, including simple cycles, simple paths and their oriented analogues. The introduction contains the history of the problem and provides an overview of the main ideas and results presented in the article. The second section gives basic concepts of the theory of Markov chains. In fact, a Markov chain is a mathematical model of some random process describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event. This section collects basic definitions needed to consider the role of graph models in the presentation and study of ergodic homogeneous Markov chains. The Markov chain can be depicted as an oriented weighted graph of transitions whose vertices correspond to the states of the chain and the arcs correspond to the transitions between them. The process will be ergodic if this weighted oriented graph is weakly connected, and the largest common divisor of the lengths of all its cycles is equal to 1. On the other hand, any connected graph can be used as a basis for building a model of the simplest Markov chain: if a vertex 𝑖 has degree 𝑘, all incident edges are converted into arcs with the weights 1/𝑘 . Moreover, in the second section the definition of the mean first passage time for an ergodic homogeneous Markov chain is given. The algorithm of finding the mean first passage time is analyzed in detail by using converging trees of the oriented graph, related to the transition matrix of the ergodic homogeneous Markov chain. At last, a mean first passage time is analyzed as the quasi-metric 𝑚 of mean first passage time on the vertices 𝑉 = {1, 2, ..., 𝑛} of the oriented graph corresponding to the transition matrix of a given ergodic homogeneous Markov chain: 𝑚(𝑖, 𝑗) is the expected number of steps (arcs) for random wandering on the oriented graph Γ, starting at 𝑖, to reach 𝑗 for the first time. This quasi-metric has a number of important theoretical and applied properties. The third section deals with the construction and research of mean first passage time quasimetrics for the undirected cycles 𝐶𝑛, 𝑛 ≥ 3. Examples of constructions of mean first passage time quasi-metrics of undirected cycles for small values of 𝑛 are considered. Illustrations of the ”graphical“ procedure of building the matrix 𝑀 are given. Properties of the resulting generalized metric structures are analyzed. In the fourth section, similar considerations for mean first passage time quasi-metrics of the undirected paths 𝑃𝑛, 𝑛 ≥ 2, are represented. The fifth section deals with the construction and research of mean first passage time quasimetrics for the directed (oriented) cycles 𝐶𝑛, 𝑛 ≥ 3. Examples of constructions of mean first passage time quasi-metrics of undirected (oriented) cycles for small values of 𝑛 are considered. Illustrations of the ”graphical“ procedure of building the matrix 𝑀 are given. Properties of the resulting generalized metric structures are analyzed. In the sixth section, similar considerations for mean first passage time quasi-metrics of the directed (oriented) paths 𝑃𝑛, 𝑛 ≥ 2, are represented. In the conclusion, the results of the work are summed up and possible directions of further research are outlined.

О специальных свойствах некоторых квазиметрик

В статье рассмотрены свойства квазиметрики среднего времени первого прохода - обобщенной метрической структуры, тесно связанной с эргодическими однородными цепями Маркова.Во введении представлена история вопроса, дан обзор основных идей и результатов работы.В первом разделе собраны основные понятия теориицепей Маркова - последовательностей случайных событий с конечным или счетным числом исходов, характеризующихся тем, что, говоря нестрого, при фиксированномнастоящем будущее независимо от прошлого. Точнее, математическая модель некоторого случайного процесса представляет собой марковскую цепь, если распределение вероятностей параметров процесса в следующий момент времени зависиттолько от параметров процесса в предыдущий момент.Во втором разделе собраны базовые определения, необходимые для рассмотрения роли графовых моделейв представлении и исследовании эргодических однородных цепей Маркова.Марковская цепь может быть изображена в виде ориентированного взвешенного графа переходов, вершины которого соответствуют состояниям цепи, а дуги - переходам между ними.Процесс будет эргодическим, если построенный взвешенный орграф является слабо связным, и наибольший общий делитель длин всех его циклов равен единице.С другой стороны, любой связный граф может служить базой для построения модели простейшей цепи Маркова: если вершина $i$ имеетстепень $k$, то все выходящие из нее ребра превращаются в дуги с весами $\frac{1}{k}$.%Основное место занимают понятия, связанные с теорией остовного леса сходящихся деревьев ориентированного %графа, соответствующего матрице перехода однородной эргодической цепи Маркова.В третьем разделе дано определение среднего времени первого прохода для однородной эргодической цепи Маркова. Представлены несколько способов построения соответствующей матрицы $M$. Подробно проанализирован алгоритм нахождения среднего времени первого прохода с помощью использования сходящихся деревьев ориентированного графа, связанного с матрицей перехода эргодической однородной цепи Маркова. Описана родственная рекуррентная процедура.В четвертом разделе матрица среднего времени первого прохода рассмотрена как {\it квазиметрика } $m$среднего времени первого прохода на множестве вершин $V=\{1, 2, ..., n\}$ ориентированного графа, соответствующего матрице перехода эргодической однородной цепи Маркова: $m(i,j)$ - ожидаемое количество шагов (дуг) для случайного блуждания на орграфе $\Gamma$, начинающегося с $i$, для достижения $j$ в первый раз. Эта квазиметрика обладает рядом важных теоретических и прикладных свойств.В частности, квазиметрика среднего времени первого прохода для простого случайного блуждания по связному невзвешенному графу $G$, в котором из любой вершины графа существует равная вероятность перемещения в любую соседнюю вершину,является взвешиваемой квазиметрикой, т.е. существует весовая функция $w: V\rightarrow\mathbb{R}_{\ge 0}$,%$w=(w_1, w_2,\ldots, w_n)$,такая, что для всех $i,j\in V$ имеет место сотношение %& Было бы здорово найти для $m_i$ наглядную интерпретацию.$m(i,j)+w_i=m(j,i)+w_j.$Менее изучены, но не менее интересны связи квазиметрики среднего времени первого прохода с другими метрическими структурами на графах, в частности, с $\alpha$-метрикой леса и ее вариациями.Наконец, в пятом разделе рассмотрены примеры построения и исследования квазиметрики среднего времени первого прохода. Помимо иллюстрации ''графовой`` процедуры построения матрицы $M$, представлены рекуррентные алгоритмы исследования и проанализированы получающиеся при этом обобщенные метрические структуры.

On Convergence Rates for Homogeneous Markov Chains

Improved rates of convergence for ergodic homogeneous Markov chains are studied. In comparison to the earlier papers the setting is also generalised to the case without a unique dominated measure. Examples are provided where the new bound is compared with the classical Markov -- Dobrushin inequality and with the second eigenvalue of the transition matrix for finite state spaces.

Local stationarity and time-inhomogeneous Markov chains

A primary motivation of this contribution is to define new locally stationary Markov models for categorical or integer-valued data. For this initial purpose, we propose a new general approach for dealing with time-inhomogeneity that extends the local stationarity notion developed in the time series literature. We also introduce a probabilistic framework which is very flexible and allows us to consider a much larger class of Markov chain models on arbitrary state spaces, including most of the locally stationary autoregressive processes studied in the literature. We consider triangular arrays of time-inhomogeneous Markov chains, defined by some families of contracting and slowly-varying Markov kernels. The finite-dimensional distribution of such Markov chains can be approximated locally with the distribution of ergodic Markov chains and some mixing properties are also available for these triangular arrays. As a consequence of our results, some classical geometrically ergodic homogeneous Markov chain models have a locally stationary version, which lays the theoretical foundations for new statistical modeling. Statistical inference of finite-state Markov chains can be based on kernel smoothing and we provide a complete and fast implementation of such models, directly usable by the practitioners. We also illustrate the theory on a real data set. A central limit theorem for Markov chains on more general state spaces is also provided and illustrated with the statistical inference in INAR models, Poisson ARCH models and binary time series models. Additional examples such as locally stationary regime-switching or SETAR models are also discussed.

Hitting time quasi-metric and its forest representation

Let $\hat m_{ij}$ be the hitting (mean first passage) time from state $i$ to state $j$ in an $n$-state ergodic homogeneous Markov chain with transition matrix $T$. Let $\Gamma$ be the weighted digraph whose vertex set coincides with the set of states of the Markov chain and arc weights are equal to the corresponding transition probabilities. It holds that $$ \hat m_{ij}= q_j^{-1}\cdot \begin{cases} f_{ij},&\text{if }\;\; i\ne j,\\ q, &\text{if }\;\; i=j, \end{cases} $$ where $f_{ij}$ is the total weight of 2-tree spanning converging forests in $\Gamma$ that have one tree containing $i$ and the other tree converging to $j$, $q_j$ is the total weight of spanning trees converging to $j$ in $\Gamma,$ and $q=\sum_{j=1}^nq_j$ is the total weight of all spanning trees in $\Gamma.$ Moreover, $f_{ij}$ and $q_j$ can be calculated by an algebraic recurrent procedure. A forest expression for Kemeny's constant is an immediate consequence of this result. Further, we discuss the properties of the hitting time quasi-metric $m$ on the set of vertices of $\Gamma$: $m(i,j)=\hat m_{ij}$, $i\neq j$, and $m(i,i)=0$. We also consider a number of other metric structures on the set of graph vertices related to the hitting time quasi-metric $m$---along with various connections between them. The notions and relationships under study are illustrated by two examples.

Comparative Construction of Plug-in Estimators of the Entropy Rate of Two-state Markov Chains

The entropy rate of an ergodic homogeneous Markov chain taking only two values is an explicit function of its transition probabilities. We study plug-in estimators of this entropy rate based on the observation of either one trajectory with long length or several trajectories with fixed length. Their exact asymptotic distributions are given. A detailed numerical study using simulation results is provided. Real data of rainfall occurrences are studied.

Estimation of the Entropy Rate of a Countable Markov Chain

We consider here ergodic homogeneous Markov chains with countable state spaces. The entropy rate of the chain is an explicit function of its transition and stationary distributions. We construct estimators for this entropy rate and for the entropy of the stationary distribution of the chain, in the parametric and nonparametric cases. We study estimation from one sample with long length and from many independent samples with given length. In the parametric case, the estimators are deduced by plug-in from the maximum likelihood estimator of the parameter. In the nonparametric case, the estimators are deduced by plug-in from the empirical estimators of the transition and stationary distributions. They are proven to have good asymptotic properties.

On the stability of the computation of the stationary probabilities of Markov chains using Perron complements

AbstractFor an n‐state ergodic homogeneous Markov chain whose transition matrix is T∈ℝn,n, it has been shown by Meyer on the one hand and by Kirkland, Neumann, and Xu on the other hand that the stationary distribution vector and that the mean first passage matrix, respectively, can be computed by a divide and conquer parallel method from the Perron complements of T. This is possible due to the facts, shown by Meyer, that the Perron complements of T are themselves transition matrices for finite ergodic homogeneous Markov chains with fewer states and that their stationary distribution vectors are multiples of the corresponding subvectors of the stationary distribution vector of the entire chain.Here we examine various questions concerning the stability of computing the stationary distribution vectors of the Perron complements and compare them with the stability of computing the stationary distribution vector for the entire chain. In particular, we obtain that condition numbers which are related to the coefficients of ergodicity improve as we pass from the entire chain to the chains associated with its Perron complements. Copyright © 2003 John Wiley & Sons, Ltd.

Ergodic Markov chains with finite convergence time

A necessary and sufficient condition for a finite ergodic homogeneous Markov chain to converge to the stationary distribution in a finite number of steps is given. This generalizes a recent result of Brosh and Gerchak, who consider only the irreducible case.