Stationary Distribution Vector Research Articles

Fixed points of doubly stochastic operators with application to Markov chains

In this work, we characterize the fixed points of the doubly stochastic operators defined on discrete spaces ℓ p ( I ) , where I is a nonempty set and p ∈ [ 1 , + ∞ ) . We use our results to characterize all Markov chains with infinite state space and doubly stochastic transition probability matrix that have at least one stationary distribution vector.

Oriented spanning trees and stationary distribution of digraphs

By using biclique partitions of digraphs, this paper gives reduction formulas for the number of oriented spanning trees, stationary distribution vector and Kemeny's constant of digraphs. As applications, we give a method for enumerating spanning trees of undirected graphs by vertex degrees and biclique partitions. The biclique partition formula also extends the results of Knuth and Levine from line digraphs to general digraphs.

APPLICATION OF POWER NUMERICAL METHOD FOR THE STATIONARY DISTRIBUTION OF MARKOV CHAIN

The evolution of this model is represented by transitions from one state to the next. Also, the physical or mathematical behavior of this system can also be illustrated by identifying all of the possible states and explaining how it transitions between them. The iterative solution approaches for the stationary distribution of Markov chains, which begin with an initial estimate of the solution vector and it becomes closer and closer to the true solution with each iteration are investigated. Our goal is to compute solutions of stationary distribution of Markov chain by utilizing the power iterative method which leaves the transition matrices unchanged and saves time by considering the discretization effect, and the convergency. Matrices operations such as multiplication with one or more vectors, lower, diagonal and upper concepts of matrix, with the help of several existing Markov chain laws, theorems, formulas, and the normalization principle are applied. For the illustrative examples, the stationary distribution vectors and table of convergence are obtained.

Application of Jacobi and Gauss–Seidel Numerical Iterative Solution Methods for the Stationary Distribution of Markov Chain

The Physical or Mathematical behaviour of this model may be represented by describing all the different states it may occupy and by indicating how it moves among these states. In this study, the stationary distribution of Markov chains was solved using iterative methods that begin with an initial estimate of the solution vector and then modified it in a way that brings it closer and closer to the real solution with each step or iteration. These methods also involved matrix operations like multiplication with one or more vectors, which preserves the transition matrices while speeding up the process. We computed the solutions using Jacobi iterative method and Gauss-Seidel iterative method in order to shed more light on the solutions of stationary distribution in Markov chain. This was done with the aid of several already-existing laws, theorems, and formulas of Markov chain and the application of normalization principle and matrix operations such as lower, upper, and diagonal matrices. The stationary distribution vector’s 𝜋𝑖,𝑖=1,2,…,4 are obtained for the illustrative example one as 𝜋(3) = (0.078125,0.109375,0.21875,0.59375) as well as the four eigenvalues of the matrix as 𝜆1=1.0, 𝜆2=−0.7718, 𝜆3,4=−0.1141±0.5576𝑖 using Jacobi iterative technique, and for illustrative example two using Gauss-Siedel method as 𝝅(𝟑) = (0.090909, 0.181818, 0.363636, 0.363636). The research shown that Gauss Siedel method converged faster than Jacobi method.

Concentration optimization of combinatorial drugs using Markov chain-based models

BackgroundCombinatorial drug therapy for complex diseases, such as HSV infection and cancers, has a more significant efficacy than single-drug treatment. However, one key challenge is how to effectively and efficiently determine the optimal concentrations of combinatorial drugs because the number of drug combinations increases exponentially with the types of drugs.ResultsIn this study, a searching method based on Markov chain is presented to optimize the combinatorial drug concentrations. In this method, the searching process of the optimal drug concentrations is converted into a Markov chain process with state variables representing all possible combinations of discretized drug concentrations. The transition probability matrix is updated by comparing the drug responses of the adjacent states in the network of the Markov chain and the drug concentration optimization is turned to seek the state with maximum value in the stationary distribution vector. Its performance is compared with five stochastic optimization algorithms as benchmark methods by simulation and biological experiments. Both simulation results and experimental data demonstrate that the Markov chain-based approach is more reliable and efficient in seeking global optimum than the benchmark algorithms. Furthermore, the Markov chain-based approach allows parallel implementation of all drug testing experiments, and largely reduces the times in the biological experiments.ConclusionThis article provides a versatile method for combinatorial drug screening, which is of great significance for clinical drug combination therapy.

Directed forests and the constancy of Kemeny’s constant

Consider a discrete–time, time–homogeneous Markov chain on states $$1, \ldots , n$$ whose transition matrix is irreducible. Denote the mean first passage times by $$m_{jk}, j,k=1,\ldots , n,$$ and stationary distribution vector entries by $$v_k, k=1, \ldots , n$$ . A result of Kemeny reveals that the quantity $$\sum _{k=1}^n m_{jk}v_k$$ , which is the expected number of steps needed to arrive at a randomly chosen destination state starting from state j, is–surprisingly–independent of the initial state j. In this note, we consider $$\sum _{k=1}^n m_{jk}v_k$$ from the perspective of algebraic combinatorics and provide an intuitive explanation for its independence on the initial state j. The all minors matrix tree theorem is the key tool employed.

VOS: A new outlier detection model using virtual graph

Outlier detection has been well studied due to its wide applications in both academia and industry, among which graph-based methods have drawn extensive attention in recent years because of their robust expressiveness for various types of datasets. Combining the local information with the implicit connections in the graph representation of the original dataset, in this study, we propose a new outlier detection model named Virtual Outlier Score (VOS). The proposed model constructs a similarity graph by using the top-k similar neighbors of each object, and introduces a virtual node coupling with a collection of virtual edges to generate a k-virtual graph. A tailored Markov random walk process is then performed on the strongly connected virtual graph under the principle that a potential outlier should get more weight to be visited. After reaching equilibrium, the stationary distribution vector is utilized to deduce the virtual outlier score. Furthermore, we provide a theoretical analysis of the proposed VOS model. Experiments on both synthetic and real-world datasets showed that the proposed model obtains an improvement over eight state-of-the-art algorithms under the measures of ROC AUC as well as the outlier discovery curve.

Correction to: A sequential update algorithm for computing the stationary distribution vector in upper block-Hessenberg Markov chains

The original version of this article unfortunately contained the following errors: In Introduction, below Eq. (1.3), the sentence should read “From (1.2) and (1.3), we have” instead of “From (1.2) and (1.6), we have”.

A sequential update algorithm for computing the stationary distribution vector in upper block-Hessenberg Markov chains

This paper proposes a new algorithm for computing the stationary distribution vector in continuous-time upper block-Hessenberg Markov chains. To this end, we consider the last-block-column-linearly-augmented (LBCL-augmented) truncation of the (infinitesimal) generator of the upper block-Hessenberg Markov chain. The LBCL-augmented truncation is a linearly-augmented truncation such that the augmentation distribution has its probability mass only on the last block column. We first derive an upper bound for the total variation distance between the respective stationary distribution vectors of the original generator and its LBCL-augmented truncation. Based on the upper bound, we then establish a series of linear fractional programming (LFP) problems to obtain augmentation distribution vectors such that the bound converges to zero. Using the optimal solutions of the LFP problems, we construct a matrix-infinite-product (MIP) form of the original (i.e., not approximate) stationary distribution vector and develop a sequential update algorithm for computing the MIP form. Finally, we demonstrate the applicability of our algorithm to BMAP/M/$\infty$ queues and M/M/$s$ retrial queues.

A Reinforcement Learning Approach for Solving the Mean Variance Customer Portfolio in Partially Observable Models

In problems involving control of financial processes, it is usually complicated to quantify exactly the state variables. It could be expensive to acquire the exact value of a given state, even if it may be physically possible to do so. In such cases it may be interesting to support the decision-making process on inaccurate information pertaining to the system state. In addition, for modeling real-world application, it is necessary to compute the values of the parameters of the environment (transition probabilities and observation probabilities) and the reward functions, which are typically, hand-tuned by experts in the field until it has acquired a satisfactory value. This results in an undesired process. To address these shortcomings, this paper provides a new Reinforcement Learning (RL) framework for computing the mean-variance customer portfolio with transaction costs in controllable Partially Observable Markov Decision Processes (POMDPs). The solution is restricted to finite state, action, observation sets and average reward problems. For solving this problem, a controller/actor-critic architecture is proposed, which balance the difficult tasks of exploitation and exploration of the environment. The architecture consists of three modules: controller, fast-tracked portfolio learning and an actor-critic module. Each module involves the design of a convergent Temporal Difference (TD) learning algorithm. We employ three different learning rules to estimate the real values of: (a) the transition matrices [Formula: see text], (b) the rewards [Formula: see text] and (c) the resources destined for carrying out a promotion [Formula: see text]. We present a proof for the estimated transition matrix rule [Formula: see text] and showing that it converges when t → ∞. For solving the optimization programming problem we extend the c-variable method for partially observable Markov chains. The c-variable is conceptualized as joint strategy given by the product of the control policy, the observation kernel Q(y|s) and the stationary distribution vector. A major advantage of this procedure is that it can be implemented efficiently for real settings in controllable POMDP. A numerical example illustrates the results of the proposed method.

Minimising the largest mean first passage time of a Markov chain: The influence of directed graphs

For a Markov chain described by an irreducible stochastic matrix A of order n, the mean first passage time mi,j measures the expected time for the Markov chain to reach state j for the first time given that the system begins in state i, thus quantifying the short-term behaviour of the chain. In this article, a lower bound for the maximum mean first passage time is found in terms of the stationary distribution vector of A, and some matrices for which equality is attained are produced. The main objective of this article is to characterise the directed graphs for which any stochastic matrix A respecting this directed graph attains equality in this lower bound, producing a class of Markov chains with optimal short-term behaviour.

ERROR BOUNDS FOR LAST-COLUMN-BLOCK-AUGMENTED TRUNCATIONS OF BLOCK-STRUCTURED MARKOV CHAINS

This paper discusses the error estimation of the last-column-block-augmented northwest-corner truncation (LC-block-augmented truncation, for short) of block-structured Markov chains (BSMCs) in continuous time. We first derive upper bounds for the absolute difference between the time-averaged functionals of a BSMC and its LC-block-augmented truncation, under the assumption that the BSMC satisfies the general $f$-modulated drift condition. We then establish computable bounds for a special case where the BSMC is exponentially ergodic. To derive such computable bounds for the general case, we propose a method that reduces BSMCs to be exponentially ergodic. We also apply the obtained bounds to level-dependent quasi-birth-and-death processes (LD-QBDs), and discuss the properties of the bounds through the numerical results on an M/M/$s$ retrial queue, which is a representative example of LD-QBDs. Finally, we present computable perturbation bounds for the stationary distribution vectors of BSMCs.

Continuous-time block-monotone Markov chains and their block-augmented truncations

This paper considers continuous-time block-monotone Markov chains (BMMCs) and their block-augmented truncations. We first introduce the block monotonicity and block-wise dominance relation for continuous-time Markov chains, and then provide some fundamental results on the two notions. Using these results, we show that the stationary distribution vectors obtained by the block-augmented truncation converge to the stationary distribution vector of the original BMMC. We also show that the last-column-block-augmented truncation (LC-block-augmented truncation) provides the best (in a certain sense) approximation to the stationary distribution vector of a BMMC among all the block-augmented truncations. Furthermore, we present computable upper bounds for the total variation distance between the stationary distribution vectors of a Markov chain and its LC-block-augmented truncation, under the assumption that the original Markov chain itself may not be block-monotone but is block-wise dominated by a BMMC with exponential ergodicity. Finally, we apply the obtained bounds to a queue with a batch Markovian arrival process and state-dependent departure rates.

Fairness Analysis of Routing in Opportunistic Mobile Networks

Multicopy utility-based forwarding algorithms are popular in opportunistic mobile networks. They aim to gain high system throughput while keeping the cost low. However, most of them ignore the fairness issue on the successful delivery rate among users. In this paper, we analyze the fairness evaluation of the success rate distribution, and we propose a new fair packet-forwarding strategy based on packet priority. We formulate the opportunistic packet-forwarding process as a discrete-time Markov chain and deduce a stationary probability distribution vector. Instead of taking the hill-climbing heuristic on utility comparison, we introduce a lower utility tolerance mechanism for the decision-making process of each node, and we theoretically demonstrate that the proposed mechanism may be used to control the success rate of packet delivery by changing a simple parameter. In addition, we adopt a message-duplication restricting mechanism to adjust the number of replications based on packet priority. Our proposed protocol can work as a plug-in for traditional utility-based forwarding algorithms. The performance is compared with several well-known opportunistic routing protocols via both a synthetic model and real human mobility traces. The results show that our protocol improves the fairness in successful delivery rates among users while maintaining almost the same system throughput. Moreover, it reduces the cost of traditional utility-based forwarding algorithms, rendering our proposed scheme not only fair but efficient as well.

Load balancing for Markov chains with a specified directed graph

Given a strongly directed graph , let be the set of stochastic matrices whose directed graph is a spanning subgraph of . We consider the problem of finding the infimum of as ranges over the set of stationary distribution vectors of irreducible matrices in Using techniques from non-linear programming, combinatorial matrix theory and non-negative matrix theory, we find this infimum, which is given in terms the cardinality of a certain collection of vertex–disjoint cycles in . The situation that the infimum is attained as a minimum for the stationary distribution vector of some irreducible matrix in is also considered.

On the Kemeny constant and stationary distribution vector for a Markov chain

Suppose that A is an irreducible stochastic matrix of order n, and denote its eigenvalues by 1, λ2, . . . , λn. The Kemeny constant, K(A) for the Markov chain associated with A is defined as K(A) = ∑n j=2 1 1−λj , and can be interpreted as the mean first passage from an unknown initial state to an unknown destination state in the Markov chain. Let w denote the stationary distribution vector for A, and suppose that w1 ≤ w2 ≤ · · · ≤ wn. In this paper, we show that K(A) ≥ ∑n j=1(j − 1)wj , and we characterise the matrices yielding equality in that bound. The results are established using techniques from matrix theory and the theory of directed graphs.

The minimum coefficient of ergodicity for a Markov chain with a given directed graph

Suppose that T is an n×n stochastic matrix, and denote its directed graph by D(T). The function τ(T)=12maxi,j=1,…,n{‖(ei−ej)⊤T‖1} is known as a coefficient of ergodicity for T, and measures the rate at which the iterates of a Markov chain with transition matrix T converge to the stationary distribution vector. Many Markov chains are equipped with an underlying combinatorial structure that is described by a directed graph, and in view of that fact, we consider the following problem: given a directed graph D, find τmin(D)≡minτ(T), where the minimum is taken over all stochastic matrices T such that D(T) is a spanning subgraph of D.In this paper, we describe τmin(D) as the solution to a linear programming problem. We then go on to provide an upper bound on τmin(D) in terms of the maximum outdegree of D, and a lower bound on τmin(D) in terms of the maximum indegree of D, characterising the equality cases in both bounds. Connections are established between the equality case in the lower bound and certain balanced incomplete block designs.

On Perturbation Bounds for the Joint Stationary Distribution of Multivariate Markov Chain Models

AbstractMultivariate Markov chain models have previously been proposed in for studying dependent multiple categorical data sequences. For a given multivariate Markov chain model, an important problem is to study its joint stationary distribution. In this paper, we use two techniques to present some perturbation bounds for the joint stationary distribution vector of a multivariate Markov chain with s categorical sequences. Numerical examples demonstrate the stability of the model and the effectiveness of our perturbation bounds.

Fourier Analysis of the Aggregation Based Algebraic Multigrid for Stochastic Matrices

We introduce new theoretical results on the convergence of the algebraic multigrid methods for obtaining stationary probability distribution vectors of stochastic matrices. We focus on sparse and nonsymmetric stochastic matrices. Our approach is based on the Fourier transform of the error propagation operator. For some special classes of stochastic matrices it allows one to find the optimal parameters of the algorithm and to estimate the rate of convergence. Examples related to the computation of stable gene profiles are presented.

Dynamic spectrum allocation with admission control based on cognitive radio for QoS support in multiple wireless network

Cognitive radio is an emerging technology in wireless access, aimed at vastly improving the way radio spectrum is utilized. In this article, we investigate the performance improvement gained by applying cognitive radio to a multiple wireless service providers (WSPs). We consider several WSPs and two types of users: primary (licensed) and secondary (unlicensed) users. Two different schemes are proposed for unlicensed users to manage their handoff. These two schemes are different in the sense that one allows to each WSP to give more priority to his own unlicensed users by dropping low priority users, while the other does not allow that. The system is modeled by a Markov process, with continuous time and finite state space. Due to the very large number of states, we propose a robust method to approximate the stationary probability distribution vector of Markov process. Based on this approximation, we develop several performance metrics, blocking and dropping probabilities for both kinds of unlicensed users. Numerical results show that this approximation is very close to the exact solution. Finally, we show that spectrum utilization of cognitive users increases with the proposed schemes especially in medium and high traffic.