Time-homogeneous Markov Chain Research Articles

Practical Modeling and Analysis of Blockchain Radio Access Network

The continually rising demand for wireless services and applications in the era of Internet of things (IoT) and artificial intelligence (AI) presents a significant number of unprecedented challenges to existing network structures. To meet the rapid growth need of mobile data services, blockchain radio access network (B-RAN) has emerged as a decentralized, trustworthy radio access paradigm spurred by blockchain technologies. However, many characteristics of B-RAN remain unclear and hard to characterize. In this study, we develop an analytical framework to model B-RAN and provide some basic fundamental analysis. Starting from block generation, we establish a queuing model based on a time-homogeneous Markov chain. From the queuing model, we evaluate the performance of B-RAN with respect to latency and security considerations. We uncover a more comprehensive picture of the achievable performance of B-RAN by connecting latency and security. At last, we present experimental results via an innovative prototype and validate the proposed model.

Wiener-Hopf factorization technique for time-inhomogeneous finite Markov chains

ABSTRACT This work contributes to the theory of Wiener-Hopf type factorization for finite Markov chains. This theory originated in the seminal paper [Barlow et al., Wiener-Hopf factorization for matrices, Séminaire de Probabilités de Strasbourg 14 (1980), pp. 324–331], which treated the case of finite time-homogeneous Markov chains. Since then, several works extended the results of [Barlow et al., Wiener-Hopf factorization for matrices, Séminaire de Probabilités de Strasbourg 14 (1980), pp. 324–331] in many directions. However, all these extensions were dealing with time-homogeneous Markov case. The first work dealing with the time-inhomogeneous situation was [Bielecki et al., Wiener-Hopf factorization for time-inhomogeneous Markov chains and its application, Probab. Math. Stat. 2019, forthcoming], where Wiener-Hopf type factorization for time-inhomogeneous finite Markov chain with piecewise constant generator matrix function was derived. In the present paper we go further: we derive and study Wiener-Hopf type factorization for time-inhomogeneous finite Markov chain with the generator matrix function being a fairly general matrix valued function of time.

Embedded Markov chain approximations in Skorokhod topologies

We prove a J1-tightness condition for embedded Markov chains and discuss four Skorokhod topologies in a unified manner. To approximate a continuous time stochastic process by discrete time Markov chains, one has several options to embed the Markov chains into continuous time processes. On the one hand, there is a Markov embedding which uses exponential waiting times. On the other hand, each Skorokhod topology naturally suggests a certain embedding. These are the step function embedding for J1, the linear interpolation embedding forM1, the multistep embedding for J2 and a more general embedding for M2. We show that the convergence of the step function embedding in J1 implies the convergence of the other embeddings in the corresponding topologies. For the converse statement, a J1-tightness condition for embedded time-homogeneous Markov chains is given.Additionally, it is shown that J1 convergence is equivalent to the joint convergence in M1 and J2.

Directed forests and the constancy of Kemeny’s constant

Consider a discrete–time, time–homogeneous Markov chain on states $$1, \ldots , n$$ whose transition matrix is irreducible. Denote the mean first passage times by $$m_{jk}, j,k=1,\ldots , n,$$ and stationary distribution vector entries by $$v_k, k=1, \ldots , n$$ . A result of Kemeny reveals that the quantity $$\sum _{k=1}^n m_{jk}v_k$$ , which is the expected number of steps needed to arrive at a randomly chosen destination state starting from state j, is–surprisingly–independent of the initial state j. In this note, we consider $$\sum _{k=1}^n m_{jk}v_k$$ from the perspective of algebraic combinatorics and provide an intuitive explanation for its independence on the initial state j. The all minors matrix tree theorem is the key tool employed.

PNS337 IMPROVING ON CYCLE CORRECTIONS FOR TIME-HOMOGENEOUS MARKOV MODELS

Markov models, which specify transition probabilities between discrete health states at each time step, are widely used within health economic modelling. Decision problems are usually assessed by attaching utilities and costs to states and summing the total utility and cost of decision options over some time-frame. Frequently, a correction method (e.g. half-cycle correction (HCC)) is applied to naïve discrete-time outputs to yield a closer approximation to an underlying continuous-time Markov chain. In this study, we note that an underlying continuous-time Markov chain that corresponds to the proposed discrete time analogue may not exist, question the rationale for corrections in this case, and introduce a novel approximation method based on Gaussian Quadrature (GQ). We considered a simple model with three health states – well, unwell and dead. We exploited analytical results for time-homogeneous Markov chains in terms of matrix exponentials and inverses to compactly express calculations, and to introduce a new n-order GQ-based numerical integration method, which is applied to naïve discrete-time output. An n-order GQ method approximates the continuous-time Markov chain result by a weighted sum of function values at specified points within the range of integration and yields exact values for polynomials of degree up to 2n – 1. We conducted a simulation study to compare the GQ methodology and other existing cycle correction methods (HCC, trapezoidal, and Simpson 1/3 and 3/8 methods), to the exact continuous-time process outcomes (gold standard). At first order, we found that the GQ method replicated HCC. In our simulation study, the third-order GQ method outperformed other existing methods (HCC, trapezoidal and Simpson 1/3 and 3/8) in approximating the gold standard results. Simpson 1/3 turned out to be the second-best method. The third-order GQ method, applied to naïve discrete-time output, can outperform other cycle correction methods for homogeneous models. Defining models in continuous-time avoids potential inconsistencies.

QML Estimation of Asymmetric Markov Switching GARCH(p, q) Processes

In this paper, I propose a natural extension of time-invariant coefficients threshold GARCH (TGARCH) processes to time-varying one, in which the associated volatility switch between different regimes due to dependency of its coefficients on unobservable (latent) time homogeneous Markov chain with finite state space (MS-TGARCH). These models are showed to be capable to capture some phenomena observed for most financial time series, among others, the asymmetric patterns, leverage effects, dependency without correlation and tail heaviness. So, some theoretical probabilistic properties of such models are discussed, in particular, we establish firstly necessary and sufficient conditions ensuring the strict stationarity and ergodicity for solution process of MS-TGARCH. Secondary, we extend the standard results for the limit theory of the popular quasi-maximum likelihood estimator (QMLE) for estimating the unknown parameters involved in model and we examine thus the strong consistency of such estimates. The finite-sample properties of QMLE are illustrated by a Monte Carlo study. Our proposed model is applied to model the exchange rates of the Algerian Dinar against the single European currency (Euro).

Performance evaluation of deterministic flow lines considering multiclass customer and random setups

Inspired by flexible manufacturing systems with setups, in this study, we investigate the steady-state behavior of flow line models with deterministic service durations. In such systems, different types of products with different batch sizes are produced in a single production line, and a setup is required when switching from one type of product to another. In this study, we aim at analyzing the effect of different setups on a system’s performance metrics, i.e., the maximum production rate and delay distribution of a product experienced inside the system. The maximum production rate of a deterministic flow line is the reciprocal of its bottleneck process time; however, the different setups existing in a system complicate the analysis. Furthermore, setups are necessary in flexible manufacturing systems. The systems that are the subject of our model are steel pipe manufacturing process and semiconductor wafer fabrications. Additionally, we obtain and analyze a two-dimensional, discrete-time, time-homogeneous Markov chain model to derive the system’s performance measures.

On the rate of convergence for the length of the longest common subsequences in hidden Markov models

AbstractLet (X, Y) = (Xn, Yn)n≥1 be the output process generated by a hidden chain Z = (Zn)n≥1, where Z is a finite-state, aperiodic, time homogeneous, and irreducible Markov chain. Let LCn be the length of the longest common subsequences of X1,..., Xn and Y1,..., Yn. Under a mixing hypothesis, a rate of convergence result is obtained for E[LCn]/n.

Estimating random walk centrality in networks

Random walk centrality (equivalently, the accessibility index) for the states of a time-homogeneous irreducible Markov chain on a finite state space is considered. It is known that the accessibility index for a particular state can be written in terms of the first and second moments of the first return time to that state. Based on that observation, the problem of estimating the random walk centrality of a state is approached by taking realizations of the Markov chain, and then statistically estimating the first two moments of the corresponding first return time. In addition to the estimate of the random walk centrality, this method also yields the standard error, the bias and a confidence interval for that estimate. For the case that the directed graph of the transition matrix for the Markov chain has a cut-point, an alternate strategy for computing the random walk centrality is outlined that may be of use when the centrality values are of interest for only some of the states. In order to illustrate the effectiveness of the results, estimates of the random walk centrality arising from random walks for several directed and undirected graphs are discussed.

Estimating the Entropy Rate of Finite Markov Chains With Application to Behavior Studies

Predictability of behavior is an important characteristic in many fields including biology, medicine, marketing, and education. When a sequence of actions performed by an individual can be modeled as a stationary time-homogeneous Markov chain the predictability of the individual’s behavior can be quantified by the entropy rate of the process. This article compares three estimators of the entropy rate of finite Markov processes. The first two methods directly estimate the entropy rate through estimates of the transition matrix and stationary distribution of the process. The third method is related to the sliding-window Lempel–Ziv compression algorithm. The methods are compared via a simulation study and in the context of a study of interactions between mothers and their children.

The Exit Time Finite State Projection Scheme: Bounding Exit Distributions and Occupation Measures of Continuous-Time Markov Chains

We introduce the exit time finite state projection (ETFSP) scheme, a truncation-based method that yields approximations to the exit distribution and occupation measure associated with the time of exit from a domain (i.e., the time of first passage to the complement of the domain) of time-homogeneous continuous-time Markov chains. We prove that: (i) the computed approximations bound the measures from below; (ii) the total variation distances between the approximations and the measures decrease monotonically as states are added to the truncation; and (iii) the scheme converges, in the sense that, as the truncation tends to the entire state space, the total variation distances tend to zero. Furthermore, we give a computable bound on the total variation distance between the exit distribution and its approximation, and we delineate the cases in which the bound is sharp. We also revisit the related finite state projection scheme and give a comprehensive account of its theoretical properties. We demonstrate the use of the ETFSP scheme by applying it to two biological examples: the computation of the first passage time associated with the expression of a gene, and the fixation times of competing species subject to demographic noise.

Thermodynamic formalism for topological Markov chains on standard Borel spaces

We develop a Thermodynamic Formalism for bounded continuous potentials defined on the sequence space $X\equiv E^{\mathbb{N}}$, where $E$ is a general Borel standard space. In particular, we introduce meaningful concepts of entropy and pressure for shifts acting on $X$ and obtain the existence of equilibrium states as additive probability measures for any bounded continuous potential. Furthermore, we establish convexity and other structural properties of the set of equilibrium states, prove a version of the Perron-Frobenius-Ruelle theorem under additional assumptions on the regularity of the potential and show that the Yosida-Hewitt decomposition of these equilibrium states do not have a purely additive part. We then apply our results to the construction of invariant measures of time-homogeneous Markov chains taking values on a general Borel standard space and obtain exponential asymptotic stability for a class of Markov operators. We also construct conformal measures for an infinite collection of interacting random paths which are associated to a potential depending on infinitely many coordinates. Under an additional differentiability hypothesis, we show how this process is related after a proper scaling limit to a certain infinite dimensional diffusion.

Production Control with Price, Cost, and Demand Uncertainty

An optimal production flow control problem of a make-to-stock manufacturing firm with price, cost, and demand uncertainty is studied. The objective of the flow rate control problem is maximizing the average profit that is the difference between the expected revenue and the expected production, inventory holding, and backlog costs. The uncertainties in the system are captured jointly in discrete environment states. In each environment state, the price, cost, and demand take different levels. The transitions between different environment states evolve according to a time-homogenous Markov chain. By using a continuous flow model, the optimal production policy is stated as a state-dependent hedging policy. The performance of the system where the production cost alternates between a high and a low cost level and the demand is either constant or also alternates between a high and a low level is analyzed under the double-hedging policy. According to this policy, the producer produces only when the cost is low and the surplus is between the two hedging levels. However when the backlog is below the lower hedging level, the producer produces with the maximum capacity regardless of the cost. The effects of production cost, production capacity, demand variability, and the dependence of the demand and the cost on the performance of the system are analyzed analytically and numerically. It is shown that controlling the production rate optimally allows producers respond to the fluctuations in price, cost, and demand in an effective way and maximize their profits.

State Estimation Over Delayed Mutihop Network

In this paper, we consider the properties for Kalman filtering in a multi-hop network. A time-homogeneous Markov chain is used to characterize the stochastic packet delay and loss phenomenon of each relay. Since, after several relays, the average expected estimation error covariance at the terminal is difficult to calculate, we derive lower and upper bounds for the average value. Using these bounds, it is found that the stability is related to one special entry in the channel's transition probability matrix, from which we obtain a necessary and sufficient condition for the stability of the single-hop case and extend the result to the multihop one.

Clustering behaviour in Markov chains with eigenvalues close to one

Finite, discrete, time-homogeneous Markov chains are frequently used as a simple mathematical model of real-world dynamical systems. In many such applications, an analysis of clustering behaviour in the states of the system is desirable, and it is well-known that the eigendecomposition of the transition matrix A of the Markov chain can provide such insight. Clustering methods based on the sign pattern in the second eigenvector of A are frequently used when A has dominant eigenvalues that are real. In this paper, we present a method to include an analysis for complex eigenvalues of A which are close to 1. Since a real spectrum is not guaranteed in most applications, this is a valuable result in the area of spectral clustering in Markov chains.

Spatiotemporal analysis of strain localization in dense granular materials

Predicting localized failure in granular materials is a problem of great interest from both the scientific and technological perspectives. The initiation and growth of strain localization have been studied using laboratory experiments and particle-based numerical simulations, and their findings have been preliminarily implemented into continuum constitutive models. In this study, we revisit strain localization in granular materials using the spatiotemporal data analysis technique, which has been extensively employed in data mining science and social science for the sake of different applications. A dense packing of granular material subjected to biaxial compression was simulated using the combined finite and discrete element method. The large amount of particle-level kinematical data, including the translational and rotational particle motion, fluctuating velocity, granular temperature, and local strain, are collected for subsequent spatiotemporal data analysis. The spatiotemporal data analysis provides a new perspective on the strain localization in dense granular materials and a rich body of new insights are presented for the first time. The spatial autocorrelation analysis results in Moran’s I values close to 1, and positive Z scores and statistically significant p values, which indicate that a dense granular system features clustered patterns during shearing. This finding proves yet again that dense granular materials have an inherent short range order. Eventually, correspondence between localized modes with different particle kinematics and spatial distributions of local Moran statistics and quadrant location map are investigated. By assuming shearing of dense granular materials is a first-order Markov chain process, the convergence and time homogeneity of this process are analyzed. Both the maximum likelihood and Pearson test statistics clearly demonstrate that shearing of dense granular materials is a time-homogenous Markov chain process.

M/M/1 queue in two alternating environments and its heavy traffic approximation

We investigate an M/M/1 queue operating in two switching environments, where the switch is governed by a two-state time-homogeneous Markov chain. This model allows to describe a system that is subject to regular operating phases alternating with anomalous working phases or random repairing periods. We first obtain the steady-state distribution of the process in terms of a generalized mixture of two geometric distributions. In the special case when only one kind of switch is allowed, we analyze the transient distribution, and investigate the busy period problem. The analysis is also performed by means of a suitable heavy-traffic approximation which leads to a continuous random process. Its distribution satisfies a partial differential equation with randomly alternating infinitesimal moments. For the approximating process we determine the steady-state distribution, the transient distribution and a first-passage-time density.

Observer-Based Output Feedback MPC for T-S Fuzzy System With Data Loss and Bounded Disturbance.

This paper investigates the output feedback model predictive control (OFMPC) for Takagi-Sugeno fuzzy networked control systems with bounded disturbance, where data quantization and data loss occur simultaneously. The quantization error is treated as sector bound uncertainties by using the sector bound approach and the data loss process is modeled as a time-homogeneous Markov chain. Invoking S -procedure and the notion of quadratic boundedness which can specify closed-loop stability for system with disturbance, the state observer is offline designed and the networked output feedback model predictive controller is provided which explicitly considers the satisfaction of input constraints. Two online synthesis algorithms of OFMPC are presented, one parameterizing the infinite horizon control moves into a single feedback law, the other into one free control move followed by the single feedback law based on the state observer. A new formula is introduced to refresh the ellipsoidal bound of estimation error which can guarantee the recursive feasibility of optimization problem. An example is given to demonstrate the effectiveness of the proposed new design techniques.

Synchronization of Nonlinearly and Stochastically Coupled Markovian Switching Networks via Event-Triggered Sampling.

This paper studies the exponential synchronization problem for a new array of nonlinearly and stochastically coupled networks via events-triggered sampling (ETS) by self-adaptive learning. The networks include the following features: 1) a Bernoulli stochastic variable is introduced to describe the random structural coupling; 2) a stochastic variable with positive mean is used to model the coupling strength; and 3) a continuous time homogeneous Markov chain is employed to characterize the dynamical switching of the coupling structure and pinned node sets. The proposed network model is capable to capture various stochastic effect of an external environment during the network operations. In order to reduce networks' workload, different ETS strategies for network self-adaptive learning are proposed under continuous and discrete monitoring, respectively. Based on these ETS approaches, several sufficient conditions for synchronization are derived by employing stochastic Lyapunov-Krasovskii functions, the properties of stochastic processes, and some linear matrix inequalities. Numerical simulations are provided to demonstrate the effectiveness of the theoretical results and the superiority of the proposed ETS approach.

Infinite Horizon Optimal Transmission Power Control for Remote State Estimation Over Fading Channels

This paper studies the joint design over an infinite horizon of the transmission power controller and remote estimator for state estimation over fading channels. A sensor observes a dynamic process and sends its observations to a remote estimator over a wireless fading channel characterized by a time-homogeneous Markov chain. The successful transmission probability depends on both the channel gains and the transmission power used by the sensor. The transmission power control rule and the remote estimator should be jointly designed, aiming to minimize an infinite-horizon cost consisting of the power usage and the remote estimation error. We formulate the joint optimization problem as an average cost belief-state Markov decision process and prove that there exists an optimal deterministic and stationary policy. We then show that when the monitored dynamic process is scalar or the system matrix is orthogonal, the optimal remote estimates depend only on the most recently received sensor observation, and the optimal transmission power is symmetric and monotonically increasing with respect to the norm of the innovation error.