Infinite Markov Chains Research Articles

Bayesian tensor factorisations for time series of counts

We propose a flexible nonparametric Bayesian modelling framework for multivariate time series of count data based on tensor factorisations. Our models can be viewed as infinite state space Markov chains of known maximal order with non-linear serial dependence through the introduction of appropriate latent variables. Alternatively, our models can be viewed as Bayesian hierarchical models with conditionally independent Poisson distributed observations. Inference about the important lags and their complex interactions is achieved via MCMC. When the observed counts are large, we deal with the resulting computational complexity of Bayesian inference via a two-step inferential strategy based on an initial analysis of a training set of the data. Our methodology is illustrated using simulation experiments and analysis of real-world data.

Oscillatory banded Hessenberg matrices, multiple orthogonal polynomials and Markov chains

A spectral Favard theorem is derived for bounded banded lower Hessenberg matrices that possess a positive bidiagonal factorization. The rich knowledge concerning the spectral and factorization properties of oscillatory matrices forms the basis of this theorem, which is formulated in terms of sequences of multiple orthogonal polynomials of types I and II, associated with a set of positive Lebesgue-Stieltjes measures. Additionally, a multiple Gauss quadrature is established, and the corresponding degrees of precision are determined. The spectral Favard theorem finds application in the context of Markov chains with transition matrices having (p + 2) diagonals, extending beyond the birth and death scenario, while still maintaining a positive stochastic bidiagonal factorization. In the finite case, the Karlin–McGregor spectral representation is provided, along with the demonstration of recurrent behavior and explicit expressions for the stationary distributions in terms of the orthogonal polynomials. Analogous results are obtained for countably infinite Markov chains. In this case, the Markov chain may not be recurrent, and its characterization is expressed in relation to the first measure. The ergodicity of the Markov chain is explored, taking into consideration the presence of a mass at 1, which corresponds to eigenvalues associated with the right and left eigenvectors.

Infinite Markov pooling of predictive distributions

This paper introduces novel approaches to forecast pooling methods based on a nonparametric prior for a weight vector combining predictive densities. The first approach places a Dirichlet process prior on the weight vector and generalizes the static linear pool. The second approach uses a hierarchical Dirichlet process prior to allow the weight vector to follow an infinite hidden Markov chain. This generalizes dynamic prediction pools to the nonparametric setting. Efficient posterior simulation based on MCMC methods is also examined. Detailed applications to short-term interest rates, realized covariance matrices and asset pricing models demonstrate that the nonparametric pool forecasts well. The paper concludes with extensions and an application for calibrating and combining predictive densities.

The bead process for beta ensembles

The bead process introduced by Boutillier is a countable interlacing of the {text {Sine}}_2 point processes. We construct the bead process for general {text {Sine}}_{beta } processes as an infinite dimensional Markov chain whose transition mechanism is explicitly described. We show that this process is the microscopic scaling limit in the bulk of the Hermite beta corner process introduced by Gorin and Shkolnikov, generalizing the process of the minors of the Gaussian Unitary and Orthogonal Ensembles. In order to prove our results, we use bounds on the variance of the point counting of the circular and the Gaussian beta ensembles, proven in a companion paper (Najnudel and Virág in Some estimates on the point counting of the Circular and the Gaussian Beta Ensemble, 2019).

Optimal event‐triggered transmission scheduling for privacy‐preserving wireless state estimation

SummaryWe consider a remote state estimation problem in the presence of an eavesdropper over packet dropping links. A smart sensor transmits its local estimates to a legitimate remote estimator, in the course of which an eavesdropper can randomly overhear the transmission. This problem has been well studied for unstable dynamical systems, but seldom for stable systems. In this article, we target at stable and marginally stable systems and aim to design an event‐triggered scheduling strategy by minimizing the expected error covariance at the remote estimator and keeping that at the eavesdropper above a user‐specified lower bound. To this end, we model the evolution of the error covariance as an infinite recurrent Markov chain and develop a recurrence relation to describe the stationary distribution of the state at the eavesdropper. Monotonicity and convergence properties of the expected error covariance are further investigated and employed to solve the optimization problem. Numerical examples are provided to validate the theoretical results.

Bayesian Nonparametric Forecast Pooling

This paper introduces a new approach to forecast pooling methods based on a nonparametric prior for the weight vector combining predictive densities. The first approach places a Dirichlet process prior on the weight vector and generalizes the static linear pool. The second approach uses a hierarchical Dirichlet process prior to allow the weight vector to follow an infinite hidden Markov chain. This generalizes dynamic prediction pools to the nonparametric setting. We discuss efficient posterior simulation based on MCMC methods. Detailed applications to short-term interest rates, realized covariance matrices and asset pricing models show the nonparametric pool forecasts well.

Tighter Lyapunov Truncation for Multi-Dimensional Continuous Time Markov Chains with Known Moments

Continuous Time Markov chains (CTMCs) are widely used to model and analyze networked systems. A common analysis approach is to solve the system of balance equations governing the state transitions of a CTMC to obtain its steadystate probability distribution, and use the state probabilities to derive or compute various performance measures. In many systems, the state space of the underlying CTMC is infinite and multi-dimensional with state-dependent transitions; exact analysis of such models is challenging. For example, the exact probability distribution of the number of jobs in the Discriminatory Processor Sharing (DPS) system, first proposed by Kleinrock in 1967 [4], is still an open challenge. Likewise, obtaining the exact state probabilities of quasi-birth-and-death (QBD) processes with leveldependent transitions is known to be challenging [1]; QBDs are infinite state space multi-dimensional Markov chains in which states are organized into levels and transitions are skip-free between the levels.

Shelf Space Selection to Control Product Expiration

Product expiration is an important problem in the consumer‐packaged‐goods industry eroding profits and generating substantial waste. We propose that shelf space selection can be an operational lever to control expiration of perishable inventory. To this end, we first explain the conditions under which shelf space impacts expiration then develop a method to determine the appropriate level of shelf space which incorporates this impact. We formulate a shelf space selection problem for a single product using an infinite horizon Markov chain model. For the special case where demand is constant across periods, we find closed‐form expressions for the average net profit, order levels, and expiration levels as a function of the product’s shelf space under different shelf rotation assumptions. We show that when inventory is not rotated, expiration increases quickly when shelf space exceeds the demand per period. In contrast, when inventory is rotated, the increase occurs when shelf space is larger than the product demand throughout its shelf life. Since computing the optimum shelf space is computationally‐challenging for large shelf life values, we approximate the net profit function leveraging the constant demand analysis. Our approximation method is easy to implement and performs well with a median optimality gap of 0.41% across 160 scenarios. We compare our method with two alternative heuristics and find that it performs better. As a byproduct of this approximation, we develop several managerial insights for the shelf space decision at the tactical level. In addition, we extend our analysis to consider shelf space‐dependent demand.

The Impact of Shelf Space on Product Expiration

Product expiration is an important problem in the consumer-packaged-goods industry eroding profits and generating substantial waste. We propose that shelf space selection can be an operational lever to control expiration of perishable inventory. To this end, we first explain the conditions under which shelf space impacts expiration then develop a method to determine the appropriate level of shelf space which incorporates this impact. We formulate a shelf space selection problem for a single product using an infinite horizon Markov chain model. For the special case where demand is constant across periods, we find closed-form expressions for the average net profit, order levels, and expiration levels as a function of the product’s shelf space under different shelf rotation assumptions. We show that when inventory is not rotated, expiration increases quickly when shelf space exceeds the demand per period. In contrast, when inventory is rotated, the increase occurs when shelf space is larger than the product demand throughout its shelf life. Since computing the optimum shelf space is computationally-challenging for large shelf life values, we approximate the net profit function leveraging the constant demand analysis. Our approximation method is easy to implement and performs well with a median optimality gap of 0.41% across 160 scenarios. We compare our method with two alternative heuristics and find that it performs well. As a byproduct of this approximation, we develop several managerial insights for the shelf space decision at the tactical level. In addition, we extend our analysis to consider shelf space dependent demand.

Probability distributions for Markov chain based quantum walks

We analyze the probability distributions of the quantum walks induced from Markov chains by Szegedy (2004). The first part of this paper is devoted to the quantum walks induced from finite state Markov chains. It is shown that the probability distribution on the states of the underlying Markov chain is always convergent in the Cesaro sense. In particular, we deduce that the limiting distribution is uniform if the transition matrix is symmetric. In the case of a non-symmetric Markov chain, we exemplify that the limiting distribution of the quantum walk is not necessarily identical with the stationary distribution of the underlying irreducible Markov chain. The Szegedy scheme can be extended to infinite state Markov chains (random walks). In the second part, we formulate the quantum walk induced from a lazy random walk on the line. We then obtain the weak limit of the quantum walk. It is noted that the current quantum walk appears to spread faster than its counterpart-quantum walk on the line driven by the Grover coin discussed in literature. The paper closes with an outlook on possible future directions.

Energy Harvesting in Multi-relay Multiuser Networks based on Two-step Selection Scheme

In this paper, we analyze average capacity of an amplify-and-forward (AF) cooperative communication system model in multi-relay multiuser networks. In contrast to conventional cooperative networks, relays in the considered network have no embedded energy supply. They need to rely on the energy harvested from the signals broadcasted by the source for their cooperative information transmission. Based on this structure, a two-step selection scheme is proposed considering both channel state information (CSI) and battery status of relays. Assuming each relay has infinite or finite energy storage for accumulating the energy, we use the infinite or finite Markov chain to capture the evolution of relay batteries and certain simplified assumptions to reduce computational complexity of the Markov chain analysis. The approximate closed-form expressions for the average capacity of the proposed scheme are derived. All theoretical results are validated by numerical simulations. The impacts of the system parameters, such as relay or user number, energy harvesting threshold and battery size, on the capacity performance are extensively investigated. Results show that although the performance of our scheme is inferior to the optimal joint selection scheme, it is still a practical scheme because its complexity is much lower than that of the optimal scheme.

Cooperative Non-Orthogonal Multiple Access with Energy Harvesting

In this paper, we analyze the outage probability of an amplify-and-forward (AF) cooperative non-orthogonal multiple access (NOMA) model in multi-relay multiuser networks. In contrast to conventional cooperative networks, relays in the considered network have no embedded energy supply; they need to rely on the energy harvested from the signals broadcasted by the source for their cooperative NOMA transmission. Based on this structure, a new relay selection scheme is proposed, considering both channel state information (CSI) and battery status of relays. Assuming each relay has infinite or finite energy storage for accumulating energy, we use the infinite or finite Markov chain to capture the evolution of relay batteries and certain simplified assumptions to reduce computational complexity of the Markov chain analysis. The approximate closed-form expressions for the outage probability of the proposed scheme are derived therefrom. All theoretical results are validated by numerical simulations. The impacts of the system parameters, such as relay number, energy harvesting threshold and battery size, on the performance are extensively investigated.

Performance analysis of prioritized broadcast service in WAVE/IEEE 802.11p

In this paper, we propose a more accurate model to capture the prioritized broadcast service in WAVE/IEEE 802.11p and comprehensively analyze the related performance indicators accordingly. We construct a 2-D Markov chain to characterize the IEEE 802.11p EDCA backoff process and subsequently build a 1-D infinite discrete-time Markov chain to identify the contention period for establishing the relation between the transmission probabilities and the channel state. We also consider the impacts of the multichannel operation defined by IEEE 1609.4 in the modeling. Unlike most previous work, we define the transmission probability as a function of the fluctuating numbers of continuous idle slots, introduce a proactive backoff stage into the 2-D Markov chain to profile the backoff procedure where the transmission queue is empty, and characterize the access delay extension and transmission synchronization caused by channel switching. We perform extensive numerical analyses and investigate the access delay, packet delivery rate, and other performance indicators. The results uncover the relations among the metrics of concern underlying various priority access categories as a function of the experienced traffic loads. The effectiveness of the proposed performance model is faithfully verified by the simulation results.

An Analytical Solution for Probabilistic Guarantees of Reservation Based Soft Real-Time Systems

We show a methodology for the computation of the probability of deadline miss for a periodic real-time task scheduled by a resource reservation algorithm. We propose a modelling technique for the system that reduces the computation of such a probability to that of the steady state probability of an infinite state Discrete Time Markov Chain with a periodic structure. This structure is exploited to develop an efficient numeric solution where different accuracy/computation time trade-offs can be obtained by operating on the granularity of the model. More importantly we offer a closed form conservative bound for the probability of a deadline miss. Our experiments reveal that the bound remains reasonably close to the experimental probability in one real-time application of practical interest. When this bound is used for the optimisation of the overall Quality of Service for a set of tasks sharing the CPU, it produces a good sub-optimal solution in a small amount of time.

Determination of robotic melon harvesting efficiency: a probabilistic approach

To automate the harvesting of melons, a mobile Cartesian robot is developed that traverses at a constant velocity over a row of precut melons whose global coordinates are known. The motion planner is programmed to have the robot harvest as many melons as possible. Numerous simulations of the robot over a field with different sets of randomly distributed melons resulted in nearly identical percentages of melons harvested. This result holds true over a wide range of robot dimensions, motor capabilities, velocities and melon distributions. Using probabilistic methods, we derive these results by modelling the robotic harvesting procedure as a stochastic process. In this simplified model, a harvest ratio is predicted analytically using Poisson and geometric distributions. Further analysis demonstrates that this model of robotic harvesting is an example of an infinite length Markov chain. Applying the mathematical tools of Markov processes to our model yields a formula for the harvest percentage that is in strong agreement with the results of the simulation. The significance of the approach is demonstrated in two of its applications: to select the most efficient actuators for maximal melon harvesting and determine the set of optimal velocities along a row of melons of varying densities.

Bounded truncation error for long-run averages in infinite Markov chains

We consider long-run averages of additive functionals on infinite discrete-state Markov chains, either continuous or discrete in time. Special cases include long-run average costs or rewards, stationary moments of the components of ergodic multi-dimensional Markov chains, queueing network performance measures, and many others. By exploiting Foster-Lyapunov-type criteria involving drift conditions for the finiteness of long-run averages we determine suitable finite subsets of the state space such that the truncation error is bounded. Illustrative examples demonstrate the application of this method.

Bounded truncation error for long-run averages in infinite Markov chains

We consider long-run averages of additive functionals on infinite discrete-state Markov chains, either continuous or discrete in time. Special cases include long-run average costs or rewards, stationary moments of the components of ergodic multi-dimensional Markov chains, queueing network performance measures, and many others. By exploiting Foster-Lyapunov-type criteria involving drift conditions for the finiteness of long-run averages we determine suitable finite subsets of the state space such that the truncation error is bounded. Illustrative examples demonstrate the application of this method.

M/M/c Queue with Two Priority Classes

This paper provides the first exact analysis of a preemptive M/M/c queue with two priority classes having different service rates. To perform our analysis, we introduce a new technique to reduce the two-dimensionally infinite Markov chain (MC), representing the two class state space, into a one-dimensionally infinite MC, from which the generating function (GF) of the number of low-priority jobs can be derived in closed form. (The high-priority jobs form a simple M/M/c system and are thus easy to solve.) We demonstrate our methodology for the c = 1, 2 cases; when c > 2, the closed-form expression of the GF becomes cumbersome. We thus develop an exact algorithm to calculate the moments of the number of low-priority jobs for any c ≥ 2. Numerical examples demonstrate the accuracy of our algorithm and generate insights on (i) the relative effect of improving the service rate of either priority class on the mean sojourn time of low-priority jobs; (ii) the performance of a system having many slow servers compared with one having fewer fast servers; and (iii) the validity of the square root staffing rule in maintaining a fixed service level for the low-priority class. Finally, we demonstrate the potential of our methodology to solve other problems using the M/M/c queue with two priority classes, where the high-priority class is completely impatient.

Limits of sequences of Markov chains

We study the limiting object of a sequence of Markov chains analogous to the limits of graphs, hypergraphs, and other objects which have been studied. Following a suggestion of Aldous, we assign to a convergent sequence of finite Markov chains with bounded mixing times a unique limit object: an infinite Markov chain with a measurable state space. The limits of the Markov chains we consider have discrete spectra, which makes the limit theory simpler than the general graph case, and illustrates how the discrete spectrum setting (sometimes called random-free or product measurable) is simpler than the general case.

An H2-Type Norm of a Discrete-Time Linear Stochastic System with Periodic Coefficients Simultaneously Affected by an Infinite Markov Chain and Multiplicative White Noise Perturbations

This article investigates the problem of the definition and computation of an H2-type norm for discrete-time time-varying periodic stochastic linear systems simultaneously affected by multiplicative white noise perturbations and random jumping according to a Markov chain with an infinite countable number of states. Also, we solve an optimization problem that contains, as a special case, the H2 optimal control problem for the considered class of stochastic systems under the assumption of perfect state measurements.