Continuous-state Space Markov Chain Research Articles

Adaptive Multiuser Cooperative NOMA Scheme With Energy Buffer-Aided Near-Users for High Spectral and Energy Efficiency

In this article, a multiuser cooperative nonorthogonal multiple access (NOMA) network is considered. In order to improve their battery lifetime, the near-user (NU) Internet of Things (IoT) nodes relay information to the far-user (FU) IoT nodes using only the energy harvested from the ambience. The harvested energy at all the NU IoT nodes is stored in energy buffers. To improve the throughput performance, the best NU and best FU (BNBF) user selection scheme is employed. Moreover, unlike conventional orthogonal multiple access (OMA) networks, an adaptive NOMA network is considered in this article that switches between direct NOMA, NOMA with relaying, and OMA modes to maximize throughput. To improve accuracy and reduce the computational complexity, the energy buffer states at the NU IoT nodes are modeled using a continuous state-space Markov chain (CSMC) instead of discrete state-space Markov chain (DSMC). We derive the limiting distributions of the stored energy in energy buffers with either the best effort policy (BEP) or the on–off policy (OOP) applied for buffer energy management. Expressions for throughput are derived for both harvest-store-use (HSU) and harvest-use (HU) architectures. The Monte Carlo simulations are used to validate the derived analytical expressions.

Partially observed optimal stopping problem for discrete-time Markov processes

This paper is dedicated to the investigation of a new numerical method to approximate the optimal stopping problem for a discrete-time continuous state space Markov chain under partial observations. It is based on a two-step discretization procedure based on optimal quantization. First, we discretize the state space of the unobserved variable by quantizing an underlying reference measure. Then we jointly discretize the resulting approximate filter and the observation process. We obtain a fully computable approximation of the value function with explicit error bounds for its convergence towards the true value function.

Estimación de la Esperanza, Varianza y Cuantiles en Simulaciones de Estado Estable

In this article three methods (multiple replications, non-overlapping batches and spaced batches) to estimate the expectation, the variance and the 90%-quantile of the steady-state waiting time in an M/M/1 queue are compared. Our comparison is based on the empirical coverage, bias and relative error of 90% asymptotic confidence intervals from 1000 independent replications of a simulation-based estimation. Experimental results show that all three methods provide similar (and valid) empirical coverage for the estimation of the expectation, variance and quantile; and the method of non-overlapping batches provided smaller bias and relative errors than the other two methods. In addition to this, an example of a continuous state-space Markov chain that is ergodic but non-geometric, is presented. For this case asymptotically valid confidence intervals by using the methods considered in this paper are also obtained.

An Introduction to Markov Chain Monte Carlo

This paper introduces the method of Markov Chain Monte Carlo (MCMC). An outline of the methods is given together with some preliminary tools. The Bayesian approach to statistics is introduced, and the necessary continuous state space Markov chain theory is summarized. Two common algorithms for generating random draws from complex joint distribution are presented; The Gibbs sampler and the Metropolis-Hastings algorithm. We discuss implementational issues and demonstrate the method by a simple empirical example on a generalized linear mixed model. The reader is assumed to have background in probability theory and to be familiar with discrete time Markov chains on a finite state space.

Capacity of Finite State Channels Based on Lyapunov Exponents of Random Matrices

The finite-state Markov channel (FSMC) is a time-varying channel having states that are characterized by a finite-state Markov chain. These channels have infinite memory, which complicates their capacity analysis. We develop a new method to characterize the capacity of these channels based on Lyapunov exponents. Specifically, we show that the input, output, and conditional entropies for this channel are equivalent to the largest Lyapunov exponents for a particular class of random matrix products. We then show that the Lyapunov exponents can be expressed as expectations with respect to the stationary distributions of a class of continuous-state space Markov chains. This class of Markov chains, which is closely related to the prediction filter in hidden Markov models, is shown to be nonirreducible. Hence, much of the standard theory for continuous state-space Markov chains cannot be applied to establish the existence and uniqueness of stationary distributions, nor do we have direct access to a central limit theorem (CLT). In order to address these shortcomings, we utilize several results from the theory of random matrix products and Lyapunov exponents. The stationary distributions for this class of Markov chains are shown to be unique and continuous functions of the input symbol probabilities, provided that the input sequence has finite memory. These properties allow us to express mutual information and channel capacity in terms of Lyapunov exponents. We then leverage this connection between entropy and Lyapunov exponents to develop a rigorous theory for computing or approximating entropy and mutual information for finite-state channels with dependent inputs. We develop a method for directly computing entropy of finite-state channels that does not rely on simulation and establish its convergence. We also obtain a new asymptotically tight lower bound for entropy based on norms of random matrix products. In addition, we prove a new functional CLT for sample entropy and apply this theorem to characterize the error in simulated estimates of entropy. Finally, we present numerical examples of mutual information computation for intersymbol interference (ISI) channels and observe the capacity benefits of adding memory to the input sequence for such channels

The radial spanning tree of a Poisson point process

We analyze a class of spatial random spanning trees built on a realization of a homogeneous Poisson point process of the plane. This tree has a simple radial structure with the origin as its root. We first use stochastic geometry arguments to analyze local functionals of the random tree such as the distribution of the length of the edges or the mean degree of the vertices. Far away from the origin, these local properties are shown to be close to those of a variant of the directed spanning tree introduced by Bhatt and Roy. We then use the theory of continuous state space Markov chains to analyze some nonlocal properties of the tree, such as the shape and structure of its semi-infinite paths or the shape of the set of its vertices less than k generations away from the origin. This class of spanning trees has applications in many fields and, in particular, in communications.

Distribution Dynamics and Cross‐Country Convergence: A New Approach

Distribution dynamics is a method for studying the evolution in time of an entire cross‐section distribution and has been initially employed to assess cross‐country convergence of per capita incomes. It has subsequently seen a widespread application in many different economic areas. When describing the law of motion of the distribution as a Markovian stochastic process, working in a discrete state‐space set up has several advantages, but the arbitrary discretisation of a continuous state‐space process has the undesired effect of removing the Markov property. This paper outlines a rigorous method for discretising a continuous state‐space Markov chain. The method is then applied to the distribution of per capita income across countries to reassess the (non‐) convergence phenomenon. It is found that the long run polarisation of per capita incomes across countries emerges even more dramatically than in previous studies.

Randomized neural networks for learning stochastic dependences

We consider the problem of learning the dependence of one random variable on another, from a finite string of independently identically distributed (i.i.d.) copies of the pair. The problem is first converted to that of learning a function of the latter random variable and an independent random variable uniformly distributed on the unit interval. However, this cannot be achieved using the usual function learning techniques because the samples of the uniformly distributed random variables are not available. We propose a novel loss function, the minimizer of which results in an approximation to the needed function. Through successive approximation results (suggested by the proposed loss function), a suitable class of functions represented by combination feedforward neural networks is selected as the class to learn from. These results are also extended for countable as well as continuous state-space Markov chains. The effectiveness of the proposed method is indicated through simulation studies.

Discretization of Continuous Markov Chains and Markov Chain Monte Carlo Convergence Assessment

We show that continuous state-space Markov chains can be rigorously discretized into finite Markov chains. The idea is to subsample the continuous chain at renewal times related to small sets that control the discretization. Once a finite Markov chain is derived from the Markov chain Monte Carlo output, general convergence properties on finite state spaces can be exploited for convergence assessment in several directions. Our choice is based on a divergence criterion derived from Kemeny and Snell, which is first evaluated on parallel chains with a stopping time and then implemented, more efficiently, on two parallel chains only, using Birkhoff's pointwise ergodic theorem for stopping rules. The performance of this criterion is illustrated on three standard examples.

Replacement policy for a partially observable Markov decision process model using fuzzy data

When the deterioration is governed by a Markov process, such processes are known Partially Observable Markov Decision Processes (POMDP) which eliminate the assumption that the state or level of deterioration of the system is known exactly. This research investigates a two state partially observable Markov chain in which only deterioration can occur and for which only actions possible are to replace or to leave alone. Because the cost structure is defined on the process can not be obtained exactly, a fuzzy concept is proposed for an optimal replacement policy which leads to the operation cost savings of a machine. The goal of this research is to develop optimal replacement policies under a new approach which has the potential for solving other problems dealing with continuous state space Markov chains using fuzzy data.