Markov Transition Kernels Research Articles

Unique ergodicity in stochastic electroconvection

We consider a stochastic electroconvection model describing the nonlinear evolution of a surface charge density in a two-dimensional fluid with additive stochastic forcing. We prove the existence and uniqueness of solutions, we define the corresponding Markov semigroup, and we study its Feller properties. When the noise forces enough modes in phase space, we obtain the uniqueness of the smooth invariant measure for the Markov transition kernels associated with the model.

Entropy production in random billiards

We introduce a class of random mechanical systems called random billiards to study the problem of quantifying the irreversibility of nonequilibrium macroscopic systems. In a random billiard model, a point particle evolves by free motion through the interior of a spatial domain, and reflects according to a reflection operator, specified in the model by a Markov transition kernel, upon collision with the boundary of the domain. We derive a formula for entropy production rate that applies to a general class of random billiard systems. This formula establishes a relation between the purely mathematical concept of entropy production rate and textbook thermodynamic entropy, recovering in particular Clausius' formulation of the second law of thermodynamics. We also study an explicit class of examples whose reflection operator, referred to as the Maxwell-Smolukowski thermostat, models systems with boundary thermostats kept at possibly different temperatures. We prove that, under certain mild regularity conditions, the class of models are uniformly ergodic Markov chains and derive formulas for the stationary distribution and entropy production rate in terms of geometric and thermodynamic parameters.

Stochastic Gradient Population Monte Carlo

The population Monte Carlo (PMC) algorithm is a powerful adaptive importance sampling (AIS) methodology used for estimating expected values of random quantities w.r.t. some target probability distribution. At each iteration, a Markov transition kernel is used to propagate a set of particles. Importance weights of the particles are computed and then used to resample the particles that are most representative of the target distribution. At the end of the algorithm, the set of all particles and weights can be used to perform estimation. The resampling step is an adaptive mechanism of the PMC algorithm that allows for particles to locate the most significant regions of the sampling space. In this letter, we generalize the adaptation procedure of PMC sampling by providing a perspective based on stochastic optimization rather than resampling. The proposed method is more flexible than standard PMC as it allows the parameter adaptation to be resolved using any stochastic optimization method. We show that under certain conditions, the standard PMC algorithm is a special case of the proposed approach.

Quantitative contraction rates for Markov chains on general state spaces

We investigate the problem of quantifying contraction coefficients of Markov transition kernels in Kantorovich (L1 Wasserstein) distances. For diffusion processes, relatively precise quantitative bounds on contraction rates have recently been derived by combining appropriate couplings with carefully designed Kantorovich distances. In this paper, we partially carry over this approach from diffusions to Markov chains. We derive quantitative lower bounds on contraction rates for Markov chains on general state spaces that are powerful if the dynamics is dominated by small local moves. For Markov chains on Rd with isotropic transition kernels, the general bounds can be used efficiently together with a coupling that combines maximal and reflection coupling. The results are applied to Euler discretizations of stochastic differential equations with non-globally contractive drifts, and to the Metropolis adjusted Langevin algorithm for sampling from a class of probability measures on high dimensional state spaces that are not globally log-concave

Approximation of Discounted Minimax Markov Control Problems and Zero-Sum Markov Games Using Hausdorff and Wasserstein Distances

This paper is concerned with a minimax control problem (also known as a robust Markov decision process (MDP) or a game against nature) with general state and action spaces under the discounted cost optimality criterion. We are interested in approximating numerically the value function and an optimal strategy of this general discounted minimax control problem. To this end, we derive structural Lipschitz continuity properties of the solution of this robust MDP by imposing suitable conditions on the model, including Lipschitz continuity of the elements of the model and absolute continuity of the Markov transition kernel with respect to some probability measure $$\mu $$ . Then, we are able to provide an approximating minimax control model with finite state and action spaces, and hence computationally tractable, by combining these structural properties with a suitable discretization procedure of the state space (related to a probabilistic criterion) and the action spaces (associated to a geometric criterion). Finally, it is shown that the corresponding approximation errors for the value function and the optimal strategy can be controlled in terms of the discretization parameters. These results are also extended to a two-player zero-sum Markov game.

Hitting Times in Markov Chains with Restart and their Application to Network Centrality

Motivated by applications in telecommunications, computer scienceand physics, we consider a discrete-time Markov process withrestart. At each step the process eitherwith a positive probability restarts from a given distribution, orwith the complementary probability continues according to a Markovtransition kernel. The main contribution of the present work is thatwe obtain an explicit expression for the expectation of the hittingtime (to a given target set) of the process with restart.The formula is convenient when considering the problem of optimizationof the expected hitting time with respect to the restart probability.We illustrate our results with two examplesin uncountable and countable state spaces andwith an application to network centrality.

A Markov chain representation of the multiple testing problem.

The problem of multiple hypothesis testing can be represented as a Markov process where a new alternative hypothesis is accepted in accordance with its relative evidence to the currently accepted one. This virtual and not formally observed process provides the most probable set of non null hypotheses given the data; it plays the same role as Markov Chain Monte Carlo in approximating a posterior distribution. To apply this representation and obtain the posterior probabilities over all alternative hypotheses, it is enough to have, for each test, barely defined Bayes Factors, e.g. Bayes Factors obtained up to an unknown constant. Such Bayes Factors may either arise from using default and improper priors or from calibrating p-values with respect to their corresponding Bayes Factor lower bound. Both sources of evidence are used to form a Markov transition kernel on the space of hypotheses. The approach leads to easy interpretable results and involves very simple formulas suitable to analyze large datasets as those arising from gene expression data (microarray or RNA-seq experiments).

On the use of Markov chain Monte Carlo methods for the sampling of mixture models: a statistical perspective

In this paper we study asymptotic properties of different data-augmentation-type Markov chain Monte Carlo algorithms sampling from mixture models comprising discrete as well as continuous random variables. Of particular interest to us is the situation where sampling from the conditional distribution of the continuous component given the discrete component is infeasible. In this context, we advance Carlin & Chib's pseudo-prior method as an alternative way of infering mixture models and discuss and compare different algorithms based on this scheme. We propose a novel algorithm, the Frozen Carlin & Chib sampler, which is computationally less demanding than any Metropolised Carlin & Chib-type algorithm. The significant gain of computational efficiency is however obtained at the cost of some asymptotic variance. The performance of the algorithm vis-a-vis alternative schemes is, using some recent results obtained in Maire et al. (Ann Stat 42: 1483---1510, 2014) for inhomogeneous Markov chains evolving alternatingly according to two different $$\pi ^{*}$$??-reversible Markov transition kernels, investigated theoretically as well as numerically.

Approximation of average cost Markov decision processes using empirical distributions and concentration inequalities

We consider a discrete-time Markov decision process with Borel state and action spaces, and possibly unbounded cost function. We assume that the Markov transition kernel is absolutely continuous with respect to some probability measure . By replacing this probability measure with its empirical distribution for a sample of size n, we obtain a finite state space control problem, which is used to provide an approximation of the optimal value and an optimal policy of the original control model. We impose Lipschitz continuity properties on the control model and its associated density functions. We measure the accuracy of the approximation of the optimal value and an optimal policy by means of a non-asymptotic concentration inequality based on the 1-Wasserstein distance between and . Obtaining numerically the solution of the approximating control model is discussed and an application to an inventory management problem is presented.

Comparison of asymptotic variances of inhomogeneous Markov chains with application to Markov chain Monte Carlo methods

In this paper, we study the asymptotic variance of sample path averages for inhomogeneous Markov chains that evolve alternatingly according to two different $\pi$-reversible Markov transition kernels $P$ and $Q$. More specifically, our main result allows us to compare directly the asymptotic variances of two inhomogeneous Markov chains associated with different kernels $P_i$ and $Q_i$, $i\in\{0,1\}$, as soon as the kernels of each pair $(P_0,P_1)$ and $(Q_0,Q_1)$ can be ordered in the sense of lag-one autocovariance. As an important application, we use this result for comparing different data-augmentation-type Metropolis-Hastings algorithms. In particular, we compare some pseudo-marginal algorithms and propose a novel exact algorithm, referred to as the random refreshment algorithm, which is more efficient, in terms of asymptotic variance, than the Grouped Independence Metropolis-Hastings algorithm and has a computational complexity that does not exceed that of the Monte Carlo Within Metropolis algorithm.

Optimal measures and Markov transition kernels

We study optimal solutions to an abstract optimization problem for measures, which is a generalization of classical variational problems in information theory and statistical physics. In the classical problems, information and relative entropy are defined using the Kullback-Leibler divergence, and for this reason optimal measures belong to a one-parameter exponential family. Measures within such a family have the property of mutual absolute continuity. Here we show that this property characterizes other families of optimal positive measures if a functional representing information has a strictly convex dual. Mutual absolute continuity of optimal probability measures allows us to strictly separate deterministic and non-deterministic Markov transition kernels, which play an important role in theories of decisions, estimation, control, communication and computation. We show that deterministic transitions are strictly sub-optimal, unless information resource with a strictly convex dual is unconstrained. For illustration, we construct an example where, unlike non-deterministic, any deterministic kernel either has negatively infinite expected utility (unbounded expected error) or communicates infinite information.

Convergence of adaptive and interacting Markov chain Monte Carlo algorithms

Adaptive and interacting Markov chain Monte Carlo algorithms (MCMC) have been recently introduced in the literature. These novel simulation algorithms are designed to increase the simulation efficiency to sample complex distributions. Motivated by some recently introduced algorithms (such as the adaptive Metropolis algorithm and the interacting tempering algorithm), we develop a general methodological and theoretical framework to establish both the convergence of the marginal distribution and a strong law of large numbers. This framework weakens the conditions introduced in the pioneering paper by Roberts and Rosenthal [J. Appl. Probab. 44 (2007) 458--475]. It also covers the case when the target distribution $\pi$ is sampled by using Markov transition kernels with a stationary distribution that differs from $\pi$.

CLTs and Asymptotic Variance of Time-Sampled Markov Chains

For a Markov transition kernel P and a probability distribution μ on nonnegative integers, a time-sampled Markov chain evolves according to the transition kernel \(P_{\mu} = \sum_k \mu(k)P^k.\) In this note we obtain CLT conditions for time-sampled Markov chains and derive a spectral formula for the asymptotic variance. Using these results we compare efficiency of Barker’s and Metropolis algorithms in terms of asymptotic variance.

On the Markov Transition Kernels for First Passage Percolation on the Ladder

We consider the first passage percolation problem on the random graph with vertex set N x {0, 1}, edges joining vertices at a Euclidean distance equal to unity, and independent exponential edge weights. We provide a central limit theorem for the first passage times ln between the vertices (0, 0) and (n, 0), thus extending earlier results about the almost-sure convergence of ln / n as n → ∞. We use generating function techniques to compute the n-step transition kernels of a closely related Markov chain which can be used to explicitly calculate the asymptotic variance in the central limit theorem.

On the Markov Transition Kernels for First Passage Percolation on the Ladder

We consider the first passage percolation problem on the random graph with vertex set N x {0, 1}, edges joining vertices at a Euclidean distance equal to unity, and independent exponential edge weights. We provide a central limit theorem for the first passage times l n between the vertices (0, 0) and (n, 0), thus extending earlier results about the almost-sure convergence of l n / n as n → ∞. We use generating function techniques to compute the n-step transition kernels of a closely related Markov chain which can be used to explicitly calculate the asymptotic variance in the central limit theorem.

Nonparametric model validations for hidden Markov models with applications in financial econometrics

We address the nonparametric model validation problem for hidden Markov models with partially observable variables and hidden states. We achieve this goal by constructing a nonparametric simultaneous confidence envelope for transition density function of the observable variables and checking whether the parametric density estimate is contained within such an envelope. Our specification test procedure is motivated by a functional connection between the transition density of the observable variables and the Markov transition kernel of the hidden states. Our approach is applicable for continuous-time diffusion models, stochastic volatility models, nonlinear time series models, and models with market microstructure noise.

Perturbation of Transition Functions and a Feynman-Kac Formula for the Incorporation of Mortality

Markov transition kernels are perturbed by output kernels with a special emphasis on building mortality into structured population models. A Feynman-Kac formula is derived which illustrates the interplay of mortality with a Markov process associated with the unperturbed kernel.

Identification of Regeneration Times in MCMC Simulation, With Application to Adaptive Schemes

Regeneration is a useful tool in Markov chain Monte Carlo simulation because it can be used to side-step the burn-in problem and to construct better estimates of the variance of parameter estimates themselves. It also provides a simple way to introduce adaptive behavior into a Markov chain, and to use parallel processors to build a single chain. Regeneration is often difficult to take advantage of because, for most chains, no recurrent proper atom exists, and it is not always easy to use Nummelin's splitting method to identify regeneration times. This article describes a constructive method for generating a Markov chain with a specified target distribution and identifying regeneration times. As a special case of the method, an algorithm which can be “wrapped” around an existing Markov transition kernel is given. In addition, a specific rule for adapting the transition kernel at regeneration times is introduced, which gradually replaces the original transition kernel with an independence-sampling Metropolis-Hastings kernel using a mixture normal approximation to the target density as its proposal density. Computational gains for the regenerative adaptive algorithm are demonstrated in examples.

Necessary conditions for geometric and polynomial ergodicity of random-walk-type

We give necessary conditions for geometric and polynomial convergence rates of randomwalk- type Markov chains to stationarity in terms of existence of exponential and polynomial moments of the invariant distribution and the Markov transition kernel. These results complement the use of Foster-Lyapunov drift conditions for establishing geometric and polynomial ergodicity. For polynomially ergodic Markov chains, the results allow us to derive exact rates of convergence and exact relations between the moments of the invariant distribution and the Markov transition kernel. In an application to Markov chain Monte Carlo we derive tight rates of convergence for symmetric random walk Metropolis.

A robustification approach to stability and to uniform particle approximation of nonlinear filters: the example of pseudo-mixing signals

We propose a new approach to study the stability of the optimal filter w.r.t. its initial condition, by introducing a “robust” filter, which is exponentially stable and which approximates the optimal filter uniformly in time. The “robust” filter is obtained here by truncation of the likelihood function, and the robustification result is proved under the assumption that the Markov transition kernel satisfies a pseudo-mixing condition (weaker than the usual mixing condition), and that the observations are “sufficiently good”. This robustification approach allows us to prove also the uniform convergence of several particle approximations to the optimal filter, in some cases of nonergodic signals.