Transition Kernel Research Articles

Robustness to incorrect models and data-driven learning in average-cost optimal stochastic control

We study continuity and robustness properties of infinite-horizon average expected cost problems with respect to (controlled) transition kernels, and applications of these results to the problem of robustness of control policies designed for approximate models applied to actual systems. We show that sufficient conditions presented in the literature for discounted-cost problems are in general not sufficient to ensure robustness for average-cost problems. However, we show that the average optimal cost is continuous in the convergences of controlled transition kernel models where convergence of models entails (i) continuous weak convergence in state and actions, and (ii) continuous setwise convergence in the actions for every fixed state variable, in addition to either uniform ergodicity or some regularity conditions. We establish that the mismatch error due to the application of a control policy designed for an incorrectly estimated model to the true model decreases to zero as the incorrect model approaches the true model under the stated convergence criteria. Our findings significantly relax related studies in the literature which have primarily considered the more restrictive total variation convergence criteria. Applications to robustness to models estimated through empirical data (where almost sure weak convergence criterion typically holds, but stronger criteria do not) are studied and conditions for asymptotic robustness to data-driven learning are established.

Quantitative Heat-Kernel Estimates for Diffusions with Distributional Drift

We consider the stochastic differential equation on mathbb {R}^{d} given bydXt=b(t,Xt)dt+dBt,\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$ \\begin{array}{@{}rcl@{}} \\mathrm{d} X_{t} = b(t,X_{t}) \\mathrm{d} t + \\mathrm{d} B_{t}, \\end{array} $$\\end{document}where B is a Brownian motion and b is considered to be a distribution of regularity > -frac 12. We show that the martingale solution of the SDE has a transition kernel Γt and prove upper and lower heat-kernel estimates for Γt with explicit dependence on t and the norm of b.

Empirical Ross recovery without discretization

AbstractWe propose a new continuous‐state method for empirically recovering the key objects in the Ross recovery theory which avoids discretization. The new method is based on a key bivariate orthogonal Hermite representation of the state price transition kernel, which leads to an elegant correspondence of the eigenvalue–eigenfunction system of the transition kernel and the eigenvalue–eigenvector system of the expansion coefficient matrix. Using S&P 500 index option prices, we demonstrate how our method can generate well‐behaved state price transition kernels and physical densities. Our method can also be used to compute the key objects in the Hansen and Scheinkman's factorization theory.

A Deterministic Approximation to Neural SDEs.

Neural Stochastic Differential Equations (NSDEs) model the drift and diffusion functions of a stochastic process as neural networks. While NSDEs are known to make accurate predictions, their uncertainty quantification properties have been remained unexplored so far. We report the empirical finding that obtaining well-calibrated uncertainty estimations from NSDEs is computationally prohibitive. As a remedy, we develop a computationally affordable deterministic scheme which accurately approximates the transition kernel, when dynamics is governed by a NSDE. Our method introduces a bidimensional moment matching algorithm: vertical along the neural net layers and horizontal along the time direction, which benefits from an original combination of effective approximations. Our deterministic approximation of the transition kernel is applicable to both training and prediction. We observe in multiple experiments that the uncertainty calibration quality of our method can be matched by Monte Carlo sampling only after introducing high computational cost. Thanks to the numerical stability of deterministic training, our method also improves prediction accuracy.

Discrete-time dynamics of structured populations via Feller kernels

<p style='text-indent:20px;'>Feller kernels are a concise means to formalize individual structural transitions in a structured discrete-time population model. An iteroparous populations (in which generations overlap) is considered where different kernels model the structural transitions for neonates and for older individuals. Other Feller kernels are used to model competition between individuals. The spectral radius of a suitable Feller kernel is established as basic turnover number that acts as threshold between population extinction and population persistence. If the basic turnover number exceeds one, the population shows various degrees of persistence that depend on the irreducibility and other properties of the transition kernels.</p>

On population growth with catastrophes

In this article, we study a particular class of Piecewise deterministic Markov processes (PDMP’s) which are semi-stochastic catastrophe versions of deterministic population growth models. In between successive jumps the process follows a flow describing deterministic population growth. Moreover, at random jump times, governed by state-dependent rates, the size of the population shrinks by a random amount of its current size, an event possibly leading to instantaneous local (or total) extinction. A special separable shrinkage transition kernel is investigated in more detail, including the case of total disasters. We discuss conditions under which such processes are recurrent (positive or null) or transient. To do so, we introduce a modified scale function which is used to compute, when relevant, the law of the height of excursions and to decide if the process is recurrent or not. The question of the finiteness of the time to extinction is investigated together with the evaluation of the mean time to extinction when the last one is finite. Some information on the embedded jump chain of the PDMP is also required when dealing with the classification of states 0 and that we exhibit.

Ensemble Metropolis Light Transport

This article proposes a Markov Chain Monte Carlo ( MCMC ) rendering algorithm based on a family of guided transition kernels. The kernels exploit properties of ensembles of light transport paths, which are distributed according to the lighting in the scene, and utilize this information to make informed decisions for guiding local path sampling. Critically, our approach does not require caching distributions in world space, saving time and memory, yet it is able to make guided sampling decisions based on whole paths. We show how this can be implemented efficiently by organizing the paths in each ensemble and designing transition kernels for MCMC rendering based on a carefully chosen subset of paths from the ensemble. This algorithm is easy to parallelize and leads to improvements in variance when rendering a variety of scenes.

Information Geometry of Reversible Markov Chains

We analyze the information geometric structure of time reversibility for parametric families of irreducible transition kernels of Markov chains. We define and characterize reversible exponential families of Markov kernels, and show that irreducible and reversible Markov kernels form both a mixture family and, perhaps surprisingly, an exponential family in the set of all stochastic kernels. We propose a parametrization of the entire manifold of reversible kernels, and inspect reversible geodesics. We define information projections onto the reversible manifold, and derive closed-form expressions for the e-projection and m-projection, along with Pythagorean identities with respect to information divergence, leading to some new notion of reversiblization of Markov kernels. We show the family of edge measures pertaining to irreducible and reversible kernels also forms an exponential family among distributions over pairs. We further explore geometric properties of the reversible family, by comparing them with other remarkable families of stochastic matrices. Finally, we show that reversible kernels are, in a sense we define, the minimal exponential family generated by the m-family of symmetric kernels, and the smallest mixture family that comprises the e-family of memoryless kernels.

Approximations of geometrically ergodic reversible markov chains

AbstractA common tool in the practice of Markov chain Monte Carlo (MCMC) is to use approximating transition kernels to speed up computation when the desired kernel is slow to evaluate or is intractable. A limited set of quantitative tools exists to assess the relative accuracy and efficiency of such approximations. We derive a set of tools for such analysis based on the Hilbert space generated by the stationary distribution we intend to sample, $L_2(\pi)$. Our results apply to approximations of reversible chains which are geometrically ergodic, as is typically the case for applications to MCMC. The focus of our work is on determining whether the approximating kernel will preserve the geometric ergodicity of the exact chain, and whether the approximating stationary distribution will be close to the original stationary distribution. For reversible chains, our results extend the results of Johndrow et al. (2015) from the uniformly ergodic case to the geometrically ergodic case, under some additional regularity conditions. We then apply our results to a number of approximate MCMC algorithms.

Comment on “Large Bayesian vector autoregressions with stochastic volatility and non-conjugate priors”

Fully Bayesian inference in a vector autoregression with stochastic volatility (VAR-SV) typically relies on simulations from a multi-step Markov chain Monte Carlo (MCMC) algorithm. Carriero et al. (2019) propose a new, faster, “triangular” algorithm (TA) to replace the systemwide algorithm (SWA) in the most time-consuming step of the VAR-SV’s standard MCMC algorithm. This paper analytically shows that the TA and SWA generally sample from different distributions, thereby disproving a central claim of Carriero et al. (2019). Replacing the SWA with the TA thus results in an ad hoc change to the MCMC algorithm’s transition kernel, leaving a priori unknown the formal relationship between the model’s posterior and simulations from the MCMC algorithm using the TA.

Predictive Constructions Based on Measure-Valued Pólya Urn Processes

Measure-valued Pólya urn processes (MVPP) are Markov chains with an additive structure that serve as an extension of the generalized k-color Pólya urn model towards a continuum of possible colors. We prove that, for any MVPP (μn)n≥0 on a Polish space X, the normalized sequence (μn/μn(X))n≥0 agrees with the marginal predictive distributions of some random process (Xn)n≥1. Moreover, μn=μn−1+RXn, n≥1, where x↦Rx is a random transition kernel on X; thus, if μn−1 represents the contents of an urn, then Xn denotes the color of the ball drawn with distribution μn−1/μn−1(X) and RXn—the subsequent reinforcement. In the case RXn=WnδXn, for some non-negative random weights W1,W2,…, the process (Xn)n≥1 is better understood as a randomly reinforced extension of Blackwell and MacQueen’s Pólya sequence. We study the asymptotic properties of the predictive distributions and the empirical frequencies of (Xn)n≥1 under different assumptions on the weights. We also investigate a generalization of the above models via a randomization of the law of the reinforcement.

The parallel convergence of income and educational achievements: joint distribution dynamics

Human capital is an important factor of economic growth. Previous studies show that convergence patterns of income and education differ on a regional level. The purpose of this article is to verify whether there is statistical relationship between the patterns of educational achievements convergence and income convergence processes in Poland on the regional and local level. The paper describes and applies an innovative way for the formal verification of similarities in convergence patterns on the basis of transition matrices and kernel density estimation. The analysis is performed on Polish data for educational achievements (lagged exam results of 15-year old pupils) and income per capita worker on the regional and local level for the period 2003–2015. Despite the occurrence of (weak) convergence for both phenomena, each has a different course. Therefore, the processes of the income convergence and educational achievements convergence in Poland were not parallel.

Automated deep abstractions for stochastic chemical reaction networks

Predicting stochastic cellular dynamics emerging from chemical reaction networks (CRNs) is a long-standing challenge in systems biology. Deep learning was recently used to abstract the CRN dynamics by a mixture density neural network, trained with traces of the original process. Such abstraction is dramatically cheaper to execute, yet it preserves the statistical features of the training data. However, in practice, the modeller has to take care of finding the suitable neural network architecture manually, for each given CRN, through a trial-and-error cycle. In this paper, we propose to further automatise deep abstractions for stochastic CRNs, through learning the neural network architecture along with learning the transition kernel of the stochastic process. The method is applicable to any given CRN, time-saving for deep learning experts and crucial for non-specialists. We demonstrate performance over a number of CRNs with multi-modal phenotypes and a multi-scale scenario where CRNs interact across a spatial grid.

Spatially Inhomogeneous Populations with Seed-Banks: I. Duality, Existence and Clustering

AbstractWe consider a system of interacting Moran models with seed-banks. Individuals live in colonies and are subject to resampling and migration as long as they are active. Each colony has a seed-bank into which individuals can retreat to become dormant, suspending their resampling and migration until they become active again. The colonies are labelled by $${\mathbb {Z}}^d$$ Z d , $$d \ge 1$$ d ≥ 1 , playing the role of a geographic space. The sizes of the active and the dormant population are finite and depend on the location of the colony. Migration is driven by a random walk transition kernel. Our goal is to study the equilibrium behaviour of the system as a function of the underlying model parameters. In the present paper, under a mild condition on the sizes of the active populations, the system is well defined and has a dual. The dual consists of a system of interacting coalescing random walks in an inhomogeneous environment that switch between an active state and a dormant state. We analyse the dichotomy of coexistence (= multi-type equilibria) versus clustering (= mono-type equilibria) and show that clustering occurs if and only if two random walks in the dual starting from arbitrary states eventually coalesce with probability one. The presence of the seed-bank enhances genetic diversity. In the dual this is reflected by the presence of time lapses during which the random walks are dormant and do not move.

Realistic Models of Financial Market and Structural Stability

The main aim of this article is to show the role of structural stability in financial modelling; that is, a specific “no-arbitrage” property is unaffected by small perturbations of the model’s dynamics. We prove that under the structural stability assumption, given a convex compact-valued multifunction, there exists a stochastic transition kernel with supports coinciding with this multifunction and one that is strong Feller in the strict sense. We also demonstrate preservation of structural stability for sufficiently small deviations of transition kernels for different probability metrics.

Inverse reinforcement learning in contextual MDPs

We consider the task of Inverse Reinforcement Learning in Contextual Markov Decision Processes (MDPs). In this setting, contexts, which define the reward and transition kernel, are sampled from a distribution. In addition, although the reward is a function of the context, it is not provided to the agent. Instead, the agent observes demonstrations from an optimal policy. The goal is to learn the reward mapping, such that the agent will act optimally even when encountering previously unseen contexts, also known as zero-shot transfer. We formulate this problem as a non-differential convex optimization problem and propose a novel algorithm to compute its subgradients. Based on this scheme, we analyze several methods both theoretically, where we compare the sample complexity and scalability, and empirically. Most importantly, we show both theoretically and empirically that our algorithms perform zero-shot transfer (generalize to new and unseen contexts). Specifically, we present empirical experiments in a dynamic treatment regime, where the goal is to learn a reward function which explains the behavior of expert physicians based on recorded data of them treating patients diagnosed with sepsis.

Quantum dynamics of the classical harmonic oscillator

A correspondence is established between measure-preserving, ergodic dynamics of a classical harmonic oscillator and a quantum mechanical gauge theory on two-dimensional Minkowski space. This correspondence is realized through an isometric embedding of the L2(μ) space on the circle associated with the oscillator’s invariant measure, μ, into a Hilbert space H of sections of a C-line bundle over Minkowski space. This bundle is equipped with a covariant derivative induced from an SO+(1, 1) gauge field on the corresponding inertial frame bundle, satisfying the Yang–Mills equations. Under this embedding, the Hamiltonian operator of a Lorentz-invariant quantum system, constructed as a geometrical Laplace-type operator on bundle sections, pulls back to the generator of the unitary group of Koopman operators governing the evolution of classical observables of the harmonic oscillator, with Koopman eigenfunctions of zero, positive, and negative eigenfrequencies corresponding to quantum eigenstates of zero (“vacuum”), positive (“matter”), and negative (“antimatter”) energies. The embedding also induces a pair of operators acting on classical observables of the harmonic oscillator, exhibiting canonical position–momentum commutation relationships. These operators have the structure of order-1/2 fractional derivatives and therefore display a form of non-locality. In a second part of this work, we study a quantum mechanical representation of the classical harmonic oscillator using a one-parameter family of reproducing kernel Hilbert spaces, K^τ, associated with the time-τ transition kernel of a fractional diffusion on the circle. As shown in recent work, these spaces are unital Banach *-algebras of functions. It is found that the evolution of classical observables in these spaces takes place via a strongly continuous, unitary Koopman evolution group, which exhibits a stronger form of classical–quantum consistency than the L2(μ) case. Specifically, for every real-valued classical observable in K^τ, there exists a quantum mechanical observable, whose expectation value is consistent with classical function evaluation. This allows for a description of classical state space dynamics, classical statistics, and quantum statistics of the harmonic oscillator within a unified framework.

Likelihood function for estimating parameters in multistate disease process with Laplace-transformation-based transition probabilities

Multistate statistical models are often used to characterize the complex multi-compartment progression of the disease such as cancer. However, the derivation of multistate transition kernels is often involved with the intractable convolution that requires intensive computation. Moreover, the estimation of parameters pertaining to transition kernel requires the individualized time-stamped history data while the traditional likelihood function forms are constructed. In this paper, we came up with a novel likelihood function derived from Laplace transformation-based transition probabilities in conjunction with Expectation–Maximization algorithm to estimate parameters. The proposed method was applied to two large population-based screening data with only aggregated count data without relying on individual time-stamped history data.

PAC-Bayes Bounds on Variational Tempered Posteriors for Markov Models.

Datasets displaying temporal dependencies abound in science and engineering applications, with Markov models representing a simplified and popular view of the temporal dependence structure. In this paper, we consider Bayesian settings that place prior distributions over the parameters of the transition kernel of a Markov model, and seek to characterize the resulting, typically intractable, posterior distributions. We present a Probably Approximately Correct (PAC)-Bayesian analysis of variational Bayes (VB) approximations to tempered Bayesian posterior distributions, bounding the model risk of the VB approximations. Tempered posteriors are known to be robust to model misspecification, and their variational approximations do not suffer the usual problems of over confident approximations. Our results tie the risk bounds to the mixing and ergodic properties of the Markov data generating model. We illustrate the PAC-Bayes bounds through a number of example Markov models, and also consider the situation where the Markov model is misspecified.

Diffusions on a space of interval partitions: Poisson–Dirichlet stationary distributions

We introduce diffusions on a space of interval partitions of the unit interval that are stationary with the Poisson–Dirichlet laws with parameters α,0 and α,α. The construction has two steps. The first is a general construction of interval partition processes obtained previously by decorating the jumps of a Lévy process with independent excursions. Here, we focus on the second step which requires explicit transition kernels and, what we call, pseudo-stationarity. This allows us to study processes obtained from the original construction via scaling and time-change. In a sequel paper we establish connections to diffusions on decreasing sequences introduced by Ethier and Kurtz (Adv. in Appl. Probab. 13 (1981) 429–452) and Petrov (Funktsional. Anal. i Prilozhen. 43 (2009) 45–66). The latter diffusions are continuum limits of up-down Markov chains on Chinese restaurant processes. Our construction is also a step toward resolving longstanding conjectures by Feng and Sun on measure-valued Poisson–Dirichlet diffusions and by Aldous on a continuum-tree-valued diffusion.