Compact State Space Research Articles

Duality and the well-posedness of a martingale problem

For two Polish state spaces EX and EY, and an operator GX, we obtain existence and uniqueness of a GX-martingale problem provided there is a bounded continuous duality function H on EX×EY together with a dual process Y on EY which is the unique solution of a GY-martingale problem. For the corresponding solutions (Xt)t≥0 and (Yt)t≥0, duality with respect to a function H in its simplest form means that the relation Ex[H(Xt,y)]=Ey[H(x,Yt)] holds for all (x,y)∈EX×EY and t≥0. While duality is well-known to imply uniqueness of the GX-martingale problem, we give here a set of conditions under which duality also implies existence without using approximating sequences of processes of a different kind (e.g. jump processes to approximate diffusions) which is a widespread strategy for proving existence of solutions of martingale problems. Given the process (Yt)t≥0 and a duality function H, to prove existence of (Xt)t≥0 one has to show that the r.h.s. of the duality relation defines for each y a measure on EX, i.e. there are transition kernels (μt)t≥0 from EX to EX such that Ey[H(x,Yt)]=∫μt(x,dx′)H(x′,y) for all (x,y)∈EX×EY and all t≥0.As examples, we treat resampling and branching models, such as the Fleming–Viot measure-valued diffusion and its spatial counterparts (with both, discrete and continuum space), as well as branching systems, such as Feller’s branching diffusion. While our main result as well as all examples come with (locally) compact state spaces, we discuss the strategy to lift our results to genealogy-valued processes or historical processes, leading to non-compact (discrete and continuum) state spaces. Such applications will be tackled in forthcoming work based on the present article.

Compensation States Approach in the Hybrid Diabatization Scheme: Extension to Multidimensional Data and Properties.

The diabatization of reactive systems for more than just a couple of states is a very demanding problem and generally requires advanced diabatization techniques. Especially for dissociative processes, the drastic changes in the adiabatic wave functions often would require large diabatic state bases, which quickly become impractical. Recently, we addressed this problem by the compensation states approach developed in the context of our hybrid diabatization scheme. This scheme utilizes wave function as well as energy data in combination with a diabatic potential model. In regions where the initial diabatic state basis becomes insufficient for an appropriate representation of the adiabatic states, new model states are generated. The new model states compensate for the state space not spanned by the initial diabatic basis. Such a compensation state is obtained by projecting the initial diabatic state space out of the adiabatic wave function. This yields a very efficient basis representation of the electronic Hamiltonian. The present work presents two new aspects. First, it is shown how other operators like the spin-orbit operator in the framework of the Effective Relativistic Coupling by Asymptotic Representation (ERCAR) can be evaluated in this compact model state space without losing the correct wave function information and accuracy. Second, the extension of the approach to multidimensional potential energy surface models is presented for methyl iodide including the C-I dissociation coordinate and the angular H3C-I bending coordinates.

Dynamics of Interacting Monomial Scalar Field Potentials and Perfect Fluids

AbstractMotivated by cosmological models of the early universe we analyse the dynamics of the Einstein equations with a minimally coupled scalar field with monomial potentials $$V(\phi )=\frac{(\lambda \phi )^{2n}}{2n}$$ V ( ϕ ) = ( λ ϕ ) 2 n 2 n , $$\lambda >0$$ λ > 0 , $$n\in {\mathbb {N}}$$ n ∈ N , interacting with a perfect fluid with linear equation of state $$p_{\textrm{pf}}=(\gamma _{\textrm{pf}}-1)\rho _{\textrm{pf}}$$ p pf = ( γ pf - 1 ) ρ pf , $$\gamma _{\textrm{pf}}\in (0,2)$$ γ pf ∈ ( 0 , 2 ) , in flat Robertson–Walker spacetimes. The interaction is a friction-like term of the form $$\Gamma (\phi )=\mu \phi ^{2p}$$ Γ ( ϕ ) = μ ϕ 2 p , $$\mu >0$$ μ > 0 , $$p\in {\mathbb {N}}\cup \{0\}$$ p ∈ N ∪ { 0 } . The analysis relies on the introduction of a new regular 3-dimensional dynamical systems’ formulation of the Einstein equations on a compact state space, and the use of dynamical systems’ tools such as quasi-homogeneous blow-ups and averaging methods involving a time-dependent perturbation parameter.

A qLPV-MPC Control Strategy for Trajectory Tracking of Quadrotors

This article proposes a model predictive control (MPC) strategy for a quadrotor drone trajectory tracking based on a compact state-space model based on a quasi-linear parameter varying (qLPV) representation of the nonlinear quadrotor. The use of a qLPV representation allows for faster execution times, which can be suitable for real-time applications and for solving the optimization problem using quadratic programming (QP). The estimation of future values of the scheduling parameters along the prediction horizon is made by using the planned trajectory based on the previous optimal control actions. The performance of the proposed approach is tested by following different trajectories in simulation to show the effectiveness of the proposed control scheme.

A Lyapunov approach to stability of positive semigroups: an overview with illustrations

The stability analysis of possibly time varying positive semigroups on non-necessarily compact state spaces, including Neumann and Dirichlet boundary conditions is a notoriously difficult subject. These crucial questions arise in a variety of areas of applied mathematics, including nonlinear filtering, rare event analysis, branching processes, physics and molecular chemistry. This article presents an overview of some recent Lyapunov-based approaches, focusing principally on practical and powerful tools for designing Lyapunov functions. These techniques include semigroup comparisons as well as conjugacy principles on non-necessarily bounded manifolds with locally Lipschitz boundaries. All the Lyapunov methodologies discussed in the article are illustrated in a variety of situations, ranging from conventional Markov semigroups on general state spaces to more sophisticated conditional stochastic processes possibly restricted to some non-necessarily bounded domains, including locally Lipschitz and smooth hypersurface boundaries, Langevin diffusions as well as coupled harmonic oscillators.

Mean Field Markov Decision Processes

We consider mean-field control problems in discrete time with discounted reward, infinite time horizon and compact state and action space. The existence of optimal policies is shown and the limiting mean-field problem is derived when the number of individuals tends to infinity. Moreover, we consider the average reward problem and show that the optimal policy in this mean-field limit is varepsilon -optimal for the discounted problem if the number of individuals is large and the discount factor close to one. This result is very helpful, because it turns out that in the special case when the reward does only depend on the distribution of the individuals, we obtain a very interesting subclass of problems where an average reward optimal policy can be obtained by first computing an optimal measure from a static optimization problem and then achieving it with Markov Chain Monte Carlo methods. We give two applications: Avoiding congestion an a graph and optimal positioning on a market place which we solve explicitly.

Discrete-time zero-sum games for Markov chains with risk-sensitive average cost criterion

We study zero-sum stochastic games for controlled discrete time Markov chains with risk-sensitive average cost criterion with countable/compact state space and Borel action spaces. The payoff function is nonnegative and possibly unbounded for countable state space case and for compact state space case it is a real-valued and bounded function. For countable state space case, under a certain Lyapunov type stability assumption on the dynamics we establish the existence of the value and a saddle point equilibrium. For compact state space case we establish these results without any Lyapunov type stability assumptions. Using the stochastic representation of the principal eigenfunction of the associated optimality equation, we completely characterize all possible saddle point strategies in the class of stationary Markov strategies. Also, we present and analyze an illustrative example.

CVaR Optimization for MDPs

We study the problem of Conditional Value-at-Risk (CVaR) optimization for a finite-state Markov Decision Process (MDP) with total discounted costs and the reduction of this problem to a stochastic game with perfect information. The CVaR optimization problem for finite and infinite-horizon MDPs can be reformulated as a zero-sum stochastic game with a compact state space. This game has the following property: while the second player has perfect information including the knowledge of the decision chosen by the first player at the current time instance, the first player does not directly observe the augmented component of the state and does not know current and past decisions chosen by the second player. Using methods of convex analysis, we show optimal policies exist for this game and an optimal policy of the first player optimizes CVaR of the total discounted costs. In addition to proving existence of optimal policies, we provide algorithms for their computation.

Large deviations for the empirical measure of the zig-zag process

The zig-zag process is a piecewise deterministic Markov process in position and velocity space. The process can be designed to have an arbitrary Gibbs type marginal probability density for its position coordinate, which makes it suitable for Monte Carlo simulation of continuous probability distributions. An important question in assessing the efficiency of this method is how fast the empirical measure converges to the stationary distribution of the process. In this paper we provide a partial answer to this question by characterizing the large deviations of the empirical measure from the stationary distribution. Based on the Feng–Kurtz approach, we develop an abstract framework aimed at encompassing piecewise deterministic Markov processes in position-velocity space. We derive explicit conditions for the zig-zag process to allow the Donsker–Varadhan variational formulation of the rate function, both for a compact setting (the torus) and one-dimensional Euclidean space. Finally we derive an explicit expression for the Donsker–Varadhan functional for the case of a compact state space and use this form of the rate function to address a key question concerning the optimal choice of the switching rate of the zig-zag process.

Markov Chain Abstractions of Electrochemical Reaction-Diffusion in Synaptic Transmission for Neuromorphic Computing.

Progress in computational neuroscience toward understanding brain function is challenged both by the complexity of molecular-scale electrochemical interactions at the level of individual neurons and synapses and the dimensionality of network dynamics across the brain covering a vast range of spatial and temporal scales. Our work abstracts an existing highly detailed, biophysically realistic 3D reaction-diffusion model of a chemical synapse to a compact internal state space representation that maps onto parallel neuromorphic hardware for efficient emulation at a very large scale and offers near-equivalence in input-output dynamics while preserving biologically interpretable tunable parameters.

A critical comparison on attitude estimation: From gaussian approximate filters to coordinate‐free dual optimal control

This paper conveys attitude and rate estimation without rate sensors by performing a critical comparison, validated by extensive simulations. The two dominant approaches to facilitate attitude estimation are based on stochastic and set-membership reasoning. The first one mostly utilizes the commonly known Gaussian-approximate filters, namely the EKF and UKF. Although more conservative, the latter seems to be more promising as it considers the inherent geometric characteristics of the underline compact state space and accounts -- from first principles -- for large model errors. We address the set-theoretic approach from a control point of view, and we show that it can overcome reported deficiencies of the Bayesian architectures related to this problem, leading to coordinate-free optimal filters. Lastly, as an example, we derive a modified predictive filter on the tangent bundle of the special orthogonal group $\mathbb{TSO}(3)$.

A general theory of coexistence and extinction for stochastic ecological communities.

We analyze a general theory for coexistence and extinction of ecological communities that are influenced by stochastic temporal environmental fluctuations. The results apply to discrete time (stochastic difference equations), continuous time (stochastic differential equations), compact and non-compact state spaces and degenerate or non-degenerate noise. In addition, we can also include in the dynamics auxiliary variables that model environmental fluctuations, population structure, eco-environmental feedbacks or other internal or external factors. We are able to significantly generalize the recent discrete time results by Benaim and Schreiber (J Math Biol 79:393-431, 2019) to non-compact state spaces, and we provide stronger persistence and extinction results. The continuous time results by Hening and Nguyen (Ann Appl Probab 28(3):1893-1942, 2018a) are strengthened to include degenerate noise and auxiliary variables. Using the general theory, we work out several examples. In discrete time, we classify the dynamics when there are one or two species, and look at the Ricker model, Log-normally distributed offspring models, lottery models, discrete Lotka-Volterra models as well as models of perennial and annual organisms. For the continuous time setting we explore models with a resource variable, stochastic replicator models, and three dimensional Lotka-Volterra models.

Limit theorems for cloning algorithms

Large deviations for additive path functionals of stochastic processes have attracted significant research interest, in particular in the context of stochastic particle systems and statistical physics. Efficient numerical ‘cloning’ algorithms have been developed to estimate the scaled cumulant generating function, based on importance sampling via cloning of rare event trajectories. So far, attempts to study the convergence properties of these algorithms in continuous time have led to only partial results for particular cases. Adapting previous results from the literature of particle filters and sequential Monte Carlo methods, we establish a first comprehensive and fully rigorous approach to bound systematic and random errors of cloning algorithms in continuous time. To this end we develop a method to compare different algorithms for particular classes of observables, based on the martingale characterization of stochastic processes. Our results apply to a large class of jump processes on compact state space, and do not involve any time discretization in contrast to previous approaches. This provides a robust and rigorous framework that can also be used to evaluate and improve the efficiency of algorithms.

Uniform Property Γ

AbstractWe further examine the concept of uniform property $\Gamma $ for $C^*$-algebras introduced in our joint work with Winter. In addition to obtaining characterisations in the spirit of Dixmier’s work on central sequences in II$_1$ factors, we establish the equivalence of uniform property $\Gamma $, a suitable uniform version of McDuff’s property for $C^*$-algebras, and the existence of complemented partitions of unity for separable nuclear $C^*$-algebras with no finite dimensional representations and a compact (non-empty) tracial state space. As a consequence, for $C^*$-algebras as in the Toms–Winter conjecture, the combination of strict comparison and uniform property $\Gamma $ is equivalent to Jiang–Su stability. We also show how these ideas can be combined with those of Matui–Sato to streamline Winter’s classification by embeddings technique.

Stochastic Approximation With Iterate-Dependent Markov Noise Under Verifiable Conditions in Compact State Space With the Stability of Iterates Not Ensured

This article compiles several aspects of the dynamics of stochastic approximation algorithms with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Markov iterate-dependent</i> noise when the iterates are not known to be stable beforehand. We achieve the same by extending the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">lock-in probability</i> (i.e., the probability of convergence of the iterates to a specific attractor of the limiting ordinary differential equation (o.d.e.) given that the iterates are in its domain of attraction after a sufficiently large number of iterations (say) <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$n_0$</tex-math></inline-formula> ) framework to such recursions. Specifically, with the more restrictive assumption of Markov iterate-dependent noise supported on a bounded subset of the Euclidean space, we give a lower bound for the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">lock-in probability</i> . We use these results to prove almost sure convergence of the iterates to the specified attractor when the iterates satisfy an asymptotic tightness condition. The novelty of our approach is that if the state space of the Markov process is compact, we prove almost sure convergence under much weaker assumptions compared to the work by Andrieu <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> , which solves the general state-space case under much restrictive assumptions by providing sufficient conditions for stability of the iterates. We also extend our single-timescale results to the case where there are two separate recursions over two different timescales. This, in turn, is shown to be useful in analyzing the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">tracking ability</i> of general adaptive algorithms. Additionally, we show that our results can be used to derive a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sample complexity estimate</i> of such recursions, which then can be used for step-size selection.

A Dual Characterization of Observability for Stochastic Systems

This paper is concerned with the definition and characterization of the observability for a continuous-time hidden Markov model where the state evolves as a continuous-time Markov process on a compact state space and the observation process is modeled as nonlinear function of the state corrupted by a Gaussian measurement noise. The main technical tool is based on the recently discovered duality relationship between minimum variance estimation and stochastic optimal control: The observability is defined as a dual of the controllability for a certain backward stochastic differential equation. Based on the dual formulation, a test for observability is presented and related to literature. The proposed duality-based framework allows one to easily relate and compare the linear and the nonlinear systems. A side-by-side summary of this relationship is given in a tabular form (Table 1).

Stability of non-linear filter for deterministic dynamics

&lt;p style='text-indent:20px;'&gt;This papers shows that nonlinear filter in the case of deterministic dynamics is stable with respect to the initial conditions under the conditions that observations are sufficiently rich, both in the context of continuous and discrete time filters. Earlier works on the stability of the nonlinear filters are in the context of stochastic dynamics and assume conditions like compact state space or time independent observation model, whereas we prove filter stability for deterministic dynamics with more general assumptions on the state space and observation process. We give several examples of systems that satisfy these assumptions. We also show that the asymptotic structure of the filtering distribution is related to the dynamical properties of the signal.&lt;/p&gt;

Display Optimization for Vertically Differentiated Locations Under Multinomial Logit Preferences

We introduce a new optimization model, dubbed the display optimization problem, that captures a common aspect of choice behavior, known as the framing bias. In this setting, the objective is to optimize how distinct items (corresponding to products, web links, ads, etc.) are being displayed to a heterogeneous audience, whose choice preferences are influenced by the relative locations of items. Once items are assigned to vertically differentiated locations, customers consider a subset of the items displayed in the most favorable locations before picking an alternative through multinomial logit choice probabilities. The main contribution of this paper is to derive a polynomial-time approximation scheme for the display optimization problem. Our algorithm is based on an approximate dynamic programming formulation that exploits various structural properties to derive a compact state space representation of provably near-optimal item-to-position assignment decisions. As a byproduct, our results improve on existing constant-factor approximations for closely related models and apply to general distributions over consideration sets. We develop the notion of approximate assortments that may be of independent interest and applicable in additional revenue management settings. Lastly, we conduct extensive numerical studies to validate the proposed modeling approach and algorithm. Experiments on a public hotel booking data set demonstrate the superior predictive accuracy of our choice model vis-à-vis the multinomial logit choice model with location bias, proposed in earlier literature. In synthetic computational experiments, our approximation scheme dominates various benchmarks, including natural heuristics—greedy methods, local search, priority rules—and state-of-the-art algorithms developed for closely related models. This paper was accepted by Yinyu Ye, optimization.

Dynamical systems in perturbative scalar field cosmology

We derive a new regular dynamical system on a three-dimensional compact state space describing linear scalar perturbations of spatially flat Robertson–Walker geometries for relativistic models with a minimally coupled scalar field with an exponential potential. This enables us to construct the global solution space, illustrated with figures, where known solutions are shown to reside on special invariant sets. We also use our dynamical systems approach to obtain new results about the comoving and uniform density curvature perturbations. Finally we show how to extend our approach to more general scalar field potentials. This leads to state spaces where the state space of the models with an exponential potential appears as invariant boundary sets, thereby illustrating their role as building blocks in a hierarchy of increasingly complex cosmological models.

Policy Evaluation in Continuous MDPs With Efficient Kernelized Gradient Temporal Difference

We consider policy evaluation in infinite-horizon discounted Markov decision problems with continuous compact state and action spaces. We reformulate this task as a compositional stochastic program with a function-valued decision variable that belongs to a reproducing kernel Hilbert space (RKHS). We approach this problem via a new functional generalization of stochastic quasi-gradient methods operating in tandem with stochastic sparse subspace projections. The result is an extension of gradient temporal difference learning that yields nonlinearly parameterized value function estimates of the solution to the Bellman evaluation equation. We call this method parsimonious kernel gradient temporal difference learning. Our main contribution is a memory-efficient nonparametric stochastic method guaranteed to converge exactly to the Bellman fixed point with probability 1 with attenuating step-sizes under the hypothesis that it belongs to the RKHS. Further, with constant step-sizes and compression budget, we establish mean convergence to a neighborhood and that the value function estimates have finite complexity. In the Mountain Car domain, we observe faster convergence to lower Bellman error solutions than existing approaches with a fraction of the required memory.