Kac Lemma Research Articles

Stochastic [formula omitted]-optimal control via forward and backward sampling

The aim of this work is to present a sampling-based algorithm designed to solve a certain class of stochastic optimal control problems, utilizing forward and backward stochastic differential equations (FBSDEs). Specifically, we address the class of problems in which the running cost of the performance index involves an L1-type minimization problem in terms of the control effort. Such problems are typically called minimum fuel problems in optimal control literature. By means of a nonlinear version of the Feynman–Kac lemma, we obtain a probabilistic representation of the solution to the nonlinear Hamilton–Jacobi–Bellman equation, expressed in the form of a system of decoupled FBSDEs. This system of FBSDEs can be solved by employing linear regression techniques.

Stochastic Differential Games: A Sampling Approach via FBSDEs

The aim of this work is to present a sampling-based algorithm designed to solve various classes of stochastic differential games. The foundation of the proposed approach lies in the formulation of the game solution in terms of a decoupled pair of forward and backward stochastic differential equations (FBSDEs). In light of the nonlinear version of the Feynman–Kac lemma, probabilistic representations of solutions to the nonlinear Hamilton–Jacobi–Isaacs equations that arise for each class are obtained. These representations are in form of decoupled systems of FBSDEs, which may be solved numerically.

Stochastic optimal control via forward and backward stochastic differential equations and importance sampling

The aim of this work is to present a novel sampling-based numerical scheme designed to solve a certain class of stochastic optimal control problems, utilizing forward and backward stochastic differential equations (FBSDEs). By means of a nonlinear version of the Feynman–Kac lemma, we obtain a probabilistic representation of the solution to the nonlinear Hamilton–Jacobi–Bellman equation, expressed in the form of a system of decoupled FBSDEs. This system of FBSDEs can be solved by employing linear regression techniques. The proposed framework relaxes some of the restrictive conditions present in recent sampling based methods within the Linearly Solvable Optimal Control framework, and furthermore addresses problems in which the time horizon is not prespecified. To enhance the efficiency of the proposed scheme when treating more complex nonlinear systems, we then derive an iterative algorithm based on Girsanov’s theorem on the change of measure, which features importance sampling. This scheme is shown to be capable of learning the optimal control without requiring an initial guess.

Generalized Kac lemma for recurrence time in iterated open quantum systems

We consider recurrence to the initial state after repeated actions of a quantum channel. After each iteration a projective measurement is applied to check recurrence. The corresponding return time is known to be an integer for the special case of unital channels, including unitary channels. We prove that for a more general class of quantum channels the expected return time can be given as the inverse of the weight of the initial state in the steady state. This statement is a generalization of the Kac lemma for classical Markov chains.

Nonlinear Stochastic Control and Information Theoretic Dualities: Connections, Interdependencies and Thermodynamic Interpretations

In this paper, we present connections between recent developments on the linearly-solvable stochastic optimal control framework with early work in control theory based on the fundamental dualities between free energy and relative entropy. We extend these connections to nonlinear stochastic systems with non-affine controls by using the generalized version of the Feynman–Kac lemma. We present alternative formulations of the linearly-solvable stochastic optimal control framework and discuss information theoretic and thermodynamic interpretations. On the algorithmic side, we present iterative stochastic optimal control algorithms and applications to nonlinear stochastic systems. We conclude with an overview of the frameworks presented and discuss limitations, differences and future directions.

An Iterative Path Integral Stochastic Optimal Control Approach for Learning Robotic Tasks

AbstractRecent work on path integral stochastic optimal control theory Theodorou et al. (2010a); Theodorou (2011) has shown promising results in planning and control of nonlinear systems in high dimensional state spaces. The path integral control framework relies on the transformation of the nonlinear Hamilton Jacobi Bellman (HJB) partial differential equation (PDE) into a linear PDE and the approximation of its solution via the use of the Feynman Kac lemma. In this work, we are reviewing the generalized version of path integral stochastic optimal control formalism Theodorou et al. (2010a), used for optimal control and planing of stochastic dynamical systems with state dependent control and diffusion matrices. Moreover we present the iterative path integral control approach, the so called Policy Improvement with Path Integrals or (PI2) which is capable of scaling in high dimensional robotic control problems. Furthermore we present a convergence analysis of the proposed algorithm and we apply the proposed framework to a variety of robotic tasks. Finally with the goal to perform locomotion the iterative path integral control is applied for learning nonlinear limit cycle attractors with adjustable land scape.

Pseudochaotic dynamics near global periodicity

In this paper, we study a piecewise linear version of kicked oscillator model: saw-tooth map. A special case of global periodicity, in which every phase point belongs to a periodic orbit, is presented. With few analytic results known for the corresponding map on torus, we numerically investigate transport properties and statistical behavior of Poincaré recurrence time in two cases of deviation from global periodicity. A non-KAM behavior of the system, as well as subdiffusion and superdiffusion, are observed through numerical simulations. Statistics of Poincaré recurrences shows Kac lemma is valid in the system and there is a relation between the transport exponent and the Poincaré recurrence exponent. We also perform careful numerical computation of capacity, information and correlation dimensions of the so-called exceptional set in both cases. Our results show that the fractal dimension of the exceptional set is strictly less than 2 and that the fractal structures are unifractal rather than multifractal.

New perspectives on Ray's theorem for the local times of diffusions

A new global isomorphism theorem is obtained that expresses the local times of transient regular diffusions under $P^{x,y}$, in terms of related Gaussian processes. This theorem immediately gives an explicit description of the local times of diffusions in terms of $0$th order squared Bessel processes similar to that of Eisenbaum and Ray's classical description in terms of certain randomized fourth order squared Bessel processes. The proofs given are very simple. They depend on a new version of Kac's lemma for $h$-transformed Markov processes and employ little more than standard linear algebra. The global isomorphism theorem leads to an elementary proof of the Markov property of the local times of diffusions and to other recent results about the local times of general strongly symmetric Markov processes. The new version of Kac's lemma gives simple, short proofs of Dynkin's isomorphism theorem and an unconditioned isomorphism theorem due to Eisenbaum.

Success or failure of a firm under different financing policies: A dynamic stochastic model

The value of a firm evolves in time, according to a standard Ito's stochastic differential equation, and under different types of financial support. In particular both the cases of legal and illegal financing are considered. The ruin/success probabilities for the firm are evaluated in closed form via special functions in the various settings. Some functionals of the process, which are relevant from the viewpoint of the financiers, (via Kac's lemma) are shown to obey a PDE, which is studied both with analytic tools (second-order ODEs of hypergeometric type) and with numerical ones. The case of insufficient payments is analyzed via stochastic calculus and a new model for loan renegotiation is offered.

Universal Data Compression Algorithm Based on Approximate String Matching

A practical source coding scheme based on approximate string matching is proposed. It is an approximate fixed-length string matching data compression combined with a block-coder based on the empirical distribution. A lemma on approximate string matching, which is an extension of the Kac Lemma, is proved. It is shown, based on the lemma, that the deterministic algorithm converts the stationary and ergodic source, u, into an output process v, and under the assumption that v is a stationary process, after the scheme has run for an infinite time, the optimal compression ratio R(D) is achieved. This reduces the problem of the universal lossy coder to the proof of stationarity of the output process ν in the proposed algorithm. The main advantages of the proposed method are the asymptotic sequential behavior of the encoder and the simplicity of the decoder.

Optimal data compression algorithm

A practical source coding scheme based on approximate string matching is proposed. It is an approximate fixed-length string matching data compression combined with a block-coder based on the empirical distribution. A lemma on approximate string matching, which is an extension of the Kac Lemma, is proved. It is shown, based on the lemma, that the deterministic algorithm converts the stationary and ergodic source u into an output process v, and under the assumption that v is a stationary process, after the scheme has run for an infinite time, the optimal compression ratio R( D) is achieved. This reduces the problem of the universal lossy coder to the proof of stationarity of the output process v in the proposed algorithm. The main advantages of the proposed method are the asymptotic sequential behavior of the encoder and the simplicity of the decoder.