Abstract
Long term average or ‘ergodic’ optimal control problems on a compact manifold are considered. The problems exhibit a special structure which is typical of control problems related to large deviations theory: Control is exerted in all directions and the control costs are proportional to the square of the norm of the control field with respect to the metric induced by the noise. The long term stochastic dynamics on the manifold will be completely characterized by the long term density ρ and the long term current density J. As such, control problems may be reformulated as variational problems over ρ and J. The density ρ is paired in the cost functional with a state dependent cost function V, and the current density J is paired with a vector potential or gauge field A. We discuss several optimization problems: the problem in which both ρ and J are varied freely, the problem in which ρ is fixed and the one in which J is fixed. These problems lead to different kinds of operator problems: linear PDEs in the first two cases and a nonlinear PDE in the latter case. These results are obtained through a variational principle using infinite dimensional Lagrange multipliers. In the case where the initial dynamics are reversible the optimally controlled diffusion is also reversible. The particular case of constraining the dynamics to be reversible of the optimally controlled process leads to a linear eigenvalue problem for the square root of the density process.
Highlights
In this paper we discuss stochastic, long term average optimal, or ‘ergodic’ control problems on compact orientable manifolds, in which control is exerted in all directions, and where the control costs are proportional to the square of the norm of the control field with respect to the metric induced by the noise
Computation of the large deviations of the generalized observables can be reduced to solving an optimization problem, similar in spirit to the discussion in Section 1.1 for the scalar potential above. In this manuscript we demonstrate that the gauge invariant approach, which has shown its capability in the context of stochastic processes can be extended to the stochastic optimal control setting while maintaining the same advantages of (i) considering optimal control of current density, in addition to classical particle density, which is achieved by introducing the gauge invariant extension of the standard Bellman equation, and (ii) formulating the gauge invariant Bellman equation as a solution of an optimization problem that involves a functional of current and particle densities
In this paper we showed how stationary long term average control problems are related to eigenvalue problems, elliptic PDEs or a non-linear eigenvalue problem
Summary
In this paper we discuss stochastic, long term average optimal, or ‘ergodic’ control problems on compact orientable manifolds, in which control is exerted in all directions, and where the control costs are proportional to the square of the norm of the control field with respect to the metric induced by the noise. Even without the inclusion of a vector potential (i.e. taking A = 0), the formulation in terms of current and density provides a new and clear perspective on the optimal control problem in relation to the large deviations theory. Computation of the large deviations of the generalized observables can be reduced to solving an optimization problem, similar in spirit to the discussion in Section 1.1 for the scalar potential above In this manuscript we demonstrate that the gauge invariant approach, which has shown its capability in the context of stochastic processes (as briefly discussed above) can be extended to the stochastic optimal control setting while maintaining the same advantages of (i) considering optimal control of current density, in addition to classical particle density, which is achieved by introducing the gauge invariant extension of the standard Bellman equation, and (ii) formulating the gauge invariant Bellman equation as a solution of an optimization problem that involves a functional of current and particle densities.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have