Bellman Operator Research Articles

Resonance Phenomena for Second-Order Stochastic Control Equations

We study the existence and the properties of solutions to the Dirichlet problem for uniformly elliptic second-order Hamilton–Jacobi–Bellman operators, depending on the principal eigenvalues of the operator.

The Dirichlet problem for the Bellman equation at resonance

We generalize the Donsker–Varadhan minimax formula for the principal eigenvalue of a uniformly elliptic operator in nondivergence form to the first principal half-eigenvalue of a fully nonlinear operator which is concave (or convex) and positively homogeneous. Examples of such operators include the Bellman operator and the Pucci extremal operators. In the case that the two principal half-eigenvalues are not equal, we show that the measures which achieve the minimum in this formula provide a partial characterization of the solvability of the corresponding Dirichlet problem at resonance.

The hardy and bellman operators in spaces connected with H( % MathType!MTEF!2!1!+- % feaagaart1ev2aaatCvAUfKttLearuqr1ngBPrgarmWu51MyVXgatC % vAUfeBSjuyZL2yd9gzLbvyNv2CaeHbd9wDYLwzYbItLDharyavP1wz % ZbItLDhis9wBH5garqqtubsr4rNCHbGeaGqiVu0Je9sqqrpepC0xbb % L8F4rqqrFfpeea0xe9Lq-Jc9vqaqpepm0xbba9pwe9Q8fs0-yqaqpe % pae9pg0FirpepeKkFr0xfr-xfr-xb9adbaqaaeGaciGaaiaabeqaam % aaeaqbaaGcbaWefv3ySLgznfgDOjdarCqr1ngBPrginfgDObcv39ga % iyaacqWFtcpvaaa!46CC! $$ \mathbb{T} $$ ) and BMO( % MathType!MTEF!2!1!+- % feaagaart1ev2aaatCvAUfKttLearuqr1ngBPrgarmWu51MyVXgatC % vAUfeBSjuyZL2yd9gzLbvyNv2CaeHbd9wDYLwzYbItLDharyavP1wz % ZbItLDhis9wBH5garqqtubsr4rNCHbGeaGqiVu0Je9sqqrpepC0xbb % L8F4rqqrFfpeea0xe9Lq-Jc9vqaqpepm0xbba9pwe9Q8fs0-yqaqpe % pae9pg0FirpepeKkFr0xfr-xfr-xb9adbaqaaeGaciGaaiaabeqaam % aaeaqbaaGcbaWefv3ySLgznfgDOjdarCqr1ngBPrginfgDObcv39ga % iyaacqWFtcpvaaa!46CC! $$ \mathbb{T} $$ )

Assume that 1 ≤ p < ∞ and a function f ∈ Lp[0, π] has the Fourier series \( \sum\limits_{n = 1}^\infty {a_n } \) cos nx. According to one result of G.H. Hardy, the series \( \sum\limits_{n = 1}^\infty {n^{ - 1} } \sum\limits_{k = 1}^n {a_k } \) cos nx is the Fourier series for a certain function \( \mathcal{H} \)(f) ∈ Lp[0, π]. But if 1 < p ≤ ∞ and f ∈ Lp[0, π], then the series \( \sum\limits_{n = 1}^\infty {\sum\limits_{k = n}^\infty {k^{ - 1} a_k } } \) cos nx is the Fourier series for a certain function \( \mathcal{B} \)(f) ∈ Lp[0, π]. Similar assertions are true for sine series. This allows one to define the Hardy operator \( \mathcal{H} \) on Lp(\( \mathbb{T} \)), 1 ≤ p < ∞, and to define the Bellman operator \( \mathcal{B} \) on Lp(\( \mathbb{T} \)), 1 < p ≤ ∞. In this paper we prove that the Bellman operator boundedly acts in VMO(\( \mathbb{T} \)), and the Hardy operator also maps a certain subspace C(\( \mathbb{T} \)) onto VMO(\( \mathbb{T} \)). We also prove the invariance of certain classes of functions with given majorants of modules of continuity or best approximations in the spaces H(\( \mathbb{T} \)), L(\( \mathbb{T} \)), VMO(\( \mathbb{T} \)) with respect to the Hardy and Bellman operators.

Non-existence of positive solutions of fully nonlinear elliptic equations in unbounded domains

In this paper we consider fully nonlinear elliptic operators of the form $F(x,u,Du,D^2u)$. Our aim is to prove that, under suitable assumptions on $F$, the only nonnegative viscosity super-solution $u$ of $F(x,u,Du,D^2u)=0$ in an unbounded domain $\Omega$ is $u\equiv 0$. We show that this uniqueness result holds for the class of nonnegative super-solutions $u$ satisfying limin$f_{x\in\Omega, |x|\to\infty}\frac{u(x)+1}{\dist(x,\partial\Omega)}=0,$ and then, in particular, for strictly sublinear super-solutions in a domain $\Omega$ containing an open cone. In the special case that $\Omega=\mathbb R^N$, or that $F$ is the Bellman operator, we show that the same result holds for the whole class of nonnegative super-solutions. Our principal assumption on the operator $F$ involves its zero and first order dependence when $|x|\to\infty$. The same kind of assumption was introduced in a recent paper in collaboration with H. Berestycki and F. Hamel [4] to establish a Liouville type result for semilinear equations. The strategy we follow to prove our main results is the same as in [4], even if here we consider fully nonlinear operators, possibly unbounded solutions and more general domains.

Performance Bounds in $L_p$‐norm for Approximate Value Iteration

Approximate value iteration (AVI) is a method for solving large Markov decision problems by approximating the optimal value function with a sequence of value function representations $V_n$ processed according to the iterations $V_{n+1}=\mathcal{AT} V_n$, where $\mathcal{T}$ is the so-called Bellman operator and $\mathcal{A}$ an approximation operator, which may be implemented by a supervised learning (SL) algorithm. Usual bounds on the asymptotic performance of AVI are established in terms of the $L_\infty$-norm approximation errors induced by the SL algorithm. However, most widely used SL algorithms (such as least squares regression) return a function (the best fit) that minimizes an empirical approximation error in $L_p$-norm ($p\geq 1$). In this paper, we extend the performance bounds of AVI to weighted $L_p$-norms, which enables us to directly relate the performance of AVI to the approximation power of the SL algorithm, hence assuring the tightness and practical relevance of these bounds. The main result is a performance bound of the resulting policies expressed in terms of the $L_p$-norm errors introduced by the successive approximations. The new bound takes into account a concentration coefficient that estimates how much the discounted future-state distributions starting from a probability measure used to assess the performance of AVI can possibly differ from the distribution used in the regression operation. We illustrate the tightness of the bounds on an optimal replacement problem.

Controlled risk processes in discrete time: Lower and upper approximations to the optimal probability of ruin

We consider the controlled random walk as a flexible model of risk process which takes into account the possibility of intervention of the insurer. We derive a method of approximating the optimal probability of ruin from below and from above. The approximations are based on iterations of the Bellman operator. To initialize the sequence of upper approximations we can use the Lundberg bound or another upper bound on the probability of ruin.

Existence and Uniqueness of Solutions to the Bellman Equation in the Unbounded Case

We study the problem of the existence and uniqueness of solutions to the Bellman equation in the presence of unbounded returns. We introduce a new approach based both on consideration of a metric on the space of all continuous functions over the state space, and on the application of some metric fixed point theorems. With appropriate conditions we prove uniqueness of solutions with respect to the whole space of continuous functions. Furthermore, the paper provides new sufficient conditions for the existence of solutions that can be applied to fairly general models. It is also proven that the fixed point coincides with the value function and that it can be approached by successive iterations of the Bellman operator.

Hardy and Bellman Transformations in Spaces Close to L∞ and L1

The replacement of coefficients of a trigonometric series by their arithmetic averages gives rise to the Hardy operator. The Bellman operator is its adjoint. The spaces Lp with p∈[1,∞) are invariant under the Hardy transformation. This result was proved by Hardy. On the other hand, the space L∞ is not invariant under the Hardy transformation, and the space L1 is not invariant under the Bellman transformation. Golubov proved that the space BMO is not invariant under the Hardy transformation and \({\text{Re}}\;^{\text{ + }} \;H\) is not invariant under the Bellman operator. In the present paper, the exact ``shift'' of the domain under the action of these operators is described for certain Orlicz, Lorenz, and Marcinkiewicz spaces and the spaces BMO and \({\text{Re}}\;^{\text{ + }} \;H\). For the Hardy operator, this shift occurs if the domain is close to L∞, and for the Bellman operator the same happens if the domain is close to L1. Bibliography: 15 titles.

On boundedness of the hardy and bellman operators in the spaces h and bmo

Hardy (1919) proved that the space Lp(T), 1 ⩽ p ⩽ ∞, is invariant under (C, 1)-transformation of Fourier coefficients. This transformation may be considered as a linear integral operator H: Lp(T) → Lp(T), 1 ⩽ p ⩽ ∞. In this paper we show that the operator H is unbounded in the space BMO(T) of periodic functions of bounded mean oscillation. For the conjugate operator B introduced by Bellman (1944) B: Lq(T) → Lq(T), 1/ρp + 1/rho;q = 1, the boundedness of B in BMO(T) is proved directly without duality argument. An example is given to show that B is unbounded in H(T).

AUBRY–MATHER THEORY AND IDEMPOTENT EIGENFUNCTIONS OF BELLMAN OPERATOR

We study the connection between Aubry–Mather theory of invariant sets of a dynamical system described by a Lagrangian L(t,x,v) = L0(v) - U(t,x) with periodic potential U(t,x), on the one hand, and idempotent spectral theory of Bellman operator of the corresponding optimization problem, on the other hand. This connection is applied to obtain a uniqueness result for an eigenfunction of Bellman operator in the case of irrational rotation number.

Thompson metric, contraction property and differentiability of policy functions

A new proof of Santos' theorem on the differentiability of policy functions arising in a general discrete-time, Ramsey-type model of capital accumulation is given. The proof is based on the contraction property of the second-order Bellman operator with respect to Thompson's metric.