Bellman Operator Research Articles

On recursive utilities with non-affine aggregator and conditional certainty equivalent

In this paper, we consider the problem of the existence and the uniqueness of a recursive utility function defined on intertemporal lotteries. The purpose of this paper is to provide the results of the existence and the uniqueness of a recursive utility function. The utility function is obtained as the limit of iterations on a nonlinear operator and is independent on initial starting points, with iterations converging at an exponential rate. We also find the maximum utility and an optimal strategy by means of iterations of the Bellman operator.

Весовые неравенства для потенциала Данкля–Рисса

Для классического потенциала Рисса или дробного интеграла $$I_{\alpha}$$ хорошо известны условия Харди-Литлвуда-Соболева-Стейна-Вейса $$(L^p, L^q)$$ -ограниченности со степенными весами. С помощью преобразования Фурье $$\mathcal{F}$$ потенциал Рисса определяется равенством $$\mathcal{F}(I_{\alpha}f)(y)=|y|^{-\alpha}\mathcal{F}(f)(y)$$. Важным обобщением преобразования Фурье стало преобразование Данкля $$\mathcal{F}_k$$, действующее в лебеговых пространствах с весом Данкля, определяемым с помощью системы корней $$R\subset\mathbb{R}^d$$, ее группы отражений G и неотрицательной функции кратности k на R, инвариантной относительно G. С. Тангавелу и Ю. Шу с помощью равенства $$\mathcal{F}_k(I_{\alpha}f)(y)=|y|^{-\alpha}\mathcal{F}_k(f)(y)$$ определили D-потенциал Рисса. Для D-потенциала Рисса также были доказаны условия ограниченности в лебеговых пространствах с весом Данкля и степенными весами, аналогичные условиям для потенциала Рисса. На конференции "Follow-up Approximation Theory and Function Spaces" в Centre de Recerca Matem`atica (CRM, Barcelona, 2017) М.Л. Гольдман поставил вопрос об условиях $$(L_p,L_q)$$-ограниченности D-потенциала Рисса с кусочно-степенными весами. Рассмотрение кусочно-степенных весов позволяет выявить влияние на ограниченность D-потенциала Рисса поведения весов в нуле и бесконечности. В настоящей работе на этот вопрос дается полный ответ. В частности,в случае потенциала Рисса получены необходимые и достаточные условия. В качестве вспомогательных результатов доказаны необходимые и достаточные условия ограниченности операторов Харди и Беллмана в лебеговых пространствах с весом Данкля и кусочно-степенными весами.

A Universal Empirical Dynamic Programming Algorithm for Continuous State MDPs

We propose universal randomized function approximation-based empirical value learning (EVL) algorithms for Markov decision processes. The “empirical” nature comes from each iteration being done empirically from samples available from simulations of the next state. This makes the Bellman operator a random operator. A parametric and a nonparametric method for function approximation using a parametric function space and a reproducing kernel Hilbert space respectively are then combined with EVL. Both function spaces have the universal function approximation property. Basis functions are picked randomly. Convergence analysis is performed using a random operator framework with techniques from the theory of stochastic dominance. Finite time sample complexity bounds are derived for both universal approximate dynamic programming algorithms. Numerical experiments support the versatility and computational tractability of this approach.

The Dual Approach to Recursive Optimization: Theory and Examples

We develop a recursive dual method for solving dynamic economic problems. The method uses a Lagrangian to pair a dynamic recursive economic problem with a dual problem. We show that such dual problems can be recursively decomposed with costates (i.e., Lagrange multipliers on laws of motion) functioning as state variables. In dynamic contracting and policy settings, the method often replaces an endogenous state space of forward†looking utilities with an exogenously given state space of costates. We provide a principle of optimality for dual problems and give conditions under which the dual Bellman operator is a contraction with the optimal dual value function its unique fixed point. We relate economic problems to their duals, address computational issues, and give examples.

Convex dynamic programming with (bounded) recursive utility

We consider convex dynamic programs with general (bounded) recursive utilities. The Contraction Mapping Theorem fails when the utility aggregator does not obey any discounting property. This failure occurs even with traditional aggregators and certainty equivalent specifications. However, the Bellman operator admits a unique fixed point when an interior policy is feasible. This happens because utility values are unique at interior consumption plans and, when an interior perturbation is feasible, drops in utility values can be avoided.

Symbolic optimal expected time reachability computation and controller synthesis for probabilistic timed automata

In this paper we consider the problem of computing the optimal (minimum or maximum) expected time to reach a target and the synthesis of an optimal controller for a probabilistic timed automaton (PTA). Although this problem admits solutions that employ the digital clocks abstraction or statistical model checking, symbolic methods based on zones and priced zones fail due to the difficulty of incorporating probabilistic branching in the context of dense time. We work in a generalisation of the setting introduced by Asarin and Maler for the corresponding problem for timed automata, where simple and nice functions are introduced to ensure finiteness of the dense-time representation. We find restrictions sufficient for value iteration to converge to the optimal expected time on the uncountable Markov decision process representing the semantics of a PTA. We formulate Bellman operators on the backwards zone graph of a PTA and prove that value iteration using these operators equals that computed over the PTA's semantics. This enables us to extract an ε-optimal controller from value iteration in the standard way.

Infinite-horizon deterministic dynamic programming in discrete time: a monotone convergence principle and a penalty method

We consider infinite-horizon deterministic dynamic programming problems in discrete time. We show that the value function of such a problem is always a fixed point of a modified version of the Bellman operator. We also show that value iteration converges increasingly to the value function if the initial function is dominated by the value function, is mapped upward by the modified Bellman operator and satisfies a transversality-like condition. These results require no assumption except for the general framework of infinite-horizon deterministic dynamic programming. As an application, we show that the value function can be approximated by computing the value function of an unconstrained version of the problem with the constraint replaced by a penalty function.

Bridging the Gap Between Imitation Learning and Inverse Reinforcement Learning.

Learning from demonstrations is a paradigm by which an apprentice agent learns a control policy for a dynamic environment by observing demonstrations delivered by an expert agent. It is usually implemented as either imitation learning (IL) or inverse reinforcement learning (IRL) in the literature. On the one hand, IRL is a paradigm relying on the Markov decision processes, where the goal of the apprentice agent is to find a reward function from the expert demonstrations that could explain the expert behavior. On the other hand, IL consists in directly generalizing the expert strategy, observed in the demonstrations, to unvisited states (and it is therefore close to classification, when there is a finite set of possible decisions). While these two visions are often considered as opposite to each other, the purpose of this paper is to exhibit a formal link between these approaches from which new algorithms can be derived. We show that IL and IRL can be redefined in a way that they are equivalent, in the sense that there exists an explicit bijective operator (namely, the inverse optimal Bellman operator) between their respective spaces of solutions. To do so, we introduce the set-policy framework that creates a clear link between the IL and the IRL. As a result, the IL and IRL solutions making the best of both worlds are obtained. In addition, it is a unifying framework from which existing IL and IRL algorithms can be derived and which opens the way for the IL methods able to deal with the environment's dynamics. Finally, the IRL algorithms derived from the set-policy framework are compared with the algorithms belonging to the more common trajectory-matching family. Experiments demonstrate that the set-policy-based algorithms outperform both the standard IRL and IL ones and result in more robust solutions.

Empirical Dynamic Programming

We propose empirical dynamic programming algorithms for Markov decision processes. In these algorithms, the exact expectation in the Bellman operator in classical value iteration is replaced by an empirical estimate to get “empirical value iteration” (EVI). Policy evaluation and policy improvement in classical policy iteration are also replaced by simulation to get “empirical policy iteration” (EPI). Thus, these empirical dynamic programming algorithms involve iteration of a random operator, the empirical Bellman operator. We introduce notions of probabilistic fixed points for such random monotone operators. We develop a stochastic dominance framework for convergence analysis of such operators. We then use this to give sample complexity bounds for both EVI and EPI. We then provide various variations and extensions to asynchronous empirical dynamic programming, the minimax empirical dynamic program, and show how this can also be used to solve the dynamic newsvendor problem. Preliminary experimental results suggest a faster rate of convergence than stochastic approximation algorithms.

Dynamic Principal-Agent Models

This paper contributes to the theoretical and numerical analysis of discrete time dynamic principal-agent problems with continuous choice sets. We first provide a new and simplified proof for the recursive reformulation of the sequential dynamic principal-agent relationship. Next we prove the existence of a unique solution for the principal's value function, which solves the dynamic programming problem in the recursive formulation. By showing that the Bellman operator is a contraction mapping, we also obtain a convergence result for the value function iteration. To compute a solution for the problem, we have to solve a collection of static principal{agent problems at each iteration. Under the assumption that the agent's expected utility is a rational function of his action, we can transform the bi-level optimization problem into a standard nonlinear program. The final results of our solution method are numerical approximations of the policy and value functions for the dynamic principal-agent model. We illustrate our solution method by solving variations of two prominent social planning models from the economics literature.

Increasing the Action Gap: New Operators for Reinforcement Learning

This paper introduces new optimality-preserving operators on Q-functions. We first describe an operator for tabular representations, the consistent Bellman operator, which incorporates a notion of local policy consistency. We show that this local consistency leads to an increase in the action gap at each state; increasing this gap, we argue, mitigates the undesirable effects of approximation and estimation errors on the induced greedy policies. This operator can also be applied to discretized continuous space and time problems, and we provide empirical results evidencing superior performance in this context. Extending the idea of a locally consistent operator, we then derive sufficient conditions for an operator to preserve optimality, leading to a family of operators which includes our consistent Bellman operator. As corollaries we provide a proof of optimality for Baird's advantage learning algorithm and derive other gap-increasing operators with interesting properties. We conclude with an empirical study on 60 Atari 2600 games illustrating the strong potential of these new operators.

An application of Kleene's fixed point theorem to dynamic programming

AbstractWe show that the least fixed point of the Bellman operator in a certain set can be computed by value iteration whether or not the fixed point is the value function. As an application, we give a simpler proof of one of Kamihigashi's main results.

Study of the Bellman equation in a production model with unstable demand

A production model with allowance for a working capital deficit and a restricted maximum possible sales volume is proposed and analyzed. The study is motivated by the urgency of analyzing well-known problems of functioning low competitive macroeconomic structures. The original formulation of the task represents an infinite-horizon optimal control problem. As a result, the model is formalized in the form of a Bellman equation. It is proved that the corresponding Bellman operator is a contraction and has a unique fixed point in the chosen class of functions. A closed-form solution of the Bellman equation is found using the method of steps. The influence of the credit interest rate on the firm market value assessment is analyzed by applying the developed model.

More on operator Bellman inequality

We present a Bellman inequality involving operator means for operators acting on a Hilbert space. We also give some Bellman inequalities concerning sesquilinear forms. Finally, we refine the Jensen's operator inequality and use it for obtaining a refinement of the Bellman operator inequality.

An order-theoretic approach to dynamic programming: an exposition

In this note, we discuss an order-theoretic approach to dynamic programming. In particular, we explain how order-theoretic fixed point theorems can be used to establish the existence of a fixed point of the Bellman operator, as well as why they are not sufficient to characterize the value function. By doing this, we present the logic behind the simple yet useful result recently obtained by Kamihigashi (Econ Theory, 2013) based on this order-theoretic approach.

Continuous dependence estimates for the ergodic problem of Bellman equation with an application to the rate of convergence for the homogenization problem

This paper is devoted to establish continuous dependence estimates for the ergodic problem for Bellman operators (namely, estimates of $$(v_1-v_2)$$ where $$v_1$$ and $$v_2$$ solve two equations with different coefficients). We shall obtain an estimate of $$\Vert v_1-v_2\Vert _\infty $$ with an explicit dependence on the $$L^\infty $$ -distance between the coefficients and an explicit characterization of the constants and also, under some regularity conditions, an estimate of $$\Vert v_1-v_2\Vert _{C^2({\mathbb {R}}^n)}$$ . Afterwards, the former result will be crucial in the estimate of the rate of convergence for the homogenization of Bellman equations. In some regular cases, we shall obtain the same rate of convergence established in the monographs by Bensoussan et al. (Asymptotic analysis for periodic structures, North-Holland, Amsterdam, 1978) and by Jikov et al. (Homogenization of differential operators and integral functionals, Springer, Berlin, 1994) for regular linear problems.

An efficient L2-norm regularized least-squares temporal difference learning algorithm

In reinforcement learning, when samples are limited in some real applications, Least-Squares Temporal Difference (LSTD) learning is prone to over-fitting, which can be overcome by the introduction of regularization. However, the solution of LSTD with regularization still depends on costly matrix inversion operations. In this paper we investigate the L2-norm regularized LSTD learning and propose an efficient algorithm to avoid expensive computational cost. We derive LSTD using Bellman operator along with projection operator. The L2-norm penalty is introduced to avoid over-fitting. We also describe the difference between Bellman residual minimization and LSTD. Then we propose an efficient recursive least-squares algorithm for L2-norm regularized LSTD, which can eliminate matrix inversion operations and decrease computational complexity effectively. We present empirical comparisons on the Boyan chain problem. The results show that the performance of the new algorithm is better than that of regularized LSTD.

A Fixed Point Theorem and an Application to Bellman Operators

This study introduces a new definition of a metric that corresponds with the topology of uniform convergence on any compact set, and shows both the existence of a unique fixed point of some operator by using this metric and that the iteration of such an operator results in convergence to this fixed point. We demonstrate that this result can be applied to Bellman operators in many situations involving economic dynamics.

Fitted value function iteration with probability one contractions

This paper studies a value function iteration algorithm based on nonexpansive function approximation and Monte Carlo integration that can be applied to almost all stationary dynamic programming problems. The method can be represented using a randomized fitted Bellman operator and a corresponding algorithm that is shown to be globally convergent with probability one. When additional restrictions are imposed, an OP(n−1/2) rate of convergence for Monte Carlo error is obtained.

Turnpike Theorems for Markov Games

This paper has a two-folded purpose. First, we attempt to outline the development of the turnpike theorems in the the last several decades. Second, we study turnpike theorems in finite-horizon two-person zero-sum Markov games on a general Borel state space. Utilising the Bellman (or Shapley) operator defined for this game, we prove the stochastic versions of the early turnpike theorem on the set of optimal strategies and the middle turnpike theorem on the distribution of the state space.