Bellman Operator Research Articles

Kernel-Based Reinforcement Learning on Representative States

Markov decision processes (MDPs) are an established framework for solving sequential decision-making problems under uncertainty. In this work, we propose a new method for batch-mode reinforcement learning (RL) with continuous state variables. The method is an approximation to kernel-based RL on a set of k representative states. Similarly to kernel-based RL, our solution is a fixed point of a kernelized Bellman operator and can approximate the optimal solution to an arbitrary level of granularity. Unlike kernel-based RL, our method is fast. In particular, our policies can be computed in O(n) time, where n is the number of training examples. The time complexity of kernel-based RL is Ω(n2). We introduce our method, analyze its convergence, and compare it to existing work. The method is evaluated on two existing control problems with 2 to 4 continuous variables and a new problem with 64 variables. In all cases, we outperform state-of-the-art results and offer simpler solutions.

Monotonicity of solutions for a class of uniformly elliptic nonlocal Bellman systems

This paper is concerned with a class of uniformly elliptic nonlocal Bellman systems in unbounded Lipschitz domains. There are two major ingredients. The first is a narrow region principle in unbounded domains. We can use this to carry out the direct method of moving planes. The second ingredient is to prove the monotonicity of positive bounded solutions for a cooperative system with uniformly elliptic nonlocal Bellman operators. Without decay assumptions on solutions, we estimate the singular integrals defining the operator along a sequence of approximate minimum points.

Distributional Reinforcement Learning via Moment Matching

We consider the problem of learning a set of probability distributions from the empirical Bellman dynamics in distributional reinforcement learning (RL), a class of state-of-the-art methods that estimate the distribution, as opposed to only the expectation, of the total return. We formulate a method that learns a finite set of statistics from each return distribution via neural networks, as in the distributional RL literature. Existing distributional RL methods however constrain the learned statistics to predefined functional forms of the return distribution which is both restrictive in representation and difficult in maintaining the predefined statistics. Instead, we learn unrestricted statistics, i.e., deterministic (pseudo-)samples, of the return distribution by leveraging a technique from hypothesis testing known as maximum mean discrepancy (MMD), which leads to a simpler objective amenable to backpropagation. Our method can be interpreted as implicitly matching all orders of moments between a return distribution and its Bellman target. We establish sufficient conditions for the contraction of the distributional Bellman operator and provide finite-sample analysis for the deterministic samples in distribution approximation. Experiments on the suite of Atari games show that our method outperforms the standard distributional RL baselines and sets a new record in the Atari games for non-distributed agents.

Bounding fixed points of set-based Bellman operator and Nash equilibria of stochastic games

Motivated by uncertain parameters encountered in Markov decision processes (MDPs) and stochastic games, we study the effect of parameter-uncertainty on Bellman operator-based algorithms under a set-based framework. Specifically, we first consider a family of MDPs where the cost parameters are in a given compact set; we then define a Bellman operator acting on a set of value functions to produce a new set of value functions as the output under all possible variations in the cost parameter. We prove the existence of a fixed point of this set-based Bellman operator by showing that the operator is contractive on a complete metric space, and explore its relationship with the corresponding family of MDPs and stochastic games. Additionally, we show that given interval set-bounded cost parameters, we can form exact bounds on the set of optimal value functions. Finally, we utilize our results to bound the value function trajectory of a player in a stochastic game.

Enhanced Exploration Least-Squares Methods for Optimal Stopping Problems

This letter presents an Approximate Dynamic Programming (ADP) least-squares based approach for solving optimal stopping problems with a large state space. By extending some previous work in the area of optimal stopping problems, it provides a framework for their formulation and resolution. The proposed method uses a combined on/off policy exploration mechanism, where states are generated by means of state transition probability distributions different from the ones dictated by the underlying Markov decision processes. The contraction mapping property of the associated projected Bellman operator is analysed as well as the convergence of the resulting algorithm.

On History-Dependent Optimization Models: A Unified Framework to Analyze Models with Habits, Satiation and Optimal Growth

In this paper, we first provide a framework for history-dependent utility models. We further consider discrete infinite horizon dynamic optimization programs in which the instantaneous payoff presents such historydependence. Habits are a particular case of history-dependence. An issue about models with habits is their lack of general framework. Our framework allows to study habit models that are either additive or multiplicative or neither, as well as satiation models. Moreover, with this unified setting one can treat the usual optimal growth models with or without habit formation and with or without satiation effects. As the way the history dependence is formalized allows us to use dynamic programming tools. We show that the value function is the unique fixed point of the Bellman operator. Such history-dependent modelizations have their motivations and applications in many areas among which decision theory, psychology, behavioral and environmental economics.

Monotonicity of solutions for the uniformly elliptic nonlocal Bellman equation on the upper half space

In this paper, we develop a sliding method for the uniformly elliptic nonlocal Bellman operator. We first establish a maximum principle on an upper half space, which plays an important role in the sliding method. Then we consider equations involving the uniformly elliptic nonlocal Bellman operator on the upper half space, and prove the monotonicity and uniqueness of solutions for the uniformly elliptic nonlocal Bellman equation. We use a new idea-estimating the singular integrals defining the operator along a sequence of approximate maximum points.

A dynamic programming framework for optimal delivery time slot pricing

We study the dynamic programming approach to revenue management in the context of attended home delivery. We draw on results from dynamic programming theory for Markov decision problems to show that the underlying Bellman operator has a unique fixed point. We then provide a closed-form expression for the resulting fixed point and show that it admits a natural interpretation. Moreover, we also show that – under certain technical assumptions – the value function, which has a discrete domain and a continuous codomain, admits a continuous extension, which is a finite-valued, concave function of its state variables, at every time step. Furthermore, we derive results on the monotonicity of prices with respect to the number of orders placed in our setting. These results open the road for achieving scalable implementations of the proposed formulation, as it allows making informed choices of basis functions in an approximate dynamic programming context. We illustrate our findings on a low-dimensional and an industry-sized numerical example using real-world data, for which we derive an approximately optimal pricing policy based on our theoretical results.

Multiagent value iteration algorithms in dynamic programming and reinforcement learning

We consider infinite horizon dynamic programming problems, where the control at each stage consists of several distinct decisions, each one made by one of several agents. In an earlier work we introduced a policy iteration algorithm, where the policy improvement is done one-agent-at-a-time in a given order, with knowledge of the choices of the preceding agents in the order. As a result, the amount of computation for each policy improvement grows linearly with the number of agents, as opposed to exponentially for the standard all-agents-at-once method. For the case of a finite-state discounted problem, we showed convergence to an agent-by-agent optimal policy. In this paper, this result is extended to value iteration and optimistic versions of policy iteration, as well as to more general DP problems where the Bellman operator is a contraction mapping, such as stochastic shortest path problems with all policies being proper.

Solving dynamic discrete choice models using smoothing and sieve methods

We propose to combine smoothing, simulations and sieve approximations to solve for either the integrated or expected value function in a general class of dynamic discrete choice (DDC) models. We use importance sampling to approximate the Bellman operators defining the two functions. The random Bellman operators, and therefore also the corresponding solutions, are generally non-smooth which is undesirable. To circumvent this issue, we introduce smoothed versions of the random Bellman operators and solve for the corresponding smoothed value functions using sieve methods. We also show that one can avoid using sieves by generalizing and adapting the “self-approximating” method of Rust (1997b) to our setting. We provide an asymptotic theory for both approximate solution methods and show that they converge with N-rate, where N is number of Monte Carlo draws, towards Gaussian processes. We examine their performance in practice through a set of numerical experiments and find that both methods perform well with the sieve method being particularly attractive in terms of computational speed and accuracy.

The rationale for the use of response functions in formalizing the tasks of economic management of the region

The paper presents studies of linear models of economic dynamics of the Neumann-Gale type, taking into account their possible stationarity, presents an analysis of existing classification approaches to the concept of optimality, presents their advantages and comparative characteristics, it is noted that the first type model open connects the concept of optimality with discounted maximization total utility. The first considers a closed system, the technological description of which includes the reproduction of all the resources necessary for development, including labor. Such a system has no external goals; its natural end in itself is development at the maximum pace. This is the most abstract and idealized scheme, but on the other hand it was it that made it possible to develop such fundamental concepts as equilibrium, a ray of (Neumann) balanced growth. Later, the apparatus of the closed model was replenished with the concepts of “direct and inverse Bellman operators”, “effective functional” (“potential”) of the model, etc. The second approach involves explicit accounting for consumption. Here the description becomes open, consumption is derived from the "technology" and described using the utility function. A new approach to the concept of “optimal development strategy” is proposed, a detailed analysis of the corresponding model is given. The article consists of three sections. 1 - staging part; 2 - analysis of the model with illustrative examples; 3 - conjugate (dual) model. The last section contains the main result on the connection of the optimal trajectories of the direct and dual problems. The paper provides an overview of literary sources in the subject area, as well as an economic interpretation of the results.

Dynamic development model using a temporary consumer scale

The paper presents studies of linear models of economic dynamics of the Neumann-Gale type, taking into account their possible stationarity, presents an analysis of existing classification approaches to the concept of optimality, presents their advantages and comparative characteristics, it is noted that the first type model - open - connects the concept of optimality with discounted maximization total utility. The first considers a closed system, the technological description of which includes the reproduction of all the resources necessary for development, including labor. Such a system has no external goals; its natural end in itself is development at the maximum pace. This is the most abstract and idealized scheme, but on the other hand it was it that made it possible to develop such fundamental concepts as equilibrium, a ray of (Neumann) balanced growth. Later, the apparatus of the closed model was replenished with the concepts of “direct and inverse Bellman operators”, “effective functional” (“potential”) of the model, etc. The second approach involves explicit accounting for consumption. Here the description becomes open, consumption is derived from the "technology" and described using the utility function. A new approach to the concept of “optimal development strategy” is proposed, a detailed analysis of the corresponding model is given. The article consists of three sections. 1 - staging part; 2 - analysis of the model with illustrative examples; 3 - conjugate (dual) model. The last section contains the main result on the connection of the optimal trajectories of the direct and dual problems. The paper provides an overview of literary sources in the subject area, as well as an economic interpretation of the results.

A Convex Optimization Approach to Dynamic Programming in Continuous State and Action Spaces

In this paper, a convex optimization-based method is proposed for numerically solving dynamic programs in continuous state and action spaces. The key idea is to approximate the output of the Bellman operator at a particular state by the optimal value of a convex program. The approximate Bellman operator has a computational advantage because it involves a convex optimization problem in the case of control-affine systems and convex costs. Using this feature, we propose a simple dynamic programming algorithm to evaluate the approximate value function at pre-specified grid points by solving convex optimization problems in each iteration. We show that the proposed method approximates the optimal value function with a uniform convergence property in the case of convex optimal value functions. We also propose an interpolation-free design method for a control policy, of which performance converges uniformly to the optimum as the grid resolution becomes finer. When a nonlinear control-affine system is considered, the convex optimization approach provides an approximate policy with a provable suboptimality bound. For general cases, the proposed convex formulation of dynamic programming operators can be modified as a nonconvex bi-level program, in which the inner problem is a linear program, without losing uniform convergence properties.

Maximum principles and the method of moving planes for the uniformly elliptic nonlocal Bellman operator and applications

In this paper, we establish various maximum principles and develop the method of moving planes and the sliding method (on general unbounded domains) for equations involving the uniformly elliptic nonlocal Bellman operator. As a consequence, we derive multiple applications of these maximum principles and the moving planes method. For instance, we prove symmetry, monotonicity and uniqueness results and asymptotic properties for solutions to various equations involving the uniformly elliptic nonlocal Bellman operator in bounded domains, unbounded domains, epigraph or $\mathbb{R}^{n}$. In particular, the uniformly elliptic nonlocal Monge-Amp\`{e}re operator introduced by Caffarelli and Charro in \cite{CC} is a typical example of the uniformly elliptic nonlocal Bellman operator.

Wasserstein Distributionally Robust Stochastic Control: A Data-Driven Approach

Standard stochastic control methods assume that the probability distribution of uncertain variables is available. Unfortunately, in practice, obtaining accurate distribution information is a challenging task. To resolve this issue, in this article we investigate the problem of designing a control policy that is robust against errors in the empirical distribution obtained from data. This problem can be formulated as a two-player zero-sum dynamic game problem, where the action space of the adversarial player is a Wasserstein ball centered at the empirical distribution. A dynamic programming solution is provided exploiting the reformulation techniques for Wasserstein distributionally robust optimization. We show that the contraction property of associated Bellman operators extends a single-stage out-of-sample performance guarantee, obtained using a measure concentration inequality, to the corresponding multistage guarantee without any degradation in the confidence level. Furthermore, we characterize an explicit form of the optimal control policy and the worst-case distribution policy for linear-quadratic problems with Wasserstein penalty.

Generalized Speedy Q-Learning

In this paper, we derive a generalization of the Speedy Q-learning (SQL) algorithm that was proposed in the Reinforcement Learning (RL) literature to handle slow convergence of Watkins' Q-learning. In most RL algorithms such as Q-learning, the Bellman equation and the Bellman operator play an important role. It is possible to generalize the Bellman operator using the technique of successive relaxation. We use the generalized Bellman operator to derive a simple and efficient family of algorithms called Generalized Speedy Q-learning (GSQL-w) and analyze its finite time performance. We show that GSQL-w has an improved finite time performance bound compared to SQL for the case when the relaxation parameter w is greater than 1. This improvement is a consequence of the contraction factor of the generalized Bellman operator being less than that of the standard Bellman operator. Numerical experiments are provided to demonstrate the empirical performance of the GSQL-w algorithm.

Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy

While maximizing expected return is the goal in most reinforcement learning approaches, risk-sensitive objectives such as conditional value at risk (CVaR) are more suitable for many high-stakes applications. However, relatively little is known about how to explore to quickly learn policies with good CVaR. In this paper, we present the first algorithm for sample-efficient learning of CVaR-optimal policies in Markov decision processes based on the optimism in the face of uncertainty principle. This method relies on a novel optimistic version of the distributional Bellman operator that moves probability mass from the lower to the upper tail of the return distribution. We prove asymptotic convergence and optimism of this operator for the tabular policy evaluation case. We further demonstrate that our algorithm finds CVaR-optimal policies substantially faster than existing baselines in several simulated environments with discrete and continuous state spaces.

Reversing Bellman operator inequality

Reversing Bellman operator inequality

Generalized Dual Dynamic Programming for Infinite Horizon Problems in Continuous State and Action Spaces

We describe a nonlinear generalization of dual dynamic programming (DP) theory and its application to value function estimation for deterministic control problems over continuous state and action spaces, in a discrete-time infinite horizon setting. We prove, using a Benders-type argument leveraging the monotonicity of the Bellman operator, that the result of a one-stage policy evaluation can be used to produce nonlinear lower bounds on the optimal value function that are valid over the entire state space. These bounds contain terms reflecting the functional form of the system's costs, dynamics, and constraints. We provide an iterative algorithm that produces successively better approximations of the optimal value function, and prove under certain assumptions that it achieves an arbitrarily low desired Bellman optimality tolerance at preselected points in the state space, in a finite number of iterations. We also describe means of certifying the quality of the approximate value function generated. We demonstrate the efficacy of the approach on systems whose dimensions are too large for conventional DP approaches to be practical.

Min–max formulas for nonlocal elliptic operators

In this work, we give a characterization of Lipschitz operators on spaces of $$C^2(M)$$ functions (also $$C^{1,1}$$, $$C^{1,\gamma }$$, $$C^1$$, $$C^\gamma $$) that obey the global comparison property—i.e. those that preserve the global ordering of input functions at any points where their graphs may touch, often called “elliptic” operators. Here M is a complete Riemannian manifold. In particular, we show that all such operators can be written as a min–max over linear operators that are a combination of drift–diffusion and integro-differential parts. In the linear (and nonlocal) case, these operators had been characterized in the 1960s, and in the local, but nonlinear case—e.g. local Hamilton–Jacobi–Bellman operators—this characterization has also been known and used since approximately since 1960s or 1970s. Our main theorem contains both of these results as special cases. It also shows any nonlinear scalar elliptic equation can be represented as an Isaacs equation for an appropriate differential game. Our approach is to “project” the operator to one acting on functions on large finite graphs that approximate the manifold, use non-smooth analysis to derive a min–max formula on this finite dimensional level, and then pass to the limit in order to lift the formula to the original operator.