Bellman Operator Research Articles

Monotonicity results of solutions to the uniformly elliptic nonlocal Bellman system

In this paper, we consider the uniformly elliptic nonlocal Bellman problem Fsu(x)=f(u(x),v(x)),Fsv(x)=g(u(x),v(x)).Firstly, we study narrow region principles for the uniformly elliptic nonlocal Bellman operators in bounded and unbounded domains, which play key roles in obtaining the main results by the process of sliding method. Then we deal with monotonicity properties of solutions to the uniformly elliptic nonlocal Bellman system.

Parameterized Projected Bellman Operator

Approximate value iteration (AVI) is a family of algorithms for reinforcement learning (RL) that aims to obtain an approximation of the optimal value function. Generally, AVI algorithms implement an iterated procedure where each step consists of (i) an application of the Bellman operator and (ii) a projection step into a considered function space. Notoriously, the Bellman operator leverages transition samples, which strongly determine its behavior, as uninformative samples can result in negligible updates or long detours, whose detrimental effects are further exacerbated by the computationally intensive projection step. To address these issues, we propose a novel alternative approach based on learning an approximate version of the Bellman operator rather than estimating it through samples as in AVI approaches. This way, we are able to (i) generalize across transition samples and (ii) avoid the computationally intensive projection step. For this reason, we call our novel operator projected Bellman operator (PBO). We formulate an optimization problem to learn PBO for generic sequential decision-making problems, and we theoretically analyze its properties in two representative classes of RL problems. Furthermore, we theoretically study our approach under the lens of AVI and devise algorithmic implementations to learn PBO in offline and online settings by leveraging neural network parameterizations. Finally, we empirically showcase the benefits of PBO w.r.t. the regular Bellman operator on several RL problems.

Symmetric Q-learning: Reducing Skewness of Bellman Error in Online Reinforcement Learning

In deep reinforcement learning, estimating the value function to evaluate the quality of states and actions is essential. The value function is often trained using the least squares method, which implicitly assumes a Gaussian error distribution. However, a recent study suggested that the error distribution for training the value function is often skewed because of the properties of the Bellman operator, and violates the implicit assumption of normal error distribution in the least squares method. To address this, we proposed a method called Symmetric Q-learning, in which the synthetic noise generated from a zero-mean distribution is added to the target values to generate a Gaussian error distribution. We evaluated the proposed method on continuous control benchmark tasks in MuJoCo. It improved the sample efficiency of a state-of-the-art reinforcement learning method by reducing the skewness of the error distribution.

Parallel and Flexible Dynamic Programming via the Mini-Batch Bellman Operator

The Bellman operator constitutes the foundation of dynamic programming (DP). An alternative is presented by the Gauss-Seidel operator, whose evaluation, differently from that of the Bellman operator where the states are all processed at once, updates one state at a time, while incorporating into the computation the interim results. The provably better convergence rate of DP methods based on the Gauss-Seidel operator comes at the price of an inherent sequentiality, which prevents the exploitation of modern multi-core systems. In this work we propose a new operator for dynamic programming, namely, the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mini-batch Bellman operator</i> , which aims at realizing the trade-off between the better convergence rate of the methods based on the Gauss-Seidel operator and the parallelization capability offered by the Bellman operator. After the introduction of the new operator, a theoretical analysis for validating its fundamental properties is conducted. Such properties allow one to successfully deploy the new operator in the main dynamic programming schemes, such as value iteration and modified policy iteration. We compare the convergence of the DP algorithm based on the new operator with its earlier counterparts, shedding light on the algorithmic advantages of the new formulation and the impact of the batch-size parameter on the convergence. Finally, an extensive numerical evaluation of the newly introduced operator is conducted. In accordance with the theoretical derivations, the numerical results show the competitive performance of the proposed operator and its superior flexibility, which allows one to adapt the efficiency of its iterations to different structures of MDPs and hardware setups.

De-Pessimism Offline Reinforcement Learning via Value Compensation.

Offline reinforcement learning (RL) has been widely used in practice due to its efficient data utilization, but it still faces the challenge of training vulnerability caused by policy deviation. Existing offline RL methods that add policy constraints or perform conservative Q -value estimation are pessimistic, making the learned policy suboptimal. In this article, we address the pessimism problem by focusing on accurate Q -value estimation. We propose the de-pessimism (DEP) operator to estimate Q values using the optimal Bellman operator or the compensation operator according to whether the actions are in the behavior support set. The compensation operator qualitatively determines the positive or negative nature of out-of-distribution (OOD) actions based on their performance compared with the behavior actions. It leverages differences in state values to compensate for the Q value of positive OOD actions, thereby alleviating pessimism. We theoretically demonstrate the convergence of DEP and its effectiveness in policy improvement. To further advance the practical application, we integrate DEP into the soft actor-critic (SAC) algorithm, yielding the value-compensated de-pessimism offline RL (DoRL-VC). Experimentally, DoRL-VC achieves state-of-the-art (SOTA) performance across mujoco locomotion, Maze 2-D, and challenging Adroit tasks, illustrating the efficacy of DEP in mitigating pessimism.

Do not Blame Bellman: It Is Koopmans' Fault

We provide a unified approach to stochastic dynamic programming with recursive utility based on an elementary application of Tarski's fixed point theorem. We establish that the exclusive source of multiple values is the presence of multiple recursive utilities consistent with the given aggregator, each yielding a legitimate value of the recursive program. We also present sufficient conditions ensuring a unique value of the recursive program in some circumstances. Overall, acknowledging the unavoidable failure of uniqueness in general, we argue that the greatest fixed point of the Bellman operator should have a privileged position.

Distributional and hierarchical reinforcement learning for physical systems with noisy state observations and exogenous perturbations

Reinforcement learning has shown remarkable success in various applications, and in some cases, even outperforms human performance. However, despite the potential of reinforcement learning, numerous challenges still exist. In this paper, we introduce a novel approach that exploits the synergies between hierarchical reinforcement learning and distributional reinforcement learning to address complex sparse-reward tasks, where noisy state observations or non-stationary exogenous perturbations are present. Our proposed method has a hierarchical policy structure, where random rewards are modeled as random variables that follow a value distribution. This approach enables the handling of complex tasks and increases robustness to uncertainties arising from measurement noise or exogenous perturbations, such as wind. To achieve this, we extend the distributional soft Bellman operator and temporal difference error to include the hierarchical structure, and we use quantile regression to approximate the reward distribution. We evaluate our method using a bipedal robot in the OpenAI Gym environment and an electric autonomous vehicle in the SUMO traffic simulator. The results demonstrate the effectiveness of our approach in solving complex tasks with the aforementioned uncertainties when compared to state-of-the-art methods. Our approach demonstrates promising results in handling uncertainties caused by noise and perturbations for challenging sparse-reward tasks, and could potentially pave the way for the development of more robust and effective reinforcement learning algorithms in real physical systems.

Model-Free λ-Policy Iteration for Discrete-Time Linear Quadratic Regulation.

This article presents a model-free λ -policy iteration ( λ -PI) for the discrete-time linear quadratic regulation (LQR) problem. To solve the algebraic Riccati equation arising from solving the LQR in an iterative manner, we define two novel matrix operators, named the weighted Bellman operator and the composite Bellman operator. Then, the λ -PI algorithm is first designed as a recursion with the weighted Bellman operator, and its equivalent formulation as a fixed-point iteration with the composite Bellman operator is shown. The contraction and monotonic properties of the composite Bellman operator guarantee the convergence of the λ -PI algorithm. In contrast to the PI algorithm, the λ -PI does not require an admissible initial policy, and the convergence rate outperforms the value iteration (VI) algorithm. Model-free extension of the λ -PI algorithm is developed using the off-policy reinforcement learning technique. It is also shown that the off-policy variants of the λ -PI algorithm are robust against the probing noise. Finally, simulation examples are conducted to validate the efficacy of the λ -PI algorithm.

Contractivity of Bellman operator in risk averse dynamic programming with infinite horizon

The paper deals with a risk averse dynamic programming problem with infinite horizon. First, the required assumptions are formulated to have the problem well defined. Then the Bellman equation is derived, which may be also seen as a standalone reinforcement learning problem. The fact that the Bellman operator is contraction is proved, guaranteeing convergence of various solution algorithms used for dynamic programming as well as reinforcement learning problems, which we demonstrate on the value iteration and the policy iteration algorithms.

On history-dependent optimization models: A unified framework to analyze models with habits, satiation and optimal growth

In this paper, we provide a general framework for history-dependent utility models. Habit formation is a particular case of history-dependence and there has been no consensus thus far on how to formalize habits. Our framework allows one to study not only these various habit models but also satiation models and other history-dependent specifications that have their motivations and applications in different areas, including decision theory, macroeconomics and environmental economics. We provide a generalization for two aspects in the modeling: the history dependence level formation and the utility function formulation. In particular, history-dependence is not restricted to linear cases and utility is not restricted to additive nor multiplicative cases. The unifying framework in the present paper thus encompasses many of the modeling ways suggested in the existing literature. We further consider discrete-time infinite horizon dynamic optimization problems in which the instantaneous utility function is history-dependent. The way history dependence is designed allows the use of dynamic programming tools. Without concavity assumptions, we show the existence of a solution and that the value function is the unique fixed point of the Bellman operator. Uniqueness of the solution is obtained via additional concavity assumptions. Many previously used optimal growth history-dependent models fit into our framework.

A unified view of configurable Markov Decision Processes: Solution concepts, value functions, and operators

In this paper, we provide a unified presentation of the Configurable Markov Decision Process (Conf-MDP) framework. A Conf-MDP is an extension of the traditional Markov Decision Process (MDP) that models the possibility to configure some environmental parameters. This configuration activity can be carried out by the learning agent itself or by an external configurator. We introduce a general definition of Conf-MDP, then we particularize it for the cooperative setting, where the configuration is fully functional to the agent’s goals, and non-cooperative setting, in which agent and configurator might have different interests. For both settings, we propose suitable solution concepts. Furthermore, we illustrate how to extend the traditional value functions for MDPs and Bellman operators to this new framework.

Risk-sensitive control for a class of diffusions with jumps

We consider a class of diffusions controlled through the drift and jump size, and driven by a jump Lévy process and a nondegenerate Wiener process, and we study infinite horizon (ergodic) risk-sensitive control problems for this model. We start with the controlled Dirichlet eigenvalue problem in smooth bounded domains, which also allows us to generalize current results in the literature on exit rate control problems. Then we consider the infinite horizon average risk-sensitive minimization and maximization problems on the whole domain. Under suitable hypotheses, we establish existence and uniqueness of a principal eigenfunction for the Hamilton–Jacobi–Bellman (HJB) operator on the whole space, and fully characterize stationary Markov optimal controls as the measurable selectors of this HJB equation.

Hardy–Littlewood-type theorems for Fourier transforms in [formula omitted

We obtain Fourier inequalities in the weighted Lp spaces for any 1<p<∞ involving the Hardy–Cesàro and Hardy–Bellman operators. We extend these results to product Hardy spaces for p⩽1. Moreover, boundedness of the Hardy-Cesàro and Hardy-Bellman operators in various spaces (Lebesgue, Hardy, BMO) is discussed. One of our main tools is an appropriate version of the Hardy–Littlewood–Paley inequality ‖fˆ‖Lp′,q≲‖f‖Lp,q.

An Algorithm to Construct Subsolutions of Convex Optimal Control Problems

We propose an algorithm that produces a nondecreasing sequence of subsolutions for a class of optimal control problems distinguished by the property that the associated Bellman operators preserve convexity. In addition to a theoretical discussion and proofs of convergence, numerical experiments are presented to illustrate the feasibility of the method.

Maximum principles and Liouville results for uniformly elliptic nonlocal Bellman System

In this paper, we are concerned with the uniformly elliptic nonlocal Bellman system. Firstly, we study different maximum principles for the uniformly elliptic nonlocal Bellman operators and obtain key ingredients for carrying on the method of moving planes. Then we derive qualitative properties of solutions to the uniformly elliptic nonlocal Bellman in bounded domains, unbounded domains, and a coercive epigraph domain Ω in Rn, respectively, such as monotonicity, symmetry, and Liouville-type results.

HARDY TYPE INEQUALITIES WITH POWER WEIGHTS FOR HAUSDORFF OPERATORS

Classical Hardy’s inequalities are concerned with the Hardy operator and its adjoint, the Bellman operator. Hausdorff operators in their various forms are natural generalizations of these two operators. Recently, the scheme used by Bradley for Hardy’s inequalities with general weights has been adjusted to the Hausdorff setting. For the Hardy and Bellman operators, the obtained necessary and sufficient conditions coincide and reduce to the classical ones. However, in this paper, we show that sufficient conditions for Hausdorff operators can be simplified. These are used for purely power weights, while for more general weights, only general necessary conditions are applied. The corresponding results are supplemented by examples.

Inhomogeneous deep Q-network for time sensitive applications

Deep Q-network (DQN) has attracted increasing attention from both industry and academic communities. Existing methods mostly formulate the decision process as discrete agent-environment interactions, while the intervals between successive interactions are largely neglected, which may otherwise reveal important signals in real-world applications. To bridge this gap, this paper proposes to explicitly model the time intervals in DQN. Specifically, we first cast the agent-environment interactions onto a continuous time dimension, and then define a time-aware learning objective and the corresponding Bellman operator. For sample efficient training, we approximate the Q-function with a neural network, where the time information is modeled by the point process. The intensity function in point process and Q-function are seamlessly integrated by sharing the same history summarization module, such that the time interval information can directly influence the model optimization process. To close the gap between the approximated and optimal Q-function, we theoretically analyze the sample complexity of our model by deriving the finite time bound in continuous time. We conduct both simulation and real-world experiments to demonstrate our model's effectiveness.

Data-Driven Optimal Control of Affine Systems: A Linear Programming Perspective

In this letter, we discuss the problem of optimal control for affine systems in the context of data-driven linear programming. First, we introduce a unified framework for the fixed point characterization of the value function, Q-function and relaxed Bellman operators. Then, in a model-free setting, we show how to synthesize and estimate Bellman inequalities from a small but sufficiently rich dataset. To guarantee exploration richness, we complete the extension of Willems&#x2019; fundamental lemma to affine systems.

Analyzing Approximate Value Iteration Algorithms

In this paper, we consider the stochastic iterative counterpart of the value iteration scheme wherein only noisy and possibly biased approximations of the Bellman operator are available. We call this counterpart the approximate value iteration (AVI) scheme. Neural networks are often used as function approximators, in order to counter Bellman’s curse of dimensionality. In this paper, they are used to approximate the Bellman operator. Because neural networks are typically trained using sample data, errors and biases may be introduced. The design of AVI accounts for implementations with biased approximations of the Bellman operator and sampling errors. We present verifiable sufficient conditions under which AVI is stable (almost surely bounded) and converges to a fixed point of the approximate Bellman operator. To ensure the stability of AVI, we present three different yet related sets of sufficient conditions that are based on the existence of an appropriate Lyapunov function. These Lyapunov function–based conditions are easily verifiable and new to the literature. The verifiability is enhanced by the fact that a recipe for the construction of the necessary Lyapunov function is also provided. We also show that the stability analysis of AVI can be readily extended to the general case of set-valued stochastic approximations. Finally, we show that AVI can also be used in more general circumstances, that is, for finding fixed points of contractive set-valued maps.

Data-driven optimal control with a relaxed linear program

The linear programming (LP) approach has a long history in the theory of approximate dynamic programming. When it comes to computation, however, the LP approach often suffers from poor scalability. In this work, we introduce a relaxed version of the Bellman operator for q-functions and prove that it is still a monotone contraction mapping with a unique fixed point. In the spirit of the LP approach, we exploit the new operator to build a relaxed linear program (RLP). Compared to the standard LP formulation, our RLP has only one family of constraints and half the decision variables, making it more scalable and computationally efficient. For deterministic systems, the RLP trivially returns the correct q-function. For stochastic linear systems in continuous spaces, the solution to the RLP preserves the minimizer of the optimal q-function, hence retrieves the optimal policy. Theoretical results are backed up in simulation where we solve sampled versions of the LPs with data collected by interacting with the environment. For general nonlinear systems, we observe that the RLP again tends to preserve the minimizers of the solution to the LP, though the relative performance is influenced by the specific geometry of the problem.