Value Function Approximation Research Articles

Off-Policy Temporal Difference Learning with Bellman Residuals

In reinforcement learning, off-policy temporal difference learning methods have gained significant attention due to their flexibility in utilizing existing data. However, traditional off-policy temporal difference methods often suffer from poor convergence and stability when handling complex problems. To address these issues, this paper proposes an off-policy temporal difference algorithm with Bellman residuals (TDBR). By incorporating Bellman residuals, the proposed algorithm effectively improves the convergence and stability of the off-policy learning process. This paper first introduces the basic concepts of reinforcement learning and value function approximation, highlighting the importance of Bellman residuals in off-policy learning. Then, the theoretical foundation and implementation details of the TDBR algorithm are described in detail. Experimental results in multiple benchmark environments demonstrate that the TDBR algorithm significantly outperforms traditional methods in terms of both convergence speed and solution quality. Overall, the TDBR algorithm provides an effective and stable solution for off-policy reinforcement learning with broad application prospects. Future research can further optimize the algorithm parameters and extend its application to continuous state and action spaces to enhance its applicability and performance in real-world problems.

Information-Theoretic Generalization Bounds for Batch Reinforcement Learning.

We analyze the generalization properties of batch reinforcement learning (batch RL) with value function approximation from an information-theoretic perspective. We derive generalization bounds for batch RL using (conditional) mutual information. In addition, we demonstrate how to establish a connection between certain structural assumptions on the value function space and conditional mutual information. As a by-product, we derive a high-probability generalization bound via conditional mutual information, which was left open and may be of independent interest.

A comparison of reinforcement learning policies for dynamic vehicle routing problems with stochastic customer requests

This paper presents directions for using reinforcement learning with neural networks for dynamic vehicle routing problems (DVRPs). DVRPs involve sequential decision-making under uncertainty where the expected future consequences are ideally included in current decision-making. A frequently used framework for these problems is approximate dynamic programming (ADP) or reinforcement learning (RL), often in conjunction with a parametric value function approximation (VFA). A straightforward way to use VFA in DVRP is linear regression (LVFA), but more complex, non-linear predictors, e.g., neural network VFAs (NNVFA), are also widely used. Alternatively, we may represent the policy directly, using a linear policy function approximation (LPFA) or neural network PFA (NNPFA). The abundance of policies and design choices complicate the use of neural networks for DVRPs in research and practice. We provide a structured overview of the similarities and differences between the policy classes. Furthermore, we present an empirical comparison of LVFA, LPFA, NNVFA, and NNPFA policies. The comparison is conducted on several problem variants of the DVRP with stochastic customer requests. To validate our findings, we study realistic extensions of the stylized problem on (i) a same-day parcel pickup and delivery case in the city of Amsterdam, the Netherlands, and (ii) the routing of robots in an automated storage and retrieval system (AS/RS). Based on our empirical evaluation, we provide insights into the advantages and disadvantages of neural network policies compared to linear policies, and value-based approaches compared to policy-based approaches.

Model-free robust reinforcement learning via Polynomial Chaos

In recent years, the Robust Markov Decision Process (RMDP) has become an important modeling framework to address the discrepancies between simulated and real-world environments in Reinforcement Learning (RL) training. The purpose of RMDP is to accommodate the uncertainty of the real-world environments, employing a conservative approach to enhance the robustness of policy decisions. However, due to the difficulty of robust value function estimation, the RMDP framework is challenging to generalize to environments with large continuous state–action spaces. Our work focuses on model-free robust RL and proposes a model-free algorithm for continuous space setting. We adopt a new perspective on uncertainty sets such that the uncertainty sets are parameterized and the parameters obey specific stochastic distributions. We present a novel approach RPC to estimate the robust value function utilizing generalized Polynomial Chaos(gPC). We provide a proof to guarantee the convergence of the algorithm. Our training framework is based on off-policy RL, which reduces the computation overhead by gPC and improves learning stability. Our algorithm can handle continuous tasks and guarantee the robustness of the algorithm without incurring excessive computational overhead. We combine RPC with the TD3 method and conduct several experiments to evaluate its performance in a continuous robot control task, and the experimental results provide further evidence of the robustness of our algorithm.

Fast Nonlinear Model Predictive Control Combining Online Trajectory Optimization and Value Function Regression

ABSTRACTModel predictive control (MPC) is a well‐developed method capable of handling complex control tasks. Implementation of an MPC requires solution of a deterministic finite horizon nonlinear optimal control (OC) problem. OC problems can be solved globally, explicitly expressing the optimal policy (and value function) as a function of the present state, or, locally, using online trajectory optimization, generating a solution that is only valid for the present state. For nonlinear problems however, neither is possible analytically, and the latter, finding a local solution online, is usually preferred. The difficulty with online trajectory optimization is that the solution must be available within a single sampling period. The main parameter affecting the computational demand of MPC is the length of the prediction horizon. The goal in this work is to reduce the length of the prediction horizon of a model predictive controller (MPC) to reduce computation time whilst preserving optimality guarantees. To this end, we propose approximation of the trajectory optimization problem for MPC by learning a finite horizon value function. The approximated value function is inserted into a truncated trajectory optimization problem so that the MPC can be attained with a reduced prediction horizon, ergo reduced computational load. By sampling the value function in a goal oriented way, we show that an effective approximate value function can be found by including both the function value and the gradients of the value function. The result is an accurate approximate MPC which leverages learning methodologies to reduce computational cost while still accounting for constraints.

Drive force allocation control based on pump displacement optimization for hydraulic hybrid motor-assisted system

To achieve rational coordination of the front and rear axle torque of commercial vehicles equipped with hydraulic hybrid motor-assisted systems (HHMAS), and further enhance the vehicle’s fuel efficiency and traction capability. A hydraulic pump displacement control strategy based on energy-efficiency-orientation (EEO) is proposed for driving force distribution. Firstly, the efficiency is analyzed from the perspective of system energy flow. Then, considering the time-varying efficiency characteristics of the hydraulic variable pump and hydraulic motor, the stochastic dynamic optimization problem is modeled as a markov decision process, and a value function estimation method is introduced to approximate the optimal energy efficiency of the entire HHMAS. Finally, the results indicate that the EEO strategy can enhance the fuel economy of vehicles, with a fuel-saving rate of 6.3% compared to the wheel-speed-following-control (WSFC) strategy. In addition, the proposed strategy exhibits superior traction capacity and climbing ability compared to the traditional WSFC strategy.

Adaptive Dynamic Programming with Reinforcement Learning on Optimization of Flight Departure Scheduling

The intricacies of air traffic departure scheduling, especially when numerous flights are delayed, frequently impede the implementation of automated decision-making for scheduling. To surmount this obstacle, a mathematical model is proposed, and a dynamic simulation framework is designed to tackle the scheduling dilemma. An optimization control strategy is based on adaptive dynamic programming (ADP), focusing on minimizing the cumulative delay time for a cohort of delayed aircraft amidst congestion. This technique harnesses an approximation of the dynamic programming value function, augmented by reinforcement learning to enhance the approximation and alleviate the computational complexity as the number of flights increases. Comparative analyses with alternative approaches, including the branch and bound algorithm for static conditions and the first-come, first-served (FCFS) algorithm for routine scenarios, are conducted. Moreover, perturbation simulations of ADP parameters validate the method’s robustness and efficacy. ADP, when integrated with reinforcement learning, demonstrates time efficiency and reliability, positioning it as a viable solution for decision-making in departure management systems.

Moment-Based Reinforcement Learning for Ensemble Control.

Problems involving controlling the collective behavior of a population of structurally similar dynamical systems, the so-called ensemble control, arise in diverse emerging applications and pose a grand challenge in systems science and control engineering. Owing to the severely under-actuated nature and the difficulty of placing large-scale sensor networks, ensemble systems are limited to being actuated and monitored at the population level. Moreover, mathematical models describing the dynamics of ensemble systems are often elusive. Therefore, it is essential to design broadcast controls that excite the entire population in such a way that the heterogeneity in system dynamics is robustly compensated. In this article, we propose a reinforcement learning (RL)-based data-driven control framework incorporating population-level aggregated measurement data to learn a global control signal for steering a dynamic population in the desired manner. In particular, we introduce the notion of ensemble moments induced by aggregated measurements and derive the associated moment system to the original ensemble system. Then, using the moment system, we learn an approximation of optimal value functions and the associated policies in terms of ensemble moments through RL. We illustrate the feasibility and scalability of the proposed moment-based approach via numerical experiments using a population of linear, bilinear, and nonlinear dynamic ensemble systems. We report that the proposed method achieves the desired control objectives of various ensemble control tasks and obtains significantly better averaged-reward when compared with three existing methods.

Decentralized Adaptive temporal-difference learning over time-varying networks and its finite-time analysis

In reinforcement learning, centralized temporal-difference (TD) learning is commonly used to solve the policy evaluation problem. However, the decentralized adaptive variant of the TD learning algorithm has rarely been investigated in multi-agent reinforcement learning. To fill this gap, based on linear function approximation, we propose a decentralized adaptive TD learning algorithm over time-varying networks, referred to as D-AdaTD. We rigorously analyze the convergence performance of D-AdaTD, i.e., the explicit finite-time analysis is established under different step-sizes. Specifically, under constant step-sizes, the average estimated value function can converge to a neighborhood of the optimal value at rate O(1/(k+1)) and the estimated parameter of each agent can converge to a neighborhood of the optimal parameter at rate O(ξk), where k is the number of iterations and ξ∈(0,1). Under diminishing step-sizes, the average estimated value function can converge to the optimal value and the average estimated parameter can converge to the optimal parameter at rate O((1+log(k+1))/k) and O((1+log(k+1))/(k+1)), respectively. In addition, we evaluate the performance of D-AdaTD via simulation experiments, which are commonly insufficient in the existing decentralized temporal-difference learning. The experimental results also validate the effectiveness of D-AdaTD.

Deep Model-Based Reinforcement Learning for Predictive Control of Robotic Systems with Dense and Sparse Rewards

Sparse rewards and sample efficiency are open areas of research in the field of reinforcement learning. These problems are especially important when considering applications of reinforcement learning to robotics and other cyber-physical systems. This is so because in these domains many tasks are goal-based and naturally expressed with binary successes and failures, action spaces are large and continuous, and real interactions with the environment are limited. In this work, we propose Deep Value-and-Predictive-Model Control (DVPMC), a model-based predictive reinforcement learning algorithm for continuous control that uses system identification, value function approximation and sampling-based optimization to select actions. The algorithm is evaluated on a dense reward and a sparse reward task. We show that it can match the performance of a predictive control approach to the dense reward problem, and outperforms model-free and model-based learning algorithms on the sparse reward task on the metrics of sample efficiency and performance. We verify the performance of an agent trained in simulation using DVPMC on a real robot playing the reach-avoid game. Video of the experiment can be found here: https://youtu.be/0Q274kcfn4c.

Variational quantum circuit learning-enabled robust optimization for AI data center energy control and decarbonization

As the demand for artificial intelligence (AI) models and applications continues to grow, data centers that handle AI workloads are experiencing a rise in energy consumption and associated carbon footprint. This work proposes a variational quantum computing-based robust optimization (VQC-RO) framework for control and energy management in large-scale data centers to address the computational challenges and overcome limitations of conventional model-based and model-free strategies. The VQC-RO framework integrates variational quantum circuits (VQCs) with classical optimization to enable efficient and uncertainty-aware control of energy systems in AI data centers. Quantum algorithms executed on noisy intermediate-scale quantum (NISQ) devices are used for value function estimation trained with Q-learning, leading to the formulation of a robust optimization problem with uncertain coefficients. The quantum computing-based robust control strategy is designed to address uncertainties associated with weather conditions and renewable energy generation while optimizing energy consumption in AI data centers. This work also outlines the computational experiments conducted at various AI data center locations in the United States to analyze the reduction in power consumption and carbon emission levels associated with the proposed quantum computing-based robust control framework. This work contributes a novel approach to energy-efficient and sustainable data center operation, promising to reduce carbon emissions and energy consumption in large-scale data centers handling AI workloads by 9.8 % and 12.5 %, respectively.

Novel Discounted Adaptive Critic Control Designs With Accelerated Learning Formulation.

Inspired by the successive relaxation method, a novel discounted iterative adaptive dynamic programming framework is developed, in which the iterative value function sequence possesses an adjustable convergence rate. The different convergence properties of the value function sequence and the stability of the closed-loop systems under the new discounted value iteration (VI) are investigated. Based on the properties of the given VI scheme, an accelerated learning algorithm with convergence guarantee is presented. Moreover, the implementations of the new VI scheme and its accelerated learning design are elaborated, which involve value function approximation and policy improvement. A nonlinear fourth-order ball-and-beam balancing plant is used to verify the performance of the developed approaches. Compared with the traditional VI, the present discounted iterative adaptive critic designs greatly accelerate the convergence rate of the value function and reduce the computational cost simultaneously.

Optimal feedback control of dynamical systems via value-function approximation

Optimal feedback control of dynamical systems via value-function approximation

Derivative-free bound-constrained optimization for solving structured problems with surrogate models

We propose and analyze a model-based derivative-free (DFO) algorithm for solving bound-constrained optimization problems where the objective function is the composition of a smooth function and a vector of black-box functions. We assume that the black-box functions are smooth and the evaluation of them is the computational bottleneck of the algorithm. The distinguishing feature of our algorithm is the use of approximate function values at interpolation points which can be obtained by an application-specific surrogate model that is cheap to evaluate. As an example, we consider the situation in which a sequence of related optimization problems is solved and present a regression-based approximation scheme that uses function values that were evaluated when solving prior problem instances. In addition, we propose and analyze a new algorithm for obtaining interpolation points that handles unrelaxable bound constraints. Our numerical results show that our algorithm outperforms a state-of-the-art DFO algorithm for solving a least-squares problem from a chemical engineering application when a history of black-box function evaluations is available.

Accelerating value function approximations for dynamic dial-a-ride problems via dimensionality reductions

As the success of ride-sharing mobility service providers shows, customer demand for shared mobility services is increasing. The availability of mobile devices enables the constant accessibility of mobility apps and the immediate placement of transport requests. To provide such a dynamic dial-a-ride service, an effective control of the fleet is necessary. One promising solution approach is the value function approximation (VFA), which on the one hand convinces through good performance, but on the other hand also stands out through fast response times for a request. Training a VFA can be a challenging task since, among other things, the dimensionality of the state space plays a decisive role. If many variables to describe a state are used, a high amount of information can produce good performance after completion of the learning process. If the state space is too high-dimensional, there is also a risk that the method will not be able to find a reasonable solution. In contrast, if the number of variables is reduced, the learning speed can be accelerated, but the eventual performance may suffer from the associated loss of information. Furthermore, not all variables are equally relevant, as they contain different amounts of information. This paper presents a hybrid strategy, temporarily lowering the dimensionality of the problem using dimension reduction methods and subsequently increasing it by mapping the lower-dimensional state representations back onto a high-dimensional state space in order to exploit the advantages of both space dimensionalities. VFA in itself results in competitive performance for the dynamic dial-a-ride problem with shared rides. The proposed hybrid state representation can outperform the reference state representations by 3%, which corresponds to a meaningful acceleration in VFA learning speed.

Asymptotic Methods for Transaction Costs

We propose a general approximation method for the determination of optimal trading strategies in markets with proportional transaction costs, with a polynomial approximation of the residual value function. The method is exemplified by several problems, from optimally tracking benchmarks and hedging the log contract to maximizing utility from terminal wealth. Strategies are also approximated by practically executable, discrete trades. We identify the necessary trade-off between the trading frequency and trade size to ensure satisfactory agreement with the theoretically optimal, continuous strategies of infinite activity.

Parameterized Projected Bellman Operator

Approximate value iteration (AVI) is a family of algorithms for reinforcement learning (RL) that aims to obtain an approximation of the optimal value function. Generally, AVI algorithms implement an iterated procedure where each step consists of (i) an application of the Bellman operator and (ii) a projection step into a considered function space. Notoriously, the Bellman operator leverages transition samples, which strongly determine its behavior, as uninformative samples can result in negligible updates or long detours, whose detrimental effects are further exacerbated by the computationally intensive projection step. To address these issues, we propose a novel alternative approach based on learning an approximate version of the Bellman operator rather than estimating it through samples as in AVI approaches. This way, we are able to (i) generalize across transition samples and (ii) avoid the computationally intensive projection step. For this reason, we call our novel operator projected Bellman operator (PBO). We formulate an optimization problem to learn PBO for generic sequential decision-making problems, and we theoretically analyze its properties in two representative classes of RL problems. Furthermore, we theoretically study our approach under the lens of AVI and devise algorithmic implementations to learn PBO in offline and online settings by leveraging neural network parameterizations. Finally, we empirically showcase the benefits of PBO w.r.t. the regular Bellman operator on several RL problems.

ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioning

Decision Transformer (DT), which employs expressive sequence modeling techniques to perform action generation, has emerged as a promising approach to offline policy optimization. However, DT generates actions conditioned on a desired future return, which is known to bear some weaknesses such as the susceptibility to environmental stochasticity. To overcome DT's weaknesses, we propose to empower DT with dynamic programming. Our method comprises three steps. First, we employ in-sample value iteration to obtain approximated value functions, which involves dynamic programming over the MDP structure. Second, we evaluate action quality in context with estimated advantages. We introduce two types of advantage estimators, IAE and GAE, which are suitable for different tasks. Third, we train an Advantage-Conditioned Transformer (ACT) to generate actions conditioned on the estimated advantages. Finally, during testing, ACT generates actions conditioned on a desired advantage. Our evaluation results validate that, by leveraging the power of dynamic programming, ACT demonstrates effective trajectory stitching and robust action generation in spite of the environmental stochasticity, outperforming baseline methods across various benchmarks. Additionally, we conduct an in-depth analysis of ACT's various design choices through ablation studies. Our code is available at https://github.com/LAMDA-RL/ACT.

Balancing resources for dynamic vehicle routing with stochastic customer requests

We consider a service provider performing pre-planned service for initially known customers with a fleet of vehicles, e.g., parcel delivery. During execution, new dynamic service requests occur, e.g., for parcel pickup. The goal of the service provider is to serve as many dynamic requests as possible while ensuring service of all initial customers. The allocation of initial services impacts the potential of serving dynamic requests. An allocation aiming on a time-efficient initial routing leads to minimal overall workload regarding the initial solution but may congest some vehicles that are unable to serve additional requests along their routes. An even workload division is less efficient but grants all vehicles flexibility for additional services. In this paper, we investigate the balance between efficiency and flexibility. For the initial customers, we modify a routing algorithm to allow a shift between efficient initial routing and evenly balanced workloads. For effective dynamic decision making with respect to the dynamic requests, we present value function approximations with different feature sets capturing vehicle workload in different levels of detail. We show that sacrificing some initial routing efficiency in favor of a balanced vehicle workload is a key factor for a flexible integration of later customer requests that leads to an average improvement of 10.75%. Further, we show when explicitly depicting heterogeneity in the vehicle workload by features of the value function approximation provides benefits and that the best choice of features leads to an average improvement of 5.71% compared to the worst feature choice.

Reinforcement learning‐based event‐triggered optimal control for unknown nonlinear systems with input delay

AbstractThe optimal control issue of discrete‐time nonlinear unknown systems with time‐delay control input is the focus of this work. In order to reduce communication costs, a reinforcement learning‐based event‐triggered controller is proposed. By applying the proposed control method, closed‐loop system's asymptotic stability is demonstrated, and a maximum upper bound for the infinite‐horizon performance index can be calculated beforehand. The event‐triggered condition requires the next time state information. In an effort to forecast the next state and achieve optimal control, three neural networks (NNs) are introduced and used to approximate system state, value function, and optimal control. Additionally, a M NN is utilized to cope with the time‐delay term of control input. Moreover, taking the estimation errors of NNs into account, the uniformly ultimately boundedness of state and NNs weight estimation errors can be guaranteed. Ultimately, the validity of proposed approach is illustrated by simulations.