Value Function Approximation Research Articles

Continuous and impulse controls differential game in finite horizon with Nash-equilibrium and application

This paper considers a new class of deterministic finite-time horizon, two-player, zero-sum differential games (DGs) in which the maximizing player is allowed to take continuous and impulse controls whereas the minimizing player is allowed to take impulse control only. We seek to approximate the value function, and to provide a verification theorem for this class of DGs. By means of dynamic programming principle (DPP) in viscosity solution (VS) framework, we first characterize the value function as the unique VS to the related Hamilton–Jacobi-Bellman–Isaacs (HJBI) double-obstacle equation. Next, we prove that an approximate value function exists, that it is the unique solution to an approximate HJBI double-obstacle equation, and converges locally uniformly towards the value function of each player when the time discretization step goes to zero. Moreover, we provide a verification theorem which characterizes a Nash-equilibrium (NE) for the DG control problem considered. Finally, by applying our results, we derive a new continuous-time portfolio optimization model, and we provide related computational algorithms and numerical results.

Online Learning of Minmax Solutions for Distributed Estimation and Tracking Control of Sensor Networks in Graphical Games

In this article, a new target tracking algorithm that is based on a distributed estimation-based control protocol is developed for multiple moving sensors subject to adversarial inputs. An augmented system composed of both the estimation error dynamics and the tracking error dynamics is introduced for the distributed estimation and target tracking control problem. We integrate estimation and control to simultaneously minimize infinite-horizon estimation and tracking errors in sensor networks with disturbances. A Minmax strategy as an alternative goal to Nash equilibrium is proposed to compute the sensor’s optimal motion controls in the presence of worst-case neighbor controls and disturbances. In contrast to Nash, Minmax yields a distributed algebraic Riccati equation that is easily solved locally by each sensor. A method based on reinforcement learning is given to compute sensor motion inputs so that all sensors estimate and track the target states. Value function approximation using neural networks is used to solve the distributed Bellman equations online based on the real-time observed data. Proofs are given to guarantee the performance of the combined distributed estimation and tracking algorithms. Finally, the simulation examples show the effectiveness of the proposed algorithm.

State-Following-Kernel-Based Online Reinforcement Learning Guidance Law Against Maneuvering Target

In this article, a state-following-kernel-based reinforcement learning method with an extended disturbance observer is proposed, whose application to a missile-target interception system is considered. First, the missile-target engagement is formulated as a vertical planar pursuit–evasion problem. The target maneuver is then estimated by an extended disturbance observer in real time, which leads to an infinite-horizon optimal regulation problem. Next, utilizing the local state approximation ability of state-following kernels, the critic neural network (NN) and actor NN for synchronous iteration are constructed to calculate the approximate optimal guidance policy. The states and NN weights are proven to be uniformly ultimately bounded using the Lyapunov method. Finally, numerical simulations against different types of nonstationary targets are effectively tested, and the results highlight the role of state-following kernels in the value function and policy approximation.

Convergence of Finite Memory Q Learning for POMDPs and Near Optimality of Learned Policies Under Filter Stability

In this paper, for partially observed Markov decision problems (POMDPs), we provide the convergence of a Q learning algorithm for control policies using a finite history of past observations and control actions, and consequentially, we establish near optimality of such limit Q functions under explicit filter stability conditions. We present explicit error bounds relating the approximation error to the length of the finite history window. We establish the convergence of such Q learning iterations under mild ergodicity assumptions on the state process during the exploration phase. We further show that the limit fixed point equation gives an optimal solution for an approximate belief Markov decision problem (MDP). We then provide bounds on the performance of the policy obtained using the limit Q values compared with the performance of the optimal policy for the POMDP, in which we also present explicit conditions using recent results on filter stability in controlled POMDPs. Whereas there exist many experimental results, (i) the rigorous asymptotic convergence (to an approximate MDP value function) for such finite memory Q learning algorithms and (ii) the near optimality with an explicit rate of convergence (in the memory size) under filter stability are results that are new to the literature to our knowledge. Funding: This research was supported in part by the Natural Sciences and Engineering Research Council (NSERC) of Canada.

Online Adaptive Dynamic Programming-Based Solution of Networked Multiple-Pursuer and Single-Evader Game

This paper presents a new scheme for the online solution of a networked multi-agent pursuit–evasion game based on an online adaptive dynamic programming method. As a multi-agent in the game can form an Internet of Things (IoT) system, by incorporating the relative distance and the control energy as the performance index, the expression of the policies when the agents reach the Nash equilibrium is obtained and proved by the minmax principle. By constructing a Lyapunov function, the capture conditions of the game are obtained and discussed. In order to enable each agent to obtain the policy for reaching the Nash equilibrium in real time, the online adaptive dynamic programming method is used to solve the game problem. Furthermore, the parameters of the neural network are fitted by value function approximation, which avoids the difficulties of solving the Hamilton-Jacobi–Isaacs equation, and the numerical solution of the Nash equilibrium is obtained. Simulation results depict the feasibility of the proposed method for use on multi-agent pursuit–evasion games.

Q-Learning With Uniformly Bounded Variance

Sample complexity bounds are a common performance metric in the reinforcement learning literature. In the discounted cost, infinite horizon setting, all of the known bounds can be arbitrarily large, as the discount factor approaches unity. These results seem to imply that a very large number of samples is required to achieve an epsilon-optimal policy. The objective of the present work is to introduce a new class of algorithms that have sample complexity <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">uniformly bounded over all discount factors</i> . One may argue that this is impossible, due to a recent min–max lower bound. The explanation is that these prior bounds concern value function approximation and not policy approximation. We show that the asymptotic covariance of the tabular Q-learning algorithm with an optimized step-size sequence is a quadratic function of a factor that goes to infinity, as discount factor approaches 1; an essentially known result. The new <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">relative Q-learning</i> algorithm proposed here is shown to have asymptotic covariance that is uniformly bounded over all discount factors.

Approximate Optimal Curve Path Tracking Control for Nonlinear Systems with Asymmetric Input Constraints

This paper proposes an approximate optimal curve-path-tracking control algorithm for partially unknown nonlinear systems subject to asymmetric control input constraints. Firstly, the problem is simplified by introducing a feedforward control law, and a dedicated design for optimal control with asymmetric input constraints is provided by redesigning the control cost function in a non-quadratic form. Then, the optimality and stability of the derived optimal control policy is demonstrated. To solve the underlying tracking Hamilton–Jacobi–Bellman (HJB) equation in consideration of partially unknown systems, an integral reinforcement learning (IRL) algorithm is utilized using the neural network (NN)-based value function approximation. Finally, the effectiveness and generalization of the proposed method is verified by experiments carried out on a high-fidelity hardware-in-the-loop (HIL) simulation system for fixed-wing unmanned aerial vehicles (UAVs) in comparison with three other typical path-tracking control algorithms.

Efficient compact linear programs for network revenue management

We are concerned with computing bid prices in network revenue management using approximate linear programming. It is well-known that affine value function approximations yield bid prices which are not sensitive to remaining capacity. The analytic reduction to compact linear programs allows the efficient computation of such bid prices. On the other hand, capacity-dependent bid prices can be obtained using separable piecewise linear value function approximations. Even though compact linear programs have been derived for this case also, they are still computationally much more expensive compared to using affine functions. We propose compact linear programs requiring substantially smaller computing times while, simultaneously, significantly improving the performance of capacity-independent bid prices. This simplification is achieved by taking into account remaining capacity only if it becomes scarce. Although our proposed linear programs are relaxations of the unreduced approximate linear programs, we conjecture equivalence and provide according numerical support. We measure the quality of an approximation by the difference between the expected performance of an induced policy and the corresponding theoretical upper bound. Using this paradigm in numerical experiments, we demonstrate the competitiveness of our proposed linear programs.

Deep Reinforcement Learning for Crowdshipping Last-Mile Delivery with Endogenous Uncertainty

In this work, we study a flexible compensation scheme for last-mile delivery where a company outsources part of the activity of delivering products to its customers to occasional drivers (ODs), under a scheme named crowdshipping. All deliveries are completed at the minimum total cost incurred with their vehicles and drivers plus the compensation paid to the ODs. The company decides on the best compensation scheme to offer to the ODs at the planning stage. We model our problem based on a stochastic and dynamic environment where delivery orders and ODs volunteering to make deliveries present themselves randomly within fixed time windows. The uncertainty is endogenous in the sense that the compensation paid to ODs influences their availability. We develop a deep reinforcement learning (DRL) algorithm that can deal with large instances while focusing on the quality of the solution: we combine the combinatorial structure of the action space with the neural network of the approximated value function, involving techniques from machine learning and integer optimization. The results show the effectiveness of the DRL approach by examining out-of-sample performance and that it is suitable to process large samples of uncertain data, which induces better solutions.

Primal-Dual Value Function Approximation for Stochastic Dynamic Intermodal Transportation with Eco-Labels

Eco-labels are a way to benchmark transportation shipments with respect to their environmental impact. In contrast to an eco-labeling of consumer products, emissions in transportation depend on several operational factors like the mode of transportation (e.g., train or truck) or a vehicle’s current and potential future capacity utilization when new orders are added for consolidation. Thus, satisfying eco-labels and doing this cost efficiently is a challenging task when dynamically routing orders in an intermodal network. In this paper, we model the problem as a multiobjective sequential decision process and propose a reinforcement learning method: value function approximation (VFA). VFAs frequently simulate trajectories of the problem and store observed values (violated eco-labels and costs) for states aggregated to a set of features. The observations are used for improved decision making in the next trajectory. For our problem, we face two additional challenges when applying a VFA, the multiple objectives and the “delayed” realization of eco-label satisfaction due to future consolidation. For the first, we propose different feature sets dependent on the objective function’s focus: costs or eco-labels. For the latter, we propose enhancing the suboptimal decision making and observed pessimistic primal values within the VFA trajectories with optimistic dual decision making when all information of a trajectory is known ex post. This enhancement is a general methodological contribution to the literature of approximate dynamic programming and will likely improve learning for other problems as well. We show the advantages of both components in a comprehensive study for intermodal transport via trains and trucks in Europe.

Joint Beamforming Design for Multiuser MISO Downlink Aided by a Reconfigurable Intelligent Surface and a Relay

Reconfigurable intelligent surfaces (RISs) have drawn considerable attention due to their ability to direct electromagnetic waves into desirable directions. Although RISs share some similarities with relays, the two have fundamental differences impacting their performance. To harness the benefits of both, we propose a downlink system wherein a relay and an RIS improve performance in terms of energy-efficiency. Using singular value decomposition (SVD), semidefinite programming (SDP), and function approximations, we propose different solutions for optimizing the beamforming matrices at the base-station (BS), the relay, and the phase shifts at the RIS to minimize the total power under quality-of-service (QoS) constraints. The problem is solved when the relay operates in half-duplex and full-duplex modes and when the reflecting elements have continuous and discrete phase shifts. Simulation results compare the performance of the system with and without the RIS or the relay, under different optimization solutions. The results show that the system with full-duplex relay and RIS outperforms the other scenarios, and the contribution of full-duplex relay is higher than that of the RIS. However, an RIS outperforms a half-duplex relay when the required QoS is high. The results also show that increasing the number of reflecting elements improves the performance better in the presence of a relay than in its absence.

Reinforcement Learning Control of Robotic Knee With Human-in-the-Loop by Flexible Policy Iteration.

We are motivated by the real challenges presented in a human-robot system to develop new designs that are efficient at data level and with performance guarantees, such as stability and optimality at system level. Existing approximate/adaptive dynamic programming (ADP) results that consider system performance theoretically are not readily providing practically useful learning control algorithms for this problem, and reinforcement learning (RL) algorithms that address the issue of data efficiency usually do not have performance guarantees for the controlled system. This study fills these important voids by introducing innovative features to the policy iteration algorithm. We introduce flexible policy iteration (FPI), which can flexibly and organically integrate experience replay and supplemental values from prior experience into the RL controller. We show system-level performances, including convergence of the approximate value function, (sub)optimality of the solution, and stability of the system. We demonstrate the effectiveness of the FPI via realistic simulations of the human-robot system. It is noted that the problem we face in this study may be difficult to address by design methods based on classical control theory as it is nearly impossible to obtain a customized mathematical model of a human-robot system either online or offline. The results we have obtained also indicate the great potential of RL control to solving realistic and challenging problems with high-dimensional control inputs.

Stochastic one-way carsharing systems with dynamic relocation incentives through preference learning

The development of carsharing services is expected to achieve greater resource efficiency and provide a sustainable solution for future mobility systems. However, operators inevitably face the imbalance between demand and supply in one-way carsharing systems (CSSs). Also, it is challenging for them to make quick and efficient operational decisions when both travel time and trip requests are uncertain. This study leverages historical and online data and proposes a learning-based methodology to quickly make real-time decisions for CSSs, including vehicle assignment, relocation, and user incentive decisions. Compared to the literature in approximate dynamic programming (ADP), which mostly focuses on uncertainty in trip requests, we propose an offline–online ADP approach to consider the temporal and spatial uncertainty in both trip requests and travel time. To tackle our high-dimensional problem, we integrate an online look-ahead policy into the offline value function approximation (VFA) policy to produce a computational tractable high-quality dynamic fleet management policy. Furthermore, a user-based relocation strategy is investigated to rebalance the fleet distribution to meet the demand better. Specifically, we aim to solve the optimal incentives the operator could offer to users to relocate cars, while user preferences towards relocation incentives are generally unknown in practice. We further enhance our anticipatory policy by developing an online module via Bayesian learning that learns the preference model on the fly using users’ revealed preference data collected online. The numerical experiments in Singapore demonstrate our offline–online ADP approach improves the solution quality and significantly compared to offline VFA policy. The results also confirm the importance of incorporating uncertainty in travel time. The benefits of the user-based relocation scheme using the Bayesian learning method using online data to enhance anticipatory decisions and learn unknown user preferences are also illustrated. It quickly and efficiently learns the user preferences, which further reduces the relocation cost and increases the profit.

Battery Dispatching for End Users With On-Site Renewables and Peak Demand Charges—An Approximate Dynamic Programming Approach

Battery energy storage systems (BESSs) have the potential to reduce end users’ electricity bills by shifting their grid demand in response to price incentives. Some electricity retailers charge end users for their peak demand in the billing cycle regardless of the time of occurrence. Yet, using BESSs to reduce demand charges is challenging, since the net demand of end users can be highly uncertain due to low aggregation and on-site renewable generation. In this article, an approximate dynamic programming (ADP) methodology is developed to control a BESS to minimize an end user’s electricity bill, which includes both an energy charge and monthly peak demand charge. To address time-varying uncertainty, the net demand is modeled using a periodic autoregressive (PAR) model, which is then used to formulate a Markov Decision Process whose objective is to minimize the sum of energy, peak demand, and battery usage costs. A backward ADP strategy is developed which is enabled by new closed-form expressions for the probability distribution of the expected stage and tail costs when a radial basis function (RBF) value function approximation (VFA) is employed. The approach is applied to real net demand data from a small Australian residential community and compared to two benchmark policies: a lookup table VFA and a model predictive control (MPC) approach. The results show that the proposed approach reduces the average monthly peak demand by about 25% (yielding an 8% reduction in the electricity bill), whereas the best-performing benchmark policy reduced the peak demand by about 17%.

An approximate dynamic programming algorithm for short-term electric vehicle fleet operation under uncertainty

This paper considers the dynamic problem of optimally operating a fleet of plug-in hybrid electric vehicles in a market environment. With uncertainty in future electricity prices and driving demands, we formulate a Markov decision process and determine a cost-minimizing policy for using the engine and charging and discharging the battery. As such, the policy is based on the trade-off between the costs of gasoline and electricity and between current and future power prices. To accommodate an inhomogeneous fleet composition and overcome the computational challenges of stochastic and dynamic optimization, including large-scale state and action spaces, we adopt the methodology of approximate dynamic programming. More specifically, using simulation and value function approximation by linear regression, we apply a least squares Monte Carlo method. This methodology allows for scaling with respect to fleet size and we are able to establish convergence of our algorithm for 100 vehicles by using 5000 samples in the simulation. Our results show that the vehicles should generally discharge the battery rather than using the engine unless battery capacity is insufficient to fully cover driving demand, but the timing of battery charging should be according to power prices. When comparing our policy to the simple policy of immediate charging, we demonstrate superiority for small and medium-sized fleets, with 2%–4% cost differences.

An approximate dynamic programming method for unit-based small hydropower scheduling

Hydropower will become an important power source of China’s power grids oriented to carbon neutral. In order to fully exploit the potential of water resources and achieve low-carbon operation, this paper proposes an approximate dynamic programming (ADP) algorithm for the unit-based short-term small hydropower scheduling (STSHS) framework considering the hydro unit commitment, which can accurately capture the physical and operational characteristics of individual units. Both the non-convex and non-linearization characteristics of the original STSHS model are retained without any linearization to accurately describe the hydropower production function and head effect, especially the dependence between the net head and the water volume in the reservoir, thereby avoiding loss of the actual optimal solution due to the large error introduced by the linearization process. An approximate value function of the original problem is formulated via the searching table model and approximate policy value iteration process to address the “curse of dimensionally” in traditional dynamic programming, which provides an approximate optimal strategy for the STSHS by considering both algorithm accuracy and computational efficiency. The model is then tested with a real-world instance of a hydropower plant with three identical units to demonstrate the effectiveness of the proposed method.

Real-Time Schedule of Microgrid for Maximizing Battery Energy Storage Utilization

This paper proposes a real-time schedule model of a microgrid (MG) for maximizing battery energy storage (BES) utilization. To this end, a BES life model is linearized using piece-wise linearization and big-M method to assess the BES life loss (BLL) in a real-time manner. The cost-effective schedule model of the MG with multiple energy resources aims to maximize BES utilization while ensuring its sufficient lifespan. Corresponding to the optimization model, approximate dynamic programming (ADP) for maximizing BES utilization (ADP-MBU) in the real-time schedule is proposed to optimize the system under stochastic environments. In ADP-MBU, a new value function approximation method employing the BES cumulative life loss (BCLL) is developed to improve the optimality and applicability. The proposed ADP-MBU algorithm can achieve satisfactory approximate optimality while reflecting the variation of real-time BLL. Case studies validate the applicability of the proposed MG schedule model and the advantages of the proposed ADP-MBU algorithm.

Proximal Policy Optimization With Policy Feedback

Proximal policy optimization (PPO) is a deep reinforcement learning algorithm based on the actor–critic (AC) architecture. In the classic AC architecture, the Critic (value) network is used to estimate the value function while the Actor (policy) network optimizes the policy according to the estimated value function. The efficiency of the classic AC architecture is limited due that the policy does not directly participate in the value function update. The classic AC architecture will make the value function estimation inaccurate, which will affect the performance of the PPO algorithm. For improvement, we designed a novel AC architecture with policy feedback (AC-PF) by introducing the policy into the update process of the value function and further proposed the PPO with policy feedback (PPO-PF). For the AC-PF architecture, the policy-based expected (PBE) value function and discount reward formulas are designed by drawing inspiration from expected Sarsa. In order to enhance the sensitivity of the value function to the change of policy and to improve the accuracy of PBE value estimation at the early learning stage, we proposed a policy update method based on the clipped discount factor. Moreover, we specifically defined the loss functions of the policy network and value network to ensure that the policy update of PPO-PF satisfies the unbiased estimation of the trust region. Experiments on Atari games and control tasks show that compared to PPO, PPO-PF has faster convergence speed, higher reward, and smaller variance of reward.

A Generalized Bootstrap Target for Value-Learning, Efficiently Combining Value and Feature Predictions

Estimating value functions is a core component of reinforcement learning algorithms. Temporal difference (TD) learning algorithms use bootstrapping, i.e. they update the value function toward a learning target using value estimates at subsequent time-steps. Alternatively, the value function can be updated toward a learning target constructed by separately predicting successor features (SF)—a policy-dependent model—and linearly combining them with instantaneous rewards. We focus on bootstrapping targets used when estimating value functions, and propose a new backup target, the ?-return mixture, which implicitly combines value-predictive knowledge (used by TD methods) with (successor) feature-predictive knowledge—with a parameter ? capturing how much to rely on each. We illustrate that incorporating predictive knowledge through an ??-discounted SF model makes more efficient use of sampled experience, compared to either extreme, i.e. bootstrapping entirely on the value function estimate, or bootstrapping on the product of separately estimated successor features and instantaneous reward models. We empirically show this approach leads to faster policy evaluation and better control performance, for tabular and nonlinear function approximations, indicating scalability and generality.

What about Inputting Policy in Value Function: Policy Representation and Policy-Extended Value Function Approximator

We study Policy-extended Value Function Approximator (PeVFA) in Reinforcement Learning (RL), which extends conventional value function approximator (VFA) to take as input not only the state (and action) but also an explicit policy representation. Such an extension enables PeVFA to preserve values of multiple policies at the same time and brings an appealing characteristic, i.e., value generalization among policies. We formally analyze the value generalization under Generalized Policy Iteration (GPI). From theoretical and empirical lens, we show that generalized value estimates offered by PeVFA may have lower initial approximation error to true values of successive policies, which is expected to improve consecutive value approximation during GPI. Based on above clues, we introduce a new form of GPI with PeVFA which leverages the value generalization along policy improvement path. Moreover, we propose a representation learning framework for RL policy, providing several approaches to learn effective policy embeddings from policy network parameters or state-action pairs. In our experiments, we evaluate the efficacy of value generalization offered by PeVFA and policy representation learning in several OpenAI Gym continuous control tasks. For a representative instance of algorithm implementation, Proximal Policy Optimization (PPO) re-implemented under the paradigm of GPI with PeVFA achieves about 40% performance improvement on its vanilla counterpart in most environments.