Value Function Approximation Research Articles

Real-Time Schedule of Microgrid for Maximizing Battery Energy Storage Utilization

This paper proposes a real-time schedule model of a microgrid (MG) for maximizing battery energy storage (BES) utilization. To this end, a BES life model is linearized using piece-wise linearization and big-M method to assess the BES life loss (BLL) in a real-time manner. The cost-effective schedule model of the MG with multiple energy resources aims to maximize BES utilization while ensuring its sufficient lifespan. Corresponding to the optimization model, approximate dynamic programming (ADP) for maximizing BES utilization (ADP-MBU) in the real-time schedule is proposed to optimize the system under stochastic environments. In ADP-MBU, a new value function approximation method employing the BES cumulative life loss (BCLL) is developed to improve the optimality and applicability. The proposed ADP-MBU algorithm can achieve satisfactory approximate optimality while reflecting the variation of real-time BLL. Case studies validate the applicability of the proposed MG schedule model and the advantages of the proposed ADP-MBU algorithm.

Proximal Policy Optimization With Policy Feedback

Proximal policy optimization (PPO) is a deep reinforcement learning algorithm based on the actor–critic (AC) architecture. In the classic AC architecture, the Critic (value) network is used to estimate the value function while the Actor (policy) network optimizes the policy according to the estimated value function. The efficiency of the classic AC architecture is limited due that the policy does not directly participate in the value function update. The classic AC architecture will make the value function estimation inaccurate, which will affect the performance of the PPO algorithm. For improvement, we designed a novel AC architecture with policy feedback (AC-PF) by introducing the policy into the update process of the value function and further proposed the PPO with policy feedback (PPO-PF). For the AC-PF architecture, the policy-based expected (PBE) value function and discount reward formulas are designed by drawing inspiration from expected Sarsa. In order to enhance the sensitivity of the value function to the change of policy and to improve the accuracy of PBE value estimation at the early learning stage, we proposed a policy update method based on the clipped discount factor. Moreover, we specifically defined the loss functions of the policy network and value network to ensure that the policy update of PPO-PF satisfies the unbiased estimation of the trust region. Experiments on Atari games and control tasks show that compared to PPO, PPO-PF has faster convergence speed, higher reward, and smaller variance of reward.

A Generalized Bootstrap Target for Value-Learning, Efficiently Combining Value and Feature Predictions

Estimating value functions is a core component of reinforcement learning algorithms. Temporal difference (TD) learning algorithms use bootstrapping, i.e. they update the value function toward a learning target using value estimates at subsequent time-steps. Alternatively, the value function can be updated toward a learning target constructed by separately predicting successor features (SF)—a policy-dependent model—and linearly combining them with instantaneous rewards. We focus on bootstrapping targets used when estimating value functions, and propose a new backup target, the ?-return mixture, which implicitly combines value-predictive knowledge (used by TD methods) with (successor) feature-predictive knowledge—with a parameter ? capturing how much to rely on each. We illustrate that incorporating predictive knowledge through an ??-discounted SF model makes more efficient use of sampled experience, compared to either extreme, i.e. bootstrapping entirely on the value function estimate, or bootstrapping on the product of separately estimated successor features and instantaneous reward models. We empirically show this approach leads to faster policy evaluation and better control performance, for tabular and nonlinear function approximations, indicating scalability and generality.

What about Inputting Policy in Value Function: Policy Representation and Policy-Extended Value Function Approximator

We study Policy-extended Value Function Approximator (PeVFA) in Reinforcement Learning (RL), which extends conventional value function approximator (VFA) to take as input not only the state (and action) but also an explicit policy representation. Such an extension enables PeVFA to preserve values of multiple policies at the same time and brings an appealing characteristic, i.e., value generalization among policies. We formally analyze the value generalization under Generalized Policy Iteration (GPI). From theoretical and empirical lens, we show that generalized value estimates offered by PeVFA may have lower initial approximation error to true values of successive policies, which is expected to improve consecutive value approximation during GPI. Based on above clues, we introduce a new form of GPI with PeVFA which leverages the value generalization along policy improvement path. Moreover, we propose a representation learning framework for RL policy, providing several approaches to learn effective policy embeddings from policy network parameters or state-action pairs. In our experiments, we evaluate the efficacy of value generalization offered by PeVFA and policy representation learning in several OpenAI Gym continuous control tasks. For a representative instance of algorithm implementation, Proximal Policy Optimization (PPO) re-implemented under the paradigm of GPI with PeVFA achieves about 40% performance improvement on its vanilla counterpart in most environments.

Robust Action Gap Increasing with Clipped Advantage Learning

Advantage Learning (AL) seeks to increase the action gap between the optimal action and its competitors, so as to improve the robustness to estimation errors. However, the method becomes problematic when the optimal action induced by the approximated value function does not agree with the true optimal action. In this paper, we present a novel method, named clipped Advantage Learning (clipped AL), to address this issue. The method is inspired by our observation that increasing the action gap blindly for all given samples while not taking their necessities into account could accumulate more errors in the performance loss bound, leading to a slow value convergence, and to avoid that, we should adjust the advantage value adaptively. We show that our simple clipped AL operator not only enjoys fast convergence guarantee but also retains proper action gaps, hence achieving a good balance between the large action gap and the fast convergence. The feasibility and effectiveness of the proposed method are verified empirically on several RL benchmarks with promising performance.

Solving optimal stopping problems under model uncertainty via empirical dual optimisation

In this work, we consider optimal stopping problems with model uncertainty incorporated into the formulation of the underlying objective function. Typically, the robust, efficient hedging of American options in incomplete markets may be described as optimal stopping of such kind. Based on a generalisation of the additive dual representation of Rogers (Math. Financ. 12:271–286, 2002) to the case of optimal stopping under model uncertainty, we develop a novel regression-based Monte Carlo algorithm for the approximation of the corresponding value function. The algorithm involves optimising a penalised empirical dual objective functional over a class of martingales. This formulation allows us to construct upper bounds for the optimal value with reduced complexity. Finally, we carry out a convergence analysis of the proposed algorithm and illustrate its performance by several numerical examples.

Dynamic Stochastic Matching Under Limited Time

In “Dynamic Stochastic Matching Under Limited Time,” Aouad and Sarıtaç analyze the design of matching policies in dynamic markets such as carpooling platforms and kidney exchange schemes. A crucial distinction with previous literature is that the agents’ arrivals and departures are fully dynamic. The demand and supply side are constantly replenished; each market participant remains available for potential matches during a limited period of time. Specifically, the authors formulate a general dynamic matching model over edge-weighted graphs, where the agents' arrivals and abandonments are stochastic and heterogeneous. The platform controls how long each agent waits and whom s/he is matched with. These decisions are subject to a fundamental tradeoff between increasing market thickness and mitigating the risk of abandonments from certain participants. The authors’ main contribution is to devise simple matching algorithms with strong performance guarantees for a broad class of networks. In contrast, they show that widely used batching algorithms have an arbitrary bad performance on certain graph-theoretic structures. Their analysis involves novel techniques including linear programming benchmarks, value function approximations, and proxies for continuous-time Markov chains, which may be of broader interest. Extensive simulations on real-world taxi demand data demonstrate that the newly developed algorithms can significantly improve cost efficiency against batching algorithms.

Adaptive grids for the estimation of dynamic models

This paper develops a method to flexibly adapt interpolation grids of value function approximations in the estimation of dynamic models using either NFXP (Rust, Econometrica: Journal of the Econometric Society, 55, 999–1033, 1987) or MPEC (Su & Judd, Econometrica: Journal of the Econometric Society, 80, 2213–2230, 2012). Since MPEC requires the grid structure for the value function approximation to be hard-coded into the constraints, one cannot apply iterative node insertion for grid refinement; for NFXP, grid adaption by (iteratively) inserting new grid nodes will generally lead to discontinuous likelihood functions. Therefore, we show how to continuously adapt the grid by moving the nodes, a technique referred to as r-adaption. We demonstrate how to obtain optimal grids based on the balanced error principle, and implement this approach by including additional constraints to the likelihood maximization problem. The method is applied to two models: (i) the bus engine replacement model (Rust, 1987), modified to feature a continuous mileage state, and (ii) to a dynamic model of content consumption using original data from one of the world’s leading user-generated content networks in the domain of music.

Robust Actor–Critic Learning for Continuous-Time Nonlinear Systems With Unmodeled Dynamics

This article considers the robust optimal control problem for a class of nonlinear systems in the presence of unmodeled dynamics. An adaptive optimal controller is designed using the online actor–critic learning and is robustified against unmodeled dynamics. To deal with unmodeled dynamics, an auxiliary signal with the system state as its input signal is designed to capture the input-to-state stability. In addition to the critic network for value function approximation, a novel robustifying term is developed and introduced into the actor network to ensure robustness during the learning process. It is shown that both the actor and the critic weights learning converge to their optimal values while guaranteeing the boundedness of all the signals in the closed loop. Simulation examples are conducted to verify the efficacy of the presented scheme.

Offline approximate value iteration for dynamic solutions to the multivehicle routing problem with stochastic demand

The multivehicle routing problem with stochastic demand (MVRPSD) is an important issue both in theory and practice. However, solving the MVRPSD through traditional methods, such as a priori optimization or rollout-algorithm-based dynamic programming is generally limited by the issues of computation efficiency and solution quality. Under increasing demand for efficient real-time logistics, we propose a novel offline approximate value iteration (OAVI) algorithm for dynamic solutions to the MVRPSD. Our algorithm benefits from offline training and thus can provide fast and effective online dynamic routing solutions. Adopting such a novel and effective algorithm presents two challenges: first, we must define a proper cost structure for the dynamic routing decision; second, we must efficiently address the curse of dimensionality of the multivehicle problem. To solve these problems, we first describe the cost structure through the value function approximation (VFA) with basis functions involving a priori cost and a priori credibility; we then design two strategies, recourse reduction (RR) and neighborhood reduction (NR), to prune the action space. The numerical experiments show that our algorithm can substantially enhance computation efficiency and solution quality compared to traditional methods.

An approximate dynamic programming approach for solving an air combat maneuvering problem

Within visual range air combat involves execution of highly complex and dynamic activities, requiring rapid, sequential decision-making to achieve success. Fighter pilots spend years perfecting tactics and maneuvers for these types of combat engagements, yet the ongoing emergence of unmanned, autonomous vehicle technologies elicits a natural question — can an autonomous unmanned combat aerial vehicle (AUCAV) be imbued with the necessary artificial intelligence to perform challenging air combat maneuvering tasks independently? We formulate and solve the air combat maneuvering problem (ACMP) to examine this important question, developing a Markov decision process (MDP) model to control a defending AUCAV seeking to destroy an attacking adversarial vehicle. The MDP model includes a 5-degree-of-freedom, point-mass aircraft state transition model to accurately represent both kinematics and energy while maneuvering. An approximate dynamic programming (ADP) approach is proposed wherein we develop and test an approximate policy iteration algorithm that implements value function approximation via neural network regression to attain high-quality maneuver policies for the AUCAV. A representative intercept scenario is specified for testing purposes wherein the AUCAV must engage and destroy an adversary aircraft attempting to penetrate the defended airspace. Several designed experiments are conducted to determine how aircraft velocity and adversary maneuvering tactics impact the efficacy of the proposed ADP solution approach and to enable efficient algorithm parameter tuning. ADP-generated policies are compared to two benchmark maneuver policies constructed from two reward shaping functions found in the ACMP literature, attaining improved mean probabilities of kill for 24 of 36 air combat situations considered.

Dynamic Valuation of Battery Lifetime

This paper proposes a dynamic valuation framework to determine the opportunity value of battery capacity degradation in grid applications based on the internal degradation mechanism and utilization scenarios. The proposed framework follows a dynamic programming approach and includes a piecewise linear value function approximation solution that solves the optimization problem over a long planning horizon. The paper provides two case studies on price arbitrage and frequency regulation using real market and system data to demonstrate the broad applicability of the proposed framework. Results show that the battery lifetime value is critically dependent on both the external market environment and its internal state of health. On the grid service side, results show that second-life batteries can provide more than 50% of the value compared to new batteries, and frequency regulation provides two times more revenue than price arbitrage throughout the battery lifetime.

A reinforcement Learning approach to resource allocation in genomic selection

Genomic selection (GS) is a technique that plant breeders use to select individuals to mate and produce new generations of species. Allocation of resources is a key factor in GS. At each selection cycle, breeders are facing the choice of budget allocation to make crosses and produce the next generation of breeding parents. Inspired by recent advances in reinforcement learning for AI problems, we develop a reinforcement learning-based algorithm to automatically learn to allocate limited resources across different generations of breeding. We mathematically formulate the problem in the framework of Markov Decision Process (MDP) by defining state and action spaces. To avoid the explosion of the state space, an integer linear program is proposed that quantifies the trade-off between resources and time. Finally, we propose a value function approximation method to estimate the action-value function and then develop a greedy policy improvement technique to find the optimal resources. We demonstrate the effectiveness of the proposed method in enhancing genetic gain using a case study with realistic data.

Attitude control for hypersonic reentry vehicles: An efficient deep reinforcement learning method

Aiming at the attitude control problem of hypersonic reentry vehicles (HRVs), a deep reinforcement learning (DRL) based anti-disturbance control method is proposed. First, a compound control framework consisting of a DRL-based auxiliary controller and a fixed-time anti-disturbance controller is proposed to improve the control performance under the premise of ensuring stability. Then, a novel value function approximation mechanism, named experience-based value expansion (EVE), is proposed to modify the value function update equation based on a two-dimensional replay buffer, which solves the DRL convergence problem brought by the HRV’s strong nonlinearities, tight coupling, and big flight envelope. Furthermore, a result-oriented encoder (ROE) is proposed to solve the DRL generalization problem brought by the HRV’s high uncertainties and unavailable real training environment. A bottleneck shape neural network structure is used for the DRL’s network structure to extract high-dimensional features and prevent overfitting to the training environment. Finally, abundant numerical comparative simulations demonstrate the effectiveness of the proposed efficient DRL algorithms and the DRL-based attitude controller.

Study of dynamic half-cell potential signal of rock bolt under stray current interference based on probabilistic-based method

Dynamic stray current leads to severe electrochemical corrosion on rock bolt in large cross-section subway tunnel. It’s important to evaluate the corrosion risk of corroded rock bolt based on monitoring signal. In this paper, an experiment for half-cell potential measurement of rock bolt under dynamic stray current interference was carried out. Tested non-stationary half-cell potential signal was analyzed the dynamic characteristics of half-cell potential in time and frequency domain. Herein, two parameters: potential shift ΔE and time ratio rt, were put forward to capture the characteristics of half-cell potential signal. A probabilistic-based analysis method was therefore proposed to quantify the corrosion risk of non-stationary half-cell potential, in which PDF of potential shift signal was fitted by generalized extreme value (GEV) function and kernel density estimation (KDE) technique. It is found that location parameter μ in GEV and the sum of the absolute values of the PDF derivative Sd could effectively represent potential shift and time ratio in half-cell potential signal respectively.

Discovering diverse solutions in deep reinforcement learning by maximizing state–action-based mutual information

Reinforcement learning algorithms are typically limited to learning a single solution for a specified task, even though diverse solutions often exist. Recent studies showed that learning a set of diverse solutions is beneficial because diversity enables robust few-shot adaptation. Although existing methods learn diverse solutions by using the mutual information as unsupervised rewards, such an approach often suffers from the bias of the gradient estimator induced by value function approximation. In this study, we propose a novel method that can learn diverse solutions without suffering the bias problem. In our method, a policy conditioned on a continuous or discrete latent variable is trained by directly maximizing the variational lower bound of the mutual information, instead of using the mutual information as unsupervised rewards as in previous studies. Through extensive experiments on robot locomotion tasks, we demonstrate that the proposed method successfully learns an infinite set of diverse solutions by learning continuous latent variables, which is more challenging than learning a finite number of solutions. Subsequently, we show that our method enables more effective few-shot adaptation compared with existing methods.

Differentiated Pricing of Shared Mobility Systems Considering Network Effects

Over the last decades, shared mobility systems have become an integral part of inner-city mobility. Modern systems allow one-way rentals, that is, customers can drop off the vehicle at a different location to where they began their trip. A prominent example is car sharing. Indeed, this work was motivated by the insight we gained in collaborating closely with Europe’s largest car sharing provider, Share Now. In car sharing, as well as in shared mobility systems in general, pricing optimization has turned out to be a promising means of increasing profit while challenged by limited vehicle supply and asymmetric demand across time and space. Thus, in practice, providers increasingly use minute pricing that is differentiated according to where a rental originates, that is, considering its location and the time of day. In research, however, such approaches have not been considered yet. In this paper, we therefore introduce the corresponding origin-based differentiated, profit-maximizing pricing problem for shared mobility systems. The problem is to determine spatially and temporally differentiated minute prices, taking network effects on the supply side and several practice relevant aspects into account. Based on a deterministic network flow model, we formulate the problem as a mixed-integer linear program and prove that it is NP-hard. For its solution, we propose a temporal decomposition approach based on approximate dynamic programming. The approach integrates a value function approximation to incorporate future profits and account for network effects. Extensive computational experiments demonstrate the benefits of capturing such effects in pricing generally, as well as showing our value function approximation’s ability to anticipate them precisely. Furthermore, in a case study based on Share Now data from Florence in Italy, we observe profit increases of around 9% compared with constant uniform minute prices, which are still the de facto industry standard.

Portfolio choice with return predictability and small trading frictions

This paper studies a portfolio choice problem of a utility-maximizing investor with return predictability and small liquidity costs. By adopting a logarithmic-return assumption, our asymptotic expansion around small liquidity costs provides the closed-form expressions for the first-order approximation of the value function and the associated almost-optimal trading strategy. The almost-optimal trading strategy indicates that the investor should trade toward the optimal frictionless portfolio, instead of directly adopting it. The approximated value function effectively captures the utility loss derived from the investor's inability to adopt the optimal frictionless portfolio directly over time. Finally, our numerical analysis indicates that the investor's utility loss is sensitive to the specifications of the return-predicting factors and that the investor's trading behaviors under the logarithmic-return and arithmetic-return assumptions can differ remarkably over a medium to a long investment horizon.

Queueing Network Controls via Deep Reinforcement Learning

Novel advanced policy gradient (APG) methods, such as trust region policy optimization and proximal policy optimization (PPO), have become the dominant reinforcement learning algorithms because of their ease of implementation and good practical performance. A conventional setup for notoriously difficult queueing network control problems is a Markov decision problem (MDP) that has three features: infinite state space, unbounded costs, and long-run average cost objective. We extend the theoretical framework of these APG methods for such MDP problems. The resulting PPO algorithm is tested on a parallel-server system and large-size multiclass queueing networks. The algorithm consistently generates control policies that outperform state-of-art heuristics in literature in a variety of load conditions from light to heavy traffic. These policies are demonstrated to be near optimal when the optimal policy can be computed. A key to the successes of our PPO algorithm is the use of three variance reduction techniques in estimating the relative value function via sampling. First, we use a discounted relative value function as an approximation of the relative value function. Second, we propose regenerative simulation to estimate the discounted relative value function. Finally, we incorporate the approximating martingale-process method into the regenerative estimator.

Nonasymptotic Analysis of Monte Carlo Tree Search

In “Nonasymptotic Analysis of Monte Carlo Tree Search,” D. Shah, Q. Xie, and Z. Xu consider the popular tree-based search strategy, the Monte Carlo Tree Search (MCTS), in the context of the infinite-horizon discounted Markov decision process. They show that MCTS with an appropriate polynomial rather than logarithmic bonus term indeed leads to the desired convergence property. The authors derive the results by establishing a polynomial concentration property of regret for a class of nonstationary multiarm bandits. Furthermore, using this as a building block, they demonstrate that MCTS, combined with nearest neighbor supervised learning, acts as a “policy improvement” operator that can iteratively improve value function approximation.