Value Function Approximation Research Articles

Deep estimation for Q⁎ with minimax Bellman error minimization

In this paper we consider the estimation of optimal state-action value function Q⁎ with ReLU ResNet based on minimax Bellman error minimization. We construct the non-asymptotic error bounds for the minimax estimator and the estimated Q function induced by the estimated greedy policy. To bound the Bellman residual error, we guarantee the approximation errors based on deep approximation theory and the statistical ones by utilizing empirical processes taking into account the Markov decision process dependency. We provide a novel generalization bound with dependent data and an approximation bound in the Hölder class which are of independent interest. This bound depends on the sample size, the ambient dimension, the width and depth of the neural network, which can bring prior insights into tuning these hyper-parameters to achieve a desired convergence rate in practice. Furthermore, the bound circumvents the curse of dimensionality if the distribution of state-action pairs is assumed to be supported on a set of low intrinsic dimension.

Asymptotically Optimal Sampling Policy for Selecting Top-m Alternatives

We consider selecting the top-m alternatives from a finite number of alternatives via Monte Carlo simulation. Under a Bayesian framework, we formulate the sampling decision as a stochastic dynamic programming problem and develop a sequential sampling policy that maximizes a value function approximation one-step look ahead. To show the asymptotic optimality of the proposed procedure, the asymptotically optimal sampling ratios that optimize the large deviations rate of the probability of false selection for selecting the top-m alternatives have been rigorously defined. The proposed sampling policy is not only proved to be consistent but also achieve the asymptotically optimal sampling ratios. Numerical experiments demonstrate superiority of the proposed allocation procedure over existing ones. History: Accepted by Bruno Tuffin, Area Editor for Simulation. Funding: This work was supported by the National Natural Science Foundation of China [Grants 72250065, 72293582, 72022001, and 71901003], and the National Science Foundation [Grant DMS-2053489], the major project of the National Natural Science Foundation of China [Grant 72293582], and the China Scholarship Council [Grant CSC202206010152]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2021.0333 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2021.0333 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Online attentive kernel-based temporal difference learning

Kernel-based reinforcement learning has received increasing attention because it requires less prior knowledge linear approximation and neural networks. Online kernel-based updating, however, is hindered by the challenge of catastrophic forgetting or interference. Sparse representation is a key method to address this issue, but existing methods fail to satisfy four criteria: learnability, nonprior, nontruncation, and explicitness. In this paper, we present an attentive kernel-based value function approximation as a learnable, nonprior, nontruncated, and explicit sparse representation. We propose the online attentive kernel-based temporal difference (OAKTD) algorithm, which employs two-timescale optimization, and provide a convergence analysis for our proposed algorithm. Experimental results show that OAKTD outperforms online kernel-based TD learning algorithms, and the TD learning algorithm with Tile Coding on classical tasks, i.e., Mountain Car, Acrobot, CartPole and Puddle World.

Differential dynamic programming for finite‐horizon zero‐sum differential games of nonlinear systems

AbstractIn this article, we present an iterative algorithm based on differential dynamic programming (DDP) for finite‐horizon two‐person zero‐sum differential games. The technique of DDP is used to expand the Hamilton–Jacobi–Isaacs (HJI) partial differential equation into higher‐order differential equations. Using value function and saddle point approximations, the DDP expansion is transformed into algebraic matrix equation in integral form. Based on the algebraic matrix equation, a DDP iterative algorithm is developed to learn the solution to the differential games. Strict proof is proposed to guarantee the iterative convergences of the value function and saddle point. The new algorithm is fundamentally different from existing results, in the sense that it overcome the technical obstacle to address the time‐varying behavior of HJI partial differential equation. Simulation examples are given to demonstrate the effectiveness of the proposed method.

A Decentralized Learning Control Scheme for Constrained Nonlinear Interconnected Systems Based on Dynamic Event-Triggered Mechanism

This article presents a decentralized learning control method for a class of partially unknown nonlinear systems with asymmetric control input constraints and mismatched interconnections via a novel dynamic event-triggering condition. By employing an integral reinforcement learning strategy, the system drift dynamics can be avoided in the learning process. Meanwhile, a critic neural network is designed to obtain the approximated value function and tuned by using the gradient descent approach. Furthermore, a novel dynamic event-triggering condition is designed to determine the occurrence of an event by introducing a dynamic variable. By using the Lyapunov theory, all signals in the closed-loop system are proved to be uniformly ultimately bounded. Finally, we present a nonlinear interconnected system and an interconnected power system to verify the effectiveness of the proposed method.

ACC-RL: Adaptive Congestion Control Based on Reinforcement Learning in Power Distribution Networks with Data Centers

Modern data center power distribution networks place greater demands on the stability and reliability of power supply. Growing network computing demands and complex network environments can cause network congestion, which in turn leads to network traffic overload and power supply equipment overload. Therefore, network congestion is one of the most important problems faced by data center power distribution networks. In this paper, we propose an approach called ACC-RL based on reinforcement learning (RL), which can effectively avoid network congestion and improve energy performance. ACC-RL models the congestion control task as a Partially Observable Markov Decision Process (POMDP). It is independent of the estimated value function and supports deterministic policies. It also sets the reward value function using real-time network information such as the transmission rate, RTT, and switch queue length, with the target transmission rate as the target equilibrium point. ACC-RL is highly general, can be trained on datasets running in different network environments, and generates a robust congestion control policy. The experimental results show that ACC-RL can solve the congestion problem without any predefined scenarios in different network environments. It can control the network traffic well, thus ensuring the stability and reliability of the power supply in the distribution network. We conduct network simulation experiments through NS-3. We set up different scenarios for experiments and data analysis in many-to-one, all-to-all, and long–short network environments. Compared with the popular rule-based congestion control algorithms such as TIMELY, DCQCN, and HPCC, ACC-RL shows different degrees of energy performance advantages in network metrics such as fairness, link utilization, and throughput.

Spatiotemporal decomposed dispatch of integrated electricity-gas system via stochastic dual dynamic programming-based value function approximation

This paper proposes a novel stochastic dual dynamic programming-based value function approximation approach for the spatiotemporal decomposed dispatch of integrated electricity-gas systems with uncertainties. Stochastic dual dynamic programming is employed to decompose the optimal dispatch problem into several subproblems in both spatial and temporal dimensions. Then, Benders cuts are used in the real-time dispatch process to describe the interaction among these subproblems, according to which each subsystem can make decentralized and real-time decisions. In this way, both the information privacy and decision independence of each subsystem can be guaranteed, while the future variability of wind power and loads can be firmed. Moreover, the historical information can be used to obtain the Benders cuts offline, which helps to omit the time-consuming iterations in the real-time decision stage, while the near-optimal solutions can still be obtained. The effectiveness of the proposed approach is verified by case studies on a 4-bus-4-node system and a 118-bus-20-node system. Numerical results demonstrate that the proposed method reduces the average error of the operation cost by 1 order of magnitude compared with the traditional real-time methods. Moreover, the computational time is significantly reduced by up to 4 orders of magnitude compared with the traditional decentralized algorithms.

Reinforcement learning method for the multi-objective speed trajectory optimization of a freight train

With the rising urge to mitigate the green house effect, reducing the energy consumption of the freight train attracts much attention. Multiple constraints should be taken into account to solve the energy-efficient control problem, which can be reformulated as the multi-objective optimization. This paper proposes a Reinforcement Learning (RL) method for the multi-objective speed trajectory optimization with the goal of the energy-efficiency, punctuality and accurate parking simultaneously. Since the solution space for the optimization problem in this paper is large and discrete, a Gated Recurrent Unit (GRU)-based network is proposed to achieve the fast approximation of the optimal value function instead of the lookup Q-table. Meanwhile, a new architecture, including the embedding matrix, is used to model the control sequence that generates the speed trajectory. Besides, this paper constructs a Deep Q-Network (DQN) framework to train the GRU network without relying on the prior knowledge of the freight train model. Finally, the Intelligent Train Operation (ITO) algorithm is proposed and verified, using the data of Beijing–Guangzhou Railway Line and HXD1B electric locomotive. The case studies indicate that the reward function for the ITO algorithm converges rapidly and the energy consumption monotonically decreases with the trip time, which satisfies the multiple optimization objectives. In terms of saving the energy consumption, the ITO algorithm performs better than Fuzzy Predictive Control (FPC), Genetic Algorithm (GA) and the field test data. The computation time of different speed trajectories demonstrates that the ITO algorithm is applicable to generating the optimal speed trajectory off-line.

Approximation Benefits of Policy Gradient Methods with Aggregated States

Folklore suggests that policy gradient can be more robust to misspecification than its relative, approximate policy iteration. This paper studies the case of state-aggregated representations, in which the state space is partitioned and either the policy or value function approximation is held constant over partitions. This paper shows a policy gradient method converges to a policy whose regret per period is bounded by ϵ, the largest difference between two elements of the state-action value function belonging to a common partition. With the same representation, both approximate policy iteration and approximate value iteration can produce policies whose per-period regret scales as [Formula: see text], where γ is a discount factor. Faced with inherent approximation error, methods that locally optimize the true decision objective can be far more robust. This paper was accepted by Hamid Nazerzadeh, data science. Supplemental Material: Data are available at https://doi.org/10.1287/mnsc.2023.4788 .

Reducing sample size requirements by extending discrete choice experiments to indifference elicitation

Discrete choice (DC) methods provide a convenient approach for preference elicitation and they lead to unbiased estimates of preference model parameters if the parameterization of the value function allows for a good description of the preferences. On the other hand, indifference elicitation (IE) has been suggested as a direct trade-off estimator for preference elicitation in decision analysis decades ago, but has not found widespread application in statistical analysis frameworks as for discrete choice methods. We develop a hierarchical, probabilistic model for IE that allows us to do Bayesian inference similar to DC methods. A case study with synthetically generated data allows us to investigate potential bias and to estimate parameter uncertainty over a wide range of numbers of replies and elicitation uncertainties for both DC and IE. Through an empirical case study with laboratory-scale choice and indifference experiments, we investigate the feasibility of the approach and the excess time needed for indifference replies. Our results demonstrate (i) the absence of bias of the suggested methodology, (ii) a reduction in the uncertainty of estimated parameters by about a factor of three or a reduction of the required number of replies to achieve a similar accuracy as with DC by about a factor of ten, (iii) the feasibility of the approach, and (iv) a median increase in time needed for indifference reply of about a factor of three. If the set of respondents is small, the higher elicitation effort may be worth to achieve a reasonable accuracy in estimated value function parameters.

Solution for Pursuit-Evasion Game of Agents by Adaptive Dynamic Programming

The paper studies a novel method for real-time solutions of the two-player pursuit-evasion game. The min-max principle is adopted to confirm the Nash equilibrium of the game. As agents in the game can form an Internet of Things (IoT) system, the real-time control law of each agent is obtained by taking a linear-quadratic cost function in adaptive dynamic programming. By introducing the Lyapunov function, we consider the scenario when capture occurs. Since most actual systems are continuous, the policy iteration algorithm is used to make the real-time policy converge to the analytical solution of the Nash equilibrium. Furthermore, we employ the value function approximation method to calculate the neural network parameters without directly solving the Hamilton–Jacobi–Isaacs equation. Simulation results depict the method’s feasibility in different scenarios of the pursuit-evasion game.

Optimal Adaptive Control of Uncertain Nonlinear Continuous-Time Systems With Input and State Delays.

In this article, an actor-critic neural network (NN)-based online optimal adaptive regulation of a class of nonlinear continuous-time systems with known state and input delays and uncertain system dynamics is introduced. The temporal difference error (TDE), which is dependent upon state and input delays, is derived using actual and estimated value function and via integral reinforcement learning. The NN weights of the critic are tuned at every sampling instant as a function of the instantaneous integral TDE. A novel identifier, which is introduced to estimate the control coefficient matrices, is utilized to obtain the estimated control policy. The boundedness of the state vector, critic NN weights, identification error, and NN identifier weights are shown through the Lyapunov analysis. Simulation results are provided to illustrate the effectiveness of the proposed approach.

Sample Complexity and Overparameterization Bounds for Temporal-Difference Learning With Neural Network Approximation

In this paper, we study the dynamics of temporal difference learning with neural network-based value function approximation over a general state space, namely, Neural TD learning. We consider two practically used algorithms, projection-free and max-norm regularized Neural TD learning, and establish the first convergence bounds for these algorithms. An interesting observation from our results is that max-norm regularization can dramatically improve the performance of TD learning algorithms in terms of sample complexity and overparameterization. The results in this work rely on a Lyapunov drift analysis of the network parameters as a stopped and controlled random process.

Neural Temporal Difference and Q Learning Provably Converge to Global Optima

Temporal difference learning (TD), coupled with neural networks, is among the most fundamental building blocks of deep reinforcement learning. However, because of the nonlinearity in value function approximation, such a coupling leads to nonconvexity and even divergence in optimization. As a result, the global convergence of neural TD remains unclear. In this paper, we prove for the first time that neural TD converges at a sublinear rate to the global optimum of the mean-squared projected Bellman error for policy evaluation. In particular, we show how such global convergence is enabled by the overparameterization of neural networks, which also plays a vital role in the empirical success of neural TD. We establish the theory for two-layer neural networks in the main paper and extend them to multilayer neural networks in the appendix. Beyond policy evaluation, we establish the global convergence of neural (soft) Q learning. Funding: Z. Yang acknowledges the Theory of Reinforceement Learning program at Simons Institute. J. D. Lee acknowledges support of the ARO under MURI Award W911NF-11-1-0304, the Sloan Research Fellowship, NSF CCF 2002272, NSF IIS 2107304, ONR Young Investigator Award, and NSF-CAREER under award #2144994. Z. Wang acknowledges National Science Foundation [Awards 2048075, 2008827, 2015568, 1934931], Simons Institute (Theory of Reinforcement Learning), Amazon, J.P. Morgan, and Two Sigma for their supports.

Cross-docking based factory logistics unitisation process: An approximate dynamic programming approach

The factory logistics increases in importance as more processes and types of components are needed to produce a product. Due to the limited industrial space and stressful competitive environment, manufacturers increasingly outsource logistics-related activities to Third-Party Logistics (3PLs) and Supply hubs. However, the challenges, including unsynchronised decisions with other partners, non-value-added warehousing operations, and excessive space occupation, still burden manufacturers. This research proposes a cross-docking-based factory logistics unitisation process (CD-FLUP) that integrates the cross-docking technique into the assembly line feeding process and uses mobile shelves for component holding and delivery. We formulate the CD-FLUP with an integer program to minimise the total number of shelf units put into storage and the number of deliveries to assembly lines. Based on time decomposition, we propose an approximate dynamic programming (ADP) approach to solve the model efficiently. The time-varying value function approximation (VFA) developed in the proposed ADP approach reduces the approximation parameters and speeds up convergence significantly. The numerical experiment shows that the proposed ADP can generate high-quality solutions far more efficiently than a commercial solver.

A tutorial on value function approximation for stochastic and dynamic transportation

A tutorial on value function approximation for stochastic and dynamic transportation

Study on learning algorithm of transfer reinforcement for multi-agent formation control

Considering the obstacle avoidance and collision avoidance for multi-agent cooperative formation in multi-obstacle environment, a formation control algorithm based on transfer learning and reinforcement learning is proposed. Firstly, in the source task learning stage, the large storage space required by Q-table solution is avoided by using the value function approximation method, which effectively reduces the storage space requirement and improves the solving speed of the algorithm. Secondly, in the learning phase of the target task, Gaussian clustering algorithm was used to classify the source tasks. According to the distance between the clustering center and the target task, the optimal source task class was selected for target task learning, which effectively avoided the negative transfer phenomenon, and improved the generalization ability and convergence speed of reinforcement learning algorithm. Finally, the simulation results show that this method can effectively form and maintain formation configuration of multi-agent system in complex environment with obstacles, and realize obstacle avoidance and collision avoidance at the same time.

Safe batch constrained deep reinforcement learning with generative adversarial network

Batch-constrained reinforcement learning constrains the learned policy to be close to the behavior policy, which holds a tremendous promise for alleviating the distributional shift in offline reinforcement learning. Existing batch-constrained techniques rely on perturbation models to adjust the actions generated from the generative model to maximize the estimated value function. However, the perturbation model deviates from the distribution of the offline datasets and introduces a new distribution drift problem, which affects the performance of the learned policies. In addition, since offline reinforcement learning cannot learn by trial and error, the final policies are often prone to failure in reality or make unsafe decisions when trained with a noisy or small size dataset. To address the above issues, this paper employs constrained generative adversarial network to generate actions with given states. Specifically, we train the generator to maximize the estimated value and constrain the state-action pairs to follow the dataset distribution. The perturbation model is trained to maximize the probability of the perturbed actions belonging to the dataset and minimize the likelihood of taking dangerous actions. Moreover, we utilize safety critics to predict the risk of the actions under a state. Experimental results show that the proposed method is effective and can choose safe actions while maintaining a high performance in offline settings.

Physics-informed deep reinforcement learning-based integrated two-dimensional car-following control strategy for connected automated vehicles

Connected automated vehicles (CAVs) are broadly recognized as next-generation transformative transportation technologies having great potential to improve traffic safety, efficiency, and stability. Efficiently controlling CAVs on two-dimensional curvilinear roadways to follow preceding vehicles is denoted as the two-dimensional car-following process, which is highly critical; this process is challenging to implement owing to the complexity and varied nature of driving environments. This study proposes an innovative integrated two-dimensional control strategy for CAVs based on deep reinforcement learning (DRL), which efficiently regulates the two-dimensional car-following process of CAVs in terms of both stability-wise longitudinal control performance and accurate lateral path-tracking performance. Within the control framework, each CAV can receive the surrounding information from downstream vehicles and roadway geometry based on vehicle-to-everything (V2X) communication. To better utilize this information, we designed a physics-informed DRL state fusion approach and reward function, which efficiently embeds prior physics knowledge and borrows the merits of the equilibrium and consensus concepts from the control theory. Given the physics-informed information, the DRL-based controller outputs the integrated control instructions for both longitudinal and lateral control. For training, we constructed a roadway with a set of varying curvatures and embedded the ground-truth vehicle trajectory datasets to more effectively capture the realistic variations in the roadway geometry and driving environment. To facilitate value function approximation and enhance the policy iteration process in training, the distributed proximal policy optimization (DPPO) algorithm was applied, owing to its balanced performance. A series of simulated experiments were conducted to validate the controller’s lateral control accuracy and stability-wise oscillation dampening performance in diverse traffic scenarios, including extreme ones.

Solving the joint military medical evacuation problem via a random forest approximate dynamic programming approach

Members of the armed forces rely on having an effective and efficient medical evacuation (MEDEVAC) process for evacuating casualties from the battlefield to medical treatment facilities during combat operations. Determining which MEDEVAC units to task to respond to incoming requests for service, known as the MEDEVAC dispatching problem, is a natural sequential decision making problem and is the focus of this paper. Unlike previous research in this area, this paper incorporates triage classification errors and the possibility of having blood transfusion kits on board select MEDEVAC units. A discounted, infinite-horizon continuous-time Markov decision process model is formulated to examine the MEDEVAC dispatching problem and to compare generated dispatching policies to the currently practiced United States military policy. We utilize an approximate dynamic programming (ADP) technique that leverages a random forest value function approximation within an approximate policy iteration algorithmic framework to develop high-quality policies for both small-scale and large-scale problem instances. A representative planning scenario involving joint combat operations in South Korea is developed and utilized to investigate the differences between the various policies. Results from the analysis indicate that applying ADP techniques can improve current practices by as much as 29% with regard to a life-saving performance metric. This research is of particular interest to the military medical community and can inform the procedures of future military MEDEVAC operations.