Approximate Dynamic Programming Algorithm Research Articles

A kernel-based approximate dynamic programming approach: Theory and application

This article proposes a novel kernel-based Dynamic Programming (DP) approximation method to tackle the typical curse of dimensionality of stochastic DP problems over the finite time horizon. Such a method utilizes kernel functions in combination with Support Vector Machine (SVM) regression to determine an approximate cost function for the entire state space of the underlying Markov Decision Process (MDP), by leveraging cost function computed for selected representative states. Kernel functions are used to define the so-called kernel matrix, while the parameter vector of the given kernel-based cost function approximation is computed by moving backwards in time from the terminal condition and by applying SVM regression. This way, the difficulty of selecting a proper set of features is also tackled. The proposed method is then extended to the infinite time horizon case. To show the effectiveness of the proposed approach, the resulting Recursive Residual Approximate Dynamic Programming (RR-ADP) algorithm is applied to the sensor scheduling design in multi-process remote state estimation problems.

A Fully Distributed ADP Algorithm for Real-Time Economic Dispatch of Microgrid

This paper proposes a fully distributed approximate dynamic programming (FD-ADP) algorithm framework for real-time economic dispatch of a microgrid (MG). Firstly, a modified distributed consensus-based algorithm (DCBA) framework with power restriction factors is proposed to distributedly obtain the real-time ED decision of MG containing distributed energy resources with linear cost functions, which improves the convergence stability compared with the traditional DCBA framework. Then, combined with the approximate dynamic programming (ADP) theory, a FD-ADP algorithm is proposed to deal with data uncertainties. To this end, the piece-wise linear (PWL) function is applied to approximate the value function, and a distributed slope update approach is proposed to embed empirical knowledge into the value function while ensuring the distributed property in off-line training. With the well-trained PWL function assisting in decision, the distributed real-time ED strategy of MG with approximate global optimality is obtained by the proposed FD-ADP algorithm during the entire schedule horizon (e.g., 24 hours). Finally, case studies on the modified IEEE 6-bus MG system and IEEE 123-bus MG system demonstrate the superiority of solution convergence, validity of distributed training, and optimality of distributed schedule of the proposed FD-ADP algorithm.

Data-Driven Optimal Tracking Control for Discrete-Time Nonlinear Systems With Unknown Dynamics Using Deterministic ADP.

This article aims to solve the optimal tracking problem (OTP) for a class of discrete-time (DT) nonlinear systems with completely unknown dynamics. A novel data-driven deterministic approximate dynamic programming (ADP) algorithm is proposed to solve this kind of problem with only input-output (I/O) data. The proposed algorithm has two advantages compared to existing data-driven deterministic ADP algorithms for the OTP. First, our algorithm can guarantee optimality while achieving better performance in the aspects of time-saving and robustness to data. Second, the near-optimal control policy learned by our algorithm can be implemented without considering expected control and enable the system states to track the user-specified reference signals. Therefore, the tracking performance is guaranteed while simplifying the algorithm implementation. Furthermore, the convergence and stability of the proposed algorithm are strictly proved through theoretical analysis, in which the errors caused by neural networks (NNs) are considered. At the end of this article, the developed algorithm is compared with two representative deterministic ADP algorithms through a numerical example and applied to solve the tracking problem for a two-link robotic manipulator. The simulation results demonstrate the effectiveness and advantages of the developed algorithm.

Multistage reliability-constrained stochastic planning of diamond distribution network: An approximate dynamic programming approach

In this paper, a multistage reliability-constrained stochastic planning model is established for the diamond distribution network, a new type of distribution system grid structure, with the goal of optimizing the total cost of investment and operation. The uncertainty of load growth is simulated by generating multiple load growth scenarios, and the power supply loss caused by the reliability of components in the system is also considered. The multistage stochastic dynamic decision process is modeled by The Markov decision process. In order to solve the “curse of dimensionality” problem in multi-stage stochastic planning model, the approximate dynamic programming algorithm is used to solve it. The case study is based on the typical structure of the diamond distribution network and realistic data of an actual district in Shanghai, China. The planning results verify the effectiveness of the model and the solution algorithm, and demonstrate the reliability and economy of the diamond distribution network are superior to the traditional double-ring network.

Cost-aware defense for parallel server systems against reliability and security failures

Parallel server systems in transportation, manufacturing, and computing heavily rely on dynamic routing using connected cyber components for computation and communication. Yet, these components remain vulnerable to random malfunctions and malicious attacks, motivating the need for resilient dynamic routing that is both traffic-stabilizing and cost-efficient. In this paper, we consider a parallel server system with dynamic routing subject to reliability and stability failures. For the reliability setting, we consider an infinite-horizon Markov decision process where the system operator strategically activates a protection mechanism upon each job arrival based on traffic state observations. We prove that an optimal deterministic threshold protecting policy exists based on the dynamic programming recursion of the Hamilton–Jacobi–Bellman equation. For the security setting, we extend the model to an infinite-horizon stochastic game where the attacker strategically manipulates routing assignments. We show that both players follow a threshold strategy at every Markov perfect equilibrium. For both failure settings, we also analyze the stability of the traffic queues. Finally, we develop approximate dynamic programming algorithms to compute the optimal/equilibrium policies and present numerical examples/experiments for validation and illustration.

Stochastic optimal dispatch of offshore-onshore regional integrated energy system based on improved state-space approximate dynamic programming

The offshore-onshore regional integrated energy system (OORIES), which includes offshore gas production platforms and wind farms, along with the onshore gas-fired combined cooling, heating, and power supply campus, can effectively improve economic and environmental benefits by the complementary and coupling operation of multiple energies. In this paper, multi-energy coupling elements and multiple energy networks in the OORIES are modeled in detail. Considering the stochastic fluctuation of the offshore gas source and wind power output, a stochastic optimal dispatch (SOD) model of OORIES with multiple energy storages, including associated gas storage, hydrogen and heat storage tanks is proposed. Furthermore, a carbon trading mechanism is introduced to ensure the environmental benefits. An improved state-space approximate dynamic programming (SSADP) algorithm is proposed to solve the SOD model. By using the approximate value functions, the multi-period SOD model is decoupled into a series of single-period optimization models which are recursively solved period by period. The approximate analytical expression of the value function concerning states of each period is obtained based on the polynomial approximation, and combined with the transition probability matrices of stochastic quantities, an accurate and efficient solution for each single-period optimization model is obtained. A case study on an OORIES, and compared with the scenario-based algorithm, chance-constrained programming algorithm and traditional SSADP algorithm, demonstrates the effectiveness and high efficiency of the proposed method.

A simulation-based approximate dynamic programming approach to dynamic and stochastic resource-constrained multi-project scheduling problem

We consider the dynamic and stochastic resource-constrained multi-project scheduling problem which allows for the random arrival of projects and stochastic task durations. Completing projects generates rewards, which are reduced by a tardiness cost in the case of late completion. Multiple types of resource are available, and projects consume different amounts of these resources when under processing. The problem is modelled as an infinite-horizon discrete-time Markov decision process and seeks to maximise the expected discounted long-run profit. We use an approximate dynamic programming algorithm (ADP) with a linear approximation model which can be used for online decision making. Our approximation model uses project elements that are easily accessible by a decision-maker, with the model coefficients obtained offline via a combination of Monte Carlo simulation and least squares estimation. Our numerical study shows that ADP often statistically significantly outperforms the optimal reactive baseline algorithm (ORBA). In experiments on smaller problems however, both typically perform suboptimally compared to the optimal scheduler obtained by stochastic dynamic programming. ADP has an advantage over ORBA and dynamic programming in that ADP can be applied to larger problems. We also show that ADP generally produces statistically significantly higher profits than common algorithms used in practice, such as a rule-based algorithm and a reactive genetic algorithm.

RSM-Based Approximate Dynamic Programming for Stochastic Energy Management of Power Systems

The stochastic energy management (SEM) of power systems is computationally intractable due to its randomness, nonconvexity, and nonlinearity. To solve this problem, a response surface method (RSM)-based approximate dynamic programming (ADP) algorithm is proposed in this paper. Since the value function can be directly obtained by RSM, the proposed algorithm does not need to iteratively approach them as existing ADP algorithms, which facilitates reducing the computing time. In addition, an improved generalized polynomial chaos (IgPC) method (i.e., an extension of RSM) is proposed to calculate the expectation of the value function of ADP in the stochastic environment. Compared with the Monte Carlo method, which is commonly used in existing ADP algorithms, IgPC requires fewer sampling scenarios while providing similar results. Simulation results with two modified IEEE test systems and a real 2778-bus system demonstrate the effectiveness of the proposed algorithm in terms of accuracy and computation efficiency.

Quadratic Taylor Expansion-Based Approximate Dynamic Programming for Fully Decentralized AC-OPF of Multi-Area Power Systems

In this paper, a quadratic Taylor expansion-based approximate dynamic programming (QTE-ADP) algorithm is proposed for the decentralized solution of multi-area AC-optimal power flow (MA-ACOPF). Different from traditional ADP algorithms, the proposed algorithm does not need to approximate the value function via a given function structure, but directly obtains the quadratic Taylor expansion (QTE) of value function based on KKT conditions. Moreover, compared with the commonly used linearized value function approximation techniques, the information used to approximate the value function is also extended from first order to second order in the proposed algorithm, which helps to improve the accuracy and efficiency of ADP. When using QTE-ADP to solve the MA-ACOPF problem, only the boundary voltage information needs to be interchanged between adjacent areas, which facilitates preserving the information privacy and decision independence of each area. Numerical simulations on several test systems demonstrate that the proposed algorithm has good performance in terms of accuracy, efficiency, and adaptability.

Policy-Iteration-Based Finite-Horizon Approximate Dynamic Programming for Continuous-Time Nonlinear Optimal Control.

The Hamilton-Jacobi-Bellman (HJB) equation serves as the necessary and sufficient condition for the optimal solution to the continuous-time (CT) optimal control problem (OCP). Compared with the infinite-horizon HJB equation, the solving of the finite-horizon (FH) HJB equation has been a long-standing challenge, because the partial time derivative of the value function is involved as an additional unknown term. To address this problem, this study first-time bridges the link between the partial time derivative and the terminal-time utility function, and thus it facilitates the use of the policy iteration (PI) technique to solve the CT FH OCPs. Based on this key finding, the FH approximate dynamic programming (ADP) algorithm is proposed leveraging an actor-critic framework. It is shown that the algorithm exhibits important properties in terms of convergence and optimality. Rather importantly, with the use of multilayer neural networks (NNs) in the actor-critic architecture, the algorithm is suitable for CT FH OCPs toward more general nonlinear and complex systems. Finally, the effectiveness of the proposed algorithm is demonstrated by conducting a series of simulations on both a linear quadratic regulator (LQR) problem and a nonlinear vehicle tracking problem.

Optimizing vaccine distribution in developing countries under natural disaster risk

AbstractFor many developing countries, COVID‐19 vaccination roll‐out programs are not only slow but vaccination centers are also exposed to the risk of natural disaster, like flooding, which may slow down vaccination progress even further. Policy‐makers in developing countries therefore seek to implement strategies that hedge against distribution risk in order for vaccination campaigns to run smoothly and without delays. We propose a stochastic‐dynamic facility location model that allows policy‐makers to choose vaccination facilities while accounting for possible facility failure. The model is a multi‐stage stochastic variant of the classic facility location problem where disruption risk is modelled as a binary multivariate random process–a problem class that has not yet been studied in the extant literature. To solve the problem, we propose a novel approximate dynamic programming algorithm which trains the shadow price of opening a flood‐prone facility on historical data, thereby alleviating the need to fit a stochastic model. We trained the model using rainfall data provided by the local government of several major cities in the Philippines which are exposed to multiple flooding events per year. Numerical results demonstrate that the solution approach yields approximately 30%–40% lower cost than a baseline approach that does not consider the risk of flooding. Recommendations based on this model were implemented following a collaboration with two large cities in the Philippines which are exposed to multiple flooding events per year.

Energy-efficient power control strategy of the delay tolerable service based on the reinforcement learning

In recent years, the rapid development of Internet technology and its applications has led to an exponential growth in the number of Internet users and wireless terminal devices, resulting in a corresponding increase in energy consumption. This has necessitated the need to reduce energy consumption while maintaining the quality of communication services. To this end, we investigate the possibility of improving energy efficiency (EE) of delay tolerable (DT) services by allocating resources based on the time-domain water-filling algorithm. We first transform the non-convex problem of maximizing EE into a convex problem of minimizing transmission power to obtain the optimal solution, and then use the greedy algorithm to obtain an upper bound. Furthermore, to capture a more realistic scenario, an Approximate Statistical Dynamic Programming (ASDP) algorithm is introduced, but its effect on enhancing EE is limited. To overcome this limitation, three Deep Reinforcement Learning (DRL) algorithms are implemented. The simulations results show that the settings of maximum transmit power and SNR during agent training have an impact on the performance of the agent. Finally, by comparing the mean values of transmission power, outage probability, equilibrium power and performance improvement percentage of several algorithms, we conclude that the Deep Deterministic Policy Gradient (DDPG) algorithm produces the best agent performance in the environment with a fixed SNR of 2 (dB).

Proactive Security-Constrained Unit Commitment Against Typhoon Disasters: An Approximate Dynamic Programming Approach

Proactive security-constrained unit commitment (SCUC) is one of the effective resilience-oriented responses to keep the secure operation of the power system during natural disasters. The approaches proposed in the previous research works cannot guarantee high-quality response timely to various emergencies during typhoon disasters. This paper proposes a proactive SCUC model solved by an approximate dynamic programming (ADP) algorithm to improve system resilience during typhoon disasters while meeting the requirements of the computational time. Both the uncertainty of the typhoon motion path and the wind speed-dependent transmission line failure probability are considered in the scenario generation. The optimization process is formulated as a Markov decision process (MDP), and a piecewise linear function (PLF) based ADP is employed for solving the model. With the exogenous information extracted from generated scenarios and recorded in the slopes of PLFs, the ADP algorithm can result in a near-optimal solution within a short computational time. Simulations are carried out using the modified IEEE RTS-79 system. The significant computational time saving and high-quality solution demonstrate the effectiveness of the proposed approach.

Policy Iteration Based Approximate Dynamic Programming Toward Autonomous Driving in Constrained Dynamic Environment

In the area of autonomous driving, it typically brings great difficulty in solving the motion planning problem since the vehicle model is nonlinear and the driving scenarios are complex. Particularly, most of the existing methods cannot be generalized to dynamically changing scenarios with varying surrounding vehicles. To address this problem, this development here investigates the framework of integrated decision and control. As part of the modules, static path planning determines the reference candidates ahead, and then the optimal path-tracking controller realizes the specific autonomous driving task. An innovative and effective constrained finite-horizon approximate dynamic programming (ADP) algorithm is herein presented to generate the desired control policy for effective path tracking. With the generalized policy neural network that maps from the state to the control input, the proposed algorithm preserves the high effectiveness for the motion planning problem towards changing driving environments with varying surrounding vehicles. Moreover, the algorithm attains the noteworthy advantage of alleviating the typically heavy computational loads with the mode of offline training and online execution. As a result of the utilization of multi-layer neural networks in conjunction with the actor-critic framework, the constrained ADP method is capable of handling complex and multidimensional scenarios. Finally, various simulations have been carried out to show that the constrained ADP algorithm is effective.

Multiple stage stochastic planning of integrated electricity and gas system based on distributed approximate dynamic programming

With the increasing coupling of integrated electricity and gas systems, it's essential to implement a proper planning scheme. The planning of the integrated electricity and gas systems usually consists of multiple planning stages and various uncertainties, making the problem difficult to solve. In this paper, combining the approximate dynamic programming method and the alternating direction multiplier method, a distributed approximate dynamic programming based multiple stage stochastic planning scheme is proposed for the integrated electricity and gas systems. The multiple stage stochastic planning scheme realizes the coordination of new installation and expansion of power grid lines, natural gas pipelines, substations, city gates, photovoltaics, wind turbines, gas turbines and energy storage devices, while considering both the long-term and the short-term uncertainties. The multiple stage stochastic planning model is reformulated as a Markov decision process with a sequential decision strategy, in which the investment variables are decided stage by stage with long-term uncertainties gradually revealed in the planning period. The distributed approximate dynamic programming algorithm is used to solve the multiple stage stochastic planning model, which is in the form of the Markov decision process, by decoupling in both temporal and spatial dimensions. Numerical tests on the test system with a 24-node power grid and a 30-node natural gas network verify the effectiveness of the multiple stage stochastic planning model.

Output‐feedback stochastic model predictive control of chance‐constrained nonlinear systems

AbstractThis study covers the output‐feedback model predictive control (MPC) of nonlinear systems subjected to stochastic disturbances and state chance constraints. The stochastic optimal control problem is solved in a stochastic dynamic programming fashion, and the output‐feedback control is performed with the extended Kalman filter. The information state is summarized as a dynamic Gaussian belief model. Thus, the stochastic Bellman equation is transformed into a deterministic equation using this model. The resulting constrained Bellman equation is solved using the proposed constrained, approximate dynamic programming algorithm. The algorithm is proved to have a Q‐superlinear local convergence rate. Numerical experiments show that the proposed algorithm can attain good control performance and reasonable chance‐constraint satisfaction and is computationally efficient owing to its dynamic programming structure.

Nonparametric Approximate Dynamic Programming via the Kernel Method

This paper presents a novel, non-parametric approximate dynamic programming (ADP) algorithm that enjoys dimension-independent approximation and sample complexity guarantees. We obtain this algorithm by “kernelizing” a recent mathematical program for ADP (the “smoothed approximate linear program”). Loosely, our guarantees show that we can exchange the importance of choosing a good approximation architecture a priori (as required by existing approaches) with sampling effort. We also present a simple active set algorithm for solving the resulting quadratic program, and prove the correctness of this method. Via a computational study on a controlled queueing network, we show that our approach is capable of outperforming parametric linear programming approaches to ADP, as well as non-trivial, tailored heuristics for the same network, even when employing generic, polynomial kernels.

Optimal decision-making of mutual fund temporary borrowing problem via approximate dynamic programming

Temporary borrowing is a liquidity risk management tool for mutual fund managers to meet investor redemption demands. We develop a new Markov decision process model to describe the temporary borrowing process, considering the multiple lending channels, the validity period of loans, and the uncertainties of cost, demand, and maximum loan amount simultaneously. An approximate dynamic programming (ADP) algorithm is initiated to solve the cost-minimizing temporary borrowing problem. We construct the value function as a separable approximation and prove the convexity of its components with respect to the available funds in different channels. Moreover, a new value function updating formula, DMAX, is designed to overcome value function overestimation. The proved convexity and the proposed formula contribute to fast and reliable value function estimation. Numerical experiments based on actual business data show that the proposed algorithm can obtain near-optimal decisions in deterministic cases and maintain high robustness in stochastic cases.

Metalearning of time series: an approximate dynamic programming approach

A new approximate dynamic programming algorithm applied to time series forecasting

Dynamic surgery management under uncertainty

Real-time surgery management involves a complex and dynamic decision-making process. The duration of surgeries in many cases cannot be known until the surgery has actually been completed. Furthermore, disruptions such as equipment failure or the arrival of a non-elective surgery can occur simultaneously. Thus, the assignment of surgeries needs to be updated, as and when disruptions occur, to minimize their effects. In this paper, we present a stochastic dynamic programming approach to the surgery allocation problem with multiple operating rooms under uncertainty. Given an elective list for the day, the dynamic optimization model minimizes the number of surgeries not carried out by the end of the shift and the total waiting times of patients during the day weighted according to their urgency level. Due to the curse of dimensionality, we apply an approximate dynamic programming algorithm to solve the stochastic dynamic surgery management model. Computational experiments are designed to demonstrate the performance of the proposed algorithm and its applicability to practical settings. The results show that the approximate dynamic programming algorithm provides a good approximation to the optimum policy and leads to some managerial insights.