Approximate Dynamic Programming Research Articles

Differential neural network robust constrained controller using approximate dynamic programming

This study focuses on developing a continuous differential neural network (DNN) approximating min–max robust control by applying dynamic neural programming. The suggested controller is applied to a class of nonlinear perturbed systems providing satisfactory dynamics for a given cost function depending on both the trajectories of the perturbed system and the designed constrained control actions. The min–max formulation for dynamic programming offers reliable control for restricted modeling uncertainties and perturbations. The suggested design considers control norm restrictions in the optimization problem. DNN’s approximation of the worst (with respect to the admissible class of perturbations and uncertainties) value function of the Hamilton–Jacobi–Bellman (HJB) equation enables the estimation of the closed-loop formulation of the controller. The robust version of the HJB partial differential equation is studied to create the learning law class for the time-varying weights in the DNN. The controller employs a time-varying Lyapunov-like differential equation and the solution of the corresponding learning laws. A recurrent algorithm based on the Kiefer–Wolfowitz technique can be used by modifying the weights’ initial conditions to fulfill the specified cost function’s end requirements. A numerical example tests the robust control proposed in this study, validating the robust optimal solution based on the DNN approximation for Bellman’s value function.

Approximate Dynamic Programming for Control of Wave Energy Converters With Implementation and Validation on a Point Absorber Prototype

It is recognized that the application of constrained optimal control for wave energy converters (WECs), represented by model predictive control (MPC), is hindered by its computation burden resulting from online optimization, especially when the model contains nonlinearities. In this article, a novel control solution based on approximate dynamic programming (ADP) is proposed, in which an explicit controller is solved by model sampling and neural network training. In this way, the major computation load is moved offline to enable fast online execution, and the need for wave prediction is eliminated. The effectiveness of ADP is verified by a simulation case study based on a nonlinear WEC system. Using nonlinear MPC as the performance benchmark, over 90% energy efficiency is achieved by ADP under various sea states with constraint satisfaction and dramatically faster computation. The ADP controller is further experimentally validated on a newly constructed WEC prototype through wave tank testing, where successful implementation and satisfactory performance are demonstrated.

Green transport fleet renewal using approximate dynamic programming: A case study in German heavy-duty road transportation

Governments and manufacturers are starting to enforce the European transport industry’s transition to sustainable mobility. Meanwhile, transport companies have begun to set their own emissions goals. To achieve these sustainably, they must develop efficient policies to renew their fleets with alternative-fuel vehicles. However, since future trends in relevant parameters are highly uncertain, fleet managers struggle to make informed decisions. We formulate fleet renewal as a sequential optimization problem, considering multiple technologies and operational clusters. Vehicle purchase, sales, depreciation, fuel, carbon, and electric battery prices are modeled as stochastic variables. We propose approximate dynamic programming to calculate fleet renewal policies that achieve emissions goals while optimizing total costs of ownership. This approach is tested in a case study of a German logistics service provider. We investigate optimal timings of purchases and sales for a heavy-duty truck fleet, considering four drivetrain technologies. Our approach can guide decision making in various fleet renewal settings. By applying it to the case study, we derive important managerial implications. The mobility transition will significantly increase transport fleets’ total cost of ownership. To minimize costs, companies should not move prematurely to low-emissions technologies, but hold vehicles for as long as possible to benefit from fewer purchases and sinking prices. The optimal policy depends on the distance driven. For short-distance operations, diesel trucks will remain the dominant technology in the next years, but will be replaced by battery electric trucks in the medium term. In the far future, trucks powered by electricity and hydrogen will be equally important.

Maximizing Net Present Value for Resource Constraint Project Scheduling Problems with Payments at Event Occurrences Using Approximate Dynamic Programming

Resource Constraint Project Scheduling Problems with Discounted Cash Flows (RCPSPDC) focuses on maximizing the net present value by summing the discounted cash flows of project activities. An extension of this problem is the Payment at Event Occurrences (PEO) scheme, where the client makes multiple payments to the contractor upon completion of predefined activities, with additional final settlement at project completion. Numerous approximation methods such as metaheuristics have been proposed to solve this NP-hard problem. However, these methods suffer from parameter control and/or the computational cost of correcting infeasible solutions. Alternatively, approximate dynamic programming (ADP) sequentially generates a schedule based on strategies computed via Monte Carlo (MC) simulations. This saves the computations required for solution corrections, but its performance is highly dependent on its strategy. In this study, we propose the hybridization of ADP with three different metaheuristics to take advantage of their combined strengths, resulting in six different models. The Estimation of Distribution Algorithm (EDA) and Ant Colony Optimization (ACO) were used to recommend policies for ADP. A Discrete cCuckoo Search (DCS) further improved the schedules generated by ADP. Our experimental analysis performed on the j30, j60, and j90 datasets of PSPLIB has shown that ADP–DCS is better than ADP alone. Implementing the EDA and ACO as prioritization strategies for Monte Carlo simulations greatly improved the solutions with high statistical significance. In addition, models with the EDA showed better performance than those with ACO and random priority, especially when the number of events increased.

Adaptive network traffic control with approximate dynamic programming based on a non-homogeneous Poisson demand model

ABSTRACT In this study, we develop a stochastic dynamic traffic-flow model subject to practical restrictions under the non-homogeneous Poisson vehicle arrival process. Using the cell transmission strategy, we establish traffic dynamics between two intersections. We also discuss simulating the random demand of source links from an estimated intensity function. Additionally, we propose an algorithm to optimize time interval division for aggregated data, aiming to enhance estimation performance. We explore applying our traffic flow model to the adaptive traffic network management problem, which is formulated as a Markov decision process. Leveraging approximate dynamic programming with recursive least squares-temporal difference learning, we achieve adaptive optimal policies. To validate our approach, we conduct a series of numerical experiments with random demands. The results of non-homogeneous Poisson demand conducted using random numbers and a real-word dataset indicate high efficiency with the piecewise constant, I-SMOOTH, and MNO-PQRS estimators. Compared to the Webster and Max-pressure control systems, our proposed approximate dynamic programming-based model exhibits superior stability and applicability.

Policy iteration forH∞control of polynomial time‐varying systems

AbstractThis paper studies theH∞control problem for polynomial time‐varying systems. TheH∞control problem has been much less investigated for time‐varying systems in comparison to the time‐invariant systems. Approximate dynamic programming (ADP) is an optimal method to solve the control problems. Therefore, it is valuable to solve the polynomial time‐varyingH∞control problem with the ADP approach. Considering the time as an independent variable for sum‐of‐squares (SOS) optimization problems, an SOS‐based ADP method is proposed to solve this problem. A policy iteration algorithm is presented, where in its policy evaluation step it is sufficient to solve an optimization problem. Some constraints are added to this optimization problem to guarantee the closed‐loop exponential stability. The convergence and stability properties of the proposed algorithm are stated and proven. Moreover, in order to design anH∞controller with a smaller disturbance attenuation coefficient, a two‐loop algorithm is suggested. Finally, the effectiveness of the proposed method is demonstrated by simulation examples.

Approximate optimal and safe coordination of nonlinear second-order multirobot systems with model uncertainties

This paper investigates the approximate optimal coordination for nonlinear uncertain second-order multi-robot systems with guaranteed safety (collision avoidance) Through constructing novel local error signals, the collision-free control objective is formulated into an coordination optimization problem for nominal multi-robot systems. Based on approximate dynamic programming technique, the optimal value functions and control policies are learned by simplified critic-only neural networks (NNs). Then, the approximated optimal controllers are redesigned using adaptive law to handle the effects of robots’ uncertain dynamics. It is shown that the NN weights estimation errors are uniformly ultimately bounded under proper conditions, and safe coordination of multiple robots can be achieved regardless of model uncertainties. Numerical simulations finally illustrate the effectiveness of the proposed controller.

Branch-and-Price for Prescriptive Contagion Analytics

A new optimization methodology helps allocate vaccines to combat a pandemic. During the COVID-19 pandemic, the extraordinary speed of vaccine developments quickly gave rise to a critical operational problem: How to distribute a scarce stockpile of vaccines across communities? In “Branch-and-Price for Prescriptive Contagion Analytics,” A. Jacquillat, M. Li, M. Rame, and K. Wang address this question by presenting a methodology to allocate shared resources across subpopulations governed by contagion dynamics. This problem combines the difficulties of mixed-integer nonconvex optimization and those of optimization with constraints governed by ordinary differential equations. By combining novel column generation, approximate dynamic programming, and branch-and-bound elements, the authors’ methodology can solve large and otherwise-intractable instances, outperforming state-of-the-art benchmarks. From a practical standpoint, their approach can significantly enhance the effectiveness of a vaccination campaign. Ultimately, the results from the paper outline vaccine allocation—beyond vaccine development and vaccine manufacturing—as a critical lever to mitigate the impact of a pandemic on public health.

Technician routing and scheduling with employees’ learning through implicit cross-training strategy

With record high talent shortages and skill mismatches around the world, this paper investigates a variant of multi-period dynamic technician and routing problem that can be modeled as a Markov decision process. To deal with the double tradeoffs between the routing and service time costs, as well as the current and future costs, we propose an approximate dynamic programming (ADP)-based cost function approximation (CFA) algorithm — the implicit cross-training strategy (ICT). A two-phase routing and scheduling heuristic is developed to account for both employees’ learning and future information, and to facilitate an efficient implementation of CFA. Extensive computational results show that ICT can provide a better solution in the current decision with a global view in comparison with the myopic strategy. In depth analysis demonstrates that ICT trains the workforce with more balanced skillsets and workloads, which ensures the flexibility of the workforce and helps buffer against the future uncertainties with substantial routing cost savings. Additionally, ICT has much more advantages in large-scale problems with more diversified service requests and randomly distributed customers.

Data-Efficient Off-Policy Learning for Distributed Optimal Tracking Control of HMAS With Unidentified Exosystem Dynamics.

In this article, a data-efficient off-policy reinforcement learning (RL) approach is proposed for distributed output tracking control of heterogeneous multiagent systems (HMASs) using approximate dynamic programming (ADP). Different from existing results that the kinematic model of the exosystem is addressable to partial or all agents, the dynamics of the exosystem are assumed to be completely unknown for all agents in this article. To solve this difficulty, an identifiable algorithm using the experience-replay method is designed for each agent to identify the system matrices of the novel reference model instead of the original exosystem. Then, an output-based distributed adaptive output observer is proposed to provide the estimations of the leader, and the proposed observer not only has a low dimension and less data transmission among agents but also is implemented in a fully distributed way. Besides, a data-efficient RL algorithm is given to design the optimal controller offline along with the system trajectories without solving output regulator equations. An ADP approach is developed to iteratively solve game algebraic Riccati equations (GAREs) using online information of state and input in an online way, which relaxes the requirement of knowing prior knowledge of agents' system matrices in an offline way. Finally, a numerical example is provided to verify the effectiveness of theoretical analysis.

An Approximate Dynamic Programming Approach to Dynamic Stochastic Matching

Dynamic stochastic matching problems arise in a variety of recent applications, ranging from ridesharing and online video games to kidney exchange. Such problems are naturally formulated as Markov decision processes (MDPs) that are, however, intractable in general. To improve tractability, we investigate the linear programming-based approach to approximate dynamic programming. This approach can provide both feasible control policies and bounds on the MDPs’ optimal policy value, which can be used to establish optimality gaps. However, the approximate linear programs (ALPs) resulting from this approach can often be difficult to solve. To address this computational challenge, we derive novel ALP reformulations that can be used for a broad class of dynamic stochastic matching problems that incorporate, among others, possible match failures and certain restrictions on feasible matchings. We show that these ALP reformulations can be solved efficiently and applied to a broad class of dynamic matching problems. In addition, our numerical results indicate that our ALP reformulations can produce tight bounds that allow us to establish near-optimal policy performance for a broad set of problem instances. Thus, ALP reformulations can present an attractive alternative for applications that involve dynamic stochastic matching. History: Accepted by Nicola Secomandi, Area Editor for Stochastic Models & Reinforcement Learning. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2021.0203 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2021.0203 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Code and Data Repository for An Approximate Dynamic Programming Approach to Dynamic Stochastic Matching

This archive is distributed in association with the INFORMS Journal on Computing under the MIT License. The purpose of this repository is to share the source code used in the paper "An Approximate Dynamic Programming Approach to Dynamic Stochastic Matching" by F. You and T. Vossen. The code contains implementation of main model of the paper, benchmark approaches for comparision, numerical instance generation and experimentation.

A kernel-based approximate dynamic programming approach: Theory and application

This article proposes a novel kernel-based Dynamic Programming (DP) approximation method to tackle the typical curse of dimensionality of stochastic DP problems over the finite time horizon. Such a method utilizes kernel functions in combination with Support Vector Machine (SVM) regression to determine an approximate cost function for the entire state space of the underlying Markov Decision Process (MDP), by leveraging cost function computed for selected representative states. Kernel functions are used to define the so-called kernel matrix, while the parameter vector of the given kernel-based cost function approximation is computed by moving backwards in time from the terminal condition and by applying SVM regression. This way, the difficulty of selecting a proper set of features is also tackled. The proposed method is then extended to the infinite time horizon case. To show the effectiveness of the proposed approach, the resulting Recursive Residual Approximate Dynamic Programming (RR-ADP) algorithm is applied to the sensor scheduling design in multi-process remote state estimation problems.

Biobjective optimization for railway alignment fine‐grained designs with parallel existing railways

AbstractUrban high‐speed railway construction is complex due to limited land resources, high population density, and potential construction risks, especially when new tracks are parallelly aligned to operational railways. Addressing a gap in current literature on fine optimization of manual alignment in such scenarios, this paper introduces a biobjective approximate fine‐grained optimization model for railway alignments (BA‐FORA). Utilizing an approximate dynamic programming (ADP) method, BA‐FORA effectively searches the feasible region to approach a global optimum, overcoming the dimensionality challenges inherent in standard dynamic programming (DP). This paper presents a biobjective optimization framework that takes into account both construction cost and construction risk adjacent to existing operating railways (CRAEOR), offering a method for the fine‐grained design of new railways adjacent to existing railways. Finally, the proposed BA‐FORA framework is applied to practical cases, demonstrating its superior optimization performance. The findings indicate that the BA‐FORA model can autonomously investigate and enhance railway alignment. It generates cost‐effective and low‐risk solutions exceeding manual efforts, ensuring alignment constraint compliance.

Robust optimal control of the multi‐input systems with unknown disturbance based on adaptive integral reinforcement learning Q‐function

AbstractConsidering overshoot and chatter caused by the unknown interference, this article studies the adaptive robust optimal controls of continuous‐time (CT) multi‐input systems with an approximate dynamic programming (ADP) based Q‐function scheme. An adaptive integral reinforcement learning (IRL) scheme is proposed to study the optimal solutions of Q‐functions. First, multi‐input value functions are presented, and Nash equilibrium is analyzed. A complex Hamilton–Jacobi–Issacs (HJI) equation is constructed with the multi‐input system and the zero‐sum‐game‐based value function. It is a challenging task to solve the HJI equation for nonlinear system. Thus, A transformation of the HJI equation is constructed as a Q‐function. The neural network (NN) is applied to learn the solution of the transformed Q‐functions based on the adaptive IRL scheme. Moreover, an error information is added to the Q‐function for the issue of insufficient initial incentives to relax the persistent excitation (PE) condition. Simultaneously, an IRL signal of the critic networks is introduced to study the saddle‐point intractable solution, such that the system drift and NN derivatives in the HJI equation are relaxed. The convergence of weight parameters is proved, and the closed‐loop stability of the multi‐system with the proposed IRL Q‐function scheme is analyzed. Finally, a two‐engine driven F‐16 aircraft plant and a nonlinear system are presented to verify the effectiveness of the proposed adaptive IRL Q‐function scheme.

A Fully Distributed ADP Algorithm for Real-Time Economic Dispatch of Microgrid

This paper proposes a fully distributed approximate dynamic programming (FD-ADP) algorithm framework for real-time economic dispatch of a microgrid (MG). Firstly, a modified distributed consensus-based algorithm (DCBA) framework with power restriction factors is proposed to distributedly obtain the real-time ED decision of MG containing distributed energy resources with linear cost functions, which improves the convergence stability compared with the traditional DCBA framework. Then, combined with the approximate dynamic programming (ADP) theory, a FD-ADP algorithm is proposed to deal with data uncertainties. To this end, the piece-wise linear (PWL) function is applied to approximate the value function, and a distributed slope update approach is proposed to embed empirical knowledge into the value function while ensuring the distributed property in off-line training. With the well-trained PWL function assisting in decision, the distributed real-time ED strategy of MG with approximate global optimality is obtained by the proposed FD-ADP algorithm during the entire schedule horizon (e.g., 24 hours). Finally, case studies on the modified IEEE 6-bus MG system and IEEE 123-bus MG system demonstrate the superiority of solution convergence, validity of distributed training, and optimality of distributed schedule of the proposed FD-ADP algorithm.

Data-Based Optimal Switching and Control With Admissibility Guaranteed Q -Learning.

This article addresses the data-based optimal switching and control codesign for discrete-time nonlinear switched systems via a two-stage approximate dynamic programming (ADP) algorithm. Through offline policy improvement and policy evaluation, the proposed algorithm iteratively determines the optimal hybrid control policy using system input/output data. Moreover, a strict proof of the convergence is given for the two-stage ADP algorithm. Admissibility, an essential property of the hybrid control policy must be ensured for practical application. To this end, the properties of the hybrid control policies are analyzed and an admissibility criterion is obtained. To realize the proposed Q -learning algorithm, an actor-critic neural network (NN) structure that employs multiple NNs to approximate the Q -functions and control policies for different subsystems is adopted. By applying the proposed admissibility criterion, the obtained hybrid control policy is guaranteed to be admissible. Finally, two numerical simulations verify the effectiveness of the proposed algorithm.

Data-Driven Optimal Tracking Control for Discrete-Time Nonlinear Systems With Unknown Dynamics Using Deterministic ADP.

This article aims to solve the optimal tracking problem (OTP) for a class of discrete-time (DT) nonlinear systems with completely unknown dynamics. A novel data-driven deterministic approximate dynamic programming (ADP) algorithm is proposed to solve this kind of problem with only input-output (I/O) data. The proposed algorithm has two advantages compared to existing data-driven deterministic ADP algorithms for the OTP. First, our algorithm can guarantee optimality while achieving better performance in the aspects of time-saving and robustness to data. Second, the near-optimal control policy learned by our algorithm can be implemented without considering expected control and enable the system states to track the user-specified reference signals. Therefore, the tracking performance is guaranteed while simplifying the algorithm implementation. Furthermore, the convergence and stability of the proposed algorithm are strictly proved through theoretical analysis, in which the errors caused by neural networks (NNs) are considered. At the end of this article, the developed algorithm is compared with two representative deterministic ADP algorithms through a numerical example and applied to solve the tracking problem for a two-link robotic manipulator. The simulation results demonstrate the effectiveness and advantages of the developed algorithm.

Robust Approximate Dynamic Programming for Nonlinear Systems With Both Model Error and External Disturbance.

Model error and external disturbance have been separately addressed by optimizing the definite H∞ performance in standard linear H∞ control problems. However, the concurrent handling of both introduces uncertainty and nonconvexity into the H∞ performance, posing a huge challenge for solving nonlinear problems. This article introduces an additional cost function in the augmented Hamilton-Jacobi-Isaacs (HJI) equation of zero-sum games to simultaneously manage the model error and external disturbance in nonlinear robust performance problems. For satisfying the Hamilton-Jacobi inequality in nonlinear robust control theory under all considered model errors, the relationship between the additional cost function and model uncertainty is revealed. A critic online learning algorithm, applying Lyapunov stabilizing terms and historical states to reinforce training stability and achieve persistent learning, is proposed to approximate the solution of the augmented HJI equation. By constructing a joint Lyapunov candidate about the critic weight and system state, both stability and convergence are proved by the second method of Lyapunov. Theoretical results also show that introducing historical data reduces the ultimate bounds of system state and critic error. Three numerical examples are conducted to demonstrate the effectiveness of the proposed method.

An Approximate Dynamic Programming Approach to Vehicle Platooning Coordination in Networks

An Approximate Dynamic Programming Approach to Vehicle Platooning Coordination in Networks