Approximate Dynamic Programming Approach Research Articles

Policy iteration for H∞ control of polynomial time‐varying systems

AbstractThis paper studies the H∞ control problem for polynomial time‐varying systems. The H∞ control problem has been much less investigated for time‐varying systems in comparison to the time‐invariant systems. Approximate dynamic programming (ADP) is an optimal method to solve the control problems. Therefore, it is valuable to solve the polynomial time‐varying H∞ control problem with the ADP approach. Considering the time as an independent variable for sum‐of‐squares (SOS) optimization problems, an SOS‐based ADP method is proposed to solve this problem. A policy iteration algorithm is presented, where in its policy evaluation step it is sufficient to solve an optimization problem. Some constraints are added to this optimization problem to guarantee the closed‐loop exponential stability. The convergence and stability properties of the proposed algorithm are stated and proven. Moreover, in order to design an H∞ controller with a smaller disturbance attenuation coefficient, a two‐loop algorithm is suggested. Finally, the effectiveness of the proposed method is demonstrated by simulation examples.

Data-Efficient Off-Policy Learning for Distributed Optimal Tracking Control of HMAS With Unidentified Exosystem Dynamics.

In this article, a data-efficient off-policy reinforcement learning (RL) approach is proposed for distributed output tracking control of heterogeneous multiagent systems (HMASs) using approximate dynamic programming (ADP). Different from existing results that the kinematic model of the exosystem is addressable to partial or all agents, the dynamics of the exosystem are assumed to be completely unknown for all agents in this article. To solve this difficulty, an identifiable algorithm using the experience-replay method is designed for each agent to identify the system matrices of the novel reference model instead of the original exosystem. Then, an output-based distributed adaptive output observer is proposed to provide the estimations of the leader, and the proposed observer not only has a low dimension and less data transmission among agents but also is implemented in a fully distributed way. Besides, a data-efficient RL algorithm is given to design the optimal controller offline along with the system trajectories without solving output regulator equations. An ADP approach is developed to iteratively solve game algebraic Riccati equations (GAREs) using online information of state and input in an online way, which relaxes the requirement of knowing prior knowledge of agents' system matrices in an offline way. Finally, a numerical example is provided to verify the effectiveness of theoretical analysis.

A kernel-based approximate dynamic programming approach: Theory and application

This article proposes a novel kernel-based Dynamic Programming (DP) approximation method to tackle the typical curse of dimensionality of stochastic DP problems over the finite time horizon. Such a method utilizes kernel functions in combination with Support Vector Machine (SVM) regression to determine an approximate cost function for the entire state space of the underlying Markov Decision Process (MDP), by leveraging cost function computed for selected representative states. Kernel functions are used to define the so-called kernel matrix, while the parameter vector of the given kernel-based cost function approximation is computed by moving backwards in time from the terminal condition and by applying SVM regression. This way, the difficulty of selecting a proper set of features is also tackled. The proposed method is then extended to the infinite time horizon case. To show the effectiveness of the proposed approach, the resulting Recursive Residual Approximate Dynamic Programming (RR-ADP) algorithm is applied to the sensor scheduling design in multi-process remote state estimation problems.

An Approximate Dynamic Programming Approach to Vehicle Platooning Coordination in Networks

An Approximate Dynamic Programming Approach to Vehicle Platooning Coordination in Networks

An Approximate Dynamic Programming Approach to Vehicle Platooning Coordination in Networks

An Approximate Dynamic Programming Approach to Vehicle Platooning Coordination in Networks

Multistage reliability-constrained stochastic planning of diamond distribution network: An approximate dynamic programming approach

In this paper, a multistage reliability-constrained stochastic planning model is established for the diamond distribution network, a new type of distribution system grid structure, with the goal of optimizing the total cost of investment and operation. The uncertainty of load growth is simulated by generating multiple load growth scenarios, and the power supply loss caused by the reliability of components in the system is also considered. The multistage stochastic dynamic decision process is modeled by The Markov decision process. In order to solve the “curse of dimensionality” problem in multi-stage stochastic planning model, the approximate dynamic programming algorithm is used to solve it. The case study is based on the typical structure of the diamond distribution network and realistic data of an actual district in Shanghai, China. The planning results verify the effectiveness of the model and the solution algorithm, and demonstrate the reliability and economy of the diamond distribution network are superior to the traditional double-ring network.

Shortening Emergency Medical Response Time with Joint Operations of Uncrewed Aerial Vehicles with Ambulances

Problem definition: Uncrewed aerial vehicles (UAVs) are transforming emergency service logistics applications across sectors, offering easy deployment and rapid response. In the context of emergency medical services (EMS), UAVs have the potential to augment ambulances by leveraging bystander assistance, thereby reducing response times for delivering urgent medical interventions and improving EMS outcomes. Notably, the use of UAVs for opioid overdose cases is particularly promising as it addresses the challenges faced by ambulances in delivering timely medication. This study aims to optimize the integration of UAVs and bystanders into EMS in order to minimize average response times for overdose interventions. Methodology/results: We formulate the joint operation of UAVs with ambulances through a Markov decision process that captures random emergency vehicle travel times and bystander availability. We apply an approximate dynamic programming approach to mitigate the solution challenges from high-dimensional state variables and complex decisions through a neural network-based approximation of the value functions (NN-API). To design the approximation, we construct a set of basis functions based on queueing and geographic properties of the UAV-augmented EMS system. Managerial implications: The simulation results suggest that our NN-API policy tends to outperform several noteworthy rule- and optimization-based benchmark policies in terms of accumulated rewards, particularly for situations that are primarily characterized by high request arrival rates and a limited number of available ambulances and UAVs. The results also demonstrate the benefits of incorporating UAVs into the EMS system and the effectiveness of an intelligent real-time operations strategy in addressing capacity shortages, which are often a problem in rural areas of the United States. Additionally, the results provide insights into specific contributions of each dispatching or redeployment strategy to overall performance improvement. Funding: This work was supported by the National Science [Grant 1761022]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/msom.2022.0166

A simulation-based approximate dynamic programming approach to dynamic and stochastic resource-constrained multi-project scheduling problem

We consider the dynamic and stochastic resource-constrained multi-project scheduling problem which allows for the random arrival of projects and stochastic task durations. Completing projects generates rewards, which are reduced by a tardiness cost in the case of late completion. Multiple types of resource are available, and projects consume different amounts of these resources when under processing. The problem is modelled as an infinite-horizon discrete-time Markov decision process and seeks to maximise the expected discounted long-run profit. We use an approximate dynamic programming algorithm (ADP) with a linear approximation model which can be used for online decision making. Our approximation model uses project elements that are easily accessible by a decision-maker, with the model coefficients obtained offline via a combination of Monte Carlo simulation and least squares estimation. Our numerical study shows that ADP often statistically significantly outperforms the optimal reactive baseline algorithm (ORBA). In experiments on smaller problems however, both typically perform suboptimally compared to the optimal scheduler obtained by stochastic dynamic programming. ADP has an advantage over ORBA and dynamic programming in that ADP can be applied to larger problems. We also show that ADP generally produces statistically significantly higher profits than common algorithms used in practice, such as a rule-based algorithm and a reactive genetic algorithm.

Approximate dynamic programming approach to efficient metro train timetabling and passenger flow control strategy with stop-skipping

Urban metro systems continuously face high travel demand during rush hours, which brings excessive energy waste and high risk to passengers. In order to alleviate passenger congestion, improve train service levels and reduce energy consumption, a nonlinear dynamic programming (DP) model of efficient metro train timetabling and passenger flow control strategy with stop-skipping is presented, which consists of state transition equations concerning train traffic and passenger load. To overcome the curse of dimensionality, the formulated nonlinear DP problem is transformed into a discrete Markov decision process, and a novel approximate dynamic programming (ADP) approach is designed based on the lookahead policy and linear parametric value function approximation. Finally, the effectiveness of this method is verified by three groups of numerical experiments. Compared with Particle Swarm Optimization (PSO) and Simulated Annealing (SA), the designed ADP approach could obtain high-quality solutions quickly, which makes it applicable to the practical implementation of metro operations.

Decomposition and approximate dynamic programming approach to optimization of train timetable and skip-stop plan for metro networks

Carefully coordinating train timetables of different operating lines can help reduce transfer delays, which in turn reduces station crowding and improves overall service quality. This paper explores the optimization to train timetable and skip-stop plans that aims to minimize the total passenger waiting time and station crowding. The problem is formulated as a mixed-integer non-linear programming model. To effectively address the complexity of the model, a decomposition and approximate dynamic programming approach is designed to convert the original network-level problem into a series of small-scale subproblems, one for each operating line, to be solved quickly in a distributed manner. The effectiveness and practicability of the model and algorithm are demonstrated on two case networks: a small-scale synthetic network of three metro lines and a real-world network based on Beijing metro. The computational results illustrate that the proposed strategy to generate train timetables and skip-stop plans can effectively reduce passenger waiting time and station crowing. The proposed decomposition and approximate dynamic programming approach is also shown to perform more efficiently than traditional heuristic algorithms, such as genetic algorithm and simulated annealing algorithm for large-scale networks.

An approximate dynamic programming approach to network-based scheduling of chemotherapy treatment sessions

A solution approach is proposed for the interday problem of assigning chemotherapy sessions at a network of treatment centres with the goal of increasing the cost-efficiency of system-wide capacity use. This network-based scheduling procedure is subject to the condition that both the first and last sessions of a patient's treatment protocol are administered at the same centre the patient is referred to by their oncologist. All intermediate sessions may be administered at other centres. It provides a systematic way of identifying effective multi-appointment scheduling policies that exploit the total capacity of a networked system, allowing patients to be treated at centres other than their home centre. The problem is modelled as a Markov decision process which is then solved approximately using techniques of approximate dynamic programming. The benefits of the approach are evaluated and compared through simulation with the existing manual scheduling procedures at two treatment centres in Santiago, Chile. The results suggest that the approach would obtain a 20% reduction in operating costs for the whole system and cut existing first-session waiting times by half. A key conclusion, however, is that a network-based scheduling procedure brings no real benefits if it is not implemented in conjunction with a proactive assignment policy like the one proposed in this paper.

An approximate dynamic programming approach for the maintenance optimisation of networked critical infrastructures

The continuous performance of networked infrastructures, such as transportation, telecommunications, and power transmission, is critical to the economic development and social well-being of society. To ensure their reliability in continuously meeting prescribed demand, cost-effective maintenance is essential. To this end, the influence of maintenance actions measures on the reliability of an infrastructure need to be considered both during their implementation and after their completion. To address this problem and to consider the uncertainty of future states, we model the multi-period optimisation of an infrastructure’s maintenance planning as a stochastic dynamic programming problem. An approximate dynamic programming approach is developed to deal with the computational complexity and solve the described problem. The proposed approach can help deal with the uncertainty of future infrastructure states by determining the optimal maintenance plan for each possible state in advance.

Managing Operations of a Hog Farm Facing Volatile Markets: Inventory and Selling Strategies

Problem definition: We study a dynamic finishing-stage planning problem of a pork producer who at the beginning of each week gets to see how many market-ready hogs she has available for sale and the current market prices. Then, she must decide which hogs to sell to a meatpacker and on the open market and which hogs to hold until the following week. The farmer is contracted to deliver a fixed quantity of hogs to the meatpacker each week priced according to a contractually predetermined market index. If the farmer underdelivers to the meatpacker, she pays a contractually predetermined unit penalty also linked to a market index. Biosecurity protocols prevent the farmer from buying hogs on the open market and selling them to the meatpacker. The farmer can, however, use the open market to sell hogs for prevailing market prices. Methodology/Results: We treat the problem as a dynamic, multiitem, nonstationary inventory problem with multiple sources of uncertainty. The optimal policy is a threshold policy with multiple price-dependent thresholds. The computational complexity required to evaluate the thresholds is the biggest impediment to using the optimal policy as a decision-support tool. So, we utilize an approximate dynamic programming approach that exploits the optimal policy structure and produces a sharp heuristic that is easy to implement. Managerial implications: Numerical experiments calibrated to a pork producer’s data reveal that the optimal policy with the heuristically estimated thresholds substantially improves the existing practice (around 25% on average). The success of the proposed model is attributed to recognizing the value of holding underweight hogs and effectively hedging supply uncertainty and future prices—an insight missed in the planning actions of the current practice.Supplemental Material: The online appendix is available at https://doi.org/10.1287/msom.2023.1216 .

Proactive Security-Constrained Unit Commitment Against Typhoon Disasters: An Approximate Dynamic Programming Approach

Proactive security-constrained unit commitment (SCUC) is one of the effective resilience-oriented responses to keep the secure operation of the power system during natural disasters. The approaches proposed in the previous research works cannot guarantee high-quality response timely to various emergencies during typhoon disasters. This paper proposes a proactive SCUC model solved by an approximate dynamic programming (ADP) algorithm to improve system resilience during typhoon disasters while meeting the requirements of the computational time. Both the uncertainty of the typhoon motion path and the wind speed-dependent transmission line failure probability are considered in the scenario generation. The optimization process is formulated as a Markov decision process (MDP), and a piecewise linear function (PLF) based ADP is employed for solving the model. With the exogenous information extracted from generated scenarios and recorded in the slopes of PLFs, the ADP algorithm can result in a near-optimal solution within a short computational time. Simulations are carried out using the modified IEEE RTS-79 system. The significant computational time saving and high-quality solution demonstrate the effectiveness of the proposed approach.

Robust Near-Optimal Coordination in Uncertain Multiagent Networks With Motion Constraints.

This article addresses the robust coordination problem for nonlinear uncertain second-order multiagent networks with motion constraints, including velocity saturation and collision avoidance. A single-critic neural network-based approximate dynamic programming approach and exact estimation of unknown dynamics are employed to learn online the optimal value function and controller. By incorporating avoidance penalties into tracking variable, constructing a novel value function, and designing of suitable learning algorithms, multiagent coordination and collision avoidance are achieved simultaneously. We show that the developed feedback-based coordination strategy guarantees uniformly ultimately bounded convergence of the closed-loop dynamical stability and all underlying motion constraints are always strictly obeyed. The effectiveness of the proposed collision-free coordination law is finally illustrated using numerical simulations.

Cross-docking based factory logistics unitisation process: An approximate dynamic programming approach

The factory logistics increases in importance as more processes and types of components are needed to produce a product. Due to the limited industrial space and stressful competitive environment, manufacturers increasingly outsource logistics-related activities to Third-Party Logistics (3PLs) and Supply hubs. However, the challenges, including unsynchronised decisions with other partners, non-value-added warehousing operations, and excessive space occupation, still burden manufacturers. This research proposes a cross-docking-based factory logistics unitisation process (CD-FLUP) that integrates the cross-docking technique into the assembly line feeding process and uses mobile shelves for component holding and delivery. We formulate the CD-FLUP with an integer program to minimise the total number of shelf units put into storage and the number of deliveries to assembly lines. Based on time decomposition, we propose an approximate dynamic programming (ADP) approach to solve the model efficiently. The time-varying value function approximation (VFA) developed in the proposed ADP approach reduces the approximation parameters and speeds up convergence significantly. The numerical experiment shows that the proposed ADP can generate high-quality solutions far more efficiently than a commercial solver.

Solving the joint military medical evacuation problem via a random forest approximate dynamic programming approach

Members of the armed forces rely on having an effective and efficient medical evacuation (MEDEVAC) process for evacuating casualties from the battlefield to medical treatment facilities during combat operations. Determining which MEDEVAC units to task to respond to incoming requests for service, known as the MEDEVAC dispatching problem, is a natural sequential decision making problem and is the focus of this paper. Unlike previous research in this area, this paper incorporates triage classification errors and the possibility of having blood transfusion kits on board select MEDEVAC units. A discounted, infinite-horizon continuous-time Markov decision process model is formulated to examine the MEDEVAC dispatching problem and to compare generated dispatching policies to the currently practiced United States military policy. We utilize an approximate dynamic programming (ADP) technique that leverages a random forest value function approximation within an approximate policy iteration algorithmic framework to develop high-quality policies for both small-scale and large-scale problem instances. A representative planning scenario involving joint combat operations in South Korea is developed and utilized to investigate the differences between the various policies. Results from the analysis indicate that applying ADP techniques can improve current practices by as much as 29% with regard to a life-saving performance metric. This research is of particular interest to the military medical community and can inform the procedures of future military MEDEVAC operations.

Approximate dynamic programming for container stacking

At some point during transport, intermodal containers will be stored at a terminal, where they are typically stacked on top of each other. Stacking yields a higher utilization of the area but may lead to unproductive reshuffle moves when containers below another need to be retrieved. Preventing reshuffles has a financial benefit, as it not only avoids the costs of executing the reshuffle but also decreases the time needed to retrieve a container. Typically, researchers consider only the retrieval of containers and assume the retrieval order is fully known. In addition, existing studies do not consider the stacking restrictions imposed by a reach stacker, which is commonly used in smaller inland terminals. This research aims to design decision support for determining real-life applicable container stack allocations so that the expected number of reshuffles is minimized. We propose a model that includes both arrivals and departures of containers as well as a certain level of uncertainty in the order thereof. The problem is modeled as a Markov Decision Process and solved using Approximate Dynamic Programming (ADP). Through numerical experiments on real-life problem instances, we conclude that the ADP approach drastically outperforms a benchmark heuristic from the literature. All data used as well as the source code has been made publicly available.

Metalearning of time series: an approximate dynamic programming approach

A new approximate dynamic programming algorithm applied to time series forecasting

Designing an optimal ship course-keeping system for an unknown object model via adaptive dynamic programming approach

The paper discusses the problem of ship autopilot design based on the adaptive (approximate) dynamic programming approach (ADP). The main objective of this paper is to propose the design of a ship's course controller within the framework of computational intelligence techniques, i.e. (in our case) based on model-free optimal regulation. This method is attractive in that it does not require knowledge of the system model to generate optimal process control. The key idea is to iteratively solve the Hamilton-Jacobi-Bellman (HJB) equation based on a simple value function approximator structure. In this way, by introducing such basic elements as evaluation and improvement, we are able to carry out an intelligent optimization process based only on the data received online. Thus, this is a kind of data-driven control methodology. The simulations of the ship course change process confirmed the high efficiency of the proposed controller. An important feature of this approach is also the elimination of the tuning problem.