Approximate Dynamic Programming Techniques Research Articles

Approximate optimal and safe coordination of nonlinear second-order multirobot systems with model uncertainties

This paper investigates the approximate optimal coordination for nonlinear uncertain second-order multi-robot systems with guaranteed safety (collision avoidance) Through constructing novel local error signals, the collision-free control objective is formulated into an coordination optimization problem for nominal multi-robot systems. Based on approximate dynamic programming technique, the optimal value functions and control policies are learned by simplified critic-only neural networks (NNs). Then, the approximated optimal controllers are redesigned using adaptive law to handle the effects of robots’ uncertain dynamics. It is shown that the NN weights estimation errors are uniformly ultimately bounded under proper conditions, and safe coordination of multiple robots can be achieved regardless of model uncertainties. Numerical simulations finally illustrate the effectiveness of the proposed controller.

Solving the joint military medical evacuation problem via a random forest approximate dynamic programming approach

Members of the armed forces rely on having an effective and efficient medical evacuation (MEDEVAC) process for evacuating casualties from the battlefield to medical treatment facilities during combat operations. Determining which MEDEVAC units to task to respond to incoming requests for service, known as the MEDEVAC dispatching problem, is a natural sequential decision making problem and is the focus of this paper. Unlike previous research in this area, this paper incorporates triage classification errors and the possibility of having blood transfusion kits on board select MEDEVAC units. A discounted, infinite-horizon continuous-time Markov decision process model is formulated to examine the MEDEVAC dispatching problem and to compare generated dispatching policies to the currently practiced United States military policy. We utilize an approximate dynamic programming (ADP) technique that leverages a random forest value function approximation within an approximate policy iteration algorithmic framework to develop high-quality policies for both small-scale and large-scale problem instances. A representative planning scenario involving joint combat operations in South Korea is developed and utilized to investigate the differences between the various policies. Results from the analysis indicate that applying ADP techniques can improve current practices by as much as 29% with regard to a life-saving performance metric. This research is of particular interest to the military medical community and can inform the procedures of future military MEDEVAC operations.

Large-Scale Multiagent System Tracking Control Using Mean Field Games.

This article studies the tracking control problem with a large-scale group of agents. Unlike traditional control techniques used in multiagent systems (MASs), a new type of intelligent design is needed to handle the intractable "Curse of Dimensionality" caused by the extremely large number of agents. To address this challenge, the mean field game (MFG) theory has been embedded into reinforcement learning to advance intelligent tracking control with large-scale MAS. Specifically, MFG-based control can calculate the optimal strategy based on one unified fix-dimension probability density function (pdf) instead of high-dimensional large-scale MAS information collected from individual agents. Moreover, the approximate dynamic programming technique is adopted to generate a new type of MFG-based algorithm. Each agent has three neural networks (NNs) to approximate the solution of the mean field type control. In addition to the algorithm development, the performance of the NNs is also analyzed using the Lyapunov method. Finally, the linear and nonlinear tracking control simulations are given to evaluate the algorithm's performance.

Improving defensive air battle management by solving a stochastic dynamic assignment problem via approximate dynamic programming

Military air battle managers face several challenges when directing operations during quickly evolving combat scenarios. These scenarios require rapid assignment decisions to engage moving targets having dynamic flight paths. In defensive operations, the success of a sequence of air battle management decisions is reflected by the friendly force’s ability to maintain air superiority and defend friendly assets. We develop a Markov decision process (MDP) model of a stochastic dynamic assignment problem, named the Air Battle Management Problem (ABMP), wherein a set of unmanned combat aerial vehicles (UCAV) must defend an asset from cruise missiles arriving stochastically over time. Attaining an exact solution using traditional dynamic programming techniques is computationally intractable. Hence, we utilize an approximate dynamic programming (ADP) technique known as approximate policy iteration with least squares temporal differences (API-LSTD) learning to find high-quality solutions to the ABMP. We create a simulation environment in conjunction with a generic yet representative combat scenario to illustrate how the ADP solution compares in quality to a reasonable, closest-intercept benchmark policy. Our API-LSTD policy improves mean success rate by 2.8% compared to the benchmark policy and offers an 81.7% increase in the frequency with which the policy performs perfectly. Moreover, we find the increased success rate of the ADP policy is, on average, equivalent to the success rate attained by the benchmark policy when using a 20% faster UCAV. These results inform military force management and defense acquisition decisions and aid in the development of more effective tactics, techniques, and procedures.

Performance comparison of approximate dynamic programming techniques for dynamic stochastic scheduling

This paper focuses on the performance comparison of several approximate dynamic programming (ADP) techniques. In particular, we evaluate three ADP techniques through a class of dynamic stochastic scheduling problems: Lagrangian-based ADP, linear programming-based ADP, and direct search-based ADP. We uniquely implement the direct search-based ADP through basis functions that differ from those used in the relevant literature. The class of scheduling problems has the property that jobs arriving dynamically and stochastically must be scheduled to days in advance. Numerical results reveal that the direct search-based ADP outperforms others in the majority of problem sets generated.

Data-based and secure switched cyber–physical systems

In this paper, we develop a completely model-free moving target defense framework for the detection and mitigation of sensor and/or actuator attacks in cyber–physical systems with dynamics that evolve in discrete-time. We incorporate an intrusion detection mechanism based on an approximate dynamic programming technique that learns the policies for optimal regulation and optimal tracking while simultaneously defending against actuator and sensor attacks in a model-free fashion. Switching rules are leveraged to force proactive and reactive defense mechanisms as well as, guarantee the stability of the equilibrium point. Finally, as a case study, we apply the proposed moving target defense framework to a DC–DC converter that is used in electric vehicles.

Dynamic Routing with Tree Based Value Function Approximations

This paper is motivated from a collaboration with a start-up delivering on-demand health care services. In this setting, nurse practitioners need to be dynamically routed to patients' houses as service requests are received. We solve this problem using approximate dynamic programming and machine learning techniques. At every stage, an optimization problem needs to be solved to assign nurse practitioners to patients. We capture the future cost of having nurse practitioners positioned at certain locations around the city by estimating a cost-to-go function. Many previous approaches treat the cost-to-go function as a black box, which requires the enumeration of all possible actions to make a routing decision at each stage. This is an intractable problem in the reoptimization setting. Our approach overcomes this issue by approximating the cost-to-go function using a tree or tree ensemble, and leverages formulation techniques from Mixed Integer Optimization to solve the optimization problem at each stage sufficiently fast. Furthermore, this approach has the advantage of being more accurate relative to separable approximations used in the literature. We also apply this approach to several other online optimization problems in operations management which can be modeled through an underlying optimization formulation that needs to be solved at each stage, and show improvements relative to state-of-the-art methods.

A Distributed Iterative Learning Framework for DC Microgrids: Current Sharing and Voltage Regulation

With the penetration of computation intelligence, an increasing number of learning methods are developed into power engineering, such as dc microgrid applications. This paper establishes a distributed iterative learning framework to solve the current sharing and voltage regulation problem in an islanded dc microgrid from the perspective of game theory. The control objectives of dc microgrid include not only achieving the desired output current dispatch, but also regulating the voltage of dc bus to its rated value. Based on the two objectives, local performance indexes are established and an N -player game is formulated. Each source aims to minimize its own performance index and to achieve the current sharing objective simultaneously. Under the framework of game theory, a distributed iterative learning algorithm is designed based on the Bellman optimality principle and subsequently carried out using the approximate dynamic programming technique. The proposed algorithm is data based where it does not require to have the accurate model parameters of the dc microgrid and it ensures that the dc microgrid falls into a Nash equilibrium. Furthermore, a rigorous convergence analysis of the proposed algorithm is given. To demonstrate the effectiveness of the proposed method, simulation examples are presented on a tested dc microgrid.

Safe Approximate Dynamic Programming via Kernelized Lipschitz Estimation.

We develop a method for obtaining safe initial policies for reinforcement learning via approximate dynamic programming (ADP) techniques for uncertain systems evolving with discrete-time dynamics. We employ the kernelized Lipschitz estimation to learn multiplier matrices that are used in semidefinite programming frameworks for computing admissible initial control policies with provably high probability. Such admissible controllers enable safe initialization and constraint enforcement while providing exponential stability of the equilibrium of the closed-loop system.

Stochastic optimal control methodologies in risk-informed community resilience planning

The absorptive and restorative abilities of a community are two key elements of community resilience following disasters. The recovery of communities relies on an efficient restoration planning of damaged critical infrastructure systems, household units, and impaired supporting social and economic functions. These interdependent systems form a dynamic system of systems that changes continuously during restoration. Therefore, an effective and practical recovery planning process for a community can be modeled as a sequential dynamic optimization problem under uncertainty. This paper seeks to enhance our understanding of dynamic optimization concepts and their role in formulating post-disaster, community-level recovery strategies. Various methods of classic dynamic programming and reinforcement learning are examined and applied. Simulation-based approximate dynamic programming techniques are introduced to overcome the curse of dimensionality, which is characteristic of large-scale and multi-state systems of systems. The paper aims not only to study the unexplored topic of dynamic optimization in community resilience, but also to be a practical reference for policymakers, practitioners, engineers, and operations analysts to harness the power of dynamic optimization toward assessing and achieving community resilience.

Algorithms of approximate dynamic programming for hydro scheduling

In hydro scheduling, unit commitment is a complex sub-problem. This paper proposes a new approximate dynamic programming technique to solve unit commitment. A new method called Least Square Policy Iteration (LSPI) algorithm is introduced which is efficient and faster in convergence. This algorithm takes the properties of widely used algorithm least square temporal difference (LSTD), enhance it further and make it useful for optimization problems. First value function is to find a fixed policy by using least square temporal difference Q (LSTDQ) algorithm which is similar to LSTD, then LSPI is introduced for making the policy iteration algorithm by using the results of LSTDQ. It combines the data efficiency of LSTDQ and policy-search efficiency of policy iteration.

Allocating resources via price management systems: a dynamic programming-based approach

In this paper, a novel model for price management systems in resource allocation problems is proposed. Stochastic customer requests for resource allocations and releases are modelled as constrained parallel Birth–Death Processes (BDP). We address both instant (i.e. the customer requires a resource to be allocated immediately) and advance (i.e. the customer books a resource for future use) reservation requests, the latter with both bounded and unbounded time interval options. Algorithms based on Dynamic Programming (DP) principles are proposed for the calculation of suitable price profiles. At the core of such algorithms, there is the resolution of stochastic optimisation problems. In particular, the maximisation of the expected total revenue is formulated via a constrained Stochastic Dynamic Programming (SDP) approach, which becomes time-variant in case of advance reservation requests. Approximate Dynamic Programming (ADP) techniques are adopted in case of large state spaces. Simulations are performed to show the effectiveness of the proposed models and the related algorithms.

Valuation of defer and relocation options in photovoltaic generation investments by a stochastic simulation-based method

Risk management is crucial when committing investments in electricity markets. Investment projects for the generation of electricity are capital-intensive, in large part irreversible and future performance is subject to high uncertainty. Fortunately, most power generation projects have strategic flexibility for handling uncertainty and for mitigating risks under unfavorable scenarios. Modern corporate finance recognizes Real Option analysis (ROA) as the correct way to value investment projects with these characteristics. Due to both, environmental concerns and escalation of fuel prices, electricity generation from renewable sources has grown dramatically worldwide over the last decade. Renewable investment projects share many of the features mentioned. As such, option valuation methods should be applied to estimate the monetary value of flexibility in renewable energy investments. This work presents an appropriate methodology for assessing the economic value of a photovoltaic power plant under uncertainties. ROA is applied to determine the value of delaying the investment decision while waiting for better market information that would reduce acquisition costs due to progress in solar technology. The flexibility of relocating the solar facility in the future upon the appearance of a more attractive site in terms of cost, network accessibility or regulatory policies is also valued. The problem of option valuation is solved through stochastic simulation combined with recursive approximate dynamic programming techniques. The methodology developed might be used by investors for more efficient decision-making and by regulatory agencies for designing adequate support policies that encourage investment in renewable energy generation.

A data-driven approximate dynamic programming approach based on association rule learning: Spacecraft autonomy as a case study

Dynamic programming (DP) and Markov Decision Process (MDP) offer powerful tools for formulating, modeling, and solving decision making problems under uncertainty. In real-world applications, the applicability of DP is limited by severe scalability issues. These issues can be addressed by Approximate Dynamic Programming (ADP) techniques. ADP methods are based on the assumption of having either a proper estimation of the underlying state transition probability distributions or a simulation mechanism with the capability of generating samples according to such probability distributions. In this paper, we present a data-driven ADP-based approach, which can offer an alternative in case such assumption cannot be guaranteed. In particular, when varying the set-up of the MDP state transition probability matrix, different policies can be calculated through exact DP or ADP methods. Such policies are then processed by an Apriori-based algorithm to find frequent association rules within them. A pruning procedure is used to select the most suitable association rules, and finally an Association Classifier infers the optimal policy in all the possible circumstances. We show a detailed application of the proposed approach for the calculation of a proper mission operations plan for spacecrafts with a high level of on-board autonomy.

Joint Computation Offloading and Multiuser Scheduling Using Approximate Dynamic Programming in NB-IoT Edge Computing System

The Internet of Things (IoT) connects a huge number of resource-constraint IoT devices to the Internet, which generate massive amount of data that can be offloaded to the cloud for computation. As some of the applications may require very low latency, the emerging mobile edge computing (MEC) architecture offers cloud services by deploying MEC servers at the mobile base stations (BSs). The IoT devices can transmit the offloaded data to the BS for computation at the MEC server. Narrowband-IoT (NB-IoT) is a new cellular technology for the transmission of IoT data to the BS. In this paper, we propose a joint computation offloading and multiuser scheduling algorithm in NB-IoT edge computing system that minimizes the long-term average weighted sum of delay and power consumption under stochastic traffic arrival. We formulate the dynamic optimization problem into an infinite-horizon average-reward continuous-time Markov decision process (CTMDP) model. In order to deal with the curse-of-dimensionality problem, we use the approximate dynamic programming techniques, i.e., the linear value-function approximation and temporal-difference learning with post-decision state and semi-gradient descent method, to derive a simple algorithm for the solution of the CTMDP model. The proposed algorithm is semi-distributed, where the offloading algorithm is performed locally at the IoT devices, while the scheduling algorithm is auction-based where the IoT devices submit bids to the BS to make the scheduling decision centrally. Simulation results show that the proposed algorithm provides significant performance improvement over the two baseline algorithms and the MUMTO algorithm which is designed based on the deterministic task model.

APPROXIMATE DYNAMIC PROGRAMMING FOR OPTIMAL SEARCH WITH AN OBSTACLE

In this paper, we study a class of optimal search problems where the search region includes a target and an obstacle, each of which has some shape. The search region is divided into small grid cells and the searcher examines one of those cells at each time period with the objective of finding the target with minimum expected cost. The searcher may either take an action that is quick but risky, or another one that is slow but safe, and incurs different cost for these actions. We formulate these problems as Markov Decision Processes (MDPs), but because of the intractability of the state space, we approximately solve the MDPs using an Approximate Dynamic Programming (ADP) technique and compare its performance against heuristic decision rules. Our numerical experiments reveal that the ADP technique outperforms heuristics on majority of problem instances.

Incremental Approximate Dynamic Programming for Nonlinear Adaptive Tracking Control with Partial Observability

Approximate dynamic programming is a class of reinforcement learning, which solves adaptive, optimal control problems and tackles the curse of dimensionality with function approximators. Within this category, linear approximate dynamic programming provides a model-free control method by systematically using a quadratic cost-to-go function. Although efficient, linear approximate dynamic programming methods are difficult to apply to nonlinear systems or time-varying systems. To overcome the above limitations, this paper proposes an adaptive nonlinear tracking control method based on incremental approximate dynamic programming, which combines the advantages of linear approximate dynamic programming and incremental nonlinear control techniques. This is a model-free method for unknown, nonlinear systems and time-varying references. The trait of separating the local model information from the cost function approximation makes this method an option for partially observable control problems. This paper, therefore, proposes two reference tracking controllers for different observability conditions: the direct measurement of the full state, and the partially observable tracking error. In each condition, two algorithms are developed for off-line learning and online learning, respectively. These algorithms are applied to attitude control of a spacecraft disturbed by internal liquid sloshing. The results demonstrate that the proposed algorithms accurately deal with the unknown, time-varying internal dynamics while retaining efficient control, even with only partial observability.

Merchant Energy Trading in a Network

We formulate the merchant trading of energy in a network of storage and transport assets as a Markov decision process with uncertain energy prices, generalizing known models. Because of the intractability of our model, we develop heuristics and both lower and dual (upper) bounds on the optimal policy value estimated within Monte Carlo simulation. We achieve tractability using linear optimization, extending near optimal approximate dynamic programming techniques for the case of a single storage asset, versions of two of which are commercially available. We propose (i) a generalization of a deterministic reoptimization heuristic, (ii) an iterative version of the least squares Monte Carlo approach, and (iii) a perfect information dual bound. We apply our methods to a set of realistic natural gas instances. The combination of our reoptimization heuristic and dual bound emerges as a practical approach to nearly optimally solve our model. Our iterative least squares Monte Carlo heuristic is also close to optimal. Compared to our other heuristic, it exhibits slightly larger optimality gaps and requires some tuning, but is faster to execute in some cases. Our methods could enhance single energy storage asset software and have potential relevance beyond our specific application. The e-companion is available at https://doi.org/10.1287/opre.2018.1732 .

Least Squares Monte Carlo and Approximate Linear Programming: Error Bounds and Energy Real Option Application

Least squares Monte Carlo (LSM) is an approximate dynamic programming (ADP) technique commonly used for the valuation of high dimensional financial and real options, but has broader applicability. It is known that the regress-later version of this method is an approximate linear programming (ALP) relaxation that implicitly provides a potential solution to a familiar ALP deficiency. Focusing on a generic finite horizon Markov decision process, we provide both theoretical and numerical backing for the usefulness of this solution, respectively using a worst-case error bound analysis and a numerical study dealing with merchant ethanol production, an energy real option application, based on an ALP heuristic that we propose. When both methodologies are applicable, our research supports the use of regress-later LSM rather than this ALP technique to approximately solve intractable Markov decision processes. Our numerical findings motivate additional research to obtain even better methods than the regress-later version of LSM.

Output feedback Q-learning for discrete-time linear zero-sum games with application to the H-infinity control

Approximate dynamic programming techniques usually rely on the feedback of the measurement of the complete state, which is generally not available in practical situations. In this paper, we present an output feedback Q-learning algorithm towards finding the optimal strategies for the discrete-time linear quadratic zero-sum game, which encompasses the H-infinity optimal control problem. A new representation of the Q-function in the output feedback form is derived for the zero-sum game problem and the optimal output feedback policies are presented. Then, a Q-learning algorithm is developed that learns the optimal control strategies online without needing any information about the system dynamics, which makes the control design completely model-free. It is shown that the proposed algorithm converges to the optimal solution obtained by solving the game algebraic Riccati equation (GARE). Unlike the value function based approach used for output feedback, the proposed Q-learning scheme does not require a discounting factor that is generally adopted to mitigate the effect of excitation noise bias. It is known that this discounting factor may compromise the closed-loop stability. The proposed method overcomes the excitation noise bias problem without resorting to the discounting factor, and therefore, converges to the nominal GARE solution. As a result, the closed-loop stability is preserved. An application to the H-infinity autopilot controller for the F-16 aircraft is demonstrated by simulation.