Approximate Dynamic Programming Framework Research Articles

Optimization of Power Control for Autonomous Hybrid Electric Vehicles with Flexible Power Demand

Abstract Technology advancement for on-road vehicles has gained significant momentum in the past decades, particularly in the field of vehicle automation and powertrain electrification. The optimization of powertrain controls for autonomous vehicles typically involves a separated consideration of the vehicle's external dynamics and powertrain dynamics, with one key aspect often overlooked. This aspect, known as flexible power demand, recognizes that the powertrain control system does not necessarily have to precisely match the power requested by the vehicle motion controller at all times. Leveraging this feature can lead to control designs achieving improved fuel economy by adding an extra degree of freedom to the powertrain control while maintaining safety and drive comfort. The present research investigates the use of an Approximate Dynamic Programming (ADP) approach to develop a powertrain controller, which takes into account the flexibility in power demand within the ADP framework. The concept of reachable sets is incorporated into the ADP framework to ensure safety, improve ride comfort, and enhance the accuracy of the optimization solution. The formulation is based on an autonomous hybrid electric vehicle (HEV), while the methodology can also be applied to other types of vehicles. It is also found that necessary customization of the ADP algorithm is needed for this particular control problem to prevent convergence issues. Finally, a case study is presented to evaluate the effectiveness of flexible power demand, as addressed by the ADP method. The experiment demonstrates a 14.1% improvement in fuel economy compared to a scenario without flexible power demand.

An ADP framework for flexibility and cost aggregation: Guarantees and open problems

With the increasing amount of Distributed Energy Resources (DERs), coordination of Distribution Grid Operators (DSOs) and Transmission Grid Operators (TSOs) is of paramount importance. Managing a large number of DERs at the TSO level is, however, challenging. To address this problem, flexibility aggregation is a topic of frequent research activities. Aggregation means to describe the combined flexibility of the DERs at the vertical grid coupling between DSO and TSO. Existing works are often limited with respect to guaranteeing feasibility, with respect to efficient numerical implementation, and in terms of quantification of the cost of DER usage. In the present paper, we investigate aggregation based on Approximate Dynamic Programming (ADP). We propose efficient numerical aggregation schemes using tools from computational geometry thus avoiding the need to solve multiple OPF problems. We rely on different variants of the DistFlow model for radial grids, which are computationally efficient. This allows to model of current and voltage limits and enables the consideration of voltage dependencies in the aggregation. Furthermore, we propose a method for cost aggregation and identify open problems of flexibility aggregation.

Display optimization under the multinomial logit choice model: Balancing revenue and customer satisfaction

In this paper, we consider an assortment optimization problem in which a platform must choose pairwise disjoint sets of assortments to offer across a series of T stages. Arriving customers begin their search process in the first stage, and progress sequentially through the stages until their patience expires, at which point they make a multinomial logit–based purchasing decision from among all products they have viewed throughout their search process. The goal is to choose the sequential displays of product offerings to maximize expected revenue. Additionally, we impose stage‐specific constraints that ensure that as each customer progresses farther and farther through the T stages, there is a minimum level of “desirability” met by the collections of displayed products. We consider two related measures of desirability: purchase likelihood and expected utility derived from the offered assortments. In this way, the offered sequence of assortments must be both high earning and well liked, which breaks from the traditional assortment setting, where customer‐centric considerations are generally not explicitly accounted for. We show that our assortment problem of interest is strongly NP‐Hard, thus ruling out the existence of a fully polynomial‐time approximation scheme (FPTAS). From an algorithmic standpoint, as a warm‐up, we develop a simple constant factor approximation scheme in which we carefully stitch together myopically selected assortments for each stage. Our main algorithmic result consists of a polynomial‐time approximation scheme (PTAS), which combines a handful of structural results related to the make‐up of the optimal assortment sequence within an approximate dynamic programming framework. We also provide an additional approximation scheme, which, under mild assumptions, can handle a cardinality constraint that enforces that an exact number of new products are introduced at each stage. Using an extensive set of numerical experiments, we demonstrate that both algorithms exhibit excellent practical performance, producing sequences of assortments that are, on average, always within 2% of optimal.

Approximate Dynamic Programming for Event-Driven H∞ Constrained Control

We study the dynamic event-driven <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$H_{\infty}$</tex-math> </inline-formula> constrained control problem through approximate dynamic programming (ADP). Differing from the existing literature considering systems with either symmetric constraints or asymmetric constraints, we consider the two different constraints simultaneously. Initially, by constructing a generalized nonquadratic value function, we transform the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$H_{\infty}$</tex-math> </inline-formula> constrained control problem into an unconstrained two-player zero-sum game. Then, we present an event-driven Hamilton–Jacobi–Isaacs equation (ED-HJIE) corresponding to the zero-sum game for lowering down the computational load. To solve the ED-HJIE, we propose a dynamic triggering mechanism together with a sole critic neural network (CNN) being built under the ADP framework. The CNN’s weights are tuned via the gradient descent approach. After that, we prove uniform ultimate boundedness of the closed-loop system and the CNN’s weight estimation error via Lyapunov’s method. Finally, we separately use an F16 aircraft plant and an inverted pendulum system to validate the present theoretical claims.

Replacement and Repair Optimization for Production Systems Under Random Production Waits

This study considers a repairable production system operated under an age-based preventive replacement policy that is subject to independent random production waits and failures. In addition to the age-based replacement policy, we propose a maintenance model that uses production waits to schedule preventive replacement. That is, a decision maker can preventively replace the system either during a production wait or at the age threshold, whereas if a failure occurs during production, the decision maker must decide whether to perform a minimal repair or a corrective replacement to restore the system. Under the above setting, we develop a semi-Markov decision process (SMDP) to obtain the optimal maintenance policy that minimizes the long-run average maintenance cost rate. We establish the existence of the optimal maintenance policy and provide an algorithm to numerically obtain the optimal action for each state. We further generalize the model to incorporate imperfect maintenance and the nonhomogeneous arrival of production waits. In the latter case, it is computationally intractable to optimize the SMDP using a value iteration algorithm due to the curse of dimensionality. To address this challenge, we further develop an approximate dynamic programming framework to generate high-quality solutions. An attractive feature of our model is its generality, such that the model includes many existing maintenance models as special cases. A real-world example from a steel factory is used to demonstrate the proposed model.

Approximate Dynamic Programming for Nonlinear-Constrained Optimizations.

In this paper, we study the constrained optimization problem of a class of uncertain nonlinear interconnected systems. First, we prove that the solution of the constrained optimization problem can be obtained through solving an array of optimal control problems of constrained auxiliary subsystems. Then, under the framework of approximate dynamic programming, we present a simultaneous policy iteration (SPI) algorithm to solve the Hamilton-Jacobi-Bellman equations corresponding to the constrained auxiliary subsystems. By building an equivalence relationship, we demonstrate the convergence of the SPI algorithm. Meanwhile, we implement the SPI algorithm via an actor-critic structure, where actor networks are used to approximate optimal control policies and critic networks are applied to estimate optimal value functions. By using the least squares method and the Monte Carlo integration technique together, we are able to determine the weight vectors of actor and critic networks. Finally, we validate the developed control method through the simulation of a nonlinear interconnected plant.

Display Optimization Under the Multinomial Logit Choice Model: Balancing Revenue and Customer Satisfaction

In this paper, we consider an assortment optimization problem in which a platform must choose pairwise disjoint sets of assortments to offer across a series of T stages. Arriving customers begin their search process in the first stage and progress sequentially through the stages until their patience expires, at which point they make a multinomial-logit-based purchasing decision from among all products they have viewed throughout their search process. The goal is to choose the sequential displays of product offerings to maximize expected revenue. Additionally, we impose stage-specific constraints that ensure that as each customer progresses farther and farther through the T stages, there is a minimum level of “desirability” met by the collections of displayed products. We consider two related measures of desirability: purchase likelihood and expected utility derived from the offered assortments. In this way, the offered sequence of assortment must be both high earning and well-liked, which breaks from the traditional assortment setting, where customer considerations are generally not explicitly accounted for. We show that our assortment problem of interest is strongly NP-Hard, thus ruling out the existence of a fully polynomial-time approximation scheme (FPTAS). From an algorithmic standpoint, as a warm-up, we develop a simple constant factor approximation scheme in which we carefully stitch together myopically selected assortments for each stage. Our main algorithmic result consists of a polynomial-time approximation scheme (PTAS), which combines a handful of structural results related to the make-up of the optimal assortment sequence within an approximate dynamic programming framework.

A Risk-Averse Stochastic Dynamic Programming Approach to Energy Hub Optimal Dispatch

This paper studies the optimal operation problem of an energy hub with multiple energy sources to serve stochastic electricity and heat loads in the presence of uncertain prices as well as operational constraints, such as minimum uptime and downtime requirements. Price and demand uncertainties are modeled by stochastic processes. The goal is to minimize some risk functional of the energy hub operational cost. A stochastic dynamic optimization formulation is introduced for the problem. An approximate dynamic programming framework, based on cost function approximation, is proposed to obtain dynamic dispatch policies. The approach enables a risk-sensitive energy hub operator to consider a non-differentiable risk measure and various constraints. The performance of the approach for the energy hub dispatch problem and characteristics of the storage levels are numerically investigated.

Capacitated inspection scheduling of multi-unit systems

Abstract We consider the inspection scheduling problem of multi-unit systems where the inspections of individual units are coupled via a capacity constraint. Although the optimal inspection policy of the majority of single-unit systems can be characterized by a threshold policy, finding an optimal policy for multi-unit systems is significantly harder. Therefore, the current state-of-the-practice uses a periodical inspection policy for all units. Instead, we propose using a dynamic programming (DP) approach to solve small-scale problem instances to optimality and use solutions optimized for a single-unit system in an approximation scheme to obtain near-optimal solutions for large-scale problems. Our results show that taking individual properties of the units to be inspected into account and incorporating the single-unit solutions within an approximate DP framework significantly decrease the inspection cost compared to a periodical inspection policy. The proposed methods can help resource-constrained regulatory agencies such as US Food and Drug Administration (FDA) to optimize their inspection activities.

Optimal Management for Parking-Lot Electric Vehicle Charging by Two-Stage Approximate Dynamic Programming

This paper targets the day-time charging scenario for plug-in electric vehicles at parking-lots near commercial places, where most vehicles have extended parking time. Compared with night-time charge scenarios for residential buildings, commercial building parking-lot charging during day-time feature significant stochastic vehicle arrival and departure, as well as highly dynamic electricity price. A two-stage approximate dynamic programming framework is proposed to determine the optimal charging strategy, utilizing the predicted short-term future information and long-term estimation from historical data. All the vehicles are desired to be charged to full prior to the departure time specified under constrained total charging capacity. The uncharged amount is subject to a significant penalty cost. Simulation scenarios are created by modeling the vehicle arrival behavior as Poisson process, including arrival time, departure time, and arrival state of charge. The simulation results show that the proposed method can significantly decrease the energy cost.

Optimal switching with minimum dwell time constraint

The problem of optimal switching between different subsystems/configurations with minimum dwell time constraints is investigated in this study and a feedback solution, using the framework of approximate dynamic programming, is proposed. The method calls for a tuning stage in which parameters of a function approximator are tuned. Online control will then be conducted with a computational load as low as evaluating a few scalar-valued functions. Finite-horizon and infinite-horizon cost functions are investigated and convergence and stability are analyzed considering the presence of approximation errors. Moreover, the case of applying a full or partial constraint on the mode sequence, as well as having a mode-dependent minimum dwell time are investigated. Finally, performance of the scheme in handling different challenges is numerically evaluated through different examples.

Itinerary-based nesting control with upsell

In order to accept future high-yield booking requests, airlines protect seats from low-yield passengers. More seats may be reserved when passengers faced with closed fare classes can upsell to open higher fare classes. We address the airline revenue management problem with capacity nesting and customer upsell, and formulate this problem by a stochastic optimization model to determine a set of static protection levels for each itinerary. We apply an approximate dynamic programming framework to approximate the objective function by piecewise linear functions, whose slopes (marginal revenue) are iteratively updated and returned by an efficient heuristic that simultaneous handles both nesting and upsells. The resulting allocation policy is tested over a real airline network and benchmarked against the randomized linear programming bid-price policy under various demand settings. Simulation results suggest that the proposed allocation policy significantly outperforms when incremental demand or upsell probability are high. Structural analyses are also provided for special demand dependence cases.

The Price of Nonabandonment: HIV in Resource-Limited Settings

The global fight against HIV/AIDS is hindered by a lack of drugs in the developing world. When patients in these countries initiate treatment, they typically remain on it until death; thus, policy makers and physicians follow nonabandonment policies. However, treated patients develop resistance to treatment, so in many cases untreated patients might benefit more from the drugs. In this paper we quantify the opportunity cost associated with restricting attention to nonabandonment policies. For this, we use an approximate dynamic programming framework to bound the benefit from allowing premature treatment termination. Our results indicate that in sub-Saharan Africa, the price associated with restricting attention to nonabandonment policies lies between 4.4% and 8.1% of the total treatment benefit. We also derive superior treatment allocation policies, which shed light on the role behavior and health progression play in prioritizing treatment initiation and termination.

Temporal logic motion control using actor–critic methods

This paper considers the problem of deploying a robot from a specification given as a temporal logic statement about some properties satisfied by the regions of a large, partitioned environment. We assume that the robot has noisy sensors and actuators and model its motion through the regions of the environment as a Markov decision process (MDP). The robot control problem becomes finding the control policy which maximizes the probability of satisfying the temporal logic task on the MDP. For a large environment, obtaining transition probabilities for each state–action pair, as well as solving the necessary optimization problem for the optimal policy, are computationally intensive. To address these issues, we propose an approximate dynamic programming framework based on a least-squares temporal difference learning method of the actor–critic type. This framework operates on sample paths of the robot and optimizes a randomized control policy with respect to a small set of parameters. The transition probabilities are obtained only when needed. Simulations confirm that convergence of the parameters translates to an approximately optimal policy.

Solving stochastic resource-constrained project scheduling problems by closed-loop approximate dynamic programming

Project scheduling problems with both resource constraints and uncertain task durations have applications in a variety of industries. While the existing research literature has been focusing on finding an a priori open-loop task sequence that minimizes the expected makespan, finding a dynamic and adaptive closed-loop policy has been regarded as being computationally intractable. In this research, we develop effective and efficient approximate dynamic programming (ADP) algorithms based on the rollout policy for this category of stochastic scheduling problems. To enhance performance of the rollout algorithm, we employ constraint programming (CP) to improve the performance of base policy offered by a priority-rule heuristic. We further devise a hybrid ADP framework that integrates both the look-back and look-ahead approximation architectures, to simultaneously achieve both the quality of a rollout (look-ahead) policy to sequentially improve a task sequence, and the efficiency of a lookup table (look-back) approach. Computational results on the benchmark instances show that our hybrid ADP algorithm is able to obtain competitive solutions with the state-of-the-art algorithms in reasonable computational time. It performs particularly well for instances with non-symmetric probability distribution of task durations.

Locomotive Planning at Norfolk Southern: An Optimizing Simulator Using Approximate Dynamic Programming

For decades, locomotive planning has been approached using the classical tools of mathematical programming; the result has been very large-scale integer programming models that are beyond the capabilities of modern solvers but still require a host of simplifying assumptions that limit their use for analyzing important planning problems. The primary interest of Norfolk Southern was in developing a model that could assist it with fleet sizing. However, the cumulative effect of the simplifications required to produce a practical integer programming formulation resulted in models that underestimated the required fleet. We use the modeling and algorithmic framework of approximate dynamic programming, which uses an intuitive balance of simulation and optimization with feedback learning, to produce a highly detailed model that calibrates accurately against historical metrics. The result was a model that can be used to plan fleet size and mix, be sensitive to a wide range of operating parameters, and adapt to many scenarios.

Inverse procedural modeling of facade layouts

In this paper, we address the following research problem: How can we generate a meaningful split grammar that explains a given facade layout? To evaluate if a grammar is meaningful, we propose a cost function based on the description length and minimize this cost using an approximate dynamic programming framework. Our evaluation indicates that our framework extracts meaningful split grammars that are competitive with those of expert users, while some users and all competing automatic solutions are less successful.

Incorporating network considerations into pavement management systems: A case for approximate dynamic programming

The objective of infrastructure management is to provide optimal maintenance, rehabilitation and replacement (MR&R) policies for a system of facilities over a planning horizon. While most approaches in the literature have studied the decision-making process as a finite resource allocation problem, the impact of construction activities on the road network is often not accounted for. The state-of-the-art Markov decision process (MDP)-based optimization approaches in infrastructure management, while optimal for solving budget allocation problems, become internally inconsistent upon introducing network constraints. In comparison, approximate dynamic programming (ADP) enables solving complex problem formulations by using simulation techniques and lower dimension value function approximations. In this paper, an ADP framework is proposed, wherein capacity losses due to construction activities are subjected to an agency-defined network capacity threshold. A parametric study is conducted on a stylized network configuration to infer the impact of network-based constraints on the decision-making process.

SMART: A Stochastic Multiscale Model for the Analysis of Energy Resources, Technology, and Policy

We address the problem of modeling energy resource allocation, including dispatch, storage, and the long-term investments in new technologies, capturing different sources of uncertainty such as energy from wind, demands, prices, and rainfall. We also wish to model long-term investment decisions in the presence of uncertainty. Accurately modeling the value of all investments, such as wind turbines and solar panels, requires handling fine-grained temporal variability and uncertainty in wind and solar in the presence of storage. We propose a modeling and algorithmic strategy based on the framework of approximate dynamic programming (ADP) that can model these problems at hourly time increments over an entire year or several decades. We demonstrate the methodology using both spatially aggregate and disaggregate representations of energy supply and demand. This paper describes the initial proof of concept experiments for an ADP-based model called SMART; we describe the modeling and algorithmic strategy and provide comparisons against a deterministic benchmark as well as initial experiments on stochastic data sets.

Approximate dynamic programming in transportation and logistics: a unified framework

Deterministic optimization has enjoyed a rich place in transportation and logistics, where it represents a mature field with established modeling and algorithmic strategies. By contrast, sequential stochastic optimization models (dynamic programs) have been plagued by the lack of a common modeling framework, and by algorithmic strategies that just do not seem to scale to real-world problems in transportation. This paper is designed as a tutorial of the modeling and algorithmic framework of approximate dynamic programming; however, our perspective on approximate dynamic programming is relatively new, and the approach is new to the transportation research community. We present a simple yet precise modeling framework that makes it possible to integrate most algorithmic strategies into four fundamental classes of policies, the design of which represents approximate solutions to these dynamic programs. The paper then uses problems in transportation and logistics to indicate settings in which each of the four classes of policies represents a natural solution strategy, highlighting the fact that the design of effective policies for these complex problems will remain an exciting area of research for many years. Along the way, we provide a link between dynamic programming, stochastic programming and stochastic search.