Approximate Dynamic Programming Research Articles

Continuous Approximate Dynamic Programming Algorithm to Promote Multiple Battery Energy Storage Lifespan Benefit in Real-Time Scheduling

Continuous Approximate Dynamic Programming Algorithm to Promote Multiple Battery Energy Storage Lifespan Benefit in Real-Time Scheduling

Discrete‐Time Optimal Control of State‐Constrained Nonlinear Systems Using Approximate Dynamic Programming

ABSTRACTThis article investigates the optimal control problem (OCP) for a class of discrete‐time nonlinear systems with state constraints. First, to overcome the challenge caused by the constraints, the original constrained OCP is transformed into an unconstrained OCP by utilizing the system transformation technique. Second, a new cost function is designed to alleviate the effect of system transformation on the optimality of the original system. Further, a novel off‐policy deterministic approximate dynamic programming (ADP) scheme is developed to obtain a near‐optimal solution for the transformed OCP. Compared to existing off‐policy deterministic ADP schemes, the developed scheme relaxes the requirement on the learning data and saves computing resources from the perspective of training neural networks. Third, considering approximation errors, we analyze the convergence and stability of the developed ADP scheme. Finally, the developed ADP with the designed cost function is tested in two numerical cases, and simulation results confirm its effectiveness.

An approximate dynamic programming approach for solving aircraft fleet engine maintenance problem: Methodology and a case study

We consider a long-term engine maintenance planning problem for an aircraft fleet. The objective is to guarantee sufficient on-wing engines to reach service levels while effectively organizing shop visits for engines. However, complexity arises from intricate maintenance policies and uncertainty in engine deterioration. To address this problem, we propose a graph-based approach representing high-dimensional engine statuses and transitions. We then formulate the problem as a multi-stage stochastic integer program with endogenous uncertainty. We develop an approximate dynamic programming algorithm enhanced by dynamic graph generation and policy-sifting techniques so as to reduce the computational overhead in large problems. We demonstrate the efficacy of our method, compared with other popular methods, in terms of running time and solution quality. In the case study, we present an implementation in a real-world decision system in China Southern Airlines, in which the proposed method works seamlessly with other supporting modules and significantly improves the efficiency of engine maintenance management.

Robust Approximate Dynamic Programming for Large-Scale Unit Commitment With Energy Storages

Robust Approximate Dynamic Programming for Large-Scale Unit Commitment With Energy Storages

Optimization of Power Control for Autonomous Hybrid Electric Vehicles with Flexible Power Demand

Abstract Technology advancement for on-road vehicles has gained significant momentum in the past decades, particularly in the field of vehicle automation and powertrain electrification. The optimization of powertrain controls for autonomous vehicles typically involves a separated consideration of the vehicle's external dynamics and powertrain dynamics, with one key aspect often overlooked. This aspect, known as flexible power demand, recognizes that the powertrain control system does not necessarily have to precisely match the power requested by the vehicle motion controller at all times. Leveraging this feature can lead to control designs achieving improved fuel economy by adding an extra degree of freedom to the powertrain control while maintaining safety and drive comfort. The present research investigates the use of an Approximate Dynamic Programming (ADP) approach to develop a powertrain controller, which takes into account the flexibility in power demand within the ADP framework. The concept of reachable sets is incorporated into the ADP framework to ensure safety, improve ride comfort, and enhance the accuracy of the optimization solution. The formulation is based on an autonomous hybrid electric vehicle (HEV), while the methodology can also be applied to other types of vehicles. It is also found that necessary customization of the ADP algorithm is needed for this particular control problem to prevent convergence issues. Finally, a case study is presented to evaluate the effectiveness of flexible power demand, as addressed by the ADP method. The experiment demonstrates a 14.1% improvement in fuel economy compared to a scenario without flexible power demand.

Design and implementation of a universal converter for microgrid applications using approximate dynamic programming and artificial neural networks

This paper introduces a novel design for a universal DC-DC and DC-AC converter tailored for DC/AC microgrid applications using Approximate Dynamic Programming and Artificial Neural Networks (ADP-ANN). The proposed converter is engineered to operate efficiently with both low-power battery and single-phase AC supply, utilizing identical side terminals and switches for both chopper and inverter configurations. This innovation reduces component redundancy and enhances operational versatility. The converter's design emphasizes minimal switch usage while ensuring efficient conversion to meet diverse load requirements from battery or AC sources. A conceptual example illustrates the design's principles, and comprehensive analyses compare the converter's performance across various operational modes. A test bench model, rated at 3000W, demonstrates the converter's efficacy in all five operational modes with AC/DC inputs. Experimental results confirm the system's robustness and adaptability, leveraging ADP-ANN for optimal performance. The paper concludes by outlining potential applications, including microgrids, electric vehicles, and renewable energy systems, highlighting the converter's key advantages such as reduced complexity, increased efficiency, and broad applicability.

Strategic Workforce Planning in Crowdsourced Delivery With Hybrid Driver Fleets

Nowadays, logistics service providers (LSPs) increasingly consider using a crowdsourced workforce on the last mile to fulfill customers’ expectations regarding same-day or on-demand delivery at reduced costs. The crowdsourced workforce’s availability is, however, uncertain. Therefore, LSPs often hire additional fixed employees to perform deliveries when the availability of crowdsourced drivers is low. In this context, the reliability versus flexibility trade-off which LSPs face over a longer period, for example, a year, remains unstudied. Against this background, we jointly study a workforce planning problem that considers salaried drivers (SDs) and the temporal development of the crowdsourced driver (CD) fleet over a long-term time horizon. We consider two types of CDs, dedicated gig-drivers (DDs) and opportunistic gig-drivers (ODs). While DDs are not sensitive to the request’s destination and typically exhibit high availability, ODs only serve requests whose origin and destination coincide with their own private route’s origin and destination. Moreover, to account for time horizon-specific dynamics, we consider stochastic turnover for both SDs and CDs as well as stochastic CD fleet growth. We formulate the resulting workforce planning problem as a Markov decision process whose reward function reflects total costs, that is, wages and operational costs arising from serving demand with SDs and CDs, and solve it via approximate dynamic programming. Applying our approach to an environment based on real-world demand data from GrubHub, we find that in fleets consisting of SDs and CDs, approximate dynamic programming (ADP)-based hiring policies can outperform myopic hiring policies by up to [Formula: see text] and lookahead policies with perfect knowledge of future information by up to [Formula: see text] in total costs. In the studied setting, we observed that DDs reduce the LSP’s total costs more than ODs. When we account for CDs’ increased resignation probability when not being matched with enough requests, the amount of required SDs increases.

Solving optimal predictor-feedback control using approximate dynamic programming

This paper is concerned with approximately solving the optimal predictor-feedback control problem of multiplicative-noise systems with input delay in infinite horizon. The optimal predictor-feedback control, provided by the analytical method, is determined by Riccati–ZXL equations and is hard to obtain in the case of unknown system dynamics. We aim to propose a policy iteration (PI) algorithm for solving the optimal solution by approximate dynamic programming. For convergence analysis of the algorithm, we first develop a necessary and sufficient stabilizing condition, in the form of several new Lyapunov-type equations, which parameterizes all predictor-feedback controllers and can be seen as an important addition to Lyapunov stability theory. We then propose an iterative scheme for the Riccati–ZXL equations computations, along with convergence analysis, based on the condition. Inspired by this scheme, a data-driven online PI algorithm, convergence implied in that of the iterative scheme, is proposed for the optimal predictor-feedback control problem without full system dynamics. Finally, a numerical example is used to evaluate the proposed PI algorithm.

Dynamic tugboat deployment and scheduling with stochastic and time-varying service demands

Container ports serve as crucial logistics hubs in global supply chains, but navigating ships within such ports is complex due to restricted waterways. Tugboats play a critical role in ensuring safety and efficiency by escorting and towing ships under these conditions. However, the tugboat deployment and scheduling problem has received little attention. To fill the research gap, we propose a new research problem - the dynamic tugboat deployment and scheduling problem, in which not all requests are confirmed initially but dynamically confirmed over time and future tugging demands need to be anticipated when managing the utilization of tugboats. To formulate the problem, we propose an extended Markov decision process (MDP) that incorporates both reactive task assignment decisions and proactive tugboat waiting decisions, creating a reactive and proactive MDP. To solve the advanced MDP model efficiently for real-time decisions, we develop an anticipatory approximate dynamic programming method that incorporates appropriate task assignment and waiting strategies for deploying and scheduling a heterogeneous tugboat fleet and embed the method into an improved rollout algorithm to anticipate future scenarios. The effectiveness, efficiency, and performance sensitivity of the developed modeling and solution methods are demonstrated via extensive numerical experiments for the Singapore container port.

Learning-based robust output tracking control for unknown discrete-time nonlinear systems with dynamic uncertainty

Although numerous approximate dynamic programming (ADP) methods have been proposed and applied to robust control problems, research on the use of ADP methods for robust tracking control in nonlinear systems remains limited, especially for discrete time (DT) nonlinear systems. In this paper, we propose an efficient robust learning-based output tracking control method, referred to as r-LTC, for a class of unknown DT nonlinear systems with dynamic uncertainty. The objectives are achieved by: 1) using a system identification technique based on network-type coefficients Auto-Regressive with eXogenous variables (ARX) model to identify completely unknown systems; 2) reformulating the identified model into a state space model in terms of deviations from the reference trajectory, such that the robust tracking problem can be converted into a robust regulation problem; 3) transforming the robust regulation problem into an equivalent optimal regulation problem for a nominal system with a modified utility function; 4) alternately implementing the performance evaluation and control policy update through two neural networks (NNs) to numerically solve DT Hamilton-Jacobi-Bellman (HJB) equation, such that the approximated optimal feedback control inputs for nominal system can be obtained, allowing for realization of all unknown bounded uncertainties. The proposed method does not require a state observer and any steady state knowledge, only the measurable system input and output data are utilized. The convergence of NNs’ approximate weights and stability of the closed-loop uncertain systems are rigorously proven using the Lyapunov method. Lastly, a case study is performed using real data collected from the Shenzhen Metro Line 12 tunneling project to examine the performance of the proposed r-LTC algorithm in terms of the set value tracking accuracy, control system stability, and robustness against different types of disturbances.

A Novel Approximate Dynamic Programming Structure for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

A Novel Approximate Dynamic Programming Structure for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

Research on the remaining life prediction technology of environmentally friendly gas switchgear based on approximate dynamic programming

Research on the remaining life prediction technology of environmentally friendly gas switchgear based on approximate dynamic programming

Approximate dynamic programming for liner shipping network design

AbstractThe global containerised trade heavily relies on liner shipping services, facilitating the worldwide movement of large cargo volumes along fixed routes and schedules. The profitability of shipping companies hinges on how efficiently they design their shipping network; a complex optimization problem known as the liner shipping network design problem (LSNDP). In recent years, approximate dynamic programming (ADP), also known as reinforcement learning, has emerged as a promising approach for large-scale optimisation. This paper introduces a novel Markov decision process for the LSNDP and investigates the potential of ADP. We show that ADP methods based on value iteration produce optimal solutions to small instances, but their scalability is hindered by high memory demands. An ADP method based on a deep neural network requires less memory and successfully obtains feasible solutions. The quality of solutions, however, declines for larger instances, possibly due to the discrete nature of high-dimensional state and action spaces.

Approximate Dynamic Programming With Enhanced Off-Policy Learning for Coordinating Distributed Energy Resources

Approximate Dynamic Programming With Enhanced Off-Policy Learning for Coordinating Distributed Energy Resources

An ADP framework for flexibility and cost aggregation: Guarantees and open problems

With the increasing amount of Distributed Energy Resources (DERs), coordination of Distribution Grid Operators (DSOs) and Transmission Grid Operators (TSOs) is of paramount importance. Managing a large number of DERs at the TSO level is, however, challenging. To address this problem, flexibility aggregation is a topic of frequent research activities. Aggregation means to describe the combined flexibility of the DERs at the vertical grid coupling between DSO and TSO. Existing works are often limited with respect to guaranteeing feasibility, with respect to efficient numerical implementation, and in terms of quantification of the cost of DER usage. In the present paper, we investigate aggregation based on Approximate Dynamic Programming (ADP). We propose efficient numerical aggregation schemes using tools from computational geometry thus avoiding the need to solve multiple OPF problems. We rely on different variants of the DistFlow model for radial grids, which are computationally efficient. This allows to model of current and voltage limits and enables the consideration of voltage dependencies in the aggregation. Furthermore, we propose a method for cost aggregation and identify open problems of flexibility aggregation.

A fix and optimize method based approximate dynamic programming approach for the strategic fleet sizing and delivery planning problem

AbstractLogistics related costs constitute a major part in total cost of a product in general. Considering a company that delivers goods to its customers using its owned fleet, fleet ownership and operational costs together with the inventory costs compose the total logistics costs. In this study, we suggest an approximate Dynamic Programming algorithm, with a look ahead strategy, that uses the fix and optimize method as the imbedded heuristic for solving integrated fleet composition and replenishment planning problem. The total annual distribution cost factors considered in the problem are vehicle ownership costs, approximate routing costs, and inventory related costs. In this problem, we aim to minimize the total logistic cost by optimizing the fleet composition, replenishment patterns, and customers assigned to each vehicle in the fleet. We produced a set of reasonably large instances randomly and showed the efficacy of the suggested solution method.

Markov decision process framework of optimal energy dispatch in a smart data center with uninterruptible power supplies

This paper examines the use of an Uninterruptible Power Supply (UPS) to enhance the operational efficiency of data centers. It focuses on developing an optimal energy scheduling strategy for a data center equipped with UPS, using the Markov decision process (MDP) framework. The MDP framework simulates the decision-making process involved in minimizing energy costs. Each unit’s available power output in the data center is treated as a Markov state, taking into account the uncertainty associated with renewable distributed generation. This uncertainty drives the system to transition to other Markov states in subsequent decision times. A recursive optimization model is established for each Markov state at each decision time to guide state-based operations, which includes determining the unit output while considering both current and future costs. The challenge of dealing with high dimensionality, arising from a substantial number of states and actions in the model, is effectively addressed by adopting an approximate dynamic programming (ADP) method. This approach incorporates decision-state and forwards dynamic algorithms to tackle the complexity of the MDP-based model. By employing ADP, the computational burden is reduced, enabling efficient and practical solutions to be obtained.

Dynamic tuberculosis screening for healthcare employees

Regular tuberculosis (TB) screening is required for healthcare employees. TB is a potentially deadly disease. Early detection prevents the spread of the disease and helps with treatment. Two types of TB diagnostic tests, which vary in terms of cost and accuracy, are available on the market. Thus, hospitals need to carefully consider the available screening options to effectively monitor TB prevalence at low cost. We develop the first optimization model and solution approach in the literature that can be used to obtain effective TB screening policies where testing decisions are differentiated based on employee characteristics. We categorize healthcare employees based on the department they work in, the specific job they do, and their history of vaccination against TB. We develop a Markov decision process (MDP) model to determine which TB test should be utilized for each employee category to minimize the total costs related to testing, undetected infections, and employees’ time lost due to testing. Due to the size of the problem, we use approximate dynamic programming (ADP) to obtain near-optimal solutions. We analyze the ADP solutions under varying assumptions to develop a screening policy that specifies not only the type of the tests that should be used but also the frequency with which each test should be administered. We further examine the impact of system parameters such as costs and TB prevalence on our policy recommendation via a sensitivity analysis. The results indicate that our approach yields a simple policy that can be used by healthcare providers that do not have the expertise or the resources to develop and solve sophisticated optimization models on an ongoing basis.

Event-Triggered Cooperative Tracking Control of Multiagent Systems With a Dynamic Leader via Approximate Dynamic Programming

Event-Triggered Cooperative Tracking Control of Multiagent Systems With a Dynamic Leader via Approximate Dynamic Programming

Hybrid simplicial-randomized approximate stochastic dynamic programming for multireservoir optimization

Hybrid simplicial-randomized approximate stochastic dynamic programming for multireservoir optimization