Approximate Dynamic Programming Method Research Articles

Policy iteration for H∞ control of polynomial time‐varying systems

AbstractThis paper studies the H∞ control problem for polynomial time‐varying systems. The H∞ control problem has been much less investigated for time‐varying systems in comparison to the time‐invariant systems. Approximate dynamic programming (ADP) is an optimal method to solve the control problems. Therefore, it is valuable to solve the polynomial time‐varying H∞ control problem with the ADP approach. Considering the time as an independent variable for sum‐of‐squares (SOS) optimization problems, an SOS‐based ADP method is proposed to solve this problem. A policy iteration algorithm is presented, where in its policy evaluation step it is sufficient to solve an optimization problem. Some constraints are added to this optimization problem to guarantee the closed‐loop exponential stability. The convergence and stability properties of the proposed algorithm are stated and proven. Moreover, in order to design an H∞ controller with a smaller disturbance attenuation coefficient, a two‐loop algorithm is suggested. Finally, the effectiveness of the proposed method is demonstrated by simulation examples.

Biobjective optimization for railway alignment fine‐grained designs with parallel existing railways

AbstractUrban high‐speed railway construction is complex due to limited land resources, high population density, and potential construction risks, especially when new tracks are parallelly aligned to operational railways. Addressing a gap in current literature on fine optimization of manual alignment in such scenarios, this paper introduces a biobjective approximate fine‐grained optimization model for railway alignments (BA‐FORA). Utilizing an approximate dynamic programming (ADP) method, BA‐FORA effectively searches the feasible region to approach a global optimum, overcoming the dimensionality challenges inherent in standard dynamic programming (DP). This paper presents a biobjective optimization framework that takes into account both construction cost and construction risk adjacent to existing operating railways (CRAEOR), offering a method for the fine‐grained design of new railways adjacent to existing railways. Finally, the proposed BA‐FORA framework is applied to practical cases, demonstrating its superior optimization performance. The findings indicate that the BA‐FORA model can autonomously investigate and enhance railway alignment. It generates cost‐effective and low‐risk solutions exceeding manual efforts, ensuring alignment constraint compliance.

Development and Adaptive Assistance Control of the Robotic Hip Exoskeleton to Improve Gait Symmetry and Restore Normal Gait

Neurological disorders and aging induce abnormal gait kinematics such as asymmetric gait, reduced stride length, and walking speeds. Gait asymmetry increases musculoskeletal pain and joint impairment in the affected leg, increases the risk of patients falling, and induces higher energy expenditure. This paper presents a novel assistive control framework for the series elastic actuator-driven robotic hip exoskeleton that improves gait symmetry and restores normal gait. The proposed assistive control depends on virtual stiffness compensation with a series elastic actuator (SEA)-driven hip exoskeleton to assist without needing estimation, prediction of the gait phase, or human intent. In addition, adaptive adjustment of the virtual stiffness based on the approximate dynamic programming method is used to improve gait symmetry. The proposed control strategy is successfully implemented, and experiments are validated with human subjects’ gait assistance using a SEA-driven wearable hip exoskeleton. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper was motivated by the problem of hip exoskeleton assistive control. The goal of hip exoskeleton assistance is to improve gait symmetry and restore the normal gait of the patient with neurological disorders and the elderly. Existing approaches to fixturing such parts generally need to estimate or predict the gait phase or human intent. And the existing approaches generally adopted fixed control parameters. The hip exoskeleton’s assistance cannot adapt to various walking conditions and human gait differences. The proposed assistive control is based on virtual stiffness compensation with a SEA-driven hip exoskeleton to assist with walking under various walking conditions without estimating or predicting the gait phase or human intent. And the proposed method is online adjusted control parameters with human-in-the-loop to improve the gait symmetry. The human walking experiment results showed that the hip exoskeleton assistive controller not only reduced muscle activation and human effort but also improved the interjoint coordination of the impaired joint online. We also showed how the assistance controller changes the control parameters according to the gait symmetry with a human-in-the-loop strategy.

Approximate dynamic programming for constrained linear systems: A piecewise quadratic approximation approach

Approximate dynamic programming (ADP) faces challenges in dealing with constraints in control problems. Model predictive control (MPC) is, in comparison, well-known for its accommodation of constraints and stability guarantees, although its computation is sometimes prohibitive. This paper introduces an approach combining the two methodologies to overcome their individual limitations. The predictive control law for constrained linear quadratic regulation (CLQR) problems has been proven to be piecewise affine (PWA) while the value function is piecewise quadratic. We exploit these formal results from MPC to design an ADP method for CLQR problems with a known model. A novel convex and piecewise quadratic neural network with a local–global architecture is proposed to provide an accurate approximation of the value function, which is used as the cost-to-go function in the online dynamic programming problem. An efficient decomposition algorithm is developed to generate the control policy and speed up the online computation. Rigorous stability analysis of the closed-loop system is conducted for the proposed control scheme under the condition that a good approximation of the value function is achieved. Comparative simulations are carried out to demonstrate the potential of the proposed method in terms of online computation and optimality.

Research on Autonomous Reconstruction Method for Dependent Combat Networks

Cascade effects arising from the interoperability of weapon systems in a dynamic battlefield environment make the combat system-of-systems (CSoS) portfolio decision a problem. In this article, we propose an integrated CSoS autonomous decision-making method framework for solving the autonomous planning problem of CSoS. Specifically, we employ a functional dependency network analysis method to capture the interdependencies between heterogeneous equipment and construct a dependent combat network (DCN). The network structure is considered as a decision action space, the CSoS formation process is considered as a portfolio decision process of the combat network, and then, an approximate dynamic programming method is employed to obtain the optimal policy function for CSoS. Finally, the validity of the proposed method is verified through a DCN case study. The results indicate that the CSoS reconstructed according to the optimal policy function restores an average of 20.2% of the lost operational capability under four attack scenarios. This study provides a technical method for autonomous planning and decision making of CSoS.

Optimizing Operations Management and Business Analytics Strategies under Uncertainty: Dynamic Programming

Dynamic programming is a method used in mathematics, management science, computer science, economics, and bioinformatics to solve complex problems by decomposing the original problem into relatively simple sub-problems. Dynamic programming is often applicable to problems with overlapping subproblems and optimal substructure properties. This paper employs a comprehensive research approach of literature review, as well as empirical analysis and case studies to investigate the topic and demonstrate the practicality and effectiveness of dynamic programming in solving complex decision-making problems. However, the curse of dimensionality poses challenges when dealing with high-dimensional decision spaces, requiring approximate dynamic programming methods, including reinforcement learning algorithms. Therefore, when dealing with more complex dynamic programming problems, it is also essential to use program construction tools such as Python to help design a program that can optimize the problem. Therefore, in the learning of dynamic programming, in addition to the correct understanding of basic concepts and methods, specific problems must be analyzed and dealt with in detail, models should be built with rich imagination, and solutions should be solved with creative skills.

A LSTM-based approximate dynamic programming method for hydropower reservoir operation optimization

Dynamic programming (DP) is a classical method developed to address the multi-stage hydropower reservoir operation problem, but still suffers from the serious dimensionality problem where the computational burden increases exponentially with the number of state variables. To improve the DP performance, this paper proposes a LSTM-based approximate dynamic programming (ADP) method for complex hydropower reservoir operation optimization. In ADP, the long short-term memory (LSTM) is treated as the response surface model to reduce redundant computations of power outputs in DP’s recursive equation, making obvious improvements in the execution efficiency. To fully assess its feasibility, the ADP method is used to find the scheduling schemes of a real-world reservoir system in China. Simulation results show that compared with the standard DP method, ADP effectively reduces the execution time while guarantee the solution quality in different cases. In the 1000-state and wet-year scenario, the ADP method achieves approximately 86.7% and 85.8% reductions in DP’s computation time for Longyangxia and Laxiwa reservoir with the goal of maximizing power generation. Thus, the LSTM-based response surface model is an effective tool to improve the DP performance in the hydropower reservoir operation problem.

Multiple stage stochastic planning of integrated electricity and gas system based on distributed approximate dynamic programming

With the increasing coupling of integrated electricity and gas systems, it's essential to implement a proper planning scheme. The planning of the integrated electricity and gas systems usually consists of multiple planning stages and various uncertainties, making the problem difficult to solve. In this paper, combining the approximate dynamic programming method and the alternating direction multiplier method, a distributed approximate dynamic programming based multiple stage stochastic planning scheme is proposed for the integrated electricity and gas systems. The multiple stage stochastic planning scheme realizes the coordination of new installation and expansion of power grid lines, natural gas pipelines, substations, city gates, photovoltaics, wind turbines, gas turbines and energy storage devices, while considering both the long-term and the short-term uncertainties. The multiple stage stochastic planning model is reformulated as a Markov decision process with a sequential decision strategy, in which the investment variables are decided stage by stage with long-term uncertainties gradually revealed in the planning period. The distributed approximate dynamic programming algorithm is used to solve the multiple stage stochastic planning model, which is in the form of the Markov decision process, by decoupling in both temporal and spatial dimensions. Numerical tests on the test system with a 24-node power grid and a 30-node natural gas network verify the effectiveness of the multiple stage stochastic planning model.

Policy Iteration Based Approximate Dynamic Programming Toward Autonomous Driving in Constrained Dynamic Environment

In the area of autonomous driving, it typically brings great difficulty in solving the motion planning problem since the vehicle model is nonlinear and the driving scenarios are complex. Particularly, most of the existing methods cannot be generalized to dynamically changing scenarios with varying surrounding vehicles. To address this problem, this development here investigates the framework of integrated decision and control. As part of the modules, static path planning determines the reference candidates ahead, and then the optimal path-tracking controller realizes the specific autonomous driving task. An innovative and effective constrained finite-horizon approximate dynamic programming (ADP) algorithm is herein presented to generate the desired control policy for effective path tracking. With the generalized policy neural network that maps from the state to the control input, the proposed algorithm preserves the high effectiveness for the motion planning problem towards changing driving environments with varying surrounding vehicles. Moreover, the algorithm attains the noteworthy advantage of alleviating the typically heavy computational loads with the mode of offline training and online execution. As a result of the utilization of multi-layer neural networks in conjunction with the actor-critic framework, the constrained ADP method is capable of handling complex and multidimensional scenarios. Finally, various simulations have been carried out to show that the constrained ADP algorithm is effective.

Fault-Tolerant Coverage Control of Multiple Satellites: A Differential Graphical Game Approach

This paper considers the regional coverage problem of multiple satellites with full consideration of the coupled position-attitude characteristics, the optimality, and the cooperation among satellites. In the fault-free case, a configuration condition of satellite formation flying (SFF) is established under which the desired coverage configuration is designed; In the presence of completely failed satellites, a configuration adjustment scheme is provided such that the coverage tasks of failed satellites are handed over to other healthy ones; Based on the proposed configurations, a distributed near-optimal coverage controller that achieves the optimal coverage control goal is designed for each satellite by utilizing the differential graphical game theory and the approximate dynamic programming (ADP) method. Simulation results show the effectiveness of the proposed methods.

Two-stage robust optimization of power cost minimization problem in gunbarrel natural gas networks by approximate dynamic programming

In short-term operation of natural gas network, the impact of demand uncertainty is not negligible. To address this issue we propose a two-stage robust model for power cost minimization problem in gunbarrel natural gas networks. The demands between pipelines and compressor stations are uncertain with a budget parameter, since it is unlikely that all the uncertain demands reach the maximal deviation simultaneously. During solving the two-stage robust model we encounter a bilevel problem which is challenging to solve. We formulate it as a multi-dimensional dynamic programming problem and propose approximate dynamic programming methods to accelerate the calculation. Numerical results based on real network in China show that we obtain a speed gain of 7 times faster in average without compromising optimality compared with original dynamic programming algorithm. Numerical results also verify the advantage of robust model compared with deterministic model when facing uncertainties. These findings offer short-term operation methods for gunbarrel natural gas network management to handle with uncertainties.

Dynamic lateral transshipment policy of perishable foods with replenishment and recycling

Considering the perishable nature of foods, avoiding overstocking and stockout is important for retailers to reduce operating costs and promote environmental sustainability. Traditional lateral transshipment policies focus on rebalancing inventory between retailers by applying redistribution combined with replenishment, but without considering recycling, which is an important tool in the management of perishable foods including fruits, vegetables, milk, and festival foods. In this work, we propose a lateral transshipment policy with both replenishment and recycling simultaneously, and model the inventory problem as a stochastic dynamic programming. We apply two different approximate dynamic programming methods to deal with the curse of dimensionality in the proposed model, and obtain the corresponding inventory policies including the decisions on lateral transshipment, replenishment and recycling. Finally, we test the two proposed inventory policies based on quasi-myopic approximation and look forward approximation by using amounts of random demand samples. The results show that both proposed policies are efficient in improving profitability and reducing waste. In addition, the results suggest that governments should pay more attention to the food regulatory framework.

An approximate dynamic programming method for unit-based small hydropower scheduling

Hydropower will become an important power source of China’s power grids oriented to carbon neutral. In order to fully exploit the potential of water resources and achieve low-carbon operation, this paper proposes an approximate dynamic programming (ADP) algorithm for the unit-based short-term small hydropower scheduling (STSHS) framework considering the hydro unit commitment, which can accurately capture the physical and operational characteristics of individual units. Both the non-convex and non-linearization characteristics of the original STSHS model are retained without any linearization to accurately describe the hydropower production function and head effect, especially the dependence between the net head and the water volume in the reservoir, thereby avoiding loss of the actual optimal solution due to the large error introduced by the linearization process. An approximate value function of the original problem is formulated via the searching table model and approximate policy value iteration process to address the “curse of dimensionally” in traditional dynamic programming, which provides an approximate optimal strategy for the STSHS by considering both algorithm accuracy and computational efficiency. The model is then tested with a real-world instance of a hydropower plant with three identical units to demonstrate the effectiveness of the proposed method.

Data-Driven Adaptive Equivalent Consumption Minimization Strategy for Hybrid Electric and Connected Vehicles

Advanced energy management strategies (EMS) are used to control the power flow through a vehicle’s powertrain. However, the cost of high-power computational hardware and lack of a priori knowledge of future road conditions poses difficult challenges for engineers attempting to implement globally optimal frameworks. One solution is to use advanced driver assistance systems (ADAS) and connectivity to obtain a prediction of future road conditions. This paper presents a look-ahead predictive EMS which combines approximate dynamic programming (ADP) methods and an adaptive equivalent consumption minimization strategy (A-ECMS) to obtain a near-optimal solution for a future prediction horizon. ECMS is highly sensitive to the equivalence factor (EF), making it necessary to adapt during a trip to account for disturbances. A novel adaptation method is presented in this work which uses an artificial neural network to learn the nonlinear relationship between a speed and the state of charge (SOC) trajectory prediction obtained from ADP to estimate the corresponding EF. A traffic uncertainty analysis demonstrates an approximately 10% fuel economy (FE) improvement over traditional A-ECMS. Using a data-driven adaptation method for A-ECMS informed by a dynamic programming (DP) based prediction results in an EMS capable of online implementation.

Can Deep Reinforcement Learning Improve Inventory Management? Performance on Lost Sales, Dual-Sourcing, and Multi-Echelon Problems

Problem definition: Is deep reinforcement learning (DRL) effective at solving inventory problems? Academic/practical relevance: Given that DRL has successfully been applied in computer games and robotics, supply chain researchers and companies are interested in its potential in inventory management. We provide a rigorous performance evaluation of DRL in three classic and intractable inventory problems: lost sales, dual sourcing, and multi-echelon inventory management. Methodology: We model each inventory problem as a Markov decision process and apply and tune the Asynchronous Advantage Actor-Critic (A3C) DRL algorithm for a variety of parameter settings. Results: We demonstrate that the A3C algorithm can match the performance of the state-of-the-art heuristics and other approximate dynamic programming methods. Although the initial tuning was computationally demanding and time demanding, only small changes to the tuning parameters were needed for the other studied problems. Managerial implications: Our study provides evidence that DRL can effectively solve stationary inventory problems. This is especially promising when problem-dependent heuristics are lacking. Yet, generating structural policy insight or designing specialized policies that are (ideally provably) near optimal remains desirable.

A Monotone Approximate Dynamic Programming Approach for the Stochastic Scheduling, Allocation, and Inventory Replenishment Problem: Applications to Drone and Electric Vehicle Battery Swap Stations

There is a growing interest in using electric vehicles (EVs) and drones for many applications. However, battery-oriented issues, including range anxiety and battery degradation, impede adoption. Battery swap stations are one alternative to reduce these concerns that allow the swap of depleted for full batteries in minutes. We consider the problem of deriving actions at a battery swap station when explicitly considering the uncertain arrival of swap demand, battery degradation, and replacement. We model the operations at a battery swap station using a finite horizon Markov decision process model for the stochastic scheduling, allocation, and inventory replenishment problem (SAIRP), which determines when and how many batteries are charged, discharged, and replaced over time. We present theoretical proofs for the monotonicity of the value function and monotone structure of an optimal policy for special SAIRP cases. Because of the curses of dimensionality, we develop a new monotone approximate dynamic programming (ADP) method, which intelligently initializes a value function approximation using regression. In computational tests, we demonstrate the superior performance of the new regression-based monotone ADP method compared with exact methods and other monotone ADP methods. Furthermore, with the tests, we deduce policy insights for drone swap stations.

Dynamic Programming Through the Lens of Semismooth Newton-Type Methods

Policy iteration and value iteration are at the core of many (approximate) dynamic programming methods. For Markov Decision Processes with finite state and action spaces, we show that they are instances of semismooth Newton-type methods for solving the Bellman equation. In particular, we prove that policy iteration is equivalent to the exact semismooth Newton method and enjoys a local quadratic convergence rate. This finding is corroborated by extensive numerical evidence in the fields of control and operations research, which confirms that policy iteration generally requires relatively few iterations to achieve convergence even in presence of a large number of admissible policies. We then show that value iteration is an instance of the fixed-point iteration method and develop a novel locally accelerated version of value iteration with global convergence guarantees and negligible extra computational costs.

An Improved Approximate Dynamic Programming Method for The Integrated Fleet Sizing and Replenishment Planning Problem with Predetermined Delivery Frequencies

In this paper we propose an Improved Approximate Dynamic Programming approach for the integrated fleet sizing and replenishment planning problem utilizing fix and optimize heuristic. The problem is about deciding both the composition of a fleet for distributing a single item and the replenishment planning based on a predetermined delivery frequency. This problem integrates two important logistical issues namely fleet sizing and replenishment planning. The objective is minimizing all relevant costs composed of vehicle ownership, inventory, and approximate routing costs. We show the effectiveness of the proposed solution method on a set of fairly large size randomly generated problem instances.

An approximate dynamic programming approach for production‐delivery scheduling under non‐stationary demand

AbstractWe consider an integrated production and delivery scheduling problem with non‐stationary demand in a two‐stage supply chain, where orders arrive dynamically and the demand is time‐varying. Orders should be first processed on identical machines and then delivered to a single next‐stage destination by the transporters with fixed departure times. The objective is to minimize the order waiting time via production‐delivery scheduling. We formulate the problem into a Markov decision process model and develop an approximate dynamic programming (ADP) method. To shrink action (decision) space, we propose the shorter processing time first and first completion first delivery (SPTm/FCFD) principle to determine order processing sequences and order delivery, and then we establish two constraints to eliminate a fraction of inferior actions. Based on the SPTm/FCFD principle, we propose the SPT/FCFD rule, and show its optimality for two scenarios. In addition, we deploy five basis functions to approximate the value function. The superior performance of ADP policy is validated via numerical experiments, compared with four benchmark policies. We also empirically study the impact of demand features on the waiting time, and results show that these features significantly affect the performances of all polices. In practice, it is suggested to postpone the peak demand, when total demand exceeds the available production capacity.

Blood component preparation‐inventory problem with stochastic demand and supply

AbstractBlood component preparation, which is the process of separating whole blood into components that are later used in hematotherapy, plays a significant role in the blood supply chain because it dramatically affects the component inventory levels in blood banks. A dynamic programming model is proposed for jointly optimizing the preparation and inventory management of blood components under uncertainty in both the supply of whole blood and the demand for blood components to better meet demand and manage component inventory levels. The combinations of five common methods for preparing blood components with filtration and centrifugation processes are considered, which generate diverse component varieties and volumes while incurring different costs. Moreover, downward substitution among different types of red blood cells is introduced. Subsequently, an approximate dynamic programming method is proposed to solve the problem of coping with its uncertain characteristics. The results of the computational study show that the proposed model and algorithm lead to superior performance compared to the preparation method combinations currently applied in North America, Europe, and China, thus providing a helpful guide on blood component preparation and inventory management in blood banks.