Real-time Dynamic Programming Research Articles

Decision-Dependent Distributionally Robust Markov Decision Process Method in Dynamic Epidemic Control

In this paper, we present a Distributionally Robust Markov Decision Process (DRMDP) approach for addressing the dynamic epidemic control problem. The Susceptible-Exposed-Infectious-Recovered (SEIR) model is widely used to represent the stochastic spread of infectious diseases, such as COVID-19. While Markov Decision Processes (MDP) offers a mathematical framework for identifying optimal actions, such as vaccination and transmission-reducing intervention, to combat disease spreading according to the SEIR model. However, uncertainties in these scenarios demand a more robust approach that is less reliant on error-prone assumptions. The primary objective of our study is to introduce a new DRMDP framework that allows for an ambiguous distribution of transition dynamics. Specifically, we consider the worst-case distribution of these transition probabilities within a decision-dependent ambiguity set. To overcome the computational complexities associated with policy determination, we propose an efficient Real-Time Dynamic Programming (RTDP) algorithm that is capable of computing optimal policies based on the reformulated DRMDP model in an accurate, timely, and scalable manner. Comparative analysis against the classic MDP model demonstrates that the DRMDP achieves a lower proportion of infections and susceptibilities at a reduced cost.

Dual-UAV Payload Transportation Using Optimized Velocity Profiles via Real-Time Dynamic Programming

In this paper, a real-time dynamic programming (RTDP) approach was developed for the first time to jointly carry a slung load using two unmanned aerial vehicles (UAVs) with a trajectory optimized for time and energy consumption. The novel strategy applies RTDP algorithm, where the journey was discretized into horizons consisting of distance intervals, and for every distance interval, an optimal policy was obtained using a dynamic programming sweep. The RTDP-based strategy is applied for dual-UAV collaborative payload transportation using coordinated motion where UAVs act as actuators on the payload. The RTDP algorithm provides the optimal velocity decisions for the slung load transportation to either minimize the journey time or the energy consumption. The RTDP approach involves minimizing a cost function which is derived after simplifying the combined model of the dual-UAV-payload system. The cost function derivation was also accommodated to dynamically distribute the load/energy between two multi-rotor platforms during a transportation mission. The cost function is used to calculate transition costs for all stages and velocity decisions. A terminal cost is used at the last distance interval during the first phase of the journey when the velocity at the end of the current horizon is not known. In the second phase, the last stage or edge of the horizon includes the destination, hence final velocity is known which is used to calculate the transition cost of the final stage. Once all transition costs are calculated, the minimum cost is traced back from the final stage to the current stage to find the optimal velocity decision. The developed approach was validated in MATLAB simulation, software in the loop Gazebo simulation, and real experiments. The numerical and Gazebo simulations showed the successful optimization of journey time or energy consumption based on the selection of the factor λ. Both simulation and real experiments results show the effectiveness and the applicability of the proposed approach.

A Kernel-Based Real-Time Adaptive Dynamic Programming Method for Economic Household Energy Systems

Modern home energy management systems (HEMSs) have great flexibility of energy consumption for customers, but at the same time, bear a range of problems, such as the high system complexity, uncertainty and time-varying nature of load consumptions, and renewable sources generation. This has brought great challenges for the real-time control. To solve these problems, we propose an HEMS that integrates a kernel-based real-time adaptive dynamic programming (K-RT-ADP) with a new preprocessing short-term prediction technique. For the preprocessing short-term prediction, we propose a gated recurrent unit-bidirectional encoder representations from the transformer (GRU-BERT) model to improve the forecasting accuracy of electrical loads and renewable energy generation. In particular, we classify household appliances into the temperature-sensitive loads, human activity sensitive loads, and insensitive/constant loads. The GRU-BERT model can incorporate weather and human activity information to predict load consumption and solar generation. For real-time control, we propose and employ the K-RT-ADP HEMS based on the GRU-BERT prediction algorithm. The objective of the K-RT-ADP HEMS is to minimize the electricity cost and maximize the solar energy utilization. To enhance the nonlinear approximation ability and generalization ability of the adaptive dynamic programming (ADP) algorithm, the K-RT-ADP algorithm leverages kernel mapping instead of neural networks. Hardware-in-the-loop experiments demonstrate the superiority of the proposed K-RT-ADP HEMS over the traditional ADP control through comparison.

A Reinforcement Learning Approach to Price Cloud Resources With Provable Convergence Guarantees.

How to generate more revenues is crucial to cloud providers. Evidences from the Amazon cloud system indicate that "dynamic pricing" would be more profitable than "static pricing." The challenges are: How to set the price in real-time so to maximize revenues? How to estimate the price dependent demand so to optimize the pricing decision? We first design a discrete-time based dynamic pricing scheme and formulate a Markov decision process to characterize the evolving dynamics of the price-dependent demand. We formulate a revenue maximization framework to determine the optimal price and theoretically characterize the "structure" of the optimal revenue and optimal price. We apply the Q -learning to infer the optimal price from historical transaction data and derive sufficient conditions on the model to guarantee its convergence to the optimal price, but it converges slowly. To speed up the convergence, we incorporate the structure of the optimal revenue that we obtained earlier, leading to the VpQ-learning ( Q -learning with value projection) algorithm. We derive sufficient conditions, under which the VpQ-learning algorithm converges to the optimal policy. Experiments on a real-world dataset show that the VpQ-learning algorithm outperforms a variety of baselines, i.e., improves the revenue by as high as 50% over the Q -learning, speedy Q -learning, and adaptive real-time dynamic programming (ARTDP), and by as high as 20% over the fixed pricing scheme.

Rebalancing Docked Bicycle Sharing System with Approximate Dynamic Programming and Reinforcement Learning

The bicycle, an active transportation mode, has received increasing attention as an alternative in urban environments worldwide. However, effectively managing the stock levels of rental bicycles at each station is challenging as demand levels vary with time, particularly when users are allowed to return bicycles at any station. There is a need for system-wide management of bicycle stock levels by transporting available bicycles from one station to another. In this study, a bicycle rebalancing model based on a Markov decision process (MDP) is developed using a real-time dynamic programming method and reinforcement learning considering dynamic system characteristics. The pickup and return demands are stochastic and continuously changing. As a result, the proposed framework suggests the best operation option every 10 min based on the realized system variables and future demands predicted by the random forest method, minimizing the expected unmet demand. Moreover, we adopt custom prioritizing strategies to reduce the number of action candidates for the operator and the computational complexity for practicality in the MDP framework. Numerical experiments demonstrate that the proposed model outperforms existing methods, such as short-term rebalancing and static lookahead policies. Among the suggested prioritizing strategies, focusing on stations with a larger error in demand prediction was found to be the most effective. Additionally, the effects of various safety buffers were examined.

Dynamic Scheduling Model of Rail-Guided Vehicle (RGV) Based on Genetic Algorithms in the Context of Mobile Computing

Track guidance vehicle (RGV) is widely used in logistics warehousing and intelligent workshop, and its scheduling effectiveness will directly affect the production and operation efficiency of enterprises. In practical operation, central information system often lacks flexibility and timeliness. By contrast, mobile computing can balance the central information system and the distributed processing system, so that useful, accurate, and timely information can be provided to RGV. In order to optimize the RGV scheduling problem in uncertain environment, a genetic algorithm scheduling rule (GAM) using greedy algorithm as the genetic screening criterion is proposed in this paper. In the experiment, RGV scheduling of two-step processing in an intelligent workshop is selected as the research object. The experimental results show that the GAM model can carry out real-time dynamic programming, and the optimization efficiency is remarkable before a certain threshold.

Parameterized Energy-Optimal Regenerative Braking Strategy for Connected and Autonomous Electrified Vehicles: A Real-Time Dynamic Programming Approach

This paper presents a vehicle speed planning system called the energy-optimal deceleration planning system (EDPS), which aims to maximize energy recuperation of regenerative braking of connected and autonomous electrified vehicles. A recuperation energy-optimal speed profile is computed based on the impending deceleration requirements for turning or stopping at an intersection. This is computed to maximize the regenerative braking energy while satisfying the physical limits of an electrified powertrain. In automated driving, the powertrain of an electrified vehicle can be directly controlled by the vehicle control unit such that it follows the computed optimal speed profile. To obtain smooth optimal deceleration speed profiles, optimal deceleration commands are determined by a parameterized polynomial-based deceleration model that is obtained by regression analyses with real vehicle driving test data. The parameters are dependent on preview information such as residual time and distance as well as target speed. The key design parameter is deceleration time, which determines the deceleration speed profile to satisfy the residual time and distance constraints as well as the target speed requirement. The bounds of deceleration commands corresponding to the physical limits of the powertrain are deduced from realistic deceleration test driving. The state constraints are dynamically updated by considering the anticipated road load and the deceleration preference. For validation and comparisons of the EDPS with different preview distances, driving simulation tests with a virtual road environment and vehicle-to-infrastructure connectivity are presented. It is shown that the longer preview distance in the EDPS, the more energy-recuperation. In comparison with driver-in-the-loop simulation tests, EDPS-based autonomous driving shows improvements in energy recuperation and reduction in trip time.

Editorial Special Issue on Adaptive Dynamic Programming and Reinforcement Learning

The past decade has witnessed a surge in research activities related to adaptive dynamic programming (ADP) and reinforcement learning (RL), particularly for control applications. Several books [item 1)–5) in the Appendix] and survey papers [item 6)–10) in the Appendix] have been published on the subject. Both ADP and RL provide approximate solutions to dynamic programming problems. In a 1995 article by Barto <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> [item 11) in the Appendix], they introduced the so-called “adaptive real-time dynamic programming,” which was specifically to apply ADP for real-time control. Later, in 2002, Murray <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> [item 12) in the Appendix] developed an ADP algorithm for optimal control of continuous-time affine nonlinear systems. On the other hand, the most famous algorithms in RL are the temporal difference algorithm [item 13) in the Appendix] and the Q-learning algorithm [item 14) and 15) in the Appendix].

Real-time energy management of photovoltaic-assisted electric vehicle charging station by markov decision process

With the rapid development of electric vehicles (EVs), the dramatic rise in the demand for electricity is creating heavy pressure on local grids. The combination of renewable energy and EV charging stations (EVCSs) provides a promising solution for alleviating the scarcity of electricity. In this paper, a finite-horizon Markov decision process (MDP) model is proposed for the optimal control of a photovoltaic (PV)-assisted EVCS in a university campus. The proposed model employs the vehicle-to-grid (V2G) technology to provide ancillary services and takes dynamic electricity price and the uncertainty of the EV owners’ parking behaviors into consideration. To guarantee computational efficiency, an adapted bounded real-time dynamic programming (BRTDP) algorithm is developed as the solution technique of the MDP model. Numerical simulations based on the dynamic electricity price of France, real PV data, and the vehicle parking patterns in a university campus are conducted to demonstrate the effectiveness of the proposed energy management system (EMS). Simulation results show that the EMS can reduce the total costs by more than 55% and 29% in summer and winter compared to the conventional charging policy, while guaranteeing the satisfaction rate of the demand for EV-charging without knowing the departure times of EVs a priori.

Privacy Preserving Planning in Stochastic Environments

Collaborative privacy preserving planning (cppp) has gained much attention in the past decade. To date, cppp has focused on domains with deterministic action effects. In this paper, we extend cppp to domains with stochastic action effects. We show how such environments can be modeled as an mdp. We then focus on the popular Real-Time Dynamic Programming (RTDP) algorithm for computing value functions for mdps, extending it to the stochastic cppp setting. We provide two versions of RTDP: a complete version identical to executing centralized RTDP, and an approximate version that sends significantly fewer messages and computes competitive policies in practice. We experiment on domains adapted from the deterministic cppp literature.

On Implementing Optimal Energy Management for EREV Using Distance Constrained Adaptive Real-Time Dynamic Programming

Extended range electric vehicles (EREVs) operate both as an electric vehicle (EV) and as a hybrid electric vehicle (HEV). As a hybrid, the on-board range extender (REx) system provides additional energy to increase the feasible driving range. In this paper, we evaluate an experimental research EREV based on the 2016 Chevrolet Camaro platform for optimal energy management control. We use model-in-loop and software-in-loop environments to validate the data-driven power loss model of the research vehicle. A discussion on the limitations of conventional energy management control algorithms is presented. We then propose our algorithm derived from adaptive real-time dynamic programming (ARTDP) with a distance constraint for energy consumption optimization. To achieve a near real-time functionality, the algorithm recomputes optimal parameters by monitoring the energy storage system’s (ESS) state of charge deviations from the previously computed optimal trajectory. The proposed algorithm is adaptable to variability resulting from driving behavior or system limitations while maintaining the target driving range. The net energy consumption evaluation shows a maximum improvement of 9.8% over the conventional charge depleting/charge sustaining (CD/CS) algorithm used in EREVs. Thus, our proposed algorithm shows adaptability and fault tolerance while being close to the global optimal solution.

A Markov decision model with dead ends for operating room planning considering dynamic patient priority

This paper addresses an operating room planning problem with surgical demands from both the elective patients and the non-elective ones. A dynamic waiting list is established to prioritize and manage the patients according to their urgency levels and waiting times. In every decision period, sequential decisions are taken by selecting high-priority patients from the waiting list to be scheduled. With consideration of random arrivals of new patients and uncertain surgery durations, the studied problem is formulated as a novel Markov decision process model with dead ends. The objective is to optimize a combinatorial cost function involving patient waiting times and operating room over-utilizations. Considering that the conventional dynamic programming algorithms have difficulties in coping with large-scale problems, we apply several adapted real-time dynamic programming algorithms to solve the proposed model. In numerical experiments, we firstly apply different algorithms to solve the same instance and compare the computational efficiencies. Then, to evaluate the effects of dead ends on the policy and the computation, we conduct simulations for multiple instances with the same problem scale but different dead ends. Experimental results indicate that incorporating dead ends into the model helps to significantly shorten the patient waiting times and improve the computational efficiency.

UAV Payload Transportation via RTDP Based Optimized Velocity Profiles

This paper explores the application of a real-time dynamic programming (RTDP) algorithm to transport a payload using a multi-rotor unmanned aerial vehicle (UAV) in order to optimize journey time and energy consumption. The RTDP algorithm is developed by discretizing the journey into distance interval horizons and applying the RTDP sweep to the current horizon to get the optimal velocity decision. RTDP sweep requires the current state of the UAV to generate the next best velocity decision. To the best of the authors knowledge, this is the first time that such real-time optimization algorithm is applied to multi-rotor based transportation. The algorithm was first tested in simulations and then experiments were performed. The results show the effectiveness and applicability of the proposed algorithm.

Practical Resolution Methods for MDPs in Robotics Exemplified With Disassembly Planning

In this letter, we focus on finding practical resolution methods for Markov decision processes (MDPs) in robotics. Some of the main difficulties of applying MDPs to real-world robotics problems are: first, having to deal with huge state spaces; and second, designing a method that is robust enough to dead ends. These complications restrict or make more difficult the application of methods, such as value iteration, policy iteration, or labeled real-time dynamic programming (LRTDP). We see in determinization and heuristic search a way to successfully work around these problems. In addition, we believe that many practical use cases offer the opportunity to identify hierarchies of subtasks and solve smaller, simplified problems. We propose a decision-making unit that operates in a probabilistic planning setting through stochastic shortest path problems, which generalize the most common types of MDPs. Our decision-making unit combines: first, automatic hierarchical organization of subtasks; and second, on-line resolution via determinization. We argue that several applications of planning benefit from these two strategies. We exemplify our approach with a robotized disassembly application. The disassembly problem is modeled in probabilistic planning definition language, and serves to define our experiments. Our results show many advantages of our method over LRTDP, such as a better capability to handle problems with large state spaces and state definitions that change when new fluents are discovered.

Optimal energy management of a residential-based hybrid renewable energy system using rule-based real-time control and 2D dynamic programming optimization method

This paper presents a magnetically coupled hybrid renewable energy system (RES) for residential applications. The proposed system integrates the energies of a set of PV panels, a fuel cell stack, and a battery using a multi-winding magnetic link to supply a residential load. It can operate in multiple grid-connected and off-grid operation modes. An energy management unit including an off-line dynamic programming-based optimization stage and a real-time rule-based controller is designed to optimally control the power flow in the system according to the provided energy plan. The system is designed according to the required standards of the grid-connected residential RES. Different sections of the proposed system including steady-state operation, control techniques, energy management method and hardware design are studied in brief. A prototype of the proposed system is developed and experimentally tested for an energy management scenario considering both sunny and cloudy profiles of the PV generation. The energy distribution and cost analysis approved the benefits of the proposed system for residential consumers.

Advanced Real-Time Dynamic Programming in the Polygonal Approximation of ECG Signals for a Lightweight Embedded Device

Arrhythmia is less frequent than a normal heartbeat in an electrocardiogram signal, and the analysis of an electrocardiogram measurement can require more than 24 hours. Therefore, the efficient storage and transmission of electrocardiogram signals have been studied, and their importance has increased recently due to the miniaturization and weight reduction of measurement equipment. The polygonal approximation method based on dynamic programming can effectively achieve signal compression and fiducial point detection by expressing signals with a small number of vertices. However, the execution time and memory area rapidly increase depending on the length of the signal and number of vertices, which are not suitable for lightweight and miniaturized equipment. In this paper, we propose a method that can be applied in embedded environments by optimizing the processing time and memory usage of dynamic programming applied to the polygonal approximation of an ECG signal. The proposed method is divided into three steps to optimize the processing time and memory usage of dynamic programming. The first optimization step is based on the characteristics of electrocardiogram signals in the polygonal approximation. Second, the size of a data bit is used as the threshold for the time difference of each vertex. Finally, a type conversion and memory optimization are applied, which allow real-time processing in embedded environments. After analyzing the performance of the proposed algorithm for a signal length L and number of vertices N, the execution time is reduced from O(L 2 N) to O(L), and the memory usage is reduced from O(L 2 N) to O(LN). In addition, the proposed method preserve a performance of fiducial point detection. In a QT-DB experiment provided by Physionet, achieving values of -4.01 ± 7.99 ms and -5.46 ± 8.03 ms.

Nonconvex function activated zeroing neural network models for dynamic quadratic programming subject to equality and inequality constraints

Zeroing neural network (ZNN, or termed Zhang neural network after its inventor), being a special type of neurodynamic methodology, has shown powerful abilities to solve a great variety of time-varying problems with monotonically increasing odd activation functions. However, the existing results on ZNN cannot handle the inequality constraint in the optimization problem and nonconvex function cannot applied to accelerating the convergence speed of ZNN. This work breaks these limitations by proposing ZNN models, allowing nonconvex sets for projection operations in activation functions and incorporating new techniques for handing inequality constraint arising in optimizations. Theoretical analyses reveal that the proposed ZNN models are of global stability with timely convergence. Finally, illustrative simulation examples are provided and analyzed to substantiate the efficacy and superiority of the proposed ZNN models for real-time dynamic quadratic programming subject to equality and inequality constraints.

Real-time dynamic programming for Markov decision processes with imprecise probabilities

Markov Decision Processes have become the standard model for probabilistic planning. However, when applied to many practical problems, the estimates of transition probabilities are inaccurate. This may be due to conflicting elicitations from experts or insufficient state transition information. The Markov Decision Process with Imprecise Transition Probabilities (MDP-IPs) was introduced to obtain a robust policy where there is uncertainty in the transition. Although it has been proposed a symbolic dynamic programming algorithm for MDP-IPs (called SPUDD-IP) that can solve problems up to 22 state variables, in practice, solving MDP-IP problems is time-consuming. In this paper we propose efficient algorithms for a more general class of MDP-IPs, called Stochastic Shortest Path MDP-IPs (SSP MDP-IPs) that use initial state information to solve complex problems by focusing on reachable states. The (L)RTDP-IP algorithm, a (Labeled) Real Time Dynamic Programming algorithm for SSP MDP-IPs, is proposed together with three different methods for sampling the next state. It is shown here that the convergence of (L)RTDP-IP can be obtained by using any of these three methods, although the Bellman backups for this class of problems prescribe a minimax optimization. As far as we are aware, this is the first asynchronous algorithm for SSP MDP-IPs given in terms of a general set of probability constraints that requires non-linear optimization over imprecise probabilities in the Bellman backup. Our results show up to three orders of magnitude speedup for (L)RTDP-IP when compared with the SPUDD-IP algorithm.

Real Time Dynamic Programming Control of Rapid Thermal Processing Furnaces

In the rapid thermal processing of a semi-conductor wafer, the temperature of the latter has to follow a preset time evolution profile while keeping spatial uniformity. A model-based open loop control strategy is presented in all its steps. A global and sufficiently accurate thermal model, based on the component interaction network, is used for the prediction of the required instantaneous heat flux distribution gained by the wafer. The same model is used to calculate the distribution of the net heat flux gainedby the wafer for each radiant emitter. The optimal combination of input powers to the radiant emitters is then determined using real time dynamic programming. It consists in minimizing the maximum difference between the required heat flux distribution and the one corresponding to the tested combination of input powers at each time step. The objective function is corrected according to the preceding time step errors, in a way to avoid cumulative temperature deviation. The solution algorithm is described, step by step, and a test case is treated. The results show good temperature tracking and uniformity.

Trial-Based Heuristic Tree Search for Finite Horizon MDPs

Dynamic programming is a well-known approach for solving MDPs. In large state spaces, asynchronous versions like Real-Time Dynamic Programming have been applied successfully. If unfolded into equivalent trees, Monte-Carlo Tree Search algorithms are a valid alternative. UCT, the most popular representative, obtains good anytime behavior by guiding the search towards promising areas of the search tree. The Heuristic Search algorithm AO∗ finds optimal solutions for MDPs that can be represented as acyclic AND/OR graphs. We introduce a common framework, Trial-based Heuristic Tree Search, that subsumes these approaches and distinguishes them based on five ingredients: heuristic function, backup function, action selection, outcome selection, and trial length. Using this framework, we describe three new algorithms which mix these ingredients in novel ways in an attempt to combine their different strengths. Our evaluation shows that two of our algorithms not only provide superior theoretical properties to UCT, but also outperform state-of-the-art approaches experimentally.