Dynamic Programming Policy Research Articles

Homotopic ADP based [formula omitted] integral sliding mode secure control for Markov jumping cyber-physical systems

This paper addresses the sliding mode secure control problem for continuous-time cyber-physical systems (CPSs) subject to unmatched disturbances and actuator deception attacks. To capture different operating conditions and account for random component failures in CPSs, a Markov jumping model is introduced to characterize the system's component matrices. Next, a novel model-free integral sliding mode function and a corresponding sliding mode controller are presented to mitigate the impact of external disturbances and deception attacks on Markov jumping cyber-physical systems. An algorithm based on adaptive dynamic programming (ADP) and homotopic policy iteration (PI) is applied to develop the sliding mode controller, called the online homotopic ADP algorithm, which avoids the requirement for an initial stabilizing matrix compared to existing PI algorithms. Lastly, the mode-independent integral sliding mode control scheme is studied to a two-mode Markov jumping vertical take-off and landing helicopter system, providing guaranteed reachability of sliding variables and stability of sliding mode dynamics. Importantly, this is achieved without any reliance on knowledge of system dynamics.

Relative Entropy Regularized Sample-Efficient Reinforcement Learning With Continuous Actions.

In this article, a novel reinforcement learning (RL) approach, continuous dynamic policy programming (CDPP), is proposed to tackle the issues of both learning stability and sample efficiency in the current RL methods with continuous actions. The proposed method naturally extends the relative entropy regularization from the value function-based framework to the actor-critic (AC) framework of deep deterministic policy gradient (DDPG) to stabilize the learning process in continuous action space. It tackles the intractable softmax operation over continuous actions in the critic by Monte Carlo estimation and explores the practical advantages of the Mellowmax operator. A Boltzmann sampling policy is proposed to guide the exploration of actor following the relative entropy regularized critic for superior learning capability, exploration efficiency, and robustness. Evaluated by several benchmark and real-robot-based simulation tasks, the proposed method illustrates the positive impact of the relative entropy regularization including efficient exploration behavior and stable policy update in RL with continuous action space and successfully outperforms the related baseline approaches in both sample efficiency and learning stability.

Optimal Synchronization of Unidirectionally Coupled FO Chaotic Electromechanical Devices With the Hierarchical Neural Network.

This article solves the problem of optimal synchronization, which is important but challenging for coupled fractional-order (FO) chaotic electromechanical devices composed of mechanical and electrical oscillators and electromagnetic filed by using a hierarchical neural network structure. The synchronization model of the FO electromechanical devices with capacitive and resistive couplings is built, and the phase diagrams reveal that the dynamic properties are closely related to sets of physical parameters, coupling coefficients, and FOs. To force the slave system to move from its original orbits to the orbits of the master system, an optimal synchronization policy, which includes an adaptive neural feedforward policy and an optimal neural feedback policy, is proposed. The feedforward controller is developed in the framework of FO backstepping integrated with the hierarchical neural network to estimate unknown functions of dynamic system in which the mentioned network has the formula transformation and hierarchical form to reduce the numbers of weights and membership functions. Also, an adaptive dynamic programming (ADP) policy is proposed to address the zero-sum differential game issue in the optimal neural feedback controller in which the hierarchical neural network is designed to yield solutions of the constrained Hamilton-Jacobi-Isaacs (HJI) equation online. The presented scheme not only ensures uniform ultimate boundedness of closed-loop coupled FO chaotic electromechanical devices and realizes optimal synchronization but also achieves a minimum value of cost function. Simulation results further show the validity of the presented scheme.

Critic-observer-based decentralized force/position approximate optimal control for modular and reconfigurable manipulators with uncertain environmental constraints

A critic-observer decentralized force/position approximate optimal control method is presented to address the joint trajectory and contacted force tracking problem of modular and reconfigurable manipulators (MRMs) with uncertain environmental constraints. The dynamic model of the MRM systems is formulated as an integration of joint subsystems via extensive state observer (ESO) associated with the effect of interconnected dynamic coupling (IDC). A radial basis function neural network (RBF-NN) is developed to deal with the IDC effects among the independent joint subsystems. Based on adaptive dynamic programming (ADP) approach and policy iteration (PI) algorithm, the Hamilton–Jacobi–Bellman (HJB) equation is approximately solved by establishing critic NN structure and then the approximated optimal control policy can be derived. The closed-loop manipulator system is proved to be asymptotic stable by using the Lyapunov theory. Finally, simulation results are provided to demonstrate the effectiveness and advantages of the proposed control method.

Comparison of dynamic programming policies for long-term hydrothermal scheduling of single-reservoir systems in steady-state regime

Markov stochastic dynamic programming (MSDP) has been extensively used for long-term hydrothermal systems (LTHS) of single-reservoir systems and has inspired many approaches to multiple-reservoir systems, despite the lack of comprehensive studies that test their performance. This study is concerned with the evaluation of classic dynamic programming policies in LTHS of single-reservoir systems considering the operation in steady-state regime. The Markov stochastic dynamic programming was compared with deterministic dynamic programming (DDP) considering average inflows, and unconditional stochastic dynamic programming (USDP) considering uncorrelated inflows. Performance measures were obtained through simulation over long horizons using historical and synthetically generated inflows. Sensitivity analysis related to hydro plant characteristics, the hydro generation share in the system, and the inflow considered in DDP was performed. Results show that MSDP provides small performance gains whatever the hydro plant considered, that become smaller with the reduction in the share of hydro generation in the system, and the proper consideration of inflows in DDP.

Constant Depth Decision Rules for multistage optimization under uncertainty

In this paper, we introduce a new class of decision rules, referred to as Constant Depth Decision Rules (CDDRs), for multistage optimization under linear constraints with uncertainty-affected right-hand sides. We consider two uncertainty classes: discrete uncertainties which can take at each stage at most a fixed number d of different values, and polytopic uncertainties which, at each stage, are elements of a convex hull of at most d points. Given the depthμ of the decision rule, the decision at stage t is expressed as the sum of t functions of μ consecutive values of the underlying uncertain parameters. These functions are arbitrary in the case of discrete uncertainties and are poly-affine in the case of polytopic uncertainties. For these uncertainty classes, we show that when the uncertain right-hand sides of the constraints of the multistage problem are of the same additive structure as the decision rules, these constraints can be reformulated as a system of linear inequality constraints where the numbers of variables and constraints is O(1)(n+m)dμN2 with n the maximal dimension of control variables, m the maximal number of inequality constraints at each stage, and N the number of stages.As an illustration, we discuss an application of the proposed approach to a Multistage Stochastic Program arising in the problem of hydro-thermal production planning with interstage dependent inflows. For problems with a small number of stages, we present the results of a numerical study in which optimal CDDRs show similar performance, in terms of optimization objective, to that of Stochastic Dual Dynamic Programming (SDDP) policies, often at much smaller computational cost.

Dynamic-programming-based failure-tolerant control for satellite with thrusters in 6-DOF motion

In this paper, a dynamic-programming approach to the coupled translational and rotational control of thruster-driven spacecraft is studied. To reduce the complexity of the problem, dynamic-programming-based optimal policies are calculated using decoupled position and attitude dynamics with generalized forces and torques as controls. A quadratic-programming-based control allocation is then used to map the controls to actuator commands. To control the spacecraft in the event of thruster failure, both the dynamic programming policies and control allocation are reconfigured to cope with the losses in controls. The control allocation parameters are adjusted dynamically to ensure the satellite always approaches the target from the side with two operative thrusters to achieve a stable control. The effectiveness of the proposed dynamic programming control is compared with a Lyapunov-stable control method, which shows that the proposed method is more fuel-efficient in tracking the same path.

Scalable reinforcement learning for plant-wide control of vinyl acetate monomer process

This paper explores a reinforcement learning (RL) approach that designs automatic control strategies in a large-scale chemical process control scenario as the first step for leveraging an RL method to intelligently control real-world chemical plants. The huge number of units for chemical reactions as well as feeding and recycling the materials of a typical chemical process induces a vast amount of samples and subsequent prohibitive computation complexity in RL for deriving a suitable control policy due to high-dimensional state and action spaces. To tackle this problem, a novel RL algorithm: Factorial Fast-food Dynamic Policy Programming (FFDPP) is proposed. By introducing a factorial framework that efficiently factorizes the action space, Fast-food kernel approximation that alleviates the curse of dimensionality caused by the high dimensionality of state space, into Dynamic Policy Programming (DPP) that achieves stable learning even with insufficient samples. FFDPP is evaluated in a commercial chemical plant simulator for a Vinyl Acetate Monomer (VAM) process. Experimental results demonstrate that without any knowledge of the model, the proposed method successfully learned a stable policy with reasonable computation resources to produce a larger amount of VAM product with comparative performance to a state-of-the-art model-based control.

Generative Adversarial Imitation Learning using Dynamic Policy Programming and Its Applications

Generative Adversarial Imitation Learning using Dynamic Policy Programming and Its Applications

Group Maintenance: A Restless Bandits Approach

We consider a maintenance planner problem to dynamically allocate the available repairmen to a system of unreliable production facilities. Each facility has several machines that incur a linear production loss due to stochastic degradation, which we model as a continuous time Markov process with fully observable states. The objective is to schedule group maintenance interventions, in discrete time epochs, so as to minimize production losses over an infinite horizon. Direct solution procedures, such as dynamic programming value or policy iteration, are impractical due to the curse of dimensionality. An approximate scheduling procedure is developed following Whittle’s restless bandits approach. In particular, we decompose the Whittle’s relaxation of our scheduling problem by production facility (i.e., bandit) using the Lagrangian technique. Based on the structural investigation of a single-bandit problem, we prove indexability and propose a novel index computational algorithm. Our numerical study shows that, for systems with three or four facilities, the index policy has a near-zero optimality gap. For systems with 10 or more facilities, the index policy expected cost remains fairly close to a lower bound that we compute using the known linear programming (LP) formulation of Whittle’s relaxation. Furthermore, the numerical study also shows that our policy yields substantial expected cost improvements relative to a benchmark LP-based heuristic when the states are partially observable and can handle large-scale systems unlike LP-based heuristics, which have excessive memory requirements.

Learning from demonstration for locally assistive mobility aids

Active assistive systems for mobility aids are largely restricted to environments mapped a-priori, while passive assistance primarily provides collision mitigation and other hand-crafted behaviors in the platform’s immediate space. This paper presents a framework providing active short-term assistance, combining the freedom of location independence with the intelligence of active assistance. Demonstration data consisting of on-board sensor data and driving inputs is gathered from an able-bodied expert maneuvring the mobility aid around a generic interior setting, and used in constructing a probabilistic intention model built with Radial Basis Function Networks. This allows for short-term intention prediction relying only upon immediately available user input and on-board sensor data, to be coupled with real-time path generation based upon the same expert demonstration data via Dynamic Policy Programming, a stochastic optimal control method. Together these two elements provide a combined assistive mobility system, capable of operating in restrictive environments without the need for additional obstacle avoidance protocols. Experimental results in both simulation and on the University of Technology Sydney semi-autonomous wheelchair in settings not seen in training data show promise in assisting users of power mobility aids.

Least squares policy iteration with instrumental variables vs. direct policy search: comparison against optimal benchmarks using energy storage

This article studies least-squares approximate policy iteration (API) methods with parametrized value-function approximation. We study several variations of the policy evaluation phase, namely, Bellman error minimization, Bellman error minimization with instrumental variables, projected Bellman error minimization, and projected Bellman error minimization with instrumental variables. For a general discrete-time stochastic control problem, Bellman error minimization policy evaluation using instrumental variables is equivalent to both variants of the projected Bellman error minimization. An alternative to these API methods is direct policy search based on knowledge gradient. The practical performance of these three approximate dynamic programming methods, (i) least squares API with Bellman error minimization, (ii) least squares API with Bellman error minimization with instrumental variables, and (iii) direct policy search, are investigated in the context of an application in energy storage operations management. We create a library of test problems using real-world data and apply value iteration to find their optimal policies. These optimal benchmarks are then used to compare the developed approximate dynamic programming policies. Our analysis indicates that least-squares API with instrumental variables Bellman error minimization prominently outperforms least-squares API with Bellman error minimization. However, these approaches underperform our direct policy search implementation.

Real-time scheduling of semi-urgent patients under waiting time targets

Semi-urgent patients arrive at an emergency department and visit the physician after triage. Patients right after triage should be served within a maximum allowable waiting time; whereas in-process patients need to be served as soon as possible to avoid adverse events. The physician must determine which one to be served next. To deal with this problem, a Markov decision process (MDP) is proposed for real-time scheduling. The wait of patients right after triage incurs a non-decreasing marginal waiting cost in their lateness, whereas the wait of in-process patients incurs linear cost function. The objective is to minimise the total weighted waiting cost. The properties of the MDP model are analysed. In the special case of long examination time and common treatment rate for all patients, we prove the multimodularity of the value function and the optimality of state-dependent threshold policies. Based on these properties, efficient heuristic policies and an approximate dynamic programming (ADP) policy are proposed. A threshold policy, which is defined by the function of expected tardiness of patients right after triage, is found to excel in all experiments, with average gaps less than 0.7% from the optimal control in small-size instances and 0.18% from ADP in real application.

Real‐time energy management for commute HEVs using modified A‐ECMS with traffic information recognition

To further improve fuel consumption performance of hybrid electric vehicles (HEVs) running on commute route in the face of time-varying traffic information, this paper investigates a real-time energy management strategy based on the adaptive equivalent consumption minimization strategy (A-ECMS) framework with traffic information recognition. The proposed management strategy integrates the global near optimization and the real-time performance. The simple traffic recognition is constructed by utilising k-means clustering algorithm to deal with the historical traffic data to form four clusters. The adaptive equivalence factor of the A-ECMS is designed as a three-dimensional mapping on each cluster and the system states by employing stochastic dynamic programming (SDP) policy iteration to solve offline the stochastic optimal control problem formulated by each cluster statistical characteristic. In real-time energy management controller online, the instantaneous power split is performed by the ECMS with a proper equivalent factor, which is obtained from mappings according to the cluster recognised by the current traffic situation and the state-of-charge (SOC). The effectiveness of the designed control strategy is verified by the simulation test conducted on GT-suite HEV simulator over real driving cycles.

Spare Parts Inventory Management with Substitution-Dependent Reliability

Motivated to apply sustainable supply chain principles to air-pollution control systems, this paper presents a dynamic inventory-management approach where substitution is possible to maintain these systems’ equipment. An air-pollution control system’s subsequent reliability depends on the replacement equipment selected. The corresponding problem is formulated as a stochastic dynamic program. Because the state and action space are prohibitively large, the approximate policy iteration algorithm is adapted to generate high-quality solutions. Therefore, this work replaces the value function with an affine combination of nonlinear basis functions and shows that a relaxation of the policy improvement step requires the solving of a mixed integer linear program. This approach helps in the designing of an algorithm that improves the quality of the approximation by solving a convex optimization problem. To assess the quality of resulting solutions, a lower bound is developed by considering a relaxation of the problem. In addition, two classes of heuristics are proposed based on a rolling-horizon two-stage stochastic programming formulation of the problem and a standard base-stock ordering policy. The performance of proposed policies is tested on a variety of settings, and results show that the approximate dynamic programming policies are near-optimal in the settings of interest and significantly outperform available benchmarks. The following analysis reveals that the proposed inventory replenishment policies resemble a base-stock policy with occasional deviations, and assignment and substitution decisions are determined by balancing the reliability with ordering, holding, and shortage costs. The online supplement is available at https://doi.org/10.1287/ijoc.2017.0794 .

Optimal Dynamic Liquidation of an Inventory via Forward Markets

In commodity markets, a trader selling her inventory over a finite time horizon has access not only to the spot market but also to forward contracts. The trader has the choice to sell at the spot price, or to short a forward contract for later delivery, or a combination of both. While the commodity is hedged with the forward contract and carried in the inventory, the trader can dynamically adjust the maturity of the forward contract at each time step until the inventory is completely depleted. This paper investigates the optimal liquidation strategy for such a trader in the above framework, which can be considered a dynamic extension of the cash and carry arbitrage that may arise in contango markets. It is proved, independent of the underlying stochastic forward price model, that a ‘partial sale’ strategy involving the sale of the inventory across different contracts is not optimal. In addition, under a linearity assumption on the forward price differences, the optimally selected forward maturity is limited to a subset comprising the immediate (spot), next, or last time stage. The theoretical results imply that the optimal actions set is considerably smaller than the feasible set. An Approximate Dynamic Programming (ADP) policy is developed, and compared to the solution achieved by an exact grid-based technique and a Forward Dynamic Optimization (FDO) approach. The risk and return characteristics of ADP and FDO methods are contrasted. The theoretical propositions are verified numerically, and found to reduce computational times significantly.

Kernel dynamic policy programming: Applicable reinforcement learning to robot systems with high dimensional states

We propose a new value function approach for model-free reinforcement learning in Markov decision processes involving high dimensional states that addresses the issues of brittleness and intractable computational complexity, therefore rendering the value function approach based reinforcement learning algorithms applicable to high dimensional systems. Our new algorithm, Kernel Dynamic Policy Programming (KDPP) smoothly updates the value function in accordance to the Kullback–Leibler divergence between current and updated policies. Stabilizing the learning in this manner enables the application of the kernel trick to value function approximation, which greatly reduces computational requirements for learning in high dimensional state spaces. The performance of KDPP against other kernel trick based value function approaches is first investigated in a simulated n DOF manipulator reaching task, where only KDPP efficiently learned a viable policy at n=40. As an application to a real world high dimensional robot system, KDPP successfully learned the task of unscrewing a bottle cap via a Pneumatic Artificial Muscle (PAM) driven robotic hand with tactile sensors; a system with a state space of 32 dimensions, while given limited samples and with ordinary computing resources.

Pneumatic artificial muscle-driven robot control using local update reinforcement learning

Graphical

Regular Policies in Abstract Dynamic Programming

We consider challenging dynamic programming models where the associated Bellman equation, and the value and policy iteration algorithms commonly exhibit complex and even pathological behavior. Our analysis is based on the new notion of regular policies. These are policies that are well-behaved with respect to value and policy iteration, and are patterned after proper policies, which are central in the theory of stochastic shortest path problems. We show that the optimal cost function over regular policies may have favorable value and policy iteration properties, which the optimal cost function over all policies need not have. We accordingly develop a unifying methodology to address long standing analytical and algorithmic issues in broad classes of undiscounted models, including stochastic and minimax shortest path problems, as well as positive cost, negative cost, risk-sensitive, and multiplicative cost problems.

A novel nested stochastic dynamic programming (nSDP) and nested reinforcement learning (nRL) algorithm for multipurpose reservoir optimization

In this article we present two novel multipurpose reservoir optimization algorithms named nested stochastic dynamic programming (nSDP) and nested reinforcement learning (nRL). Both algorithms are built as a combination of two algorithms; in the nSDP case it is (1) stochastic dynamic programming (SDP) and (2) nested optimal allocation algorithm (nOAA) and in the nRL case it is (1) reinforcement learning (RL) and (2) nOAA. The nOAA is implemented with linear and non-linear optimization. The main novel idea is to include a nOAA at each SDP and RL state transition, that decreases starting problem dimension and alleviates curse of dimensionality. Both nSDP and nRL can solve multi-objective optimization problems without significant computational expenses and algorithm complexity and can handle dense and irregular variable discretization. The two algorithms were coded in Java as a prototype application and on the Knezevo reservoir, located in the Republic of Macedonia. The nSDP and nRL optimal reservoir policies were compared with nested dynamic programming policies, and overall conclusion is that nRL is more powerful, but significantly more complex than nSDP.