Robust Markov Decision Process Research Articles

Bounding the difference between the values of robust and non-robust Markov decision problems

Abstract In this note we provide an upper bound for the difference between the value function of a distributionally robust Markov decision problem and the value function of a non-robust Markov decision problem, where the ambiguity set of probability kernels of the distributionally robust Markov decision process is described by a Wasserstein ball around some reference kernel whereas the non-robust Markov decision process behaves according to a fixed probability kernel contained in the ambiguity set. Our derived upper bound for the difference between the value functions is dimension-free and depends linearly on the radius of the Wasserstein ball.

On the Convex Formulations of Robust Markov Decision Processes

Robust Markov decision processes (MDPs) are used for applications of dynamic optimization in uncertain environments and have been studied extensively. Many of the main properties and algorithms of MDPs, such as value iteration and policy iteration, extend directly to RMDPs. Surprisingly, there is no known analog of the MDP convex optimization formulation for solving RMDPs. This work describes the first convex optimization formulation of RMDPs under the classical sa-rectangularity and s-rectangularity assumptions. By using entropic regularization and exponential change of variables, we derive a convex formulation with a number of variables and constraints polynomial in the number of states and actions, but with large coefficients in the constraints. We further simplify the formulation for RMDPs with polyhedral, ellipsoidal, or entropy-based uncertainty sets, showing that, in these cases, RMDPs can be reformulated as conic programs based on exponential cones, quadratic cones, and nonnegative orthants. Our work opens a new research direction for RMDPs and can serve as a first step toward obtaining a tractable convex formulation of RMDPs. Funding: The work in the paper was supported, in part, by NSF [Grants 2144601 and 1815275]; and Agence Nationale de la Recherche [Grant 11-LABX-0047].

Robust Average-Reward Reinforcement Learning

Robust Markov decision processes (MDPs) aim to find a policy that optimizes the worst-case performance over an uncertainty set of MDPs. Existing studies mostly have focused on the robust MDPs under the discounted reward criterion, leaving the ones under the average-reward criterion largely unexplored. In this paper, we develop the first comprehensive and systematic study of robust average-reward MDPs, where the goal is to optimize the long-term average performance under the worst case. Our contributions are four-folds: (1) we prove the uniform convergence of the robust discounted value function to the robust average-reward function as the discount factor γ goes to 1; (2) we derive the robust average-reward Bellman equation, characterize the structure of its solution set, and prove the equivalence between solving the robust Bellman equation and finding the optimal robust policy; (3) we design robust dynamic programming algorithms, and theoretically characterize their convergence to the optimal policy; and (4) we design two model-free algorithms unitizing the multi-level Monte-Carlo approach, and prove their asymptotic convergence

Solving Non-rectangular Reward-Robust MDPs via Frequency Regularization

In robust Markov decision processes (RMDPs), it is assumed that the reward and the transition dynamics lie in a given uncertainty set. By targeting maximal return under the most adversarial model from that set, RMDPs address performance sensitivity to misspecified environments. Yet, to preserve computational tractability, the uncertainty set is traditionally independently structured for each state. This so-called rectangularity condition is solely motivated by computational concerns. As a result, it lacks a practical incentive and may lead to overly conservative behavior. In this work, we study coupled reward RMDPs where the transition kernel is fixed, but the reward function lies within an alpha-radius from a nominal one. We draw a direct connection between this type of non-rectangular reward-RMDPs and applying policy visitation frequency regularization. We introduce a policy-gradient method, and prove its convergence. Numerical experiments illustrate the learned policy's robustness and its less conservative behavior when compared to rectangular uncertainty.

Robust Average-Reward Markov Decision Processes

In robust Markov decision processes (MDPs), the uncertainty in the transition kernel is addressed by finding a policy that optimizes the worst-case performance over an uncertainty set of MDPs. While much of the literature has focused on discounted MDPs, robust average-reward MDPs remain largely unexplored. In this paper, we focus on robust average-reward MDPs, where the goal is to find a policy that optimizes the worst-case average reward over an uncertainty set. We first take an approach that approximates average-reward MDPs using discounted MDPs. We prove that the robust discounted value function converges to the robust average-reward as the discount factor goes to 1, and moreover when it is large, any optimal policy of the robust discounted MDP is also an optimal policy of the robust average-reward. We further design a robust dynamic programming approach, and theoretically characterize its convergence to the optimum. Then, we investigate robust average-reward MDPs directly without using discounted MDPs as an intermediate step. We derive the robust Bellman equation for robust average-reward MDPs, prove that the optimal policy can be derived from its solution, and further design a robust relative value iteration algorithm that provably finds its solution, or equivalently, the optimal robust policy.

Decision-Dependent Distributionally Robust Markov Decision Process Method in Dynamic Epidemic Control

In this paper, we present a Distributionally Robust Markov Decision Process (DRMDP) approach for addressing the dynamic epidemic control problem. The Susceptible-Exposed-Infectious-Recovered (SEIR) model is widely used to represent the stochastic spread of infectious diseases, such as COVID-19. While Markov Decision Processes (MDP) offers a mathematical framework for identifying optimal actions, such as vaccination and transmission-reducing intervention, to combat disease spreading according to the SEIR model. However, uncertainties in these scenarios demand a more robust approach that is less reliant on error-prone assumptions. The primary objective of our study is to introduce a new DRMDP framework that allows for an ambiguous distribution of transition dynamics. Specifically, we consider the worst-case distribution of these transition probabilities within a decision-dependent ambiguity set. To overcome the computational complexities associated with policy determination, we propose an efficient Real-Time Dynamic Programming (RTDP) algorithm that is capable of computing optimal policies based on the reformulated DRMDP model in an accurate, timely, and scalable manner. Comparative analysis against the classic MDP model demonstrates that the DRMDP achieves a lower proportion of infections and susceptibilities at a reduced cost.

Data-driven remanufacturing planning with parameter uncertainty

We consider the problem of remanufacturing planning in the presence of statistical estimation errors. Determining the optimal remanufacturing timing, first and foremost, requires modeling of the state transitions of a system. The estimation of these probabilities, however, often suffers from data inadequacy and is far from accurate, resulting in serious degradation in performance. To mitigate the impacts of the uncertainty in transition probabilities, we develop a novel data-driven modeling framework for remanufacturing planning in which decision makers can remain robust with respect to statistical estimation errors. We model the remanufacturing planning problem as a robust Markov decision process, and construct ambiguity sets that contain the true transition probabilities with high confidence. We further establish structural properties of optimal robust policies and provide insights for remanufacturing planning. A computational study on the NASA turbofan engine shows that our data-driven robust decision framework consistently yields better out-of-sample reward and higher reliability of the performance guarantee, compared to the nominal model that uses the maximum likelihood estimates of the transition probabilities without considering parameter uncertainty.

Toward theoretical understandings of robust Markov decision processes: Sample complexity and asymptotics

In this paper, we study the nonasymptotic and asymptotic performances of the optimal robust policy and value function of robust Markov Decision Processes (MDPs), where the optimal robust policy and value function are estimated from a generative model. While prior work focusing on nonasymptotic performances of robust MDPs is restricted in the setting of the KL uncertainty set and (s,a)-rectangular assumption, we improve their results and also consider other uncertainty sets, including the L1 and χ2 balls. Our results show that when we assume (s,a)-rectangular on uncertainty sets, the sample complexity is about O˜(|S|2|A| ε2ρ2(1−γ)4). In addition, we extend our results from the (s,a)-rectangular assumption to the s-rectangular assumption. In this scenario, the sample complexity varies with the choice of uncertainty sets and is generally larger than the case under the (s,a)-rectangular assumption. Moreover, we also show that the optimal robust value function is asymptotically normal with a typical rate n under the (s,a) and s-rectangular assumptions from both theoretical and empirical perspectives.

A methodical interpretation of adaptive robotics: Study and reformulation

The recent development of industrial manufacturing and social services has witnessed a significant trend of automation and intelligentization due to the wide application of robots and the technology of artificial intelligence (AI). While robots liberate humans from tedious and dangerous work in hazardous environments, AI simplifies the programming of robots by automatically inferring patterns and models from the interaction between the robots and the environment. Nevertheless, the application of robots and AI to more general manufacturing and social tasks is still limited by the lack of flexibility and adaptability to the changes in the task and the environment. Thus, a new concept, adaptive robotics, has been proposed to address the desire that an AI-powered robot should be able to properly reprogram itself to these changes without human intervention. Nevertheless, this concept is yet too abstract to provide any specific guidance to the development of robot programs. In this paper, we attempt to provide methodical redefinition and reformulation of adaptive robotics both in conceptual and mathematical manners based on the study of previous results. First of all, we introduce the essential motivation and the conceptual origination of adaptability of mechanical systems. Then, we review the previous literature and explore the related work of adaptive robotics. Based on this, we provide a uniform mathematical formulation of adaptive robotics based on adaptive and robust Markov-decision process (MDP). Through this work, we attempt to inspire the generic framework of adaptive robotics incorporating the existing immature paradigms, by which we are aiming at a clarified and well-defined context of adaptive robotics for future research on related domains.

Policy Learning for Robust Markov Decision Process with a Mismatched Generative Model

In high-stake scenarios like medical treatment and auto-piloting, it's risky or even infeasible to collect online experimental data to train the agent. Simulation-based training can alleviate this issue, but may suffer from its inherent mismatches from the simulator and real environment. It is therefore imperative to utilize the simulator to learn a robust policy for the real-world deployment. In this work, we consider policy learning for Robust Markov Decision Processes (RMDP), where the agent tries to seek a robust policy with respect to unexpected perturbations on the environments. Specifically, we focus on the setting where the training environment can be characterized as a generative model and a constrained perturbation can be added to the model during testing. Our goal is to identify a near-optimal robust policy for the perturbed testing environment, which introduces additional technical difficulties as we need to simultaneously estimate the training environment uncertainty from samples and find the worst-case perturbation for testing. To solve this issue, we propose a generic method which formalizes the perturbation as an opponent to obtain a two-player zero-sum game, and further show that the Nash Equilibrium corresponds to the robust policy. We prove that, with a polynomial number of samples from the generative model, our algorithm can find a near-optimal robust policy with a high probability. Our method is able to deal with general perturbations under some mild assumptions and can also be extended to more complex problems like robust partial observable Markov decision process, thanks to the game-theoretical formulation.

Robust Markov Decision Processes with Data-Driven, Distance-Based Ambiguity Sets

We consider finite- and infinite-horizon Markov decision processes (MDPs) with unknown state-transition probabilities. They are assumed to belong to certain ambiguity sets, and the goal is to maximize the worst-case expected total discounted reward over all probabilities from these sets. Specifically, the ambiguity set for any state-action-stage triplet is a ball---it includes all probability mass functions (pmfs) within a certain distance from the empirical pmf constructed using historical, independent observations of state transitions. We prove that optimal values in the resulting robust MDPs (RMDPs) converge to the optimal value of the true MDP if the radii of the ambiguity balls approach zero as the sample-size diverges to infinity. In addition, robust optimal policies for sufficiently large sample-sizes are optimal to the true MDP. These results rely on a sufficient condition that links convergence of pmfs with respect to the distance function with their componentwise convergence in an appropriate space. Further, for finite sample-sizes, the optimal value of the RMDP provides a lower bound on the value of the robust optimal policy in the true MDP, with a high probability. A certain concentration inequality is sufficient for this out-of-sample performance guarantee. Several well-known distances satisfy these conditions. Numerical experiments suggest that one can choose from several distance functions to build computationally tractable RMDPs that exhibit good out-of-sample performance, and balance conservativeness with probabilistic guarantees.

Distributionally Robust Inventory Control When Demand Is a Martingale

Demand forecasting plays an important role in many inventory control problems. To mitigate the potential harms of model misspecification in this context, various forms of distributionally robust optimization have been applied. Although many of these methodologies suffer from the problem of time inconsistency, the work of Klabjan et al. established a general time-consistent framework for such problems by connecting to the literature on robust Markov decision processes. Motivated by the fact that many forecasting models exhibit very special structure as well as a desire to understand the impact of positing different dependency structures in distributionally robust multistage optimization, we formulate and solve a time-consistent distributionally robust multistage newsvendor model, which naturally robustifies some of the simplest inventory models with demand forecasting. In particular, in some of the simplest such models, demand evolves as a martingale (i.e., expected demand tomorrow equals realized demand today). We consider a robust variant of such models in which the sequence of future demands may be any martingale with given mean and support. Under such a model, past realizations of demand are naturally incorporated into the structure of the uncertainty set going forward. We explicitly compute the minimax optimal policy (and worst-case distribution) in closed form by combining ideas from convex analysis, probability, and dynamic programming. We prove that, at optimality, the worst-case demand distribution corresponds to the setting in which inventory may become obsolete at a random time. To gain further insight, we prove weak convergence (as the time horizon grows large) to a simple and intuitive process. We also compare with the analogous setting in which demand is independent across periods (analyzed previously by Shapiro) and identify several differences between these models in the spirit of the price of correlations studied by Agrawal et al. Finally, we complement our results by providing both numerical experiments that illustrate the potential benefits and limitations of our approach as well as additional theoretical analyses exploring what happens when our modeling assumptions do not hold.

Distributionally Robust Markov Decision Processes and Their Connection to Risk Measures

We consider robust Markov decision processes with Borel state and action spaces, unbounded cost, and finite time horizon. Our formulation leads to a Stackelberg game against nature. Under integrability, continuity, and compactness assumptions, we derive a robust cost iteration for a fixed policy of the decision maker and a value iteration for the robust optimization problem. Moreover, we show the existence of deterministic optimal policies for both players. This is in contrast to classical zero-sum games. In case the state space is the real line, we show under some convexity assumptions that the interchange of supremum and infimum is possible with the help of Sion’s minimax theorem. Further, we consider the problem with special ambiguity sets. In particular, we are able to derive some cases where the robust optimization problem coincides with the minimization of a coherent risk measure. In the final section, we discuss two applications: a robust linear-quadratic problem and a robust problem for managing regenerative energy.

Scalable First-Order Methods for Robust MDPs

Robust Markov Decision Processes (MDPs) are a powerful framework for modeling sequential decision making problems with model uncertainty. This paper proposes the first first-order framework for solving robust MDPs. Our algorithm interleaves primal-dual first-order updates with approximate Value Iteration updates. By carefully controlling the tradeoff between the accuracy and cost of Value Iteration updates, we achieve an ergodic convergence rate that is significantly better than classical Value Iteration algorithms in terms of the number of states S and the number of actions A on ellipsoidal and Kullback-Leibler s-rectangular uncertainty sets. In numerical experiments on ellipsoidal uncertainty sets we show that our algorithm is significantly more scalable than state-of-the-art approaches. Our framework is also the first one to solve robust MDPs with s-rectangular KL uncertainty sets.

Markov decision processes with recursive risk measures

In this paper, we consider risk-sensitive Markov Decision Processes (MDPs) with Borel state and action spaces and unbounded cost. We treat both finite and infinite planning horizons. Our optimality criterion is based on the recursive application of static risk measures. This is motivated by recursive utilities in the economic literature. It has been studied before for the entropic risk measure and is extended here to general static risk measures. Under direct assumptions on the model data we derive a Bellman equation and prove the existence of optimal Markov policies. For an infinite planning horizon, the model is shown to be contractive and the optimal policy to be stationary. Our approach unifies results for a number of well-known risk measures. Moreover, we establish a connection to distributionally robust MDPs, which provides a global interpretation of the recursively defined objective function. Monotone models are studied in particular.

Robust Deep Reinforcement Learning for Quadcopter Control

Deep reinforcement learning (RL) has made it possible to solve complex robotics problems using neural networks as function approximators. However, the policies trained on stationary environments suffer in terms of generalization when transferred from one environment to another. In this work, we use Robust Markov Decision Processes (RMDP) to train the drone control policy, which combines ideas from Robust Control and RL. It opts for pessimistic optimization to handle potential gaps between policy transfer from one environment to another. The trained control policy is tested on the task of quadcopter positional control. RL agents were trained in a MuJoCo simulator. During testing, different environment parameters (unseen during the training) were used to validate the robustness of the trained policy for transfer from one environment to another. The robust policy outperformed the standard agents in these environments, suggesting that the added robustness increases generality and can adapt to non-stationary environments. Codes: https://github.com/adipandas/gym_multirotor

Stochastic Energy Management of Electric Bus Charging Stations With Renewable Energy Integration and B2G Capabilities

In this paper, the stochastic energy management of electric bus charging stations (EBCSs) is investigated, where the photovoltaic (PV) with integrated battery energy storage systems (BESS) and bus-to-grid (B2G) capabilities of electric buses (EBs) are included for cost-effective charging of EBs. Also, the day-ahead dynamic prices are derived to mitigate charging impacts on power distribution systems. This problem is formulated as a distributionally robust Markov decision process (DRMDP) with uncertain transition probabilities and costs to address the impacts of random bus loads with inaccurate probability density function estimation. An event-based ambiguity set with combined statistical distance and moment information is developed to achieve minimax-regret criteria for less-conservative and robust solutions. To facilitate practical applications with reduced computational complexity, a heuristic regret function is proposed, based on which the dynamic prices are derived. Case studies based on EB data from St. Albert Transit and IEEE test feeders indicate that the proposed method can minimize EB charging cost with mitigated impacts on power distribution systems.

Stochastic and Distributionally Robust Load Ensemble Control

Demand response (DR) programs aim to engage distributed demand-side resources in providing ancillary services for electric power systems. Previously, aggregated thermostatically controlled loads (TCLs) have been demonstrated as a technically viable and economically valuable provider of such services that can effectively compete with conventional generation resources in reducing load peaks and smoothing demand fluctuations. Yet, to provide these services at scale, a large number of TCLs must be accurately aggregated and operated in sync. This paper describes a Markov Decision Process (MDP) that aggregates and models an ensemble of TCLs. Using the MDP framework, we propose to internalize the exogenous uncertain dynamics of TCLs by means of stochastic and distributionally robust optimization. First, under mild assumptions on the underlying uncertainty, we derive analytical stochastic and distributionally robust control policies for dispatching a given TCL ensemble. Second, we further relax these mild assumptions to allow for a more delicate treatment of uncertainty, which leads to distributionally robust MDP formulations with moment- and Wasserstein-based ambiguity sets that can be efficiently solved numerically. The case study compares the analytical and numerical control policies using a simulated ensemble of 1,000 air conditioners.

Distributionally robust optimization for sequential decision-making

ABSTRACTThe distributionally robust Markov Decision Process (MDP) approach asks for a distributionally robust strategy that achieves the maximal expected total reward under the most adversarial distribution of uncertain parameters. In this paper, we study distributionally robust MDPs where ambiguity sets for the uncertain parameters are of a format that can easily incorporate in its description the generalized moment as well as statistical distance information of the uncertainty. In this way, we generalize existing works on distributionally robust MDPs with generalized-moment-based and statistical-distance-based ambiguity sets to incorporate information from the former class such as moments and dispersions to the latter class that critically depends on empirical observations of the uncertain parameters. We show that under this format of ambiguity sets, the resulting distributionally robust MDP remains tractable under mild technical conditions and a distributionally robust strategy can be constructed by solving a sequence of one-stage convex optimization subproblems through a Bellman type backward induction.

An active-set strategy to solve Markov decision processes with good-deal risk measure

This paper proposes a quasi closed-form solution for the reweighting of transition probabilities in finite state, finite action distributionally robust Markov decision processes with good-deal risk measure. The relation to the expected (risk-neutral) and minimax (worst-case) discounted cumulated cost objectives is discussed, as well as possible methods for the choice of the risk measure parameters. Numerical results illustrate the computational effectiveness of the proposed approach.