Nonstationary Markov Decision Processes Research Articles

On Average Optimality for Non-Stationary Markov Decision Processes in Borel Spaces

This paper studies Borel models of nonstationary Markov decision processes (MDPs) with average criteria. We establish a suitable fixed point theorem, which is used to show the existence of solutions to the average optimality equation (AOE), without using contractions. The existence of optimal policies follows from the obtained solutions to the AOE. Furthermore, we show that versions of the rolling horizon algorithm can be used to produce an optimal policy or an ϵ-optimal policy. Finally, we compare the optimality conditions imposed in this paper with the existing ones in the literature, and demonstrate that they can be satisfied while the previous ones are not. Funding: This research was supported by National Key Research and Development Program of China [2022YFA1004600], National Natural Science Foundation of China [No. 12301170], Guangdong Basic and Applied Research Foundation [2023A1515012644], and the Fundamental Research Funds for the Central Universities, Sun Yat-Sen University [No. 23qnpy39].

Research on Dynamic Subsidy Based on Deep Reinforcement Learning for Non-Stationary Stochastic Demand in Ride-Hailing

The ride-hailing market often experiences significant fluctuations in traffic demand, resulting in supply-demand imbalances. In this regard, the dynamic subsidy strategy is frequently employed by ride-hailing platforms to incentivize drivers to relocate to zones with high demand. However, determining the appropriate amount of subsidy at the appropriate time remains challenging. First, traffic demand exhibits high non-stationarity, characterized by multi-context patterns with time-varying statistical features. Second, high-dimensional state/action spaces contain multiple spatiotemporal dimensions and context patterns. Third, decision-making should satisfy real-time requirements. To address the above challenges, we first construct a Non-Stationary Markov Decision Process (NSMDP) based on the assumption of ride-hailing service systems dynamics. Then, we develop a solution framework for the NSMDP. A change point detection method based on feature-enhanced LSTM within the framework can identify the changepoints and time-varying context patterns of stochastic demand. Moreover, the framework also includes a deterministic policy deep reinforcement learning algorithm to optimize. Finally, through simulated experiments with real-world historical data, we demonstrate the effectiveness of the proposed approach. It performs well in improving the platform’s profits and alleviating supply-demand imbalances under the dynamic subsidy strategy. The results also prove that a scientific dynamic subsidy strategy is particularly effective in the high-demand context pattern with more drastic fluctuations. Additionally, the profitability of dynamic subsidy strategy will increase with the increase of the non-stationary level.

Model-Free Nonstationary Reinforcement Learning: Near-Optimal Regret and Applications in Multiagent Reinforcement Learning and Inventory Control

We consider model-free reinforcement learning (RL) in nonstationary Markov decision processes. Both the reward functions and the state transition functions are allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain variation budgets. We propose Restarted Q-Learning with Upper Confidence Bounds (RestartQ-UCB), the first model-free algorithm for nonstationary RL, and show that it outperforms existing solutions in terms of dynamic regret. Specifically, RestartQ-UCB with Freedman-type bonus terms achieves a dynamic regret bound of [Formula: see text], where S and A are the numbers of states and actions, respectively, [Formula: see text] is the variation budget, H is the number of time steps per episode, and T is the total number of time steps. We further present a parameter-free algorithm named Double-Restart Q-UCB that does not require prior knowledge of the variation budget. We show that our algorithms are nearly optimal by establishing an information-theoretical lower bound of [Formula: see text], the first lower bound in nonstationary RL. Numerical experiments validate the advantages of RestartQ-UCB in terms of both cumulative rewards and computational efficiency. We demonstrate the power of our results in examples of multiagent RL and inventory control across related products. This paper was accepted by Omar Besbes, revenue management and market analytics. Funding: The research of D. Simchi-Levi and R. Zhu was supported by the MIT Data Science Laboratory. The research of W. Mao, K. Zhang, and T. Başar was supported in part by the U.S. Army Research Laboratory (ARL) Cooperative Agreement W911NF-17-2-0196, in part by the Office of Naval Research (ONR) [MURI Grant N00014-16-1-2710], and in part by the Air Force Office of Scientific Research (AFOSR) [Grant FA9550-19-1-0353]. K. Zhang also acknowledges support from U.S. Army Research Laboratory (ARL) [Grant W911NF-24-1-0085]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.02533 .

Solving nonstationary Markov decision processes via contextual decomposition: A military air battle management application

Reinforcement learning for nonstationary problems is a subject of widespread research given that most realistic problems do not exist within static environments. Approaching these problems can require significant effort in feature engineering to provide a learning algorithm with enough useful information about the state space to uncover complex system dynamics. As an alternative for problems with sufficient data describing the nonstationary environment, we propose the contextual decomposition Markov decision process (CDMDP) as a collection of stationary sub-problems intended to approximate nonstationary problem dynamics using a linear combination of value functions. We demonstrate the effectiveness of the CDMDP approach with an application in military air battle management. We use a designed computational experiment and analysis of variance to show that a complex, nonstationary learning problem can be effectively approximated with a small set of stationary sub-problems, and that the CDMDP solution significantly improves solution quality over a baseline approach without the need for additional feature engineering. If a researcher suspects that a complex and continuously varying environment can be approximated by a small number of stationary contexts, the CDMDP framework may save significant computational resources and yield decision policies that are much easier to visualize and implement.

A Risk-Averse Preview-Based Q-Learning Algorithm: Application to Highway Driving of Autonomous Vehicles

A risk-averse preview-based <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -learning planner is presented for navigation of autonomous vehicles (AVs). To this end, the multilane road ahead of a vehicle is represented by a finite-state nonstationary Markov decision process (MDP). A risk assessment unit module is then presented, which leverages the preview information provided by sensors along with a stochastic reachability module to assign reward values to the MDP states and update them as scenarios develop. A sampling-based risk-averse preview-based <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -learning algorithm is finally developed, which generates samples using the preview information and reward function to learn risk-averse optimal planning strategies without actual interaction with the environment. The risk factor is imposed on the objective function to avoid fluctuation of the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> values, which can jeopardize the vehicle’s safety and/or performance. The overall hybrid automaton model of the system is leveraged to develop a feasibility check unit module that detects unfeasible plans and enables the planner system to react proactively to the changes of the environment. Finally, to verify the efficiency of the presented algorithm, its implementation on two highway driving scenarios of an AV in a varying traffic density is considered.

Non-stationary Risk-Sensitive Reinforcement Learning: Near-Optimal Dynamic Regret, Adaptive Detection, and Separation Design

We study risk-sensitive reinforcement learning (RL) based on an entropic risk measure in episodic non-stationary Markov decision processes (MDPs). Both the reward functions and the state transition kernels are unknown and allowed to vary arbitrarily over time with a budget on their cumulative variations. When this variation budget is known a prior, we propose two restart-based algorithms, namely Restart-RSMB and Restart-RSQ, and establish their dynamic regrets. Based on these results, we further present a meta-algorithm that does not require any prior knowledge of the variation budget and can adaptively detect the non-stationarity on the exponential value functions. A dynamic regret lower bound is then established for non-stationary risk-sensitive RL to certify the near-optimality of the proposed algorithms. Our results also show that the risk control and the handling of the non-stationarity can be separately designed in the algorithm if the variation budget is known a prior, while the non-stationary detection mechanism in the adaptive algorithm depends on the risk parameter. This work offers the first non-asymptotic theoretical analyses for the non-stationary risk-sensitive RL in the literature.

Real-Time Pricing Method for Spot Cloud Services with Non-Stationary Excess Capacity

Cloud operators face massive unused excess computing capacity with a stochastic non-stationary nature due to time-varying resource utilization with peaks and troughs. Low-priority spot (pre-emptive) cloud services with real-time pricing have been launched by many cloud operators, which allow them to maximize excess capacity revenue while keeping the right to reclaim capacities when resource scarcity occurs. However, real-time spot pricing with the non-stationarity of excess capacity has two challenges: (1) it faces incomplete peak–trough and pattern shifts in excess capacity, and (2) it suffers time and space inefficiency in optimal spot pricing policy, which needs to search over the large space of history-dependent policies in a non-stationary state. Our objective was to develop a real-time pricing method with a spot pricing scheme to maximize expected cumulative revenue under a non-stationary state. We first formulated the real-time spot pricing problem as a non-stationary Markov decision process. We then developed an improved reinforcement learning algorithm to obtain the optimal solution for real-time pricing problems. Our simulation experiments demonstrate that the profitability of the proposed reinforcement learning algorithm outperforms that of existing solutions. Our study provides both efficient optimization algorithms and valuable insights into cloud operators’ excess capacity management practices.

Structure Learning-Based Task Decomposition for Reinforcement Learning in Non-stationary Environments

Reinforcement learning (RL) agents empowered by deep neural networks have been considered a feasible solution to automate control functions in a cyber-physical system. In this work, we consider an RL-based agent and address the issue of learning via continual interaction with a time-varying dynamic system modeled as a non-stationary Markov decision process (MDP). We view such a non-stationary MDP as a time series of conventional MDPs that can be parameterized by hidden variables. To infer the hidden parameters, we present a task decomposition method that exploits CycleGAN-based structure learning. This method enables the separation of time-variant tasks from a non-stationary MDP, establishing the task decomposition embedding specific to time-varying information. To mitigate the adverse effect due to inherent noises of task embedding, we also leverage continual learning on sequential tasks by adapting the orthogonal gradient descent scheme with a sliding window. Through various experiments, we demonstrate that our approach renders the RL agent adaptable to time-varying dynamic environment conditions, outperforming other methods including state-of-the-art non-stationary MDP algorithms.

Automating Vehicles by Risk-Averse Preview-based Q-Learning Algorithm

A risk-averse preview-based Q-learning planner is presented for navigation of autonomous vehicles. To this end, the multi-lane road ahead of a vehicle is represented by a finite-state non-stationary Markov decision process (MDP). A sampling-based risk-averse preview-based Q-learning algorithm is finally developed that generates samples using the preview information and reward function to learn risk-averse optimal planning strategies without actual interaction with the environment. The risk factor is imposed on the objective function to avoid fluctuation of the Q values, which can jeopardize the vehicle's safety and/or performance. Theoretical results are provided to bound the number of samples required to guarantee ϵ-optimal planning with a high probability. Finally, to verify the efficiency of the presented algorithm, its implementation on highway driving of an autonomous vehicle in a varying traffic density is considered.

Discovery of Optimal Solution Horizons in Non-Stationary Markov Decision Processes with Unbounded Rewards

Infinite-horizon non-stationary Markov decision processes provide a general framework to model many real-life decision-making problems, e.g., planning equipment maintenance. Unfortunately, these problems are notoriously difficult to solve, due to their infinite dimensionality. Often, only the optimality of the initial action is of importance to the decision-maker: once it has been identified, the procedure can be repeated to generate a plan of arbitrary length. The optimal initial action can be identified by finding a time horizon so long that data beyond it has no effect on the initial decision. This horizon is known as a solution horizon and can be discovered by considering a series of truncations of the problem until a stopping rule guaranteeing initial decision optimality is satisfied. We present such a stopping rule for problems with unbounded rewards. Given a candidate policy, the rule uses a mathematical program that searches for other possibly optimal initial actions within the space of feasible truncations. If no better action can be found, the candidate action is deemed optimal. Our rule runs faster than comparable rules and discovers shorter, more efficient solution horizons.

Markov decision processes with dynamic transition probabilities: An analysis of shooting strategies in basketball

In this paper we model basketball plays as episodes from team-specific non-stationary Markov decision processes (MDPs) with shot clock dependent transition probabilities. Bayesian hierarchical models are employed in the modeling and parametrization of the transition probabilities to borrow strength across players and through time. To enable computational feasibility, we combine lineup-specific MDPs into team-average MDPs using a novel transition weighting scheme. Specifically, we derive the dynamics of the team-average process such that the expected transition count for an arbitrary state-pair is equal to the weighted sum of the expected counts of the separate lineup-specific MDPs. We then utilize these non-stationary MDPs in the creation of a basketball play simulator with uncertainty propagated via posterior samples of the model components. After calibration, we simulate seasons both on-policy and under altered policies and explore the net changes in efficiency and production under the alternate policies. Additionally, we discuss the game-theoretic ramifications of testing alternative decision policies.

Stochastic dynamic resource allocation for HIV prevention and treatment: An approximate dynamic programming approach

Abstract Human immunodeficiency virus (HIV) is a key global health concern, with 33 million people living with HIV worldwide and 2.7 million new infections occurring annually. To prevent the spread of this widely prevalent epidemic disease, prevention and treatment intervention strategies urgently need to be implemented. The goal of this study is to propose stochastic dynamic programming (SDP) and approximate dynamic programming (ADP) algorithms that will optimally allocate the limited intervention budget among the HIV disease compartments and determine the best set of interventions that should be applied to each disease compartment, while minimizing the number of HIV-infected and people diagnosed with acquired immune deficiency syndrome (AIDS) as well as related deaths over a multi-year planning horizon. A compartmental model is constructed and formulated as a nonstationary Markov decision process (MDP) in order to capture the progression of the disease among the highest risk group—African American/black men who have sex with men (BMSM). In order to alleviate the computational difficulties arising from the exponentially growing state space in the SDP, we propose ADP algorithms that determine the approximately optimal budget allocation policies over six years. Our results suggest a greater allocation of the limited budget to prevention measures rather than treatment interventions, such as antiretroviral therapy (ART). As opposed to traditional policies that allocate the budget only once at the beginning of the time horizon, the ADP model suggests using a dynamic proportional budget strategy, allocating the budget dynamically over a multi-period planning period as the uncertainty in disease transmission is revealed. Results show that our ADP approach provides significant increases in health benefits and cost savings in HIV prevention and intervention.

Answer set programming for non-stationary Markov decision processes

Non-stationary domains, where unforeseen changes happen, present a challenge for agents to find an optimal policy for a sequential decision making problem. This work investigates a solution to this problem that combines Markov Decision Processes (MDP) and Reinforcement Learning (RL) with Answer Set Programming (ASP) in a method we call ASP(RL). In this method, Answer Set Programming is used to find the possible trajectories of an MDP, from where Reinforcement Learning is applied to learn the optimal policy of the problem. Results show that ASP(RL) is capable of efficiently finding the optimal solution of an MDP representing non-stationary domains.

Acquisition of new technology information for maintenance and replacement policies

In this paper, we propose the first model that considers the option to acquire information on the profitability of a new technology that is not yet available on the market for asset maintenance and replacement decisions. We consider the uncertainty of future asset characteristics by incorporating information acquisition decisions into a non-stationary Markov decision process framework. Using this framework, we optimise asset maintenance and replacement decisions as well as the optimal timing of new technology adoption. Through mathematical analyses, the monotone properties and convexity of the value function and optimal policy are deduced. Deeper numerical analyses highlight the importance of considering the acquisition of information on future technology when formulating a maintenance and replacement policy for the asset. We also deduce a non-intuitive result: an increase in the arrival probability of new technology does not necessarily make the acquisition of additional information more attractive.

Policy iteration for robust nonstationary Markov decision processes

Policy iteration is a well-studied algorithm for solving stationary Markov decision processes (MDPs). It has also been extended to robust stationary MDPs. For robust nonstationary MDPs, however, an “as is” execution of this algorithm is not possible because it would call for an infinite amount of computation in each iteration. We therefore present a policy iteration algorithm for robust nonstationary MDPs, which performs finitely implementable approximate variants of policy evaluation and policy improvement in each iteration. We prove that the sequence of cost-to-go functions produced by this algorithm monotonically converges pointwise to the optimal cost-to-go function; the policies generated converge subsequentially to an optimal policy.

Optimal decision policy for deteriorating systems with on-line monitoring in a variable environment with selectable operations

ABSTRACTWe investigate the optimal decision-making problem for deteriorating systems operating in a variable environment with selectable operations. We assume that information about system deterioration and the surrounding environment are obtained by on-line monitoring. Using this information, the decision-maker selects the operation that will minimise the total expected cost over an infinite horizon. Since there are strong dynamic interactions among a deteriorating system, its surrounding environment, and the operation being executed, we formulate the optimal decision-making problem for this system as a non-stationary Markov decision process with both environment and operation as parameters. The properties of the resulting optimal expected cost function are examined, and the optimal decision policy for operation selection is found to be given by a two-dimenional control limit policy under certain conditions. The optimal control limit policy reduces the total operating costs and enhances the dynamic decision-making process for deteriorating systems operating in a variable environment.

Inverse optimization in countably infinite linear programs

Given the costs and a feasible solution for a linear program, inverse optimization involves finding new costs that are close to the original ones and make the given solution optimal. We develop an inverse optimization framework for countably infinite linear programs using the weighted absolute sum metric. We reformulate this as an infinite-dimensional mathematical program using duality. We propose a convergent algorithm that solves a sequence of finite-dimensional LPs to tackle it. We apply this to non-stationary Markov decision processes.

Inverse Optimization in Countably Infinite Linear Programs

Given the objective coefficients and a feasible solution for a linear program, inverse optimization involves finding a new vector of objective coefficients that (i) is as close as possible to the original vector; and (ii) would make the given feasible solution optimal. This problem is well-studied for finite-dimensional linear programs. We develop a duality-based inverse optimization framework for countably infinite linear programs (CILPs) -- problems that include a countably infinite number of variables and constraints. Using the standard weighted absolute sum metric to quantify distance between cost vectors, we provide conditions under which constraints in the inverse optimization problem can be reformulated as a countably infinite set of linear constraints. We propose a convergent algorithm to solve the resulting infinite-dimensional mathematical program. This algorithm involves solving a sequence of finite-dimensional linear programs. We apply these results to inverse optimization in infinite-horizon non-stationary Markov decision processes.

Maintaining a system subject to uncertain technological evolution

Maintenance decisions can be directly affected by the introduction of a new asset on the market, especially when the new asset technology could increase the expected profit. However new technology has a high degree of uncertainty that must be considered such as, e.g., its appearance time on the market, the expected revenue and the purchase cost. In this way, maintenance optimization can be seen as an investment problem where the repair decision is an option for postponing a replacement decision in order to wait for a potential new asset. Technology investment decisions are usually based primarily on strategic parameters such as current probability and expected future benefits while maintenance decisions are based on “functional” parameters such as deterioration levels of the current system and associated maintenance costs. In this paper, we formulate a new combined mathematical optimization framework for taking into account both maintenance and replacement decisions when the new asset is subject to technological improvement. The decision problem is modelled as a non-stationary Markov decision process. Structural properties of the optimal policy and forecast horizon length are then derived in order to guarantee decision optimality and robustness over the infinite horizon. Finally, the performance of our model is highlighted through numerical examples.

Extreme point characterization of constrained nonstationary infinite-horizon Markov decision processes with finite state space

We study infinite-horizon nonstationary Markov decision processes with discounted cost criterion, finite state space, and side constraints. This problem can equivalently be formulated as a countably infinite linear program (CILP), a linear program with countably infinite number of variables and constraints. We provide a complete algebraic characterization of extreme points of the CILP formulation and illustrate the characterization for special cases. The existence of a K-randomized optimal policy for a problem with K side constraints also follows from this characterization.