Optimal Stationary Policy Research Articles

Scalable reinforcement learning approaches for dynamic pricing in ride-hailing systems

Dynamic pricing is a widely applied strategy by ride-hailing companies, such as Uber and Lyft, to match the trip demand with the availability of drivers. Deciding proper pricing policies is challenging and existing reinforcement learning (RL)-based solutions are restricted in solving small-scale problems. In this study, we contribute to RL-based approaches that can address the dynamic pricing problem in real-world-scale ride-hailing systems. We first characterize the dynamic pricing problem with a clear distinction between historical prices and current prices. We then translate our dynamic pricing problem into Markov Decision Process (MDP) and prove the existence of a deterministic stationary optimal policy. Our solutions are based on an off-policy reinforcement learning algorithm called twin-delayed deep determinant policy gradient (TD3) that performs offline learning of the optimal pricing policy using historical data and applies the learned policy to the next time slot, e.g., one week. We enhance TD3 by creating three mechanisms to reduce our model complexity and enhance training effectiveness. Extensive numerical experiments are conducted on both small grid networks (16 zones) and the NYC network (242 zones) to demonstrate the performance of the proposed algorithm. The results show our algorithm can efficiently find the optimal pricing policy for both the small and large networks, and can significantly enhance the platform profit and service efficiency.

Optimal transmission strategy for multiple Markovian fading channels: Existence, structure, and approximation

This paper investigates the optimal transmission strategy for remote state estimation over multiple Markovian fading channels. A smart sensor is used to obtain a local state estimate of a system, and transmits it to a remote estimator. A new transmission strategy is proposed by co-designing the channel allocation and the transmission power control. The co-designing problem is modeled as a constrained Markov decision process (CMDP) to minimize the expected average estimation error covariance subject to the energy constraint over an infinite horizon. The CMDP is then relaxed as an unconstrained Markov decision process (UMDP) using the Lagrange multiplier method. Sufficient conditions for the existence of the optimal stationary policy for the UMDP are established to obtain the optimal transmission strategy. The structure of the optimal transmission power control policy for the UMDP with discounted cost is also elucidated. Taking account of the discrete-continuous hybrid action space, a parameterized deep Q-network (P-DQN) algorithm is employed to obtain an approximate optimal policy for the UMDP. Finally, a moving vehicle example is introduced to illustrate the effectiveness of the developed methods.

Maintenance optimization of a two‐component series system considering masked causes of failure

AbstractMaintenance planning of two‐component systems has been extensively studied in recent decades. In the literature, most studies assume that the failure cause of a two‐component system is self‐announcing. In some real applications, the failure cause is masked, and a diagnosis with professional equipment is needed to reveal the failed component. This study investigates a preventive replacement policy of a two‐component series system considering masked causes of failure. When an unexpected failure occurs, we can carry out a diagnosis to reveal the failed component and replace it subsequently, or we can directly replace the whole system without diagnosis. Meanwhile, when we carry out a preventive replacement on a component, the other component can be replaced opportunistically. We formulate the problem as a semi‐Markov decision process, and prove the existence of the stationary optimal policy. The optimal preventive replacement age thresholds for each component and the corresponding optimal maintenance actions upon each failure are jointly obtained to minimize the long‐term average maintenance cost per time unit. A comprehensive numerical study is provided to illustrate the effectiveness of our proposed model.

Adaptive discounted control for piecewise deterministic Markov processes

The main goal of this paper is to study the adaptive infinite-horizon discounted continuous-time optimal control problem of piecewise deterministic Markov processes (PDMPs) with the control acting continuously on the jump intensity λ and on the transition measure Q of the process. It is assumed that jump parameters (λ and Q), as well the continuous and boundary costs (Cg and Ci respectively), depend on an unknown parameter β⁎. It is shown that the principle of estimation and control holds, that is, the strategy consisting of choosing, at each stage n, an action according to an optimal stationary policy, where the true but unknown parameter β⁎ is replaced by its estimated value βˆn, is asymptotically discount optimal, provided that the sequence of estimators {βˆn} of β⁎ is strongly consistent, that is, βˆn converge to β⁎ almost surely. In the framework of PDMPs, the so-called discrepancy function depends on the derivative along the flow of the value function as well as on some boundary conditions, which brings new challenges in the analysis of this problem.

Optimal Sensor Scheduling Under Intermittent Observations Subject to Network Dynamics

Motivated by various distributed control applications, we consider a linear system with Gaussian noise observed by multiple sensors which transmit measurements over a dynamic lossy network. We characterize the stationary optimal sensor scheduling policy for the finite horizon, discounted, and long-term average cost problems and show that the value iteration algorithm converges to a solution of the average cost problem. We further show that the suboptimal policies provided by the rolling horizon truncation of the value iteration also guarantee stability and provide near-optimal average cost. Lastly, we provide qualitative characterizations of the multidimensional set of measurement loss rates for which the system is stabilizable for a static network, thus extending earlier results on intermittent observations.

A note on the existence of optimal stationary policies for average Markov decision processes with countable states

In many practical stochastic dynamic optimization problems with countable states, the optimal policy possesses certain structural properties. For example, the (s,S) policy in inventory control, the well-known cμ-rule and the recently discovered c/μ-rule (Xia et al. (2022)) in scheduling of queues. A presumption of such results is that an optimal stationary policy exists. There are many research works regarding to the existence of optimal stationary policies of Markov decision processes with countable state spaces (see, e.g., Bertsekas (2012); Hernández-Lerma and Lasserre (1996); Puterman (1994); Sennott (1999)). However, these conditions are usually not easy to verify in such optimization problems. In this paper, we study the optimization of long-run average of continuous-time Markov decision processes with countable state spaces. We provide an intuitive approach to prove the existence of an optimal stationary policy. The approach is simply based on compactness of the policy space, with a special designed metric, and the continuity of the long-run average in the space. Our method is capable to handle cost functions unbounded from both above and below, which makes a complementary contribution to the literature work where the cost function is unbounded from only one side. Examples are provided to illustrate the application of our main results.

Risk-Sensitive Average Optimality for Discrete-Time Markov Decision Processes

In this paper we study the risk-sensitive average optimality for discrete-time Markov decision processes with denumerable states and unbounded costs. We derive the multiplicative Poisson equation under the suitable ergodicity conditions via an approximation method. Moreover, we prove the existence of a unique solution to the risk-sensitive average cost optimality equation and give an equivalent characterization of the set of all optimal stationary policies. Finally, we present the policy iteration algorithm and show its convergence.

A Markov Decision Model for Managing Display-Advertising Campaigns

Problem definition: Managers in ad agencies are responsible for delivering digital ads to viewers on behalf of advertisers, subject to the terms specified in the ad campaigns. They need to develop bidding policies to obtain viewers on an ad exchange and allocate them to the campaigns to maximize the agency’s profits, subject to the goals of the ad campaigns. Academic/practical relevance: Determining a rigorous solution methodology is complicated by uncertainties in the arrival rates of viewers and campaigns, as well as uncertainty in the outcomes of bids on the ad exchange. In practice, ad hoc strategies are often deployed. Our methodology jointly determines optimal bidding and viewer-allocation strategies and obtains insights about the characteristics of the optimal policies. Methodology: New ad campaigns and viewers are treated as Poisson arrivals, and the resulting model is a Markov decision process, where the state of the system is the number of undelivered impressions in queue for each campaign type in each period. We develop solution methods for bid optimization and viewer allocation and perform a sensitivity analysis with respect to the key problem parameters. Results: We solve for the optimal dynamic, state-dependent bidding and allocation policies as a function of the number of ad impressions in queue, for both the finite horizon and steady-state cases. We show that the resulting optimization problems are strictly concave in the decision variables and develop and evaluate a heuristic method that can be applied to large problems. Managerial implications: Numerical analysis of our heuristic solution shows that its errors are generally small and that the optimal dynamic, state-dependent bidding policies obtained by our model are significantly better than optimal static policies. Our proposed approach is managerially attractive because it is easy to implement in practice. We identify the capacity of the impression queue as an important managerial control lever and show that it can be more effective than using higher bids to reduce delay penalties. We quantify potential operational benefits from the consolidation of ad campaigns, as well as merging ad exchanges. Supplemental Material: The online appendix is available at https://doi.org/10.1287/msom.2022.1142 .

Preventive replacement policy of a system considering multiple maintenance actions upon a failure

AbstractAge‐based preventive replacement policy has been extensively studied in the literature. Upon an unexpected failure, most studies assume that only one type of maintenance actions can be executed to restore the system back to work. For example, minimal repair or corrective replacement that bring the system to respective as‐bad‐as‐old and as‐good‐as‐new states. Few studies consider both two types of maintenance actions, while predetermine the structure of the maintenance policy. Modern systems are becoming increasingly complex. Besides minimal repair and corrective replacement, imperfect repair can also be carried out to restore a failed system back to work, which brings it to a state between as‐bad‐as‐old and as‐good‐as‐new. This study investigates an optimal age‐based replacement policy, where minimal repair, corrective replacement, and imperfect repair can be carried out upon an unexpected failure with different maintenance effects. We formulate the problem as a semi‐Markov decision process (SMDP), and prove the existence of the stationary optimal policy. The optimal preventive replacement age threshold and the corresponding optimal maintenance action upon each failure are jointly obtained to minimize the long‐term average maintenance cost per time unit. A numerical study is provided to illustrate the effectiveness of our proposed model.

Transmission power allocation for remote estimation with multi-packet reception capabilities

In this paper we consider the problem of transmission power allocation for remote estimation of a dynamical system in the case where the estimator is able to simultaneously receive packets from multiple interfering sensors, as it is possible e.g. with the latest wireless technologies such as 5G and WiFi. To this end we introduce a general model where packet arrival probabilities are determined based on the received Signal-to-Interference-and-Noise Ratio and with two different receivers design schemes, one implementing standard multi-packet reception technique and one implementing Successive Interference Cancellation decoding algorithm in addition. Then we cast the power allocation problem as an optimization task where the mean error covariance at the remote estimator is minimized, while penalizing the mean transmission power consumption. For the infinite-horizon problem we show the existence of a stationary optimal policy, while for the finite-horizon case we derive some structural properties under the special scenario where the overall system to be estimated can be seen as a set of independent subsystems. Numerical simulations illustrate the improvement given by the proposed receivers, especially when SIC is used, over orthogonal schemes that schedule only one sensor transmission at a time in order to avoid interference.

Sleep, Sense or Transmit: Energy-Age Tradeoff for Status Update With Two-Threshold Optimal Policy

Age-of-Information (AoI), or simply age, which measures the data freshness, is essential for real-time Internet-of-Things (IoT) applications. On the other hand, energy saving is urgently required by many energy-constrained IoT devices. This paper studies the energy-age tradeoff for status update from a sensor to a monitor over an error-prone channel. The sensor can sleep, sense and transmit a new update, or retransmit by considering both sensing energy and transmit energy. An infinite-horizon average cost problem is formulated as a Markov decision process (MDP) with the objective of minimizing the weighted sum of average AoI and average energy consumption. By solving the associated discounted cost problem and analyzing the Markov chain under the optimal policy, we prove that there exists a threshold optimal stationary policy with only two thresholds, i.e., one threshold on the AoI at the transmitter (AoIT) and the other on the AoI at the receiver (AoIR). Moreover, the two thresholds can be efficiently found by a line search. Numerical results show the performance of the optimal policies and the tradeoff curves with different parameters. Comparisons with the conventional policies show that considering sensing energy is of significant impact on the policy design, and introducing sleep mode greatly expands the tradeoff range.

Risk-Sensitive Optimal Execution via a Conditional Value-at-Risk Objective

We consider a liquidation problem in which a risk-averse trader tries to liquidate a fixed quantity of an asset in the presence of market impact and random price fluctuations. The trader encounters a trade-off between the transaction costs incurred due to market impact and the volatility risk of holding the position. Our formulation begins with a continuous-time and infinite horizon variation of the seminal model of Almgren and Chriss (2000), but we define as the objective the conditional value-at-risk (CVaR) of the implementation shortfall, and allow for dynamic (adaptive) trading strategies. In this setting, we are able to derive closed-form expressions for the optimal liquidation strategy and its value function. Our results yield a number of important practical insights. We are able to quantify the benefit of adaptive policies over optimized static policies. The relevant improvement depends only on the level of risk aversion: for moderate levels of risk aversion, the optimal dynamic policy outperforms the optimal static policy by 5-15%, and outperforms the optimal volume weighted average price (VWAP) policy by 15-25%. This improvement is achieved through dynamic policies that exhibit "aggressiveness-in-the-money": trading is accelerated when price movements are favorable, and is slowed when price movements are unfavorable. From a mathematical perspective, our analysis exploits the dual representation of CVaR to convert the problem to a continuous-time, zero-sum game. We leverage the idea of the state-space augmentation, and obtain a partial differential equation describing the optimal value function, which is separable and a special instance of the Emden-Fowler equation. This leads to a closed-form solution. As our problem is a special case of a linear-quadratic-Gaussian control problem with a CVaR objective, these results may be interesting in broader settings.

Asymptotic Optimality and Rates of Convergence of Quantized Stationary Policies in Continuous‐Time Markov Decision Processes

This paper is concerned with the asymptotic optimality of quantized stationary policies for continuous‐time Markov decision processes (CTMDPs) in Polish spaces with state‐dependent discount factors, where the transition rates and reward rates are allowed to be unbounded. Using the dynamic programming approach, we first establish the discounted optimal equation and the existence of its solutions. Then, we obtain the existence of optimal deterministic stationary policies under suitable conditions by more concise proofs. Furthermore, we discretize and incentivize the action space and construct a sequence of quantizer policies, which is the approximation of the optimal stationary policies of the CTMDPs, and get the approximation result and the rates of convergence on the expected discounted rewards of the quantized stationary policies. Also, we give an iteration algorithm on the approximate optimal policies. Finally, we give an example to illustrate the asymptotic optimality.

Managing a Hybrid RDC‐DC Inventory System

In this study, we study a hybrid RDC‐DC serial inventory system where the regional distribution center (RDC) replenishes its stock from an outside supplier (OS), while the distribution center (DC) faces random demand and replenishes its stock from the RDC. Unlike in the traditional serial system, the DC itself can replenish its inventory from outside as well. We firstly derive structural properties for the optimal long‐run average cost and the optimal stationary policy by vanishing discount approach, and then propose two simple and easy‐to‐implement policies. The first policy, which we call the three‐index policy, combines the characteristics of the echelon‐base‐stock policy for the serial system (Clark and Scarf. 1960. Management Sci.6(4): 475–490) and the dual‐index policy for the dual‐sourcing system (Veeraraghavan and Scheller‐Wolf. 2008. Oper. Res.56(4): 850–864). We show that the order‐up‐to level of the DC from the RDC can be computed by a newsboy fractile. A simulation‐based optimization procedure for the policy is provided. We then develop the approximate linear programming ( ALP) policy based on the three‐index policy and the multimodularity of the problem. This policy applies the linear programming approach to approximately solve the value function of the dynamic programming formulation. Numerical results show that both the three‐index policy and the ALP policy are comparable to the optimal policy computed via dynamic programming, and the latter performs slightly better. Moreover, the OS of the DC can draw considerable cost savings under both policies. We also conduct a numerical study with problem parameters calibrated using actual data from a consumer goods company in China to glen insights on the management of the system.

Risk-sensitive average continuous-time Markov decision processes with unbounded transition and cost rates

AbstractThis paper considers risk-sensitive average optimization for denumerable continuous-time Markov decision processes (CTMDPs), in which the transition and cost rates are allowed to be unbounded, and the policies can be randomized history dependent. We first derive the multiplicative dynamic programming principle and some new facts for risk-sensitive finite-horizon CTMDPs. Then, we establish the existence and uniqueness of a solution to the risk-sensitive average optimality equation (RS-AOE) through the results for risk-sensitive finite-horizon CTMDPs developed here, and also prove the existence of an optimal stationary policy via the RS-AOE. Furthermore, for the case of finite actions available at each state, we construct a sequence of models of finite-state CTMDPs with optimal stationary policies which can be obtained by a policy iteration algorithm in a finite number of iterations, and prove that an average optimal policy for the case of infinitely countable states can be approximated by those of the finite-state models. Finally, we illustrate the conditions and the iteration algorithm with an example.

On gradual-impulse control of continuous-time Markov decision processes with exponential utility

AbstractWe consider a gradual-impulse control problem of continuous-time Markov decision processes, where the system performance is measured by the expectation of the exponential utility of the total cost. We show, under natural conditions on the system primitives, the existence of a deterministic stationary optimal policy out of a more general class of policies that allow multiple simultaneous impulses, randomized selection of impulses with random effects, and accumulation of jumps. After characterizing the value function using the optimality equation, we reduce the gradual-impulse control problem to an equivalent simple discrete-time Markov decision process, whose action space is the union of the sets of gradual and impulsive actions.

Joint optimization of preventive maintenance and production scheduling for multi-state production systems based on reinforcement learning

Preventive maintenance and production scheduling are two important and interactive activities in production systems. In this work, the integrated optimization problem of production scheduling for multi-state single-machine production systems experiencing degradation processes is investigated. Preventive maintenance tasks and jobs scheduling are jointly considered to find the optimal production policy by considering the processing costs, the maintenance costs, and the completion rewards, simultaneously. We formulate the integrated optimization problem as Markov decision process framework. R-learning algorithm is introduced to maximize the long-run expected average rewards per time unit over infinite horizon. On the basis of the analysis of the optimal stationary policy, the appropriate condition to perform preventive maintenance following optimal stationary policy is presented. This provides the basis for the improvement in R-learning algorithm. Furthermore, a novel heuristic reinforcement learning method is proposed to deal with the integrated model more efficiently. Finally, we present the simulation results and analysis of the proposed algorithm's performance in terms of the number of job types and machine states. The simulation results and analysis show the effectiveness of the proposed approach for solving the integrated problems.

Integro-differential optimality equations for the risk-sensitive control of piecewise deterministic Markov processes

In this paper we study the minimization problem of the infinite-horizon expected exponential utility total cost for continuous-time piecewise deterministic Markov processes with the control acting continuously on the jump intensity $$\lambda $$ and on the transition measure Q of the process. The action space is supposed to depend on the state variable and the state space is considered to have a frontier such that the process jumps whenever it touches this boundary. We characterize the optimal value function as the minimal solution of an integro-differential optimality equation satisfying some boundary conditions, as well as the existence of a deterministic stationary optimal policy. These results are obtained by using the so-called policy iteration algorithm, under some continuity and compactness assumptions on the parameters of the problem, as well as some non-explosive conditions for the process.

Discounted Markov Decision Processes with Constrained Costs: the decomposition approach

In this paper we consider a constrained optimization of discrete time Markov Decision Processes (MDPs) with finite state and action spaces, which accumulate both a reward and costs at each decision epoch. We will study the problem of finding a policy that maximizes the expected total discounted reward subject to the constraints that the expected total discounted costs are not greater than given values. Thus, we will investigate the decomposition method of the state space into the strongly communicating classes for computing an optimal or a nearly optimal stationary policy. The discounted criterion has many applications in several areas such that the Forest Management, the Management of Energy Consumption, the finance, the Communication System (Mobile Networks) and the artificial intelligence.

Asymptotic Evaluations of the Stability Index for a Markov Control Process with the Expected Total Discounted Reward Criterion

In this work, for a control consumption-investment process with the discounted reward optimization criteria, a numerical estimate of the stability index is made. Using explicit formulas for the optimal stationary policies and for the value functions, the stability index is explicitly calculated and through statistical techniques its asymptotic behavior is investigated (using numerical experiments) when the discount coefficient approaches 1. The results obtained define the conditions under which an approximate optimal stationary policy can be used to control the original process.