Optimal Dynamic Policy Research Articles

Optimal Control of Service Systems with Heterogeneous Servers and Priority Customers

We study service systems with parallel servers and random customer arrivals and focus on the waiting cost of customers. Using a Markov decision process (MDP) modeling approach, we analytically characterize the structures of the optimal dynamic server assignment policies for two important systems, one consisting of multiple homogeneous servers and two classes of customers and the other consisting of two heterogeneous servers and multiple classes of customers. Based on the obtained results, we propose a threshold-type heuristic policy for the generalized system consisting of multiple heterogeneous servers and multiple classes of customers. To design such a heuristic policy, we first develop techniques for the performance evaluation of general threshold-type policies with any given threshold values. We then construct a path to search for the optimal threshold values. We compare the performance of the best threshold-type heuristic policy with that of the optimal policy and show that our proposed heuristic policy is computationally efficient yet generates great performance. To derive additional managerial insights, we compare the system under our threshold-type dynamic server assignment policy with other commonly seen and simple systems, such as the dedicated system and the work-conserving flexible priority system. The clear performance advantage observed from extensive numerical experiments demonstrates the importance and usefulness of dynamic server assignment control for systems serving multiple classes of customer arrivals. Finally, we extend our analysis to incorporate customer-dependent service rates and sojourn-time minimization performance metrics. This paper was accepted by Chung Piaw Teo, optimization. Funding: This work was supported by the Hong Kong Research Grants Council [16500921, 16505819] and the National Natural Science Foundation of China [72201233, 72301222]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2023.01228 .

Cooperative multi-agent reinforcement learning for multi-area integrated scheduling in wafer fabs

The existing scheduling methods of wafer fabs focus on single area, achieving local optimisation while failing to realise global optimisation due to neglecting the coordination of multi-area. Therefore, it is necessary to consider the complex opposing relationships between multi-area caused by constraints such as batch processing, re-entrance, and multiple residency times within and between areas to conduct integrated scheduling and shorten the production cycle time. For this issue, this paper proposes a cooperative multi-agent reinforcement learning for multi-area integrated scheduling. Aiming at the dynamic batching and scheduling considering the dynamic arrival lots in multi-area, a multi-agent reinforcement learning algorithm is presented to learn the optimal dynamic batching and scheduling policy firstly. Subsequently, a cooperative multi-agent framework is raised to achieve the global optimisation and coordination of multi-area. Furthermore, an adaptive exploration strategy is constructed to enhance the global exploration capability of the complex solution space caused by residency time constraints and re-entrant property. Moreover, a policy share enhanced Double DQN is employed to improve the generalisation and adaptability of the multi-agent. Finally, the experiments demonstrate that the proposed integrated scheduling method has better comprehensive performance compared to the previous area-separated scheduling methods.

Dynamic resource allocation and scheduling for appointment-based systems with walk-ins

Effectively managing constrained and perishable capacity amid fluctuating demands is a common challenge in service industries, often addressed through appointment-based systems that incorporate walk-ins. Balancing appointments and walk-ins is crucial for enhancing system value and maintaining stability. In this article, we study an integrated decision problem for appointment scheduling and resource allocation across multiple facilities, considering stochastic demands from three segments: priority customers, walk-in customers, and appointment customers. We formulate the problem as a sequential scheduling-allocation challenge using stochastic dynamic programming. Leveraging the anti-multimodularity of the value function, we fully characterize the optimal dynamic resource allocation policy and its monotone structural properties under any appointment schedule. Given the complexity and intractability of optimal scheduling with dynamic resource allocation, we introduce two suboptimal scheduling policies—vertical scheduling and horizontal scheduling—and a heuristic allocation policy. Additionally, we create an Upper Bound (UB) for the original problem as a surrogate, establishing an asymptotically optimal UB for the scaled problem through its objective value and a corresponding lower bound through its solution. This UB solution serves as an efficient and effective heuristic, demonstrating superior performance compared with the combined performance of suboptimal scheduling policies with optimal dynamic resource allocation, especially as problem size increases. It also holds significant practical implications for integrated pooling systems.

What Can Measured Beliefs Tell Us About Monetary Non-Neutrality?

This paper studies how measured beliefs can be used to identify monetary non-neutrality. In a general equilibrium model with both nominal rigidities and endogenous information acquisition, we analytically characterize firms’ optimal dynamic information policies and how their beliefs affect monetary non-neutrality. We then show that data on the cross-sectional distributions of uncertainty and pricing durations are both necessary and sufficient to identify monetary non-neutrality. Finally, implementing our approach in New Zealand survey data, we find that informational frictions approximately double monetary non-neutrality and endogeneity of information is important: models with exogenous information would overstate monetary non-neutrality by approximately 50%.

Curbing the Opioid Crisis: Optimal Dynamic Policies for Preventive and Mitigating Interventions

Problem statement: This paper addresses the challenge of effectively responding to the opioid epidemic stemming from prescription pills through a public health lens. It centers on the strategic distribution of resources across diverse interventions aimed at preventing and mitigating the consequences of opioid use disorder (OUD) and overdose occurrences. Methodology: This paper proposes a decision aid tool built on the expected utility theory that leverages a Susceptible-Infected-Removed compartmental model to simulate the dynamics of the epidemic in a population. This model then feeds into a Markov Decision Process (MDP) model to generate optimal policies upon the current state of the epidemic. The optimal policies allocate the intervention budget to primary preventive and mitigating interventions in each decision period by minimizing the cost of fatal overdoses relative to the population’s number of individuals with OUD, considering the impact magnitude of each intervention, based on the current state of the epidemic. A 10-year simulation of the epidemic’s progression is conducted to assess the dynamic efficacy of the proposed decision tool. Results: The findings reveal an average reduction of 29% in total costs compared to the scenario without interventions and a decrease of 12% in total costs on average compared to the scenario with a 50-50 allocation. The extensive sensitivity analysis of key parameters validates the decision aid tool. We observe that it is optimal to allocate a significant portion of the budget to prevention when the rate of opioid pill acquisition rises. Even with a heightened rate of fatal overdoses, it remains optimal to mostly invest in preventive interventions, as long as fatal overdose rates are lower than opioid access rates. Practical implications: This study provides practitioners with a tool to effectively address the opioid epidemic and enhance public health by deciding how to allocate their budget to various levels of intervention.

Data-driven inventory policy: Learning from sequentially observed non-stationary data

This paper aims to find dynamic inventory policies for retailers that have limited knowledge about future demand and sequentially observe the unprecedented demand data. We assume the demand is non-stationary; it follows different distributions for different time periods, and the data distributions and the transition behavior are unknown. Two solution approaches are presented to tackle this problem. Integrated-Bayesian (IB) approach is a parametric approach and is introduced for the case when an uncertainty set of possible demand distributions is available. A non-parametric approach, separate-lasso (SL), is proposed for the case that the uncertainty set possible demand distributions is not known. Both methods are theoretically analyzed and empirically benchmarked against several state-of-the-art heuristics. The theoretical analyses provide easy-to-implement algorithms for both approaches, while performance guarantees are derived for the separate-lasso approach. Computational studies show that the proposed methods outperform state-of-the-art heuristics–namely, sample average approximation, rolling horizon, and exponential smoothing–in nine different data environments. The optimal dynamic policy is not obtainable in this dynamic setting as reliable demand forecasts are not available. Therefore, we derive an approximated optimal policy, OPT, assuming the complete knowledge of the demand data in advance. The empirical results reveal that the cost of the proposed approaches is only 12% higher than that of OPT on average. Furthermore, we show that the proposed methods capture the hidden patterns inside the highly non-stationary real-world demand data of one of the largest e-commerce websites.

Heuristic Rules for the Dynamic Pricing Problem

This paper is devoted to the development of heuristics for the dynamic pricing problem. A discrete time model of dynamic pricing on the fixed time horizon is proposed. It is applicable to products that satisfy two properties: 1) product value expires at a certain predetermined date, and 2) consumers demand at most a single unit of the product. This type of demand structure allows deriving a simple system of recursive equations for optimal prices using dynamic programming techniques. Optimal pricing policy is expressed as a function of time to expiration and inventory levels of unsold products. An analytical solution to this problem was obtained for special cases, while for the general case, a numerical algorithm has been developed. Qualitative characteristics of the optimal pricing policy are established, and their implications for dynamics of inventories and prices are discussed. Based on these observations, a simple heuristic rule for dynamic price adjustments is proposed. Performance of this heuristic is evaluated against the optimal dynamic and fixed-price policies using Monte-Carlo experiments. Results demonstrate high efficiency of the proposed heuristic strategy and its even simpler derivatives. Heuristics’ adaptability and ease of implementation should make it suitable and attractive for small and medium businesses.

Ordering COVID-19 vaccines for social welfare with information updating: Optimal dynamic order policies and vaccine selection in the digital age

In the digital age, operations can be improved by a wise use of information and technological tools. During the COVID-19 pandemic, governments faced various choices of vaccines possessing different efficacy and availability levels at different time points. In this article, we consider a two-stage vaccine ordering problem of a government from a first and only supplier in the first stage, and either the same supplier or a new second supplier in the second stage. Between the two stages, potential demand information for the vaccine is collected to update the forecast. Using dynamic programming, we derive the government’s optimal vaccine ordering policy. We find that the government should select its vaccine supplier based on the disease’s infection rate in the society. When the infection rate is low, the government should order nothing at the first stage and order from the supplier with a higher efficacy level at the second stage. When the disease’s infection rate is high, the government should order vaccines at the first stage and switch to the other supplier with a lower efficacy level at the second stage. We extend our model to examine (i) the value of blockchain adoption and (ii) the impact of vaccines’ side effects.

Ordering COVID-19 Vaccines for Social Welfare with Information Updating: Optimal Dynamic Order Policies and Vaccine Selection in the Digital Age

Ordering COVID-19 Vaccines for Social Welfare with Information Updating: Optimal Dynamic Order Policies and Vaccine Selection in the Digital Age

Optimal admission and queuing control with reneging behavior under premature discharge decisions

AbstractQueues or waiting lines are a common phenomenon in healthcare facilities. The increasing number of admissions to hospital emergency departments (EDs) has resulted in overcrowded EDs and decreased quality of care. Patients may renege with dissatisfaction while waiting, prolonged hospital stays, or the cancelation of surgeries, especially for those with acute diseases. This paper studies patient admission and queuing control policies, considering premature discharge decisions and reneging behavior. First, we formulate this problem as an infinite‐horizon total discounted cost Markov decision process to balance bed utility and experiences of patients waiting for admission to minimize the total expected cost for the hospital. Then, we characterize an optimal admission threshold policy by analyzing the structural properties of the optimal value function. Numerical experiments are employed to discuss how the optimal dynamic policy depends on some key parameters of the system. Finally, the performance of the optimal policy is discussed through comparison with the benchmark: A myopic rule (first come, first served) mimicking the decision‐making in practice under different scenarios. The numerical results show that the reneging behavior can affect the superiority of the optimal policy compared to the benchmark, and the optimal policy can significantly reduce the number of waiting patients without increasing the total expected cost when the system is overloaded.

Dynamic pricing and inventory control of online retail of fresh agricultural products with forward purchase behavior

In this paper we formulate and analyze a novel model on a retailer’s inventory and pricing decisions for fresh agricultural products with consumers’ forward consuming behavior under online channel. We consider a dynamic stochastic setting, where every period consists of two stages, discounting pricing stage and regular sale stage. At the beginning of the period, the retailer decides how much new fresh agricultural products to order and sets discount price for leftover inventories from the previous period which will be disposed otherwise, and determines regular price for fresh products on the second stage, respectively. Since forward purchase consumers may buy the products during discount pricing stage, which may cannibalize future sales at regular price, the retailer needs to make a trade-off decision between regular price and discounting price. We bring forward a dynamic optimization model and use nonlinear programming method of Karush Kuhn Tucker condition to obtain the optimal dynamic strategy, which is comparatively analyzed to dominate the related static strategy. We also show that consumers forward buying behavior will negatively influence the retailer’s profit. When the price is set too low in regular or discounting sales, the profit will show an up-down trend if the inventory exceeds a certain threshold. Meanwhile, when fresh goods returns are allowed and resold in the secondary stage, the retailer’s profit will increase. We finally conduct numerical studies to further characterize the optimal policy, and to evaluate the loss of efficiency under static policies when compared to the optimal dynamic policy.

Technological Rivalry and Optimal Dynamic Policy in an Open Economy

Technological Rivalry and Optimal Dynamic Policy in an Open Economy

Optimal dynamic pricing and production policy for a stochastic inventory system with perishable products and inventory-level-dependent demand

In this paper, we consider a joint dynamic pricing and production policy for a stochastic inventory system with perishable products. The demand is stochastic and dependent on the price and the level of the on-hand inventory. Combined dynamic pricing and production control, a stochastic dynamic optimization model that maximizes the total discounted profit is built. Applying the stochastic optimal control theory, we formulate the problem of finding the optimal joint dynamic pricing and production schedule as the problem of solving a Hamilton-Jacobi-Bellman (HJB) equation. By solving the HJB equation, analytical solutions for optimal pricing and production rate are obtained. In addition, a numerical example is given to illustrate the validness of the theoretical results and sensitivity analysis on major system parameters is carried out.

A Markov Decision Model for Managing Display-Advertising Campaigns

Problem definition: Managers in ad agencies are responsible for delivering digital ads to viewers on behalf of advertisers, subject to the terms specified in the ad campaigns. They need to develop bidding policies to obtain viewers on an ad exchange and allocate them to the campaigns to maximize the agency’s profits, subject to the goals of the ad campaigns. Academic/practical relevance: Determining a rigorous solution methodology is complicated by uncertainties in the arrival rates of viewers and campaigns, as well as uncertainty in the outcomes of bids on the ad exchange. In practice, ad hoc strategies are often deployed. Our methodology jointly determines optimal bidding and viewer-allocation strategies and obtains insights about the characteristics of the optimal policies. Methodology: New ad campaigns and viewers are treated as Poisson arrivals, and the resulting model is a Markov decision process, where the state of the system is the number of undelivered impressions in queue for each campaign type in each period. We develop solution methods for bid optimization and viewer allocation and perform a sensitivity analysis with respect to the key problem parameters. Results: We solve for the optimal dynamic, state-dependent bidding and allocation policies as a function of the number of ad impressions in queue, for both the finite horizon and steady-state cases. We show that the resulting optimization problems are strictly concave in the decision variables and develop and evaluate a heuristic method that can be applied to large problems. Managerial implications: Numerical analysis of our heuristic solution shows that its errors are generally small and that the optimal dynamic, state-dependent bidding policies obtained by our model are significantly better than optimal static policies. Our proposed approach is managerially attractive because it is easy to implement in practice. We identify the capacity of the impression queue as an important managerial control lever and show that it can be more effective than using higher bids to reduce delay penalties. We quantify potential operational benefits from the consolidation of ad campaigns, as well as merging ad exchanges. Supplemental Material: The online appendix is available at https://doi.org/10.1287/msom.2022.1142 .

Joint Learning and Control in Stochastic Queueing Networks with Unknown Utilities

We study the optimal control problem in stochastic queueing networks with a set of job dispatchers connected to a set of parallel servers with queues. Jobs arrive at the dispatchers and get routed to the servers following some routing policy. The arrival processes of jobs and the service processes of servers are stochastic with unknown arrival rates and service rates. Upon the completion of each job from dispatcher u n at server s m , a random utility whose mean is unknown is obtained. We seek to design a control policy that makes routing decisions at the dispatchers and scheduling decisions at the servers to maximize the total utility obtained by the end of a finite time horizon T . The performance of policies is measured by regret, which is defined as the difference in total expected utility with respect to the optimal dynamic policy that has access to arrival rates, service rates and underlying utilities. We first show that the expected utility of the optimal dynamic policy is upper bounded by T times the solution to a static linear program, where the optimization variables correspond to rates of jobs from dispatchers to servers and the feasibility region is parameterized by arrival rates and service rates. We next propose a policy for the optimal control problem that is an integration of a learning algorithm and a control policy. The learning algorithm seeks to learn the optimal extreme point solution to the static linear program based on the information available in the optimal control problem. The control policy, a mixture of priority-based and Joint-the-Shortest-Queue routing at the dispatchers and priority-based scheduling at the servers, makes decisions based on the graphical structure induced by the extreme point solutions provided by the learning algorithm. We prove that our policy achieves logarithmic regret whereas application of existing techniques to the optimal control problem would lead to Ω(√ T )-regret. The theoretical analysis is further complemented with simulations to evaluate the empirical performance of our policy.

Optimal and Suboptimal Dynamic Cache Update Algorithms for Wireless Cellular Networks

In this letter, we study the dynamic cache update problem for wireless content caching networks. The cost minimization perspective is considered, wherein the system cost contains three components, including the cache cost, the retrieval cost, and the update cost. Particularly, the update cost lies at the root of the coded caching strategy. To address the non-convex optimization problem, we propose both suboptimal and optimal algorithms. More specifically, we prove that with previous caching state, the cost minimization problem for the current time interval is a 0–1 Knapsack problem, which can be optimally solved by a dynamic programming with pseudo polynomial time complexity. On this basis, the original optimization problem is solved by an iterative manner, resulting in an suboptimal solution. Thereafter, we design the optimal dynamic caching policy, which is derived from the recursive algorithm. Simulation result shows that the suboptimal solution achieves near-optimal performance. It also demonstrates the superiority of our developed strategies compared to various baselines in terms of cost saving. Moreover, counterintuitive insight is judiciously acquired, such as that the high cache capacity does not always reveal positive effect on system’s performance when update cost is taken into consideration.

Optimal dynamic multi-keyword bidding policy of an advertiser in search-based advertising

Optimal dynamic multi-keyword bidding policy of an advertiser in search-based advertising

Optimal dynamic mining policy of blockchain selfish mining through sensitivity-based optimization

Optimal dynamic mining policy of blockchain selfish mining through sensitivity-based optimization

Optimal dynamic regulation of carbon emissions market

AbstractIn this article, we deal with optimal dynamic carbon emission regulation of a set of firms. On the one hand, the regulator dynamically allocates emission allowances to each firm. On the other hand, firms face idiosyncratic, as well as common, economic shocks on emissions, and they have linear quadratic abatement costs. Firms can trade allowances so as to minimize total expected costs, which arise from abatement, trading, and terminal penalty. Using variational methods, we first exhibit in closed form the market equilibrium in function of the regulator's dynamic allocation. We then solve the Stackelberg game between the regulator and the firms. The result is a closed‐form expression of the optimal dynamic allocation policies that allow a desired expected emission reduction. The optimal policy is unique in the presence of market impact. In absence of market impact, the optimal policy is nonunique, but all the optimal policies share common properties. The optimal policies are fully responsive, and therefore induce a constant abatement effort and a constant price of allowances. Our results are robust to some extensions, like different penalty functions.

Optimal Dynamic Reinsurance Under Heterogeneous Beliefs and CARA Utility

This paper examines the optimal dynamic reinsurance policy for an insurance company under belief heterogeneity. We assume the reinsurance premium is calculated according to the mean-conditional value-at-risk principle and impose the incentive compatibility constraint to rule out moral hazard. Under the objective of maximizing the exponential utility function, we obtain the optimal strategies in closed form via a “relaxation and modification” approach. The optimal contracts have more complicated structures than the standard proportional and excess-of-loss reinsurance widely investigated in the literature. In particular, we demonstrate that the insurer may optimally choose to purchase proportional reinsurance in different layers when the reinsurer is more pessimistic about the underlying loss. Our model lends support to the observation that reinsurance contracts in practice often involve proportional reinsurance in multiple layers. We also demonstrate that belief heterogeneity can explain the inverse relationship between the purchase of reinsurance and the size of the insurer's losses observed in the reinsurance market.