Threshold-type Policy Research Articles

Age-Optimal Scheduling Over Hybrid Channels

We consider the problem of minimizing the age of information when a source can transmit status updates over two heterogeneous channels. Our work is motivated by recent developments in 5 G mmWave technology, where transmissions may occur over an unreliable but fast (e.g., mmWave) channel or a slow reliable (e.g., sub-6 GHz) channel. The unreliable channel is modeled as a time-correlated Gilbert-Elliot channel at a high rate when the channel is in the “ON” state. The reliable channel provides a deterministic but lower data rate. The scheduling strategy determines the channel to be used for transmission in each time slot, aiming to minimize the time-average age of information (AoI). The optimal scheduling problem is formulated as a Markov Decision Process (MDP), which is challenging to solve because super-modularity does not hold in a part of the state space. We address this challenge and show that a multi-dimensional threshold-type scheduling policy is optimal for minimizing the age. By exploiting the structure of the MDP and analyzing the discrete time Markov chains (DTMCs) of the threshold-type policy, we devise a low-complexity bisection algorithm to compute the optimal thresholds. We compare different scheduling policies using numerical simulations.

Managing the Personalized Order-Holding Problem in Online Retailing

Problem definition: A significant percentage of online consumers place consecutive orders within a short duration. To reduce the total order arrangement cost, an online retailer may consolidate consecutive orders from the same consumer. We investigate how long the retailer should hold the consumer’s orders before sending them to a third-party logistics provider (3PL) for processing. In this order-holding problem, we optimize the holding time to balance the total order arrangement cost and the potential delay in delivery. Methodology/results: We model the order-holding problem as a Markov decision process. We show that the optimal order-holding decisions follow a threshold-type policy that is straightforward to implement: Hold any pending orders if the holding time is within a threshold or send them to the 3PL otherwise. Whenever the consumer places a new order, the holding time is reset, and the threshold is updated based on a cumulative set of the past consecutive orders in the consumer’s shopping journey. Using a consumer’s sequential decision model, we personalize the threshold by finding its closed-form expression in the consumer’s order features. We determine the model’s coefficients and evaluate the threshold-type policy using the data of the 2020 MSOM Data Driven Research Challenge. Extensive numerical experiments suggest that the personalized threshold-type policy outperforms two commonly used benchmarks by having fewer order arrangements or shorter holding times. Furthermore, personalizing the order-holding decisions is significantly more valuable for “enterprise” customers. Managerial implications: Our research suggests a higher threshold for consumers who are more likely to place consecutive orders within a short duration. The consumers’ demographic information has a significant effect on the threshold. Specifically, the threshold is higher for “plus” consumers, female consumers, and consumers in the age group of 16–25 years. The threshold for tier 1 cities is lower than that for tier 2 to tier 4 cities but higher than that for tier 5 cities.History: This paper has been accepted as part of the 2020 MSOM Data Driven Challenge.Funding: This work was supported by the National Natural Science Foundation of China [Grants 71931009, 72201237, and 72231009], the Research Grants Council of Hong Kong [Grants 15501920 and 15501221], the Singapore Ministry of Education Academic Research Fund [Tier 1, Grant RG17/21; Tier 2, Grant MOE2019-T2-1-045], the Association of South-East Asian Nations Business Research Initiative Grant [Grant G17C20421], and the Neptune Orient Lines [Fellowship NOL21RP04].Supplemental Material: The online appendices are available at https://doi.org/10.1287/msom.2023.1201 .

Optimality of a Network Monitoring Agent and Validation in a Real Probe

The evolution of commodity hardware makes it possible to use this type of equipment to implement traffic monitoring systems. A preliminary empirical evaluation of a network traffic probe based on Linux indicates that the system performance has significant losses as the network rate increases. To assess this issue, we consider a model with two tandem queues and a moving server. In this system, we formulate a three-dimensional Markov Decision Process in continuous time. The goal of the proposed model is to determine the position of the server in each time slot so as to optimize the system performance which is measured in terms of throughput. We first formulate an equivalent discrete-time Markov Decision Process and we propose a numerical method to characterize the solution of our problem in a general setting. The solution we obtain in this problem has been tested for a wide range of scenarios and, in all the instances, we observe that the optimality is close to a threshold type policy. We also consider a real probe and we validate the good performance of threshold policies in real applications.

A signature-based approach for reliability modeling and maintenance optimization of a coherent system

In reliability context, mean residual lifetime (MRL) models and proportional hazard (PH) models are widely used for reliability analysis and failure prediction. Since maintenance actions typically are based on aforementioned reliability metrics, recent researches have been focused on the extensions of these two models for accurate decision-making. In this paper, using signature arguments, we propose a state-dependent mean residual life (SDMRL) model and demonstrate its applicability in both reliability and maintenance field. In preference to existing models, the SDMRL modeling can be applied to coherent systems and account both changes in the operating environment and for the natural ageing process of the system. Assuming a bivariate threshold-type policy, the paper aims at minimizing the long-run average maintenance cost per unit time by determining appropriate inspection intervals and optimal replacement policy. The proposed model and the behavior of the optimal solutions as the model parameters vary are illustrated through numerical examples.

An Optimal Flow Admission and Routing Control Policy for Resource Constrained Networks.

Overloaded network devices are becoming an increasing problem especially in resource limited networks with the continuous and rapid increase of wireless devices and the huge volume of data generated. Admission and routing control policy at a network device can be used to balance the goals of maximizing throughput and ensuring sufficient resources for high priority flows. In this paper we formulate the admission and routing control problem of two types of flows where one has a higher priority than the other as a Markov decision problem. We characterize the optimal admission and routing policy, and show that it is a state-dependent threshold type policy. Furthermore, we conduct extensive numerical experiments to gain more insight into the behavior of the optimal policy under different systems’ parameters. While dynamic programming can be used to solve such problems, the large size of the state space makes it untractable and too resource intensive to run on wireless devices. Therefore, we propose a fast heuristic that exploits the structure of the optimal policy. We empirically show that the heuristic performs very well with an average reward deviation of 1.4% from the optimal while being orders of magnitude faster than the optimal policy. We further generalize the heuristic for the general case of a system with n () types of flows.

Improved Crop Productivity Through Optimized Planting Schedules

Problem definition: We study the optimal seeding policy under rainfall uncertainty in rain-fed agriculture and explore its advantage over commonly used heuristics in practice. Academic/practical relevance: Our work is in the area of agriculture operations, and we focus on the improvement of farmer’s expected total profit by optimizing planting schedules. Methodology: We model a farmer’s planting problem under limited planting capacity in a finite-horizon stochastic dynamic program. Results: We show that the optimal planting policy is a time-dependent, threshold-type policy, and the optimal threshold is dependent on the soil water content and planting capacity. In our computational study, we use the well-known Decision Support System for Agrotechnology Transfer simulator used in agriculture to obtain the expected yield when planting in any given period. Utilizing field weather data from Southern Africa, in a real-size, large-scale problem, we demonstrate a significant relative profit advantage of the optimal planting schedule over commonly used heuristics in practice. We show that the relative advantage of the optimal policy increases as the climate condition becomes more severe for planting. We also develop a heuristic based on the secretary problem and demonstrate the increased efficacy of the secretary heuristic. Managerial implications: We show that the farmer should plant down to the optimal threshold of seed amount. However, in practice, farmers start to plant each year after observing enough cumulative rainfall. Utilizing field weather data, in a real-size, large-scale problem, we show significant improvement of the expected total profit if the farmer could adopt the optimal policy.

Service Rate and Admission Control in an M/G/1 Queue with State-Dependent Arrival Rates

Admission and service rate control problems in queueing systems have been studied in the literature. For an exponential service time distribution, the optimality of the threshold-type policy has been proved. However, in production systems, the production time follows a general distribution, not an exponential one. In this paper, control of the service speed according to the number of customers is considered. The analytical results of an M/G/1 queue with arrival and service rates that depend on the number of customers in the system, which is called an Mn/Gn/1 queue, are used to compute the performance measure of service rate control. In particular, for the case in which the arrival rates are the same among queue-length intervals, a computation method for deriving stationary distributions is developed. Constant, uniform, exponential, and Bernoulli distributions on the service time are considered via numerical experiments. The results show that the optimal threshold depends on the type of distribution, even if the mean value of the service time is the same. In addition, when the reward rate is small, a case in which a non-threshold-type service rate control policy outperforms all threshold-type policies is identified.

Optimal energy-efficient policies for data centers through sensitivity-based optimization

In this paper, we propose a novel dynamic decision method by applying the sensitivity-based optimization theory to find the optimal energy-efficient policy of a data center with two groups of heterogeneous servers. Servers in Group 1 always work at high energy consumption, while servers in Group 2 may either work at high energy consumption or sleep at low energy consumption. An energy-efficient control policy determines the switch between work and sleep states of servers in Group 2 in a dynamic way. Since servers in Group 1 are always working with high priority to jobs, a transfer rule is proposed to migrate the jobs in Group 2 to idle servers in Group 1. To find the optimal energy-efficient policy, we set up a policy-based Poisson equation, and provide explicit expressions for its unique solution of performance potentials by means of the RG-factorization. Based on this, we characterize monotonicity and optimality of the long-run average profit with respect to the policies under different service prices. We prove that the bang-bang control is always optimal for this optimization problem, i.e., we should either keep all servers sleep or turn on the servers such that the number of working servers equals that of waiting jobs in Group 2. As an easy adoption of policy forms, we further study the threshold-type policy and obtain a necessary condition of the optimal threshold policy. We hope the methodology and results derived in this paper can shed light to the study of more general energy-efficient data centers.

A new approach to maintenance optimisation of repairable parallel systems subject to hidden failures

Maintenance policies are developed for decision-making about repair and maintenance of deteriorating parallel systems consisting identical components whose failures are detected only by inspections. Inspections at periodic times reveal the true state of components and preventive and corrective maintenance actions are carried out in response to the observed components state. The decision process is driven by the excursion of a state process regulated through an age reduction model. The modelling approach allows a wide class of models to be considered. Assuming a threshold-type policy, the article aims at minimising the long-run average maintenance cost per unit time by determining appropriate inspection intervals and a maintenance threshold. Using the renewal-reward theorem, the expected cost per cycle and expected cycle length emerge as solutions of equations, and a recursive scheme is devised to solve them. We illustrate the procedure for the case when the components’ lifetime conforms to the Weibull distribution. Furthermore, a sensitivity analysis is performed to examine the effect of model’s parameters. The unified structure developed allows different scenarios to be explored.

A Threshold Type Policy for Trading a Mean-Reverting Asset with Fixed Transaction Costs

A mean-reverting model is often used to capture asset price movements fluctuating around its equilibrium. A common strategy trading such mean-reverting asset is to buy low and sell high. However, determining these key levels in practice is extremely challenging. In this paper, we study the optimal trading of such mean-reverting asset with a fixed transaction (commission and slippage) cost. In particular, we focus on a threshold type policy and develop a method that is easy to implement in practice. We formulate the optimal trading problem in terms of a sequence of optimal stopping times. We follow a dynamic programming approach and obtain the value functions by solving the associated HJB equations. The optimal threshold levels can be found by solving a set of quasi-algebraic equations. In addition, a verification theorem is provided together with sufficient conditions. Finally, a numerical example is given to illustrate our results. We note that a complete treatment of this problem was done recently by Leung and associates. Nevertheless, our work was done independently and focuses more on developing necessary optimality conditions.

Optimal lateral transshipment policies for a two location inventory problem with multiple demand classes

We consider an inventory model for spare parts with two stockpoints, providing repairable parts for a critical component of advanced technical systems. As downtime costs for these systems are expensive, ready–for–use spare parts are kept in stock to be able to quickly respond to a breakdown of a system. We allow for lateral transshipments of parts between the stockpoints upon a demand arrival. Each stockpoint faces demands from multiple demand classes. We are interested in the optimal lateral transshipment policy. There are three ways in which a demand can by satisfied: from own stock, via a lateral transshipment, or via an emergency procedure. Using stochastic dynamic programming, we characterize and prove the structure of the optimal policy, that is, the policy for satisfying the demands which minimizes the average operating costs of the system. This optimal policy is a threshold type policy, with state-dependent thresholds at each stockpoint for every demand class. We show a partial ordering in these thresholds in the demand classes. In addition, we derive conditions under which the so-called hold back and complete pooling policies are optimal, two policies that are often assumed in the literature. Furthermore, we study several model extensions which fit in the same modeling framework.

A probabilistic model for maintaining and optimising the life-cycle performance of deteriorating structures

Benefiting from a devised imperfect repair model and given a cost structure, this paper addresses the problem of determining an optimal inspection and threshold-type repair policy for systems whose performance is described by a Wiener process. The system is monitored at periodic times and preventive maintenance actions are carried out in response to the observed system state. The approach can deal with failures defined by performance or regulations. Precisely speaking, failure is defined by a critical set such that the first entry of the performance process to the critical set implies system failure. This approach is typically appropriate for lifecycle models as a specific performance requirement is, or is close to being violated. Since there is a random amount of maintenance, and on the other hand each maintenance incurs a cost, using the renewal-reward theorem, this paper aims at joint determination of an optimal inspection and repair policy providing a right balance between the amount of maintenance and the increasing cost. Benefiting from an imperfect repair model, the presented probabilistic model provides a framework for further developments.

Stochastic inventory system with lead time flexibility: offered by a manufacturer/transporter

This paper studies lead time flexibility in a two-stage continuous review supply chain in which the retailer uses the (R, Q) inventory system: when his inventory position reaches R, the retailer places orders with size Q to the manufacturer, who uses a transportation provider to deliver them with different lead time options. According to the contract, the manufacturer is able to expedite or postpone the delivery if the retailer makes such a request. Hence, the retailer has the flexibility to modify the lead time by using the most up-to-date demand information. The optimal lead time policy is found to be a threshold-type policy. The sensitivity analysis also shows that R is much more sensitive to the change of lead time than Q, and thus, the paper is primarily focused on finding optimal R. We also provide a cost approximation which yields unimodal cost in R. Furthermore, we analyze the order crossing problem and derive an upper bound for the probability of order crossing. Finally, we conduct an extensive sensitivity analysis to illustrate the effects of lead time flexibility on supply chain performance and discuss the managerial insights.

Linking product quality and customer behavior for performance analysis and optimization of make-to-order manufacturing systems

A queueing network model is developed for understanding the way product quality may affect the profitability of production systems, when the consumers base their future demand patterns on the quality of the products they have recently purchased. We examine a multistage make-to-order system which receives orders from regular and occasional customers, the former having a higher mean demand rate. Each outgoing item undergoes inspection and quality grading to decide whether it will be discarded as nonconforming or shipped to a customer. In the latter case, the customer who purchases the item will subsequently become a regular or occasional customer with complementary probabilities which depend on the quality level of that item. The solutions of simple test cases with dynamic programming show that the optimal policy is state-dependent, complex, and computationally intensive. A much simpler, threshold-type policy is proposed, whose performance evaluation and optimization uses closed queueing network formulas and has minimal computational requirements. Numerical results indicate that the proposed policy performs almost as well as the optimal policy.

Optimal Control of State-Dependent Service Rates in a MAP/M/1 Queue

In this paper, we study the optimal control of service rates in a queueing system with a Markovian arrival process (MAP) and exponential service times. The service rate is allowed to be state dependent, i.e., we can adjust the service rate according to the queue length and the phase of the MAP. The cost function consists of holding cost and operating cost. The goal is to find the optimal service rates that minimize the long-run average total cost. To achieve that, we use the matrix-analytic methods (MAM) together with the sensitivity-based optimization (SBO) theory. A performance difference formula is derived, which can quantify the difference of the long-run average total cost under any two different settings of service rates. Based on the difference formula, we show that the long-run average total cost is monotone in the service rate and the optimal control is a bang–bang control. We also show that, under some mild conditions, the optimal control policy of service rates is of a quasi-threshold-type. By utilizing the MAM theory, we propose a recursive algorithm to compute the value function related quantities. An iterative algorithm to efficiently find the optimal policy, which is similar to policy iteration, is proposed based on the SBO theory. We further study some special cases of the problem, such as the optimality of the threshold-type policy for the M/M/1 queue. Finally, a number of numerical examples are presented to demonstrate the main results and explore the impact of the phase of the MAP on the optimization in the MAP/M/1 queue.

Dynamic control of a closed two-stage queueing network for outfitting process in shipbuilding

The U.S. Naval shipbuilding industry faces significant challenges to build ships on-time and within budgeted cost. To achieve greater efficiency and timeliness in shipbuilding, we developed a flexible two-stage queueing model under a CONWIP job release policy to enhance the planning and control of the outfitting process, one of the key processes in shipbuilding. The model is formulated using Markov Decision Processes which can provide (1) the optimal dynamic control policy and (2) the optimal cost. The numerical results showed that the optimal control policy is a state dependent threshold type policy and very complex to analyze. Therefore, we developed a static model to simplify the dynamic model and used Mean Value Analysis to gain insights. Using both data from the dynamic model and the static model, we developed a regression model to calculate a threshold policy heuristic. Testing reveals that the performance of this heuristic is very close to the optimal.

Flexible Capacity Management with Future Information

We consider a maintenance service provider that overhauls aircraft engines in a central service facility. The service provider can choose between a low and a high capacity. However, the high capacity can only be used for a certain share of the time. We model the service facility as a queue with adaptable service rate and develop capacity control policies minimizing the time-average queue length. The reactive threshold-type policy only considers the current queue length and we show that the time-average queue length diverges as the load of the system increases. State-of-the-art aircraft engines are equipped with sensors permitting prediction of future maintenance needs. Thus, the solely forward-looking policy is based on predicted arrival times and maintenance requirements of engines arriving to the system in the future. We show that the time-average queue length converges to a finite value as the load increases. As the reactive policy outperforms the solely forward-looking policy for low arrival rates, we combine both to a proactive policy. While we assume that future information is available within an infinite lookahead window when stating the policy, we derive conditions under which a proactive capacity control policy can be employed and how the policy has to be modified if the lookahead window is finite.

Scheduling Preventive Maintenance for a Nonperiodically Inspected Deteriorating System

We consider the problem of scheduling maintenance activities for a stochastically deteriorating manufacturing system inspected aperiodically. The state of the system (which can only be revealed at inspection epochs) is characterized jointly by the system’s physical state (the level of damage or remaining capacity) and its virtual age. Assuming a threshold-type policy, the paper aims at minimizing the long-run average maintenance cost per unit time by determining appropriate inspection intervals and maintenance thresholds. Using the renewal reward theorem, the expected cost per cycle and expected cycle length emerge as solutions of an integral equation, and a recursive scheme is devised to solve them. We illustrate the procedure for the case when the physical state evolves as a Gamma process.

Dynamic replenishment from two sources with different yields, costs, and leadtimes

We study a single-item periodic-review inventory system with two sources of supply: in-house production and outsourcing. The two sources differ in yields, costs, and leadtimes. Demands in consecutive periods are independent random variables and stockout is fully backlogged. Our problem is to characterize the optimal dynamic policy that simultaneously determines the in-house production quantity and the outsourcing quantity for each period to minimize the total discounted cost. We show that the optimal policy for in-house production can be characterized by a threshold and the production quantity is decreasing in the starting inventory. The optimal outsourcing policy, however, is more complex, and a threshold-type policy is only optimal under a more restrictive assumption about the yields. Comparative statics about the effects of yield uncertainty, outsourcing on the optimal policy, and cost performance are discussed.

A Jackson network model and threshold policy for joint optimization of energy and delay in multi-hop wireless networks

Abstract This paper studies the joint optimization problem of energy and delay in a multi-hop wireless network. The optimization variables are the transmission rates, which are adjustable according to the packet queueing length in the buffer. The optimization goal is to minimize the energy consumption of energy-critical nodes and the packet transmission delay throughout the network. In this paper, we aim at understanding the well-known decentralized algorithms which are threshold based from a different research angle. By using a simplified network model, we show that we can adopt the semi-open Jackson network model and study this optimization problem in closed form. This simplified network model further allows us to establish some significant optimality properties. We prove that the system performance is monotonic with respect to (w.r.t.) the transmission rate. We also prove that the threshold-type policy is optimal, i.e., when the number of packets in the buffer is larger than a threshold, transmit with the maximal rate (power); otherwise, no transmission. With these optimality properties, we develop a heuristic algorithm to iteratively find the optimal threshold. Finally, we conduct some simulation experiments to demonstrate the main idea of this paper.