Heuristic Policies Research Articles

Optimal and near-optimal control of a capacitated assemble-to-order system with component commonality and backordered demands

We investigate the optimal policy of a capacitated assemble-to-order (ATO) system with component commonality and backordered demands. Using a Markov decision process framework, we model the problem with the objective of minimising both the expected total discounted cost and the expected average cost rate over an infinite-horizon. To characterise the structure of the optimal policy, we adopt a two-step approach. First, we demonstrate that the value function satisfies a set of preliminary properties, indicating that it is optimal to allocate the common component’s inventory to the product with a larger backorder penalty, if its demand can be immediately fulfilled. This insight simplifies the Bellman optimality equation, allowing for a lower-dimensional state representation, which we then applied to fully characterise the optimal policy. Building on the structure of the optimal policy, we develop four heuristic policies. Extensive numerical experiments demonstrate that these heuristic policies perform well. In particular, heuristics H2 and H3 exhibit, on average, only 2% and 1% performance loss compared to the optimal policy. Our results offer practical implications for addressing larger ATO systems.

Contextual factors impacting WHO Framework Convention on Tobacco Control implementation in Africa-a scoping review.

According to the World Health Organization (WHO), tobacco use causes over 8 million deaths annually including 1.3 million due to second-hand exposure. Furthermore, data from the Tobacco Atlas show that the tobacco industry continues to target new markets in the WHO African region, one of two regions where absolute numbers of smokers continue to increase. Understanding context contributes to policy formulation and implementation ensuring relevance to a country's political economy. Focusing on the WHO African region, this scoping review (i) maps the extent of academic research examining contextual factors on the WHO Framework Convention on Tobacco Control (WHO FCTC) national-level implementation, and (ii) reports on contextual factors impacting the WHO FCTC implementation. Using a stepwise structured approach, we conducted a search across four academic databases, yielding 10 342 articles and 42 were selected for full data extraction. Leichter's four categories of context (situational, structural, cultural and exogenous) and the stages of heuristic policy model guided data extraction. Study findings indicated that situational contextual factors such as the burden of disease or its impact on health can push governments toward policy formulation. Structural contextual factors included political considerations, economic interests, funding, institutional congruence, strength of policy and institutional capacity as important. Cultural contextual factors included the influence of policy entrepreneurs, current social trends and public opinion. Exogenous contextual factors included the WHO FCTC, tobacco industry influence at the national-level and bi-lateral partnerships. Further understanding contextual factors affecting the WHO FCTC national implementation can strengthen policy formulation and align required support with the WHO FCTC Secretariat and other relevant bodies.

Optimal Control of Service Systems with Heterogeneous Servers and Priority Customers

We study service systems with parallel servers and random customer arrivals and focus on the waiting cost of customers. Using a Markov decision process (MDP) modeling approach, we analytically characterize the structures of the optimal dynamic server assignment policies for two important systems, one consisting of multiple homogeneous servers and two classes of customers and the other consisting of two heterogeneous servers and multiple classes of customers. Based on the obtained results, we propose a threshold-type heuristic policy for the generalized system consisting of multiple heterogeneous servers and multiple classes of customers. To design such a heuristic policy, we first develop techniques for the performance evaluation of general threshold-type policies with any given threshold values. We then construct a path to search for the optimal threshold values. We compare the performance of the best threshold-type heuristic policy with that of the optimal policy and show that our proposed heuristic policy is computationally efficient yet generates great performance. To derive additional managerial insights, we compare the system under our threshold-type dynamic server assignment policy with other commonly seen and simple systems, such as the dedicated system and the work-conserving flexible priority system. The clear performance advantage observed from extensive numerical experiments demonstrates the importance and usefulness of dynamic server assignment control for systems serving multiple classes of customer arrivals. Finally, we extend our analysis to incorporate customer-dependent service rates and sojourn-time minimization performance metrics. This paper was accepted by Chung Piaw Teo, optimization. Funding: This work was supported by the Hong Kong Research Grants Council [16500921, 16505819] and the National Natural Science Foundation of China [72201233, 72301222]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2023.01228 .

Modelling human navigation and decision dynamics in a first-person herding task.

This study investigated whether dynamical perceptual-motor primitives (DPMPs) could also be used to capture human navigation in a first-person herding task. To achieve this aim, human participants played a first-person herding game, in which they were required to corral virtual cows, called targets, into a specified containment zone. In addition to recording and modelling participants' movement trajectories during gameplay, participants' target-selection decisions (i.e. the order in which participants corralled targets) were recorded and modelled. The results revealed that a simple DPMP navigation model could effectively reproduce the movement trajectories of participants and that almost 80% of the participants' target-selection decisions could be captured by a simple heuristic policy. Importantly, when this policy was coupled to the DPMP navigation model, the resulting system could successfully simulate and predict the behavioural dynamics (movement trajectories and target-selection decisions) of participants in novel multi-target contexts. Implications of the findings for understanding complex human perceptual-motor behaviour and the development of artificial agents for robust human-machine interaction are discussed.

Whittle index approach to the multi-class queueing systems with convex holding costs and IHR service times

AbstractWe consider the dynamic scheduling problem in a single-server multi-class queueing system, where the accumulated holding costs of a customer is a convexly increasing function of the time that the customer spends in the system. This problem was already addressed by van Mieghem in his seminal paper (Ann Appl Probab 5:808–833, 1995). He launched the so-called generalized $$c\mu $$ c μ rule and proved its asymptotic optimality in the heavy traffic limit. The optimal scheduling problem with convex holding costs have also been studied in many other papers. However, the holding costs in these papers are typically count-convex, i.e., the aggregate holding cost rate related to a class of customers increases convexly as a function of the current number of customers in this class, which is clearly different from the time-convex holding costs introduced in van Mieghem (Ann Appl Probab 5:808–833, 1995) and studied also in the present paper. We utilize the Whittle index approach to develop a novel heuristic policy for the dynamic scheduling problem with time-convex holding costs. Instead of the original open version of the continuous-time scheduling problem, we apply the Whittle index approach to the corresponding discrete-time closed version of the problem. As the main theoretical contribution, we prove that, for IHR service times, the discounted discrete-time problem is indexable and also give an explicit expression for the Whittle index itself. A special challenge in our model is the two-dimensional state space with an infinite number of possible states. By utilizing the discrete-time results, we develop a novel index policy for the original continuous-time problem and demonstrate, by numerical simulations, that this Whittle index policy outperforms the generalized $$c\mu $$ c μ rule. Thus, while the generalized $$c\mu $$ c μ rule is asymptotically optimal in the heavy traffic limit, it can still be improved in normal traffic conditions.

Condition-based switching, loading, and age-based maintenance policies for standby systems

Standby techniques are widely incorporated in structural design to enhance the inherent reliability of systems. To further leverage the system performance during operation, decision-makers can adopt operational policies to manage system degradation. Specifically, at the system level, unit switching that dynamically determines the online unit contributes to avoiding unexpected shutdowns. At the unit level, adjusting load levels to manage the trade-off between condition degradation and revenue accumulation is crucial for maximizing profit. Additionally, adopting age-based maintenance as a tactical decision, which effectively facilitates the integration of maintenance resources, can be implemented to restore a degraded system. For instance, maintenance is typically scheduled at fixed moments for multi-generator power systems located in remote areas. In between maintenance moments, the proactive switching of generators can ensure uninterrupted output, and the adjustment of load levels for online generators helps to maximize output. Motivated by such engineering practices, this paper investigates condition-based switching, loading, and age-based maintenance policies for standby systems to maximize the expected profit rate in the long-run horizon. The problem is formulated as a Markov decision process. The structural properties of the control-limit switching and monotone loading policies are analyzed for easy policy implementation and efficient problem solutions. For comparative purposes, several heuristic policies are proposed and evaluated. Finally, numerical examples are presented to validate theoretical results and illustrate the superiority of the proposed risk control policy.

Fair Incentives for Repeated Engagement

We study a decision-maker’s problem of finding optimal monetary incentive schemes for retention when faced with agents whose participation decisions (stochastically) depend on the incentive they receive. Our focus is on policies constrained to fulfill two fairness properties that preclude outcomes wherein different groups of agents experience different treatment on average. We formulate the problem as a high-dimensional stochastic optimization problem and study it through the use of a closely related deterministic variant. We show that the optimal static solution to this deterministic variant is asymptotically optimal for the dynamic problem under fairness constraints. Though solving for the optimal static solution gives rise to a nonconvex optimization problem, we uncover a structural property that allows us to design a tractable, fast-converging heuristic policy. Traditional schemes for retention ignore fairness constraints; indeed, the goal in these is to use differentiation to incentivize repeated engagement with the system. Our work (i) shows that even in the absence of explicit discrimination, dynamic policies may unintentionally discriminate between agents of different types by varying the type composition of the system, and (ii) presents an asymptotically optimal policy to avoid such discriminatory outcomes.

Group machinery intelligent maintenance: Adaptive health prediction and global dynamic maintenance decision-making

Intelligent preventive maintenance powered by health data analytics is essential to ensure operation safety and performance of diverse industrial equipment. Despite the rapid advancement of preventive maintenance methodologies in recent several decades, their implementation to global dynamic maintenance of complex systems across infinite time horizon, in particular leveraging real-time prognosis information to realize adaptive group partition and ordering, has been largely an under-explored domain. To this end, this study devises a generic global-dynamic group maintenance policy for multi-component degrading systems. As opposed to existing models, the policy automatically renews the entire system health information instantly upon the completion of each group maintenance task; therefore, it realizes the dynamic union of (a) predictive group maintenance and (b) unplanned opportunistic maintenance over an infinite time horizon for the first time, such that to promote decision precision and efficiency. A dynamic grouping algorithm is designed to explore the structure of optimal maintenance solutions. The applicability is exemplified by comparative experiments that show substantial advantages over heuristic policies.

Failure risk management: adaptive performance control and mission abort decisions.

The failure behavior of safety-critical systems typically depends on the system performance level, which offers opportunities to control system failure risk through dynamic performance adjustment. Moreover, mission abort serves as an intuitive way to mitigate safety hazards during mission execution. Our study focuses on systems that execute successive missions with random durations. To balance mission completion probability and system failure risk, we examine two decision problems: when to abort missions and how to select the performance level prior to mission abort. Our objective is to maximize the expected revenue through dynamic performance control and mission abort (PCMA) decisions. We consider condition-based PCMA decisions and formulate the joint optimization problem into a Markov decision process. We establish the monotonicity and concavity of the value function. Based on this insight, we show that optimizing the mission abort policy requires a series of control limits. In addition, we provide conditions under which the performance control policies are monotone. For comparative purposes, we analytically evaluate the performances of some heuristic policies. Finally, we present a case study involving unmanned aerial vehicles executing power line inspections. The results indicate the superiority of our proposed risk control policies in enhancing operational performance for safety-critical systems. Dynamic performance adjustment and mission abort decisions provide opportunities to reduce the failure risk and increase operational rewards of safety-criticalsystems.

Inventory rationing, admission control, and production capacity allocation in a make‐to‐stock/make‐to‐order manufacturing system

AbstractThis paper considers a manufacturing system in which products are produced in both make‐to‐stock (MTS) and make‐to‐order (MTO) modes. Production of MTS and MTO products is done in batches, incurs a setup cost, and is non‐preemptive. The inventory of MTS products fulfills the demand of multiple classes, and each class demand can be satisfied or rejected. Customer orders for MTO production can be accepted or rejected, and their size is the same as the production batch. The primary goal of this paper is to study a policy that coordinates inventory rationing, admission control, and production capacity allocation to maximize the system's profit. We formulate the problem as a Markov decision process model and identify the structure of optimal control policies. We investigate the effect of inventory rationing on the profit by comparing its performance to that of the system with a first‐come‐ first‐serve policy to allocate inventory to multiple demand classes and study the extent to which the benefit of inventory rationing can be affected by system parameter changes. We also propose a heuristic that manages control decisions from linear threshold functions. Our test results from numerical examples show that the average percentage difference between the optimal and heuristic policies is within 1.2%.

Asymptotic Scaling of Optimal Cost and Asymptotic Optimality of Base-Stock Policy in Several Multidimensional Inventory Systems

Asymptotic Optimality of Simple Heuristic Policies for Multidimensional Inventory Systems Stochastic inventory systems with multidimensional state spaces, such as lost-sales system with positive lead time and perishable inventory system, are challenging to manage because of the curse of dimensionality, and their optimal control policies are extremely complex. In “Asymptotic Scaling of Optimal Cost and Asymptotic Optimality of Base-Stock Policy in Several Multidimensional Inventory Systems,” Bu, Gong and Chao consider three classes of such systems in the regime of large unit penalty cost, and they establish the asymptotic optimality of (modified) simple base-stock policies as well as an explicit expression for the optimal cost rate in each of these systems. These results justify the applications of such policies in real-world applications, and they are achieved by constructing tight newsvendor upper and lower bounds on the systems’ costs and analyzing the asymptotic scaling of newsvendor costs with large unit penalty cost. This approach is expected to be useful in studying other multidimensional stochastic inventory systems.

Cost Optimization in Cloud Computing: Capacity Reservation for Intermittent Random Demand Surges

The adoption of cloud computing has been accelerating over the last decade, while enterprise cloud users (“firms”) are struggling to manage their growing cloud expenditures in the face of intermittent digital demand surges caused by planned or random events. To deal with such challenges, a firm can employ reserved instances with the standard contract length (e.g., one year) to meet the stationary base demand, which we refer to as the base contracts, complemented by additional reserved instances with either standard or shorter contract lengths, which we refer to as the supplementary contracts, to cope with the demand surges. We first analyze a model whereby the surge and inter-surge durations are deterministic, demand magnitude is random, and cancellation of the reservations is allowed. We develop a capacity management plan for the firm including not only the optimal capacity levels, which follow a newsvendor-type solution, but also the optimal policy for managing the purchase, renewal, cancellation, or expiration of the supplementary contracts, which can be characterized as a two-threshold policy. Due to the complexity of the structure of the optimal policy, we also construct an effective heuristic policy by excluding the renewal option from the action space, which can be applied to a more general setting where the surge and inter-surge durations are random. We examine two model extensions: (1) when trades of reserved instances are allowed in a secondary marketplace; (2) when the firm does not have exact information about the distributions of the surge magnitude and duration while it can adjust the capacity levels as data unveils. Our analysis shows that the optimal policy for managing the supplementary contracts depends on the relative magnitude of the surge and inter-surge durations in relationship to the cancellation fee. Moreover, our numerical results show that cloud platforms that offer a secondary marketplace are more attractive to firms from a cost standpoint than those that allow cancellation only. The latter, without the secondary marketplace, however, can achieve parity with the former by offering a deeper discount rate for the reserved instances, thereby bypassing the cost of administering the secondary marketplace.

Spare parts recommendation for corrective maintenance of capital goods considering demand dependency

We consider a maintenance service provider that services geographically dispersed customers with its local service engineers. Traditionally, when a system failure is reported, a service engineer makes a diagnostic visit to the customer’s location to perform corrective maintenance. If spare parts are required, they are ordered and a second visit is scheduled at a later date to complete the corrective maintenance. In this paper, the service provider can proactively send spare parts to the customer to avoid the costly second visit. Motivated by a real-world problem in the high-tech industry, our model considers the cost of a second visit, fixed shipment costs, retrieval costs for the parts that are sent to the customer, and send-back costs for the parts that are sent but not used for corrective maintenance. The uncertainty in the set of parts required for corrective maintenance is modeled with a general distribution that can capture the dependencies between demands for different spare parts. We formulate an integer linear program to find the optimal set of spare parts that minimizes the expected total cost. We derive analytical results for the structure of the optimal policy and compare the optimal policy with two benchmark policies from practice. We observe that the policies from practice often find the optimal policy, and a new heuristic policy that exploits the structure of the optimal policy, on average, performs better than the benchmark policies.

Taylor Approximation of Inventory Policies for One-Warehouse, Multi-Retailer Systems with Demand Feature Information

We consider a distribution system in which retailers replenish perishable goods from a warehouse, which, in turn, replenishes from an outside source. Demand at each retailer depends on exogenous features and a random shock, and unfulfilled demand is lost. The objective is to obtain a data-driven replenishment and allocation policy that minimizes the average inventory cost per time period. The extant data-driven methods either cannot guarantee a feasible solution for out-of-sample feature observations or generate one with excessive computational time. We propose a policy that resolves these issues in two steps. In the first step, we assume that the distributions of features and random shocks are known. We develop an effective heuristic policy by using Taylor expansion to approximate the retailer’s inventory cost. The resulting solution is closed-form, referred to as Taylor Approximation (TA) policy. We show that the TA policy is asymptotically optimal in the number of retailers. In the second step, we apply the linear quantile regression and kernel density estimation to the TA solution to obtain the data-driven policy called Data-Driven Taylor Approximation (DDTA) policy. We prove that the DDTA policy is consistent with the TA policy. A numerical study shows that the DDTA policy is very effective. Using a real data set provided by Fresh Hema, we show that the DDTA policy reduces the average cost by 11.0% compared with Hema’s policy. Finally, we show that the main results still hold in the cases of correlated demand features, positive lead times, and censored demand. This paper was accepted by J. George Shanthikumar, data science. Funding: Y. Yang acknowledges financial support from the NSFC [Grants 72125004, 71821002]. W. Zhou acknowledges financial support from the NSFC [Grants 72192823, 71821002]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2021.04241 .

Stochastic Economic Lot Scheduling via Self-Attention Based Deep Reinforcement Learning

The Stochastic Economic Lot Scheduling Problem (SELSP) is a difficult dynamic optimization problem with wide industrial applications. Traditional methods such as hyper-heuristics are manually designed based on substantial expert knowledge, which may limit their optimization performance. Recently, Deep Reinforcement Learning (DRL) is shown to be promising in automatically learning scheduling policies for SELSP. However, its performance is still quite far from that of hyper-heuristics, due to the lack of suitable deep models. In this paper, we propose a novel DRL method to learn dynamic scheduling policies for SELSP in an end-to-end fashion. Based on self-attention, our method can effectively extract useful features from raw state information, and is flexible in handling different numbers of products, which is not viable for previous methods. Experiments on a complex biopharmaceutical manufacturing process show that our method outperforms a recent DRL method and state-of-the-art hyper-heuristics. Moreover, the trained policy performs better in environments different from training with demand forecast errors and varying number of products, showing its strong robustness and generalization ability. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The Stochastic Economic Lot Scheduling Problem (SELSP) is an important problem for manufacturing enterprises, which is to optimally balance the production and inventory so as to minimize the total cost. However, SELSP is very challenging to solve due to the involvement of uncertain factors such as customer demands and machine failures. Traditional methods for solving SELSP, such as heuristic policies and hyper-heuristics, heavily rely on human experiences to design and hence the performance could be limited. This paper proposes a Deep Reinforcement Learning (DRL) based method to automatically learn scheduling policy for solving SELSP, which could alleviate the above limitation through a self-attention based feature extraction mechanism and reward based training. Experimental results on a realistic manufacturing process show that our method can deliver higher revenue than conventional manual policy and an existing DRL based method.

Optimizing inland container shipping through reinforcement learning

In this study, we investigate the container delivery problem and explore ways to optimize the complex and nuanced system of inland container shipping. Our aim is to fulfill customer demand while maximizing customer service and minimizing logistics costs. To address the challenges posed by an unpredictable and rapidly-evolving environment, we examine the potential of leveraging reinforcement learning (RL) to automate the decision-making process and craft agile, efficient delivery schedules. Through a rigorous and comprehensive numerical study, we evaluate the efficacy of this approach by comparing the performance of several high-performance heuristic policies with that of agents trained using reinforcement learning, under various problem settings. Our results demonstrate that a reinforcement learning approach is robust and particularly useful for decision makers who must match logistics demand with capacity dynamically and have multiple objectives.

Minimizing File Transfer Time in Opportunistic Spectrum Access Model

We study the file transfer problem in opportunistic spectrum access (OSA) model, which has been widely studied in throughput-oriented applications for max-throughput strategies and in delay-related works that commonly assume identical channel rates and fixed file sizes. Our work explicitly considers minimizing the file transfer time for a given file in a set of heterogeneous-rate Bernoulli channels, showing that max-throughput policy doesn't minimize file transfer time in general. We formulate a mathematical framework for static extend to dynamic policies by mapping our file transfer problem to a stochastic shortest path problem. We analyze the performance of our proposed static and dynamic optimal policies over the max-throughput policy. We propose a mixed-integer programming formulation as an efficient alternative way to obtain the dynamic optimal policy and show a huge reduction in computation time. Then, we propose a heuristic policy that takes into account the performance-complexity tradeoff and consider the online implementation with unknown channel parameters. Furthermore, we present numerical simulations to support our analytical results and discuss the effect of switching delay on different policies. Finally, we extend the file transfer problem to Markovian channels and demonstrate the impact of the correlation of each channel.

Global Health in the Grip of Neoliberalism: A Combined Retrospective Comparative Stages Heuristic Policy Analysis

Background: The COVID-19 pandemic revealed critical weaknesses in global health systems, many of which have roots in the neoliberal policies that have dominated global health governance since the 1980s. Neoliberalism, characterized by market-driven policies, privatization, and reduced government intervention, has profoundly impacted healthcare access, equity, and quality worldwide. Objective: This study aims to analyze the impact of neoliberal reforms on health systems in various countries, focusing on the interplay between governance, policy formulation, and stewardship in the health sector. By examining the outcomes of these reforms, the study seeks to understand how different approaches to neoliberalism have shaped health system performance and equity. Methods: The study employs the Stages Heuristic Model (SHM) to conduct a retrospective comparative analysis of health policy reforms in ten countries, including the United States, Chile, New Zealand, Ecuador, and China. The research synthesizes data from peer-reviewed articles to assess the outcomes of neoliberal policies on healthcare systems. Results: The analysis reveals that in countries such as the United States and Chile, neoliberal reforms led to increased health inequities and a tiered healthcare system, where access to quality care became increasingly dependent on socioeconomic status. In contrast, countries like Taiwan, which balanced neoliberal reforms with strong public health initiatives, managed to maintain more equitable health systems. China and Venezuela, both of which initially adopted more interventionist healthcare policies, made notable early strides in improving access to healthcare and addressing inequities, particularly for underserved populations. However, Venezuela's progress was undermined by economic challenges and insufficient support for higher-level care, leading to a decline in health outcomes over time. Conclusion: The study underscores the need for governance models that prioritize equity and public health in healthcare systems. It suggests that while neoliberal policies can drive economic efficiency, they often do so at the cost of health equity, necessitating a re-evaluation of global health governance and policy approaches.

Production control problem for multi-product multi-resource make-to-stock systems

Most of today's production systems are working with parallel production resources to increase throughput rate due to the increase in high variability in demand and product mix. Effective control and performance evaluation of such systems is of paramount importance to minimize production and inventory-related costs. We examine a production-inventory system featuring parallel production resources capable of producing various products. In many industries such as automotive, white goods, electronics, and paint, multiple/parallel production resources are widely used to produce the ideal amount and satisfy incoming demands for distinct products. In this study, shortage cost is not restricted to only one type and both lost sales and backordering cases are analyzed. In order to analyze the optimal production policies' behavior, we initially formulate dynamic programming models for both lost sales and backordering systems, treating them as Markov Decision Processes. Subsequently, we solve these models using the value iteration algorithm. Given the challenges posed by the curse of dimensionality in the value iteration algorithm, we suggest alternative heuristic production policies. These policies extend the existing ones described for multi-item single-resource make-to-stock (MTS) systems to accommodate multiple resources. We construct simulation models to assess the efficacy of the heuristic policies, conducting comparisons of their performance against both the optimal policy and among one another. To the best of our knowledge, there has been no exploration of scenarios involving multiple production resources concurrently manufacturing distinct products in a MTS environment. Hence, this study serves as an extension to the examination of multi-item, multi-production resource MTS systems.

Managing inventory in customizable multi-echelon assembly systems

We consider a general assembly system that produces variations on a single product. Customers can choose to customize any aspect of the product which corresponds to one or more stages of the assembly system. We assume that demand for the original product and the demand for any customization at any stage are all random with known probability distributions. We model this problem as a multi-echelon inventory management problem. This problem has not been studied before. Allowing for customization at any stage in an assembly system makes this problem significantly more complex than similar problems considered in the past. To solve this problem, we ‘split' the assembly line into two parallel assembly lines which we call uncommitted and committed lines. We show that the usual method of reducing the assembly system to a series is not possible in general. We develop a heuristic policy which can be used to then reduce the system into a series system. We also present numerical calculations which highlight the efficacy of the heuristic policy.