Decision Epoch Research Articles

Dynamic planning for urban drayage operations

This paper develops approaches for incorporating informational and operational dynamics within models for intermodal drayage operations with flexible tasks. Tasks with flexible origins or destinations create an important complexity. We develop models to minimize the cost of drayage moves when flexibility exists. The solution approach generates plans at several decision epochs during the day. The optimization problem at each epoch is a complex multistage stochastic optimization problem; we find good solutions using a two-stage approximation. We show that with such a model, a drayage firm can find high-quality plans even when many tasks are not known initially.

An integrated product planning model for pricing and bundle selection using Markov decision processes and data envelope analysis

Abstract When bundling products during the product planning stage, there are a number of possible combinations that can be offered to customers. Consider a firm that offers N distinctive products, then there are 2 N −( N +1) possible bundling combinations. Now, if we wish to make pricing and bundle selection decisions, keeping in my mind limited inventory and a finite time horizon, then the size of the state space could be very large and finding an optimal solution could be impossible. To tackle this issue, we formulate an integrated model that utilizes a Markov decision process and data envelope analysis. Bundle selections are made using data envelope analysis in each decision epoch. Once the efficient bundles are selected they are priced by solving a Markov decision process using dynamic programming. Numerical examples are solved to exhibit the model's potential in solving real-world problems.

Optimal Control of an Assembly System with Multiple Stages and Multiple Demand Classes

We consider an assembly system with multiple stages, multiple items, and multiple customer classes. The system consists of m production facilities, each producing a different item. Items are produced in variable batch sizes, one batch at a time, with exponentially distributed batch production times. Demand from each class takes place continuously over time according to a compound Poisson process. At each decision epoch, we must determine whether or not to produce an item and should demand from a particular class arise whether or not to satisfy it from existing inventory, if any is available. We formulate the problem as a Markov decision process and use it to characterize the structure of the optimal policy. In contrast to systems with exogenous and deterministic production leadtimes, we show that the optimal production policy for each item is a state-dependent base-stock policy with the base-stock level non-increasing in the inventory level of items that are downstream and non-decreasing in the inventory level of all other items. For inventory allocation, we show that the optimal policy is a multi-level state-dependent rationing policy with the rationing level for each demand class non-increasing in the inventory level of all non-end items. We also show how the optimal control problem can be reformulated in terms of echelon inventory and how the essential features of the optimal policy can be reinterpreted in terms of echelon inventory.

Use of a Markov Decision Process for Improving Treatment Strategies for Brain Metastases

To construct a decision model for studying optimal treatment strategies for patients with brain metastases. A Markov Decision Process (MDP) was used as the framework for modeling the sequence of events resulting from the process of imaging followed by treatment. MDPs calculate the optimal strategy of actions (therapies) when the outcomes are stochastic. The therapy process was modeled as a series of decision epochs in which the state of the patient is determined, a therapy is delivered, and the state of the patient evolves stochastically. The possible actions were whole brain radiation, stereotactic radiosurgery, surgery and waiting. The probabilities of different outcomes for each type of action were obtained from the literature and from a clinical database. The RTOG recursive partitioning analysis and subsequent literature were used to determine the distribution of life expectancies. Outcomes focused on quality of life. Optimal treatment strategies were calculated for different initial states. A model was constructed and evaluated against expert judgment and clinical experience. Each version of the model was then modified to obtain better agreement between the model recommendations and clinical experience for a set of standard scenarios. The effects of different input variables, such as the distribution of life expectancies, were investigated and the sensitivity of the model output was determined. This process demonstrated that a reliable and stable model could be constructed; recommendations for improved data such as quality of life data and outcome probabilities were obtained. The broad range of life expectancies of patients with brain metastases, coupled with improving systemic therapies, has resulted in a significant number of patients undergoing multiple treatments. Previous studies have shown the difficulty of improving survival. We have provided a method for studying optimal treatment strategies that incorporates the possibility of patients undergoing several treatments for their brain metastases. This work has focused on quality of life as an endpoint, though the model could be modified for other goals such as improved survival. The current model would benefit from improved data and future work will focus on these improvements.

SU-GG-T-445: Sequential Treatment of Brain Metastases Using a Markov Decision Process

Purpose: To use a Markov Decision Process (MDP) to model the sequential treatment of recurrent brain metastases. The purpose of the model is to calculate the optimal set of treatment decisions given the state of the patient. Improved systemic cancer control has resulted in growing numbers of patients who undergo multiple treatments for brain metastases. Given this situation, it is important to determine the optimal treatment decision at any point in time given the condition of the patient and the possibility of future treatments. Methods: Sequential stochastic decisions can be optimized with respect to a given reward using dynamic programming methods. A model of patients with brain metastases was constructed using a definition of state, a set of actions (whole brain RT, surgery, stereotactic radiosurgery, waiting), transition probabilities between states dependent on a given action, and a reward based on quality of life. Transition probabilities were obtained from the literature and clinical data. The problem was solved for both an infinite horizon, resulting in a stationary policy, and a fixed horizon, resulting in optimal decisions for each decision epoch. Results: The model was developed and validated by comparison with clinical judgment. Selection of adequate state variables and suitable rewards was made by comparing optimal policies with standard clinical situations. After the development phase, the model was used to study the role that possible subsequent treatments have on optimal decisions. The effects of knowledge regarding expected lifespan on optimal policies was investigated. In addition, comparisons of stationary policies with finite horizon policies were made. Conclusion: A first attempt at applying MDPs to radiation therapy decision making focussed on improving treatment strategies for metastatic braincancer. An important application of this model will be to determine the extent to which improvements in our knowledge of a patient's prognosis will alter recommended therapies.

Coordinated Look-ahead Control of Multiple CSPS System by Multi-agent Reinforcement Learning

The optimal control problem of a multiple conveyor-serviced production station (CSPS) system is concerned.The objective is to maximize the part-processing rate of the entire system by choosing a suitable look-ahead control strategy for each CSPS.According to the reaction-diffusion mechanism of multi-agent systems,the original performance function of each agent is first modified by introducing an item with a diffusion function that denotes the interaction of local information (The original item is assumed to have a reaction function).Then,combined with the concept of performance potentials,a multi-agent algorithm,i.e.,Wolf-PHC algorithm,is proposed to derive the coordinated look-ahead control strategy for systems with either discounted or average performance criteria,where the decision epoch of each agent is asynchronous.Finally,a simulation example is used to illustrate the effectiveness of the algorithm,and the simulation results show that due to the modification of the performance functions,the contributions of all the stations are well balanced,and the part-processing rate of the entire system is increased significantly.

Optimal control of a production‐inventory system with both backorders and lost sales

AbstractWe consider the optimal control of a production inventory‐system with a single product and two customer classes where items are produced one unit at a time. Upon arrival, customer orders can be fulfilled from existing inventory, if there is any, backordered, or rejected. The two classes are differentiated by their backorder and lost sales costs. At each decision epoch, we must determine whether or not to produce an item and if so, whether to use this item to increase inventory or to reduce backlog. At each decision epoch, we must also determine whether or not to satisfy demand from a particular class (should one arise), backorder it, or reject it. In doing so, we must balance inventory holding costs against the costs of backordering and lost sales. We formulate the problem as a Markov decision process and use it to characterize the structure of the optimal policy. We show that the optimal policy can be described by three state‐dependent thresholds: a production base‐stock level and two order‐admission levels, one for each class. The production base‐stock level determines when production takes place and how to allocate items that are produced. This base‐stock level also determines when orders from the class with the lower shortage costs (Class 2) are backordered and not fulfilled from inventory. The order‐admission levels determine when orders should be rejected. We show that the threshold levels are monotonic (either nonincreasing or nondecreasing) in the backorder level of Class 2. We also characterize analytically the sensitivity of these thresholds to the various cost parameters. Using numerical results, we compare the performance of the optimal policy against several heuristics and show that those that do not allow for the possibility of both backordering and rejecting orders can perform poorly.© 2010 Wiley Periodicals, Inc. Naval Research Logistics 2010

Optimal Control of a Production-Inventory System with Both Backorders and Lost Sales

We consider the optimal control of a production inventory-system with a single product and two customer classes where items are produced one unit at a time. Upon arrival, customer orders can be fulfilled from existing inventory, if there is any, backordered, or rejected. The two classes are differentiated by their backorder and lost sales costs. At each decision epoch, we must determine whether or not to produce an item and if so, whether to use this item to increase inventory or to reduce backlog. At each decision epoch, we must also determine whether or not to satisfy demand from a particular class (should one arise), backorder it, or reject it. In doing so, we must balance inventory holding costs against the costs of backordering and lost sales. We formulate the problem as a Markov decision process and use it to characterize the structure of the optimal policy. We show that the optimal policy can be described by three state-dependent thresholds: a production base-stock level and two order-admission levels, one for each class. The production base-stock level determines when production takes place and how to allocate items that are produced. This base-stock level also determines when orders from the class with the lower shortage costs (class 2) are backordered and not fulfilled from inventory. The order-admission levels determine when orders should be rejected. We show that the threshold levels are monotonic (either non-increasing or non-decreasing) in the backorder level of class 2. We also characterize analytically the sensitivity of these thresholds to the various cost parameters. Using numerical results, we compare the performance of the optimal policy against several heuristics and show that those that do not allow for the possibility of both backordering and rejecting orders can perform poorly.

New product development projects evaluation under time uncertainty

The development time is one of the key factors that contribute to the new product development success. In spite of that, the impact of the time uncertainty on the development has been not fully exploited, as far as decision supporting models to evaluate this kind of projects is concerned. In this context, the objective of the present paper is to evaluate the development process of new technologies under time uncertainty. We introduce a model which captures this source of uncertainty and develop an algorithm to evaluate projects that incorporates Monte Carlo Simulation and Dynamic Programming. The novelty in our approach is to thoroughly blend the stochastic time with a formal approach to the problem, which preserves the Markov property. We base our model on the distinction between the decision epoch and the stochastic time. We discuss and illustrate the applicability of our model through an empirical example.

A Generalized Gittins Index for a Class of Multiarmed Bandits with General Resource Requirements

We generalise classical multiarmed bandits to allow for the distribution of a (fixed amount of a) divisible resource among the constituent bandits at each decision point. Bandit activation consumes amounts of the available resource, which may vary by bandit and state. Any collection of bandits may be activated at any decision epoch, provided they do not consume more resource than is available. We propose suitable bandit indices that reduce to those proposed by Gittins [Gittins, J. C. 1979. Bandit processes and dynamic allocation indices (with discussion). J. R. Statist. Soc. B41 148–177] for the classical models. The index that emerges is an elegant generalization of the Gittins index, which measures in a natural way the reward earnable from a bandit per unit of resource consumed. The paper discusses both how such indices may be computed and how they may be used to construct heuristics for resource distribution. We also describe how to develop bounds on the closeness to optimality of index heuristics and demonstrate a form of asymptotic optimality for a greedy index heuristic in a class of simple models. A numerical study testifies to the strong performance of a weighted index heuristic.

A semi-Markov decision algorithm for the optimal maintenance of a multistage deteriorating two-unit standby system

The semi-Markov decision model is a powerful tool in analyzing sequential decision processes with random decision epochs for a multi-state deteriorating system subject to aging and fatal shocks. In this paper, we propose a model for a two-unit standby system where a cold standby unit is attached to an operating (active) one. For this model, the active unit goes through a finite number of states of successive degradation preceding the failure, while the other one is in cold standby state. At each deterioration state of the active unit, two types of maintenance are considered, minimal and major, depending on the degrading level. The minimal maintenance aims to improve the degradation of the unit by recovering it to the previous degradation stage. The maximum allowable number of minimal maintenances for all states of the active unit must not exceed a certain limit. On the other hand, the major maintenance is necessary when the active unit fails. Once this maintenance is completed, the unit is restored to as good as new. To make the system operate more time without any interruption, the standby unit can be switched online until the active unit finishes its minimal or major maintenance. The switch between the two units is perfect and switchover is instantaneous. After using the standby unit, it is serviced or overhauled to maintain it in as good as new state. We use an iterative numerical approach, based on the policy iteration method, to drive the optimal state-dependent maintenance policy that minimizes the long-run expected cost rate of the system. Finally, numerical examples are given to illustrate and evaluate the performance of the proposed policy.

Simultaneous determination of production and maintenance schedules using in‐line equipment condition and yield information

AbstractIn many manufacturing environments, equipment condition has a significant impact on product quality, or yield. This paper presents a semi‐Markov decision process model of a single‐stage production system with multiple products and multiple maintenance actions. The model simultaneously determines maintenance and production schedules, accounting for the fact that equipment condition affects the yield of each product differently. It extends earlier work by allowing the expected time between decision epochs to vary by both action and machine state, by allowing multiple maintenance actions, and by treating the outcome of maintenance as less than certain. Sufficient conditions are developed that ensure the monotonicity of both the optimal production and maintenance actions. While the maintenance conditions closely resemble previously studied conditions for this type of problem, the production conditions represent a significant departure from earlier results. The simultaneous solution method is compared to an approach commonly used in industry, where the maintenance and production problems are treated independently. Solving more than one thousand test problems confirms that the combination of both features of the model—accounting for product differences and solving the problems simultaneously—has a significant impact on performance. © 2007 Wiley Periodicals, Inc. Naval Research Logistics, 2008

Capacity Management for Contract Manufacturing

Contract manufacturers sell capacity under different terms to different buyers. In a common practice, the supplier offers a market standard price and lead-time combination for products listed in its catalog. At the same time, it strikes contracts with some high-volume customers who require recurring delivery of a custom product at a short notice, usually timed to meet the buyer’s production schedule. Whereas the manufacturer is obligated to satisfy demand from these customers, it can dynamically choose which transactional orders to accept. In this paper, we analyze two scenarios. In the first scenario, the manufacturer produces contractual orders on a make-to-order basis, and in the second, it maintains finished goods inventory for such orders. At each decision epoch, the manufacturer decides which transactional orders to accept to maximize its long-run expected profit. When contractual items are made to stock, the manufacturer also chooses how much extra capacity to allocate to contractual items, over and above what is needed to meet realized demand. We establish the structure of the optimal policies and propose scalable heuristics when optimal policies are hard to compute/implement. Our models are then used to study the effect of demand variability on the optimal profit. We also show how the amount of capacity available, relative to demand, changes the desirability of serving either transactional-only or both markets.

Admission control of a service station when the arrivals are internal

All studies in the admission control of a service station make decisions at arrival epochs. When arrivals are internal and are rejected from a queue, the rejected jobs have to be routed to other stations in the system. However the system will not know whether a job will be admitted to a queue or not until its arrival epoch to that queue. Thus, the system has to react dynamically and agilely to the decisions made at a specific queue and may try several queues before finding a queue that admits the job. This paper remedies these difficulties by changing the decision epochs of the admission control from arrival epochs to departure epochs with the actions of switching (keeping) the arrival stream on or off. Thus upstream stations will have information on the admission status of their downstream stations all the time. It is proved that the optimal policy for this revised admission control system is of control limit type for an M/G/1 queue. Comparisons of the optimal values and optimal policies for the admission controls made at arrival epochs and at departure epochs are included in the paper.

Asynchronous action-reward learning for nonstationary serial supply chain inventory control

Action-reward learning is a reinforcement learning method. In this machine learning approach, an agent interacts with non-deterministic control domain. The agent selects actions at decision epochs and the control domain gives rise to rewards with which the performance measures of the actions are updated. The objective of the agent is to select the future best actions based on the updated performance measures. In this paper, we develop an asynchronous action-reward learning model which updates the performance measures of actions faster than conventional action-reward learning. This learning model is suitable to apply to nonstationary control domain where the rewards for actions vary over time. Based on the asynchronous action-reward learning, two situation reactive inventory control models (centralized and decentralized models) are proposed for a two-stage serial supply chain with nonstationary customer demand. A simulation based experiment was performed to evaluate the performance of the proposed two models.

Field-level customer support decision model for high-tech products in India

Product support offers competitive differentiation for high-tech products in developing markets like India. Highly fluctuating numbers of daily customer field-service requests mandate 'optimal field staffing decision'. Despite an operational decision, it assumes strategic significance influencing long run customer service level. Companies decide on the field staff strength (fixed and outsourced) to comply with the fluctuating field service demand. This study models a real-time customer support system featuring restricted daily working hours and decision making epochs. The study assesses the daily field service demand based on the cumulative sales developing a methodology. The study uses stochastic principles to derive mathematical expressions for average customer waiting time and cost of customer service in a service system with decision-making epochs. Real time data that validate the model show that customer waiting time minimises with increased number of decision epochs without affecting the cost of service.

The stochastic joint replenishment problem: A new policy, analysis, and insights

AbstractIn this study, we propose a new parsimonious policy for the stochastic joint replenishment problem in a single‐location, N‐item setting. The replenishment decisions are based on both group reorder point‐group order quantity and the time since the last decision epoch. We derive the expressions for the key operating characteristics of the inventory system for both unit and compound Poisson demands. In a comprehensive numerical study, we compare the performance of the proposed policy with that of existing ones over a standard test bed. Our numerical results indicate that the proposed policy dominates the existing ones in 100 of 139 instances with comparably significant savings for unit demands. With batch demands, the savings increase as the stochasticity of demand size gets larger. We also observe that it performs well in environments with low demand diversity across items. The inventory system herein also models a two‐echelon setting with a single item, multiple retailers, and cross docking at the upper echelon. © 2006 Wiley Periodicals, Inc. Naval Research Logistics, 2006

Packet delay in optical circuit-switched networks

A framework is provided for evaluation of packet delay distribution in an optical circuit-switched network. The framework is based on a fluid traffic model, packet queueing at edge routers, and circuit-switched transmission between edge routers. Packets are assigned to buffers according to their destination, delay constraint, physical route and wavelength. At every decision epoch, a subset of buffers is allocated to end-to-end circuits for transmission, where circuit holding times are based on limited and exhaustive circuit allocation policies. To ensure computational tractability, the framework approximates the evolution of each buffer independently. Slack variables are introduced to decouple amongst buffers in a way that the evolution of each buffer remains consistent with all other buffers in the network. The delay distribution is derived for a single buffer and an approximation is given for a network of buffers. The approximation entails finding a fixed point for the functional relation between the slack variables and a specific circuit allocation policy. An analysis of a specific policy, in which circuits are probabilistically allocated based on buffer size, is given as an illustrative example. The framework is shown to be in good agreement with a discrete event simulation model.

Dynamic Column Generation for Dynamic Vehicle Routing with Time Windows

We consider a dynamic vehicle routing problem with hard time windows, in which a set of customer orders arrives randomly over time to be picked up within their time windows. The dispatcher does not have any deterministic or probabilistic information on the location and size of a customer order until it arrives. The objective is to minimize the sum of the total distance of the routes used to cover all the orders. We propose a column-generation-based dynamic approach for the problem. The approach generates single-vehicle trips (i.e., columns) over time in a real-time fashion by utilizing existing columns, and solves at each decision epoch a set-partitioning-type formulation of the static problem consisting of the columns generated up to this time point. We evaluate the performance of our approach by comparing it to an insertion-based heuristic and an approach similar to ours, but without computational time limit for handling the static problem at each decision epoch. Computational results on various test problems generalized from a set of static benchmark problems in the literature show that our approach outperforms the insertion-based heuristic on most test problems.

Optimal staffing policies for customer support system with uncertain demand

'Customer support' includes all value-added product services after the product sale, such as installation, maintenance and repairs, etc. A high-level of uncertainty and fluctuations accompanies the demand for customer support services. Hence, the system needs optimal field-staffing decisions with the least risk. The problem becomes all the more challenging under restricted working hour conditions. Daily working hour duration embedded with 'decision epochs' partly mitigates demand uncertainty. This research work uses this framework and assesses the volatile demand with a probability distribution. Further, this research develops a decision model using the Markov Decision Process (MDP) in a Dynamic Programmin (DP) framework to optimise the field-staffing decisions considering demand fluctuations and risk. Results of the numerical analysis show that more decision epochs reduces cost. Also, the neutral risk policies become optimal with the increase in the number of decision epochs.