Brownian Control Problem Research Articles

Asymptotically optimal energy consumption and inventory control in a make-to-stock manufacturing system

We study a make-to-stock manufacturing system in which a single server makes the production. The server consumes energy, and its power consumption depends on the server state: a busy server consumes more power than an idle server, and an idle server consumes more power than a turned-off server. When a server is turned on, it completes a costly set-up process that lasts a while. We jointly control the finished goods inventory and the server’s energy consumption. The objective is to minimize the long-run average inventory holding, backorder, and energy consumption costs by deciding when to produce, when to idle or turn off the server, and when to turn on a turned-off server. Because the exact analysis of the problem is challenging, we consider the asymptotic regime in which the server is in the conventional heavy-traffic regime. We formulate a Brownian control problem (BCP) with impulse and singular controls. In the BCP, the impulse control appears due to server shutdowns, and the singular control appears due to server idling. Depending on the system parameters, the optimal BCP solution is either a control-band or barrier policy. We propose a simple heuristic control policy from the optimal BCP solution that can easily be implemented in the original (non-asymptotic) system. Furthermore, we prove the asymptotic optimality of the proposed control policy in a Markovian setting. Finally, we show that our proposed policy performs close to optimal in numerical experiments.

An Approximate Analysis of Dynamic Pricing, Outsourcing, and Scheduling Policies for a Multiclass Make-to-Stock Queue in the Heavy Traffic Regime

A new study by Ata and Barjesteh proposes an effective joint dynamic pricing, outsourcing, and scheduling policy for a multiclass make-to-stock manufacturing system. The authors approximate this control problem with a Brownian control problem. They solve the Brownian problem explicitly by exploiting the solution to a Riccati equation and propose a policy for the manufacturing system based on its solution. The proposed policy is a two-sided barrier policy. Outsourcing and idling processes are used to maintain the workload above the lower and below the upper barriers, respectively. Dynamic prices are used to control the workload process between the two barriers. The authors show using a simulation study that the optimality gap of the proposed policy is small, and their proposed policy outperforms a long list of static pricing policies. Moreover, the gap between their proposed policy and the static pricing policies increases with the server utilization and the outsourcing cost.

Joint Inventory and Scheduling Control in a Repair Facility

We study inventory and repair scheduling decisions of a maintenance service provider for repairable capital goods. Due to high downtime costs, the service provider keeps spare parts on stock to replace broken parts quickly. The service provider should determine the inventory level of spare parts for each component and the repair scheduling policy. Furthermore, in case of a stock-out, the service provider should decide whether to backorder the demand or execute an emergency repair, which is an urgent but expensive repair operation for a broken part followed by a fast form of installation. The objective is to minimize the long-run average inventory holding, backorder, and emergency repair costs. We formulate the repairable network as a closed queueing system and consider an asymptotic regime in which the repair facility is in the conventional heavy-traffic regime. Then, we formulate and solve a Brownian control problem (BCP). From the optimal BCP solution, we derive a simple and intuitive decision rule stating whether emergency repairs are necessary or not to achieve a close-to-optimal system performance. Moreover, we propose a simple, intuitive, and easy-to-implement heuristic control policy and demonstrate its close-to-optimal performance via numerical experiments.

Scheduling Queues with Customer Transfers

We consider a multiclass single-server system wherein customers of each class form their own queue and can either abandon the system or transfer to other queues while waiting. A system manager makes dynamic scheduling decisions to minimize system-wide waiting cost. We solve the stochastic control problem under both infinite-horizon discounted and average cost criteria. By studying a Brownian control problem that arises in the conventional heavy traffic regime, we identify (under each cost criterion) an asymptotically optimal policy that prioritizes across different classes according to a workload-dependent dynamic index rule. The proposed policies acknowledge the impact of customer transfers on the optimal allocation of service resources and generate rich managerial insights. One of the technical difficulties in the proof of the asymptotic optimality result is to identify a natural stability condition and formally show that the sequence of diffusion-scaled processes is uniformly stable under that condition.

On the Taylor Expansion of Value Functions

We introduce a framework for approximate dynamic programming that we apply to discrete time chains on $\mathbb{Z}_+^d$ with countable action sets. Our approach is grounded in the approximation of the (controlled) chain's generator by that of another Markov process. In simple terms, our approach stipulates applying a second-order Taylor expansion to the value function to replace the Bellman equation with one in continuous space and time where the transition matrix is reduced to its first and second moments. In some cases, the resulting equation (which we label {\bf TCP}) can be interpreted as corresponding to a Brownian control problem. When tractable, the TCP serves as a useful modeling tool. More generally, the TCP is a starting point for approximation algorithms. We develop bounds on the optimality gap---the sub-optimality introduced by using the control produced by the Taylored equation. These bounds can be viewed as a conceptual underpinning, analytical rather than relying on weak convergence arguments, for the good performance of controls derived from Brownian control problems. We prove that, under suitable conditions and for suitably large initial states, (i) the optimality gap is smaller than a $1-\alpha$ fraction of the optimal value, where $\alpha\in (0,1)$ is the discount factor, and (ii) the gap can be further expressed as the infinite horizon discounted value with a lower-order per period reward. Computationally, our framework leads to an aggregation approach with performance guarantees. While the guarantees are grounded in PDE theory, the practical use of this approach requires no knowledge of that theory.

Dynamic Dispatch and Centralized Relocation of Cars in Ride-Hailing Platforms

We consider a ride-hailing platform that seeks to maximize its profit by dynamically dispatching cars to pick up customers and centrally relocating cars from one area to another. We model the ride-hailing platform as a closed stochastic processing network. Because the problem appears intractable, we resort to an approximate analysis in the heavy-traffic regime and consider the resulting Brownian control problem. This problem is simplified considerably and reduced to a lower-dimensional singular control problem called the workload formulation. We develop a novel algorithm to solve the workload problem numerically. We apply this algorithm to the workload problem derived from the New York City taxi data set. The solution helps us derive a dynamic control policy for the New York City application. In doing so, we prescribe the ride-hailing platform to first solve an offline linear program, whose optimal solution can be interpreted as the optimal static control policy. This solution helps partition the areas of the city into pools of areas. The platform only uses the information on the fraction of cars in the various pools, which reduces the state space dimension significantly, making the problem computationally tractable. When the distribution of cars among the pools is balanced, the platform follows the optimal static control policy. Otherwise, the platform intervenes to move the system to a more balanced state by either dropping demand or using a dispatch or relocation activity that is not used under the optimal static control policy. We demonstrate the effectiveness of the proposed dynamic control policy for the New York City application using a simulation study.

Brownian Control Problems for a Multiclass M/M/1 Queueing Problem with Model Uncertainty

We consider a multidimentional Brownian control problem (BCP) with model uncertainty that formally emerges from a multiclass M/M/1 queueing control problem under heavy traffic with model uncertainty. The BCP is formulated as a multidimensional stochastic differential game with two players: a minimizer who has an equivalent role to the decision maker in the queueing control problem and a maximizer whose role is to set up the uncertainty of the model. The dynamics are driven by a Brownian motion. We show that a state-space collapse properly holds. That is, the multidimensional BCP can be reduced to a one-dimensional BCP with model uncertainty that also takes the form of a two-player stochastic differential game. Then, the value function of both games is characterized as the unique solution to a free-boundary problem from which we extract equilibria for both games. Finally, we analyze the dependence of the value function and the equilibria on the ambiguity parameters.

The economic average cost Brownian control problem

AbstractIn this paper we introduce and solve a generalization of the classic average cost Brownian control problem in which a system manager dynamically controls the drift rate of a diffusion process X. At each instant, the system manager chooses the drift rate from a pair {u, v} of available rates and can invoke instantaneous controls either to keep X from falling or to keep it from rising. The objective is to minimize the long-run average cost consisting of holding or delay costs, processing costs, costs for invoking instantaneous controls, and fixed costs for changing the drift rate. We provide necessary and sufficient conditions on the cost parameters to ensure the problem admits a finite optimal solution. When it does, a simple control band policy specifying economic buffer sizes (α, Ω) and up to two switching points is optimal. The controller should invoke instantaneous controls to keep X in the interval (α, Ω). A policy with no switching points relies on a single drift rate exclusively. When there is no cost to change the drift rate, a policy with a single switching point s indicates that the controller should change to the slower drift rate when X exceeds s and use the faster drift rate otherwise. When there is a cost to change the drift rate, a policy with two switching points s < S indicates that the controller should maintain the faster drift rate until X exceeds S and maintain the slower drift rate until X falls below s.

Dynamic Pricing of a Multiclass Make-to-Stock Queue

We consider a make-to-stock manufacturing system selling multiple products to price-sensitive customers. The system manager seeks to maximize the long-run average profit by making dynamic pricing, outsourcing, and scheduling decisions: First, she adjusts prices dynamically depending on the system state. Second, when the backlog of work is judged excessive, she may outsource (or reject) new orders thereby incurring outsourcing costs. Third, she decides dynamically on which product to prioritize in the manufacturing process, i.e., she makes dynamic scheduling decisions. This problem appears analytically intractable. Thus, we resort to an approximate analysis in the heavy-traffic regime and consider the resulting Brownian control problem. We solve this problem explicitly by exploiting the solution to a particular Riccati equation. The optimal solution to the Brownian control problem is a two-sided barrier policy with drift rate control: Outsourcing and idling processes are used to keep the workload process above the lower reflecting barrier and below the upper reflecting barrier, respectively. Between the two barriers, a state-dependent drift rate is used to control the workload process. By interpreting this solution in the context of the original model, we propose a joint dynamic pricing, outsourcing, and scheduling policy, and demonstrate its effectiveness through a simulation study.

Dynamic Scheduling for Parallel Server Systems in Heavy Traffic: Graphical Structure, Decoupled Workload Matrix and some Sufficient Conditions for Solvability of the Brownian Control Problem

We consider a dynamic scheduling problem for parallel server systems. J. M. Harrison has proposed a scheme for using diffusion control problems to approximately solve such control problems for heavily loaded systems. This approach has been very successfully used in the special case when the diffusion control problem can be reduced to an equivalent one for a one-dimensional workload process. However, it remains a challenging open problem to make substantial progress on using Harrison’s scheme when the workload process is more than one-dimensional. Here we present some new structural results concerning the diffusion control problem for parallel server systems with arbitrary workload dimension. Specifically, we prove that a certain server-buffer graph associated with a parallel server system is a forest of trees. We then exploit this graphical structure to prove that there exists a matrix, used to define the workload process, that has a block diagonal-like structure. An important feature of this matrix is that, except when the workload is one-dimensional, this matrix is frequently different from a choice of workload matrix proposed by Harrison. We demonstrate that our workload matrix simplifies the structure of the control problem for the workload process by proving that when the original diffusion control problem has linear holding costs, the equivalent workload formulation also has a linear cost function. We also use this simplification to give sufficient conditions for a certain least control process to be an optimal control for the diffusion control problem with linear holding costs. Under these conditions, we propose a continuous review threshold-type control policy for the original parallel server system that exploits pooling of servers within trees in the server-buffer graph and uses non-basic activities connecting different trees in a critical manner. We call this partial pooling. We conjecture that this threshold policy is asymptotically optimal in the heavy traffic limit. We illustrate the solution of the diffusion control problem and our proposed threshold control policy for a three-buffer, three-server example.

Dynamic scheduling for parallel server systems in heavy traffic: Graphical structure, decoupled workload matrix and some sufficient conditions for solvability of the Brownian control problem

We consider a dynamic scheduling problem for parallel server systems. J. M. Harrison has proposed a scheme for using diffusion control problems to approximately solve such control problems for heavily loaded systems. This approach has been very successfully used in the special case when the diffusion control problem can be reduced to an equivalent one for a one-dimensional workload process. However, it remains a challenging open problem to make substantial progress on using Harrison’s scheme when the workload process is more than one-dimensional. Here we present some new structural results concerning the diffusion control problem for parallel server systems with arbitrary workload dimension. Specifically, we prove that a certain server-buffer graph associated with a parallel server system is a forest of trees. We then exploit this graphical structure to prove that there exists a matrix, used to define the workload process, that has a block diagonal-like structure. An important feature of this matrix is that, except when the workload is one-dimensional, this matrix is frequently different from a choice of workload matrix proposed by Harrison. We demonstrate that our workload matrix simplifies the structure of the control problem for the workload process by proving that when the original diffusion control problem has linear holding costs, the equivalent workload formulation also has a linear cost function. We also use this simplification to give sufficient conditions for a certain least control process to be an optimal control for the diffusion control problem with linear holding costs. Under these conditions, we propose a continuous review threshold-type control policy for the original parallel server system that exploits pooling of servers within trees in the server-buffer graph and uses non-basic activities connecting different trees in a critical manner. We call this partial pooling. We conjecture that this threshold policy is asymptotically optimal in the heavy traffic limit. We illustrate the solution of the diffusion control problem and our proposed threshold control policy for a three-buffer, three-server example.

Resource Sharing Networks: Overview and an Open Problem

This paper provides an overview of the resource sharing networks introduced by Massoulié and Roberts [20] to model the dynamic behavior of Internet flows. Striving to separate the model class from the applications that motivated its development, we assume no prior knowledge of communication networks. The paper also presents an open problem, along with simulation results, a formal analysis, and a selective literature review that provide context and motivation. The open problem is to devise a policy for dynamic resource allocation that achieves what we call hierarchical greedy ideal (HGI) performance in the heavy traffic limit. The existence of such a policy is suggested by formal analysis of an approximating Brownian control problem, assuming that there is “local traffic” on each processing resource.

Scheduling parallel servers in the nondegenerate slowdown diffusion regime: Asymptotic optimality results

We consider the problem of minimizing queue-length costs in a system with heterogenous parallel servers, operating in a many-server heavy-traffic regime with nondegenerate slowdown. This regime is distinct from the well-studied heavy traffic diffusion regimes, namely the (single server) conventional regime and the (many-server) Halfin-Whitt regime. It has the distinguishing property that waiting times and service times are of comparable magnitudes. We establish an asymptotic lower bound on the cost and devise a sequence of policies that asymptotically attain this bound. As in the conventional regime, the asymptotics can be described by means of a Brownian control problem, the solution of which exhibits a state space collapse.

Dynamic Scheduling of a Two-Server Parallel Server System with Complete Resource Pooling and Reneging in Heavy Traffic: Asymptotic Optimality of a Two-Threshold Policy

We consider a dynamic control problem for a parallel server system commonly known as the N-system. An N-system is a two-server parallel server system with two job classes, one server that can serve both classes, and one server that can only serve one class. We assume that jobs within each class arrive according to a renewal process. The random service time of a job has a general distribution that may depend on both the job's class and the server providing the service. Each job independently reneges, or abandons the queue without receiving service, if service does not begin within an exponentially distributed amount of time. The objective is to minimize the expected infinite horizon discounted cost of holding jobs in the system and having customers abandon, by dynamically scheduling waiting jobs to available servers.It is not possible to solve this control problem exactly, and so, we consider an asymptotic regime in which the system satisfies both a heavy traffic and a resource pooling condition. Then, we solve the limiting Brownian control problem, and interpret its solution as a policy in the original N-system. We label the servers and job classes so that server 1 can only serve class 1 and server 2 can serve both classes. The policy we propose has two thresholds. There is one threshold on the total number of jobs in the system, and one threshold on the number of class 1 jobs in the system. These thresholds are used to determine which job class server 2 should serve. We show that this proposed policy is asymptotically optimal within a specified class of admissible policies in the heavy traffic limit, and has the same limiting cost as the Brownian control problem solution.

On scheduling a multiclass queue with abandonments under general delay costs

We consider a multiclass queueing system with abandonments and general delay costs. A system manager makes dynamic scheduling decisions to minimize long-run average delay and abandonment costs. We consider the three types of delay cost: (i) linear, (ii) convex, and (iii) convex---concave, where the last one corresponds to settings where customers may have a particular deadline in mind but once that deadline passes there is increasingly little difference in the added delay. The dynamic control problem for the queueing system is not tractable analytically. Therefore, we consider the system in the conventional heavy traffic regime and study the approximating Brownian control problem (BCP). We observe that the approximating BCP does not admit a pathwise solution due to abandonments. In particular, the celebrated cμ rule and its extension, the generalized cμ rule, which is asymptotically optimal under convex delay costs with no abandonments, are not optimal in this case. Consequently, we solve the associated Bellman equation, which yields a dynamic index policy (derived from the value function) as the optimal control for the approximating BCP. Interpreting that control in the context of the original queueing system, we propose practical policies for each of the three cases considered and demonstrate their effectiveness through a simulation study.

Dynamic scheduling of a GI/GI/1+GI queue with multiple customer classes

We consider a dynamic control problem for a GI/GI/1+GI queue with multiclass customers. The customer classes are distinguished by their interarrival time, service time, and abandonment time distributions. There is a cost c k >0 for every class k?{1,2,?,N} customer that abandons the queue before receiving service. The objective is to minimize average cost by dynamically choosing which customer class the server should next serve each time the server becomes available (and there are waiting customers from at least two classes). It is not possible to solve this control problem exactly, and so we formulate an approximating Brownian control problem. The Brownian control problem incorporates the entire abandonment distribution of each customer class. We solve the Brownian control problem under the assumption that the abandonment distribution for each customer class has an increasing failure rate. We then interpret the solution to the Brownian control problem as a control for the original dynamic scheduling problem. Finally, we perform a simulation study to demonstrate the effectiveness of our proposed control.

Drift Control with Changeover Costs

We model the problem of managing capacity in a build-to-order environment as a Brownian drift control problem and seek a policy that minimizes the long-term average cost. We assume the controller can, at some cost, shift the processing rate among a finite set of alternatives, for example by adding or removing staff, increasing or reducing the number of shifts, or opening or closing production lines. The controller incurs a cost for capacity per unit time and a delay cost that reflects the opportunity cost of revenue waiting to be recognized or the customer service impacts of delaying delivery of orders. Furthermore, he incurs a cost per unit to reject orders or idle resources as necessary to keep the workload of waiting orders within a prescribed range. We introduce a practical restriction on this problem, called the 𝒮-restricted Brownian control problem, and show how to model it via a structured linear program. We demonstrate that an optimal solution to the 𝒮-restricted problem can be found among a special class of policies called deterministic nonoverlapping control band policies. These results exploit apparently new relationships between complementary dual solutions and relative value functions that allow us to obtain a lower bound on the average cost of any nonanticipating policy for the problem, even without the 𝒮 restriction. Under mild assumptions on the cost parameters, we show that our linear programming approach is asymptotically optimal for the unrestricted Brownian control problem in the sense that by appropriately selecting the 𝒮-restricted problem, we can ensure its solution is within an arbitrary finite tolerance of a lower bound on the average cost of any nonanticipating policy for the unrestricted Brownian control problem.

Optimal buffer size and dynamic rate control for a queueing system with impatient customers in heavy traffic

We address a rate control problem associated with a single server Markovian queueing system with customer abandonment in heavy traffic. The controller can choose a buffer size for the queueing system and also can dynamically control the service rate (equivalently the arrival rate) depending on the current state of the system. An infinite horizon cost minimization problem is considered here. The cost function includes a penalty for each rejected customer, a control cost related to the adjustment of the service rate and a penalty for each abandoning customer. We obtain an explicit optimal strategy for the limiting diffusion control problem (the Brownian control problem or BCP) which consists of a threshold-type optimal rejection process and a feedback-type optimal drift control. This solution is then used to construct an asymptotically optimal control policy, i.e. an optimal buffer size and an optimal service rate for the queueing system in heavy traffic. The properties of generalized regulator maps and weak convergence techniques are employed to prove the asymptotic optimality of this policy. In addition, we identify the parameter regimes where the infinite buffer size is optimal.

Dynamic Control of a Make-to-Order, Parallel-Server System with Cancellations

Motivated by make-to-order production systems, we consider a dynamic control problem for a multiclass, parallel-server queueing system. The production system serves multiple classes of customers who require rigid due-date lead times and may cancel their order subject to a cancellation penalty. To meet the due-date constraints, a system manager may outsource orders when the backlog of work is judged excessive, thereby incurring outsourcing costs. The system manager strives to minimize long-run average costs by dynamically making outsourcing and resource allocation decisions. Under heavy-traffic conditions, the scheduling problem is approximated by a Brownian control problem. Interpreting the solution of the Brownian control problem in the context of the original queueing system, a nongreedy outsourcing and resource allocation policy is proposed. A simulation experiment is performed to demonstrate the effectiveness of this policy.

Asymptotically Optimal Control for an Assemble-to-Order System with Capacitated Component Production and Fixed Transport Costs

This paper examines a two-tier assemble-to-order system. Customer orders for various products must be filled within the product-specific target lead time, or become lost sales. A product can be assembled instantaneously if its required components are in stock at the assembly facility. The production facility for each component is geographically distant from the assembly facility, and the transportation lead time is deterministic. Each shipment of components incurs a fixed cost and a variable cost per unit. The system manager must initially commit to the production capacity for each component. Then, in response to customer orders, he must dynamically manage production (expediting and salvaging) and shipping for each component, and the sequence of customer orders for assembly (how scarce components are allocated to outstanding orders). The objective is to minimize expected discounted costs for lost sales, production, and shipping. This discounted formulation accounts for financial inventory holding costs but not physical inventory holding costs. The main result is that as the order arrival rate for each product becomes large and the discount rate becomes small, a simple threshold policy with independent control of each component is asymptotically optimal. The policy is parameterized by five numbers for each component. Expressions for these parameters, the expected discounted cost, and the long-run average rates of salvaging and expediting are obtained by solving an approximating Brownian control problem. In a numerical example from the computer industry, the Brownian approximation is remarkably accurate.