Dynamic Programming Formulation Research Articles

A global shooting algorithm for the facility location and capacity acquisition problem on a line with dense demand

This paper describes the development of an exact allocation-based solution algorithm for the facility location and capacity acquisition problem (LCAP) on a line with dense demand data. Initially, the n-facility problem on a line is studied and formulated as a dynamic programming model in the allocation decision space. Next, we cast this dynamic programming formulation as a two-point boundary value problem and provide conditions for the existence and uniqueness of solutions. We derive sufficient conditions for non-empty service regions and necessary conditions for interior facility locations. We develop an efficient exact shooting algorithm to solve the problem as an initial value problem and illustrate on an example. A computational study is conducted to study the effect of demand density and other problem parameters on the solutions.

Alternative Layout and Automation of Optimality Results for Machine Replacement Problems Based On Stationary Data and Age Transition Perspectives

Aim: This investigation aimed at automating the computations of optimal replacement policies and rewards for a class of equipment replacement problems based on time perspectives and stationary pertinent data. Methodology: The aim was achieved by the exploitation of the structure of the states given as functions of decision periods, in age-transition dynamic programming recursions. Results: Alternative Excel solution implementation templates were designed and automated for the determination of the optimal replacement policies in machine replacement problems, with pertinent data given only as functions of machines’ ages. Conclusion: The automation of these templates obviates the need for manual inputs of the states and stage numbering, as well as the inherent tedious and prohibitive manual computations associated with dynamic programming formulations and may be optimally exploited for sensitivity analyses on such models.

Design and Full Automation of Excel Solution Templates for a Time-perspective Class of Machine Replacement Problems with Pertinent Dynamic Data

Aim: The aim of this paper is to design and automate optimal policy prescriptions and returns for a time perspective class of machine replacement problems with pertinent dynamic data. Methodology: The aim was achieved by the exploitation of the structure of the states in timeperspective dynamic programming recursions and the coding functionality in Microsoft Excel. Results: The paper designed and fully automated the solution templates for the determination of the optimal time replacement policies in machine replacement problems, with pertinent data given as dynamic functions of new machine purchase year and machine age. Conclusion: The automation of these templates obviates the need for manual inputs of the states and stage numbering as well as the inherent tedious and prohibitive manual computations associated with dynamic programming formulations and may be optimally appropriated for sensitivity analyses on such model, ensuring that problems that could take days to solve are solved in a matter of just a few minutes.

Starting Age Zero-Based Excel Automation of Optimal Policy Prescriptions and Returns for Machine Replacement Problems with Stationary Data and Age Transition Perspectives

Aim: This investigation aimed at automating the computations of optimal replacement policies and rewards for a class of equipment replacement problems with zero starting age, based on age perspectives and stationary pertinent data. Methodology: The aim was achieved by the exploitation of the structure of the states given as functions of the decision periods, in age-transition dynamic programming recursions. Results: Optimal Excel interface and solution implementation templates were designed and automated for the determination of the optimal replacement policies in machine replacement problems, with pertinent data given only as a function of machine age. Conclusion: The automation of these templates obviates the need for manual inputs of the states and stage numbering, as well as the inherent tedious and prohibitive manual computations associated with dynamic programming formulations and may be optimally exploited for sensitivity analyses on such models, paving the way for the solutions of practical and large-scale problems.

Delayed Purchase Options in Single-Leg Revenue Management

Many airline reservation systems offer the commitment option to their potential passengers. This option allows passengers to reserve a seat for a fixed duration before making a final purchase decision. In this study, we develop single-leg revenue management models that consider such contingent commitment decisions. We start with a dynamic programming model of this problem. This model is computationally intractable as it requires storing a multidimensional state space because of bookkeeping of the committed seats. To alleviate this difficulty, we propose an alternate dynamic programming formulation that uses an approximate model of how the contingent commitments behave and we show how to extract a capacity allocation policy from the approximate dynamic programming formulation. In addition, we present a deterministic linear programming model that gives an upper bound on the optimal expected revenue from the intractable dynamic programming model. As the problem size becomes large in terms of flight capacity and the expected number of arrivals, we demonstrate an asymptotic lower bound for the deterministic linear programming model. Our extensive numerical study indicates that offering commitment options can noticeably increase potential revenue even though offering a contingent commitment option may not always be in the best interest of the airline. Also, our results show that the proposed approximate dynamic programming model coordinates capacity allocation and commitment decisions quite well.

The Feedback Capacity of the Binary Erasure Channel With a No-Consecutive-Ones Input Constraint

The input-constrained erasure channel with feedback is considered, where the binary input sequence contains no consecutive ones, i.e., it satisfies the $(1,\infty )$ -RLL constraint. We derive the capacity for this setting, which can be expressed as $C_{\epsilon }=\max _{0\leq p\leq 0.5} \frac {(1-\epsilon )H_{b}(p)}{1+(1-\epsilon )p}$ , where $\epsilon $ is the erasure probability and $ H_{b}(\cdot )$ is the binary entropy function. Moreover, we prove that a priori knowledge of the erasure at the encoder does not increase the feedback capacity. The feedback capacity was calculated using an equivalent dynamic programming (DP) formulation with an optimal average-reward that is equal to the capacity. Furthermore, we obtained an optimal encoding procedure from the solution of the DP, leading to a capacity-achieving, zero-error coding scheme for our setting. DP is, thus, shown to be a tool not only for solving optimization problems, such as capacity calculation, but also for constructing optimal coding schemes. The derived capacity expression also serves as the only non-trivial upper bound known on the capacity of the input-constrained erasure channel without feedback, a problem that is still open.

Coupled bisection for root ordering

We consider the problem of solving multiple “coupled” root-finding problems at once, in that we can evaluate every function at the same point simultaneously. Using a dynamic programming formulation, we show that a sequential bisection algorithm is a close-to-optimal method for finding a ranking with respect to the zeros of these functions. We show the ranking can be found in linear time, prove an asymptotic approximation guarantee of 1.44, and conjecture that this policy is near-optimal.

Optimal Power Management for Remote Estimation With an Energy Harvesting Sensor

This paper studies the design of an estimation system where a remotely observed source sequence is to be communicated through a noisy channel to an estimator. The remote node is assumed to have the capability of harvesting, and, subject to a capacity limit, storing energy from its ambient environment. The focus is on various transmit power-allocation strategies that minimize the mean square error at the estimator for such an energy harvesting estimation system as the fluctuation of harvested energy presents a unique challenge compared with a traditional battery powered system. We first establish the optimality of uncoded transmission for such a system. Two types of side information (SI) at the transmitter are then considered in this paper: noncausal SI (energy harvested in the past, present, and future) and causal SI (energy harvested in the past). For the case where noncausal SI is available and battery storage is unlimited, it is shown that the optimal power allocation amounts to a simple “staircase-climbing” procedure, where the power level follows a nondecreasing staircase function. For the case where battery storage has a finite capacity, the optimal power-allocation policy can also be obtained via standard convex optimization techniques. Dynamic programming (DP) is used to optimize the allocation policy when only causal SI is available. The issue of unknown transmit power at the receiver is also addressed for both the causal and noncausal SI cases. Finally, to make the proposed solutions practically more meaningful, two heuristic schemes are proposed; these schemes are largely motivated by the structure of the solution to the DP formulation but with much reduced computational complexity. Numerical examples are provided to examine the complexity–performance tradeoff of various power-allocation strategies.

Production control problem integrating overhaul and subcontracting strategies for a quality deteriorating manufacturing system

We study an unreliable deteriorating manufacturing system that produces conforming and non-conforming parts to satisfy a constant demand product rate. The manufacturing system is comprised of a failure-prone machine. Due to the combined effect of random availability variations and deterioration, the system is not able to fulfill long-term product demand. In particular, when finished goods inventory is positive, clients demand are fulfilled on-time and without delay. When backlog exists, subcontracting options are available at a higher cost to supplement the limited production capacity of the manufacturing system. The effect of deterioration is observed mainly in the quality of the parts produced, since the rate of defectives increases as the machines deteriorate. Overhaul activities can be conducted to mitigate the effect of deterioration. We propose a joint feedback control policy based on a stochastic dynamic programming formulation which aims simultaneously to determine the production and overhaul rates, and the rate at which subcontractors are requested. The proposed joint control policy minimizes the total cost, including the inventory, backlog, repair, overhaul, defectives, production and subcontracting costs, over an infinite planning horizon. To determine the optimal control parameters, we adopt a numerical scheme to solve the optimality equations and a numerical example is presented as an illustration of the approach. The structure of the joint control policy is validated through an extensive sensitivity analysis.

The routed inventory pooling problem with multiple lateral transshipments

We consider a single item, single-period inventory problem with two identical retailers who can pool stocks at multiple predetermined points in time. Since routing decisions are often taken at the multi-item level in practice, we use a predetermined route for redistribution at the single-item level. The objective is to determine order-up-to levels and transshipment decisions that minimise the sum of inventory holding, backorder and fixed as well as variable transshipment costs. We present a Dynamic Programming formulation to solve the problem and we compare the performance of our routed pooling policy to situations without pooling and with complete pooling in an extensive numerical study. The results reveal when routed pooling performs well and when multiple pooling moments provide significant benefits.

Optimal Bayesian Learning of Dose-Response Parameters from a Cohort

There has been a surge of clinical interest in the idea of response-guided dosing (RGD). The goal in RGD is to tailor drug-doses to the stochastic evolution of each individual patient's disease condition over the treatment course. The hope is that this form of individualized therapy will deliver the right dose to the right patient at the right time. Several expert panels have observed that despite the excitement surrounding RGD, quantitative, data-driven decision-making approaches that learn patients' dose-response and incorporate this information into adaptive dosing strategies are lagging behind. This situation is particularly exacerbated in clinical trials. For instance, fixed design clinical studies for estimating the key parameter of a dose-response function might not treat trial patients optimally. Similarly, the dosing strategies employed in clinical trials for RGD often appear ad-hoc. In this paper, we study the problem of finding optimal RGD policies while learning the distribution of a dose-response parameter from a cohort of patients. We provide a Bayesian stochastic dynamic programming (DP) formulation of this problem. Exact solution of Bellman's equations for this problem is computationally intractable. We therefore present two approximate control schemes and mathematically analyze the monotonicity, stationarity, and separability structures of the resulting dosing strategies. These structures are then exploited in efficient, approximate solution of the optimal learning problem. Computer simulations using the Michaelis-Menten dose-response function are included as an example wherein we study the effect of cohort size and prior misspecification on learning, and also compare the performance of various dosing policies.

Revenue Maximization with Optimal Capacity Control in Infrastructure as a Service Cloud Markets

Infrastructure-as-a-Service cloud providers offer diverse purchasing options and pricing plans, namely on-demand, reservation, and spot market plans. This allows them to efficiently target a variety of customer groups with distinct preferences and to generate more revenue accordingly. An important consequence of this diversification however, is that it introduces a non-trivial optimization problem related to the allocation of the provider's available data center capacity to each pricing plan. The complexity of the problem follows from the different levels of revenue generated per unit of capacity sold, and the different commitments consumers and providers make when resources are allocated under a given plan. In this work, we address a novel problem of maximizing revenue through an optimization of capacity allocation to each pricing plan by means of admission control for reservation contracts, in a setting where aforementioned plans are jointly offered to customers. We devise both an optimal algorithm based on a stochastic dynamic programming formulation and two heuristics that trade-off optimality and computational complexity. Our evaluation, which relies on an adaptation of a large-scale real-world workload trace of Google, shows that our algorithms can significantly increase revenue compared to an allocation without capacity control given that sufficient resource contention is present in the system. In addition, we show that our heuristics effectively allow for online decision making and quantify the revenue loss caused by the assumptions made to render the optimization problem tractable.

Maximising profit for multiple-product, single-period, single-machine manufacturing under sequential set-up constraints that depend on lot size

The classical problem of order acceptance/rejection in make-to-order environments, when aiming to maximise profit with machine set-ups is extended in this paper to multiple set-ups depending on manufacturing batch size. In this case, if the manufacturing batch is larger than certain product-dependent bounds, not only is the initial set-up required but also periodic reset-ups are in order, generating sub-batches of the same order, such as tool resharpening and machine recalibration. A network formulation provides the basis for identifying effective algorithms to obtain a solution to the problem. A binary programming model (BPM) and a dynamic programming formulation (DPF) are proposed to solve the problem to optimality. In addition, two heuristics are developed to obtain lower bounds on maximum profit: each attempt to maximise customer satisfaction under production time restrictions, and to provide an extension to the classical knapsack problem. Numerical experimentation shows that computational time is not an issue when BPM and heuristics are applied, but the cost of commercial solvers for BPM algorithms might be problematic. However, if the aim is to code the DPF in-house, the curse of dimensionality in dynamic programming must be addressed, although dynamic programming does yield a full sensitivity analysis, which is useful for decision-making.

An adaptive crashing policy for stochastic time-cost tradeoff problems

We consider a stochastic time-cost tradeoff problem that determines how much to crash activities with the purpose of minimizing the expected summation of crashing and tardiness costs. We propose a threshold policy that makes a crashing decision contingent on a project׳s current status; crashing an activity to compensate delayed starting time from a predetermined threshold. First, we present a dynamic programming (DP) formulation to find the threshold values, and prove that the resulting threshold policy is optimal for a serial-graph project. Since the above optimality does not hold generally, we develop a DP-based procedure to construct a threshold policy for arbitrary-graph projects.We show through the computational experiments that our threshold policy is rapidly constructed by this procedure, and its cost is not very far from the theoretical lower bound.

Revenue management under customer choice behaviour with cancellations and overbooking

In many application areas such as airlines and hotels a large number of bookings are typically cancelled. Explicitly taking into account cancellations creates an opportunity for increasing revenue. Motivated by this we propose a revenue management model based on Talluri and van Ryzin (2004) that takes cancellations into account in addition to customer choice behaviour. Moreover, we consider overbooking limits as these are influenced by cancellations. We model the problem as a Markov decision process and propose three dynamic programming formulations to solve the problem, each appropriate in a different setting. We show that in certain settings the problem can be solved exactly using a tractable solution method. For other settings we propose tractable heuristics, since the problem faces the curse of dimensionality. Numerical results show that the heuristics perform almost as good as the exact solution. However, the model without cancellations can lead to a revenue loss of up to 20 percent. Lastly we provide a parameter estimation method based on Newman et al. (2014). This estimation method is fast and provides good parameter estimates. The combination of the model, the tractable and well-performing solution methods, and the parameter estimation method ensures that the model can efficiently be applied in practice.

The economic speed of an oceangoing vessel in a dynamic setting

The optimal (economic) speed of oceangoing vessels has become of increased importance due to the combined effect of low freight rates and volatile bunker prices. We examine the problem for vessels operating in the spot market in a tramp mode. In the case of known freight rates between origin destination combinations, a dynamic programming formulation can be applied to determine both the optimal speed and the optimal voyage sequence. Analogous results are derived for random freight rates of known distributions. In the case of independent rates the economic speed depends on fuel price and the expected freight rate, but is independent of the revenue of the particular voyage. For freight rates that depend on a state of the market Markovian random variable, economic speed depends on the market state as well, with increased speed corresponding to good states of the market. The dynamic programming equations in our models differ from those of Markovian decision processes so we develop modifications of standard solution methods, and apply them to small examples.

Capacity of Electron-Based Communication Over Bacterial Cables: The Full-CSI Case

Motivated by recent discoveries of microbial communities that transfer electrons across centimeter-length scales, this paper studies the information capacity of bacterial cables via electron transfer, which coexists with molecular diffusion, under the assumption of full causal channel state information (CSI). The bacterial cable is modeled as an electron queue that transfers electrons from the encoder at the electron donor source, which controls the desired input electron intensity, to the decoder at the electron acceptor sink. Local clogging due to ATP saturation of the cells is modeled, as well as interference and leakage due to electron donors and acceptors along the cable. A discrete-time scheme is investigated, enabling the computation of an achievable rate. The regime of asymptotically small time-slot duration is analyzed, and the optimality of binary input distributions is proved, i.e., the encoder transmits at either maximum or minimum intensity, as dictated by the physical constraints of the cable. A dynamic programming formulation of the capacity is proposed, and the optimal binary signaling is determined via policy iteration. It is proved that the optimal signaling has smaller intensity than that given by the myopic policy , which greedily maximizes the instantaneous information rate but neglects its effect on the steady-state cable distribution. In contrast, the optimal scheme balances the tension between achieving high instantaneous information rate, and inducing a favorable steady-state distribution, such that those states characterized by high information rates are visited more frequently, thus revealing the importance of CSI. This work represents a first contribution towards the design of electron signaling schemes in complex microbial communities, e.g., bacterial cables and biofilms, where the tension between maximizing the transfer of information and guaranteeing the well-being of the overall bacterial community arises, and motivates further research on the design of more practical schemes, where CSI is only partially available.

Deterministic single exponential time algorithms for connectivity problems parameterized by treewidth

It is well known that many local graph problems, like Vertex Cover and Dominating Set, can be solved in time 2O(tw)|V|O(1) for graphs G=(V,E) with a given tree decomposition of width tw. However, for nonlocal problems, like the fundamental class of connectivity problems, for a long time we did not know how to do this faster than twO(tw)|V|O(1). Recently, Cygan et al. (FOCS 2011) presented Monte Carlo algorithms for a wide range of connectivity problems running in time ctw|V|O(1) for a small constant c, e.g., for Hamiltonian Cycle and Steiner Tree. Naturally, this raises the question whether randomization is necessary to achieve this runtime; furthermore, it is desirable to also solve counting and weighted versions (the latter without incurring a pseudo-polynomial cost in the runtime in terms of the weights).We present two new approaches rooted in linear algebra, based on matrix rank and determinants, which provide deterministic ctw|V|O(1) time algorithms, also for weighted and counting versions. For example, in this time we can solve Traveling Salesman or count the number of Hamiltonian cycles. The rank based ideas provide a rather general approach for speeding up even straightforward dynamic programming formulations by identifying “small” sets of representative partial solutions; we focus on the case of expressing connectivity via sets of partitions, but the essential ideas should have further applications. The determinant-based approach uses the Matrix Tree Theorem for deriving closed formulas for counting versions of connectivity problems; we show how to evaluate those formulas via dynamic programming.

Optimization of a Stormwater Quality Management Pond System

Stormwater system management in urban watersheds is often achieved by effective design and use of detention ponds which help mitigate impacts of urban drainage from water quantity and quality perspectives. The costs associated with these tasks for management can be minimized considering the fulfillment of objectives of environmental and regulatory compliance. Nonlinear and mixed integer nonlinear programming (MINLP) formulations with discrete and binary variables is developed in this study to obtain an optimal design for a multiple stormwater detention pond system. The main objective considered is the minimization of cost constrained on system performance related to pollution control. Analytical probabilistic expressions in mathematically closed form for system performance depicting watershed hydrology, control system hydraulics and pollution removal processes are integrated into the optimization formulations. Gradient-based NLP and genetic algorithm-based solvers are used to obtain optimal solutions. Application of the methodology is demonstrated with a hypothetical case study system with realistic hydrologic and water quality parameter values and the benefits of solutions for effective pollution control are reported. Results from the solutions of the formulations provide optimal design parameters considering the runoff control and pollutant reduction considering environmental and regulatory constraints. A comparison of results from these formulations to those from a dynamic programming (DP) formulation developed in an earlier study indicates that limitations associated with discretization within DP can be overcome with the proposed optimal formulations.

Understanding Quota Dynamics in Wireless Networks

In designing new service plans, network service providers need to understand how consumption of voice or data service will change in response to pricing signals. It is difficult to acquire such information from customer usage data because voice minutes and data bandwidth are typically sold in the form of large quotas. We address this issue by studying how end-users consume their quotas, both in a prepaid setting (where users pay in advance and refill as needed) and a postpaid setting (where users pay each month for a fixed amount of quota). Our presentation has three main parts. In the first we present data on quota usage for prepaid voice/text services and show that users reduce their voice usage when their balances become low. Moreover, when balances are low there is a tendency to shift from voice to SMS. In the second part, we provide descriptive models of both prepaid and postpaid services. The main feature of these models is that there is a background level of potential demand and the rate at which this potential demand is realized depends on the amount of quota balance available. In the third part, we propose utility maximizing models that can account for this type of behavior. In the prepaid case the main feature of the model is a discount function that represents the perceived cost to the user of a quota refill that will occur sometime in the future. In the postpaid case, where the end-user is attempting to get the maximum amount of utility from his monthly quota, we present a dynamic programming formulation in which utility functions are time varying and not known to the user in advance.