Dynamic Programming Formulation Research Articles

Maximum probabilistic all-or-nothing paths

We consider the problem of a maximum probabilistic all-or-nothing network path. Each arc is associated with a profit and a probability and the objective is to select a path with maximum value for the product of probabilities multiplied by the sum of arc profits. The problem can be motivated by applications including serial-system design or subcontracting of key project activities that may fail. When subcontracting such critical success activities, each must be completed on time, according to the specs, and in a satisfactory manner in order for the entire project to be deemed successful. We develop a dynamic programming (DP) method for this problem in the acyclic graph setting, under an independence assumption. Two different fully-polynomial approximation schemes are developed based on the DP formulations, one of which applies repeated rounding and scaling to the input data, while the other uses only rounding. In experiments we compare the DP approach with mixed-integer nonlinear programming (MINLP) using a branch-and-cut method, on synthetic randomly generated instances as well as realistic ones.

Comprehensive Analysis and Optimal Configurations of the EVT Powertrain

This paper investigates the design of a series-parallel hybrid electric powertrain that includes an electric variable transmission (EVT) and two mechanical gears that connect the EVT to the rest of the powertrain. The EVT is implemented as a dual rotor electric machine (EM), which from a design perspective can be studied as two separate EMs. The goal is to optimize the size of the two EMs and the two mechanical gears, while minimizing fuel consumption, subject to performance requirements on top vehicle speed and acceleration time. A practical optimization methodology is proposed that decouples the multi-objective problem into three different control subproblems: a slow dynamic subproblem for fuel economy, a fast dynamic subproblem for acceleration performance and a static optimization subproblem for the top speed performance. Efficient dynamic programming formulations are presented to solve the two dynamic subproblems. Two different scenarios are discussed and analyzed, where the first one presents the influence of the EMs’ sizes on the vehicle performance and the second one delivers near optimal configurations of the EVT powertrain for different gear ratios. A case study is also carried out to compare the sizing results between the proposed method and an existing benchmark method, showing that the two EMs could be reduced by 45% and 11.87% respectively, while powertrain performance could still be maintained at the same level.

Control of M/Cox-2/s make-to-stock systems

This study considers a make-to-stock production system with multiple identical parallel servers, fixed production start-up costs and lost sales. Processing times are assumed to be two-phase Coxian random variables that allows us to model the systems having rework or remanufacturing operations. First, the dynamic programming formulation is developed and the structure of the optimal production policy is characterized. Due to the highly dynamic nature of the optimal policy, as a second contribution we propose an easy-to-apply production policy. The proposed policy makes use of the dynamic state information and controlled by only two parameters. We test the performance of the proposed policy at several instances and reveal that it is near optimal. We also assess the value of dynamic state information in general by comparing the proposed policy with the well-known static inventory position based policy.

Admission control in a two-class loss system with periodically varying parameters and abandonments

Motivated by service systems, such as telephone call centers and emergency departments, we consider admission control for a two-class multi-server loss system with periodically varying parameters and customers who may abandon from service. Assuming mild conditions for the parameters, a dynamic programming formulation is developed. We show that under the infinite horizon discounted problem, there exists an optimal threshold policy and provide conditions for a customer class to be preferred for each fixed time, extending stationary results to the non-stationary setting. We approximate the non-stationary problem by discretizing the time horizon into equally spaced intervals and examine how policies for this approximation change as a function of time and parameters numerically. We compare the performance of these approximations with several admission policies used in practice in a discrete-event simulation study. We show that simpler admission policies that ignore non-stationarity or abandonments lead to significant losses in rewards.

Dynamic pricing in airline revenue management

In this study, an exact Dynamic Programming formulation is developed for price-based revenue management in the airline industry. Structural properties of the optimal pricing policy are presented first. Then, the optimal policy is investigated for four different price-demand relationships and the characteristic properties of the airline demand are discussed. Performance of the proposed Dynamic Programming formulation is numerically compared with an alternative approximate approach based on the repeated use of a Mixed Integer Programming model and also with intuitive methods of temporal price discrimination. The numerical results show that significant revenue improvements are obtained by the proposed dynamic pricing methodology.

A newsvendor model with autocorrelated demand under a time-consistent dynamic CVaR measure

As a result of autocorrelation, static risk measures such as value at risk and Conditional Value at Risk (CVaR) are time inconsistent and can thus result in inconsistent decisions over time. In this article, we present a time-consistent dynamic CVaR measure and examine it in the context of a newsvendor problem with autocorrelated demand. Due to the concavity of our CVaR measure, the dynamic program formulation associated with our dynamic newsvendor problem is not immediately separable. However, by exploring certain properties of the dynamic CVaR measure and underlying profit function, our dynamic program can be transformed into a sequence of (single-period) risk-averse newsvendor problems that depend on the observed demand history. By examining the structure of the optimal order quantities, we find both intuitive and counterintuitive results. When demands are positively correlated, the optimal order quantity is monotonically increasing in the degree of risk aversion. However, when demands are negatively correlated and the underlying cost structure satisfies certain conditions, the optimal order quantity is no longer monotonically increasing in the degree of risk aversion. Instead, the optimal order quantity is a decreasing (increasing) function of the degree of risk aversion when it is below (above) a certain threshold. We also show that these results continue to hold when demands follow a general ARMA process, and when inventory carryover is considered.

Optimal Energy Management for a Mild Hybrid Vehicle With Electric and Hybrid Engine Boosting Systems

This paper investigates the utilization of two low-voltage hybrid mechanisms on fuel economy improvement of a vehicle with a downsized engine. The first mild hybrid system, configured with a motor, a planetary gear set, and a supercharger, permits switching between parallel hybrid operation and boosting but not both. This constrained flexibility offers additional degrees of freedom, yet it requires a highly optimized decision of which functionality, torque assist, or boost assist, in addition to how much energy to be used by each. The second mild hybrid system also supports supercharging and torque assist but with two separate motors, hence is more flexible. Both systems enable the use of a downsized engine as a competitive counterpart to widely used turbocharged and full hybrid electric powertrains. The optimal control problem for energy management of both configurations on a vehicle with an automatic transmission is formulated and solved using dynamic programming (DP) over standard drive cycles. The results indicate around 25% fuel consumption improvement compared to a baseline turbo-charged engine over the FTP75 cycle, which is around 80% of the full hybridization benefit. The supercharging versus torque assist share and its sensitivity to the drive cycle, battery size, and compressor efficiency, which influence the availability of electric energy and the losses of electrical and mechanical paths, is also investigated. Few causally implementable rules for selecting the single motor system operating mode are found by analysis of the DP solution, and their effectiveness is shown by incorporating into the DP formulation and comparing the resulting fuel consumption.

An Approximation Algorithm for Capacity Allocation Over a Single Flight Leg with Fare-Locking

In this paper, we study a revenue management model over a single flight leg, where the customers are allowed to lock an available fare. Each customer arrives into the system with an interest in purchasing a ticket for a particular fare class. If this fare class is available, the customer immediately purchases the ticket by paying the fare or locks the fare by paying a fee. If the customer locks the fare, then the airline reserves the capacity for the customer for a certain duration of time. At the end of this duration of time, the customer makes her ultimate purchase decision at the locked fare. The goal of the airline is to find a policy to decide which set of fare classes to make available at each time period to maximize the total expected revenue. Such fare locking options are commonly offered by airlines; the dynamic programming formulation of the revenue management problem with the option to lock an available fare has a high-dimensional state variable that keeps track of the locked fares. We develop an approximate policy that is guaranteed to obtain at least half of the optimal total expected revenue. Our approach is based on leveraging a linear programming approximation to decompose the problem by the seats on the flight and solving a dynamic program that separately controls the capacity on each seat. We also show that our results continue to hold when the airline makes pricing decisions instead of fare class availability decisions. Our numerical experiments show that the practical performance of our approximate policy is remarkably good compared to a tractable upper bound on the optimal total expected revenue. The online supplement is available at https://doi.org/10.1287/ijoc.2018.0816 .

Exact and heuristic approaches for joint maintenance and spare parts planning

In this study, we consider the joint problem of preventive replacement and spare parts inventory planning. We present an exact dynamic programming formulation to minimize the total expected cost over a finite planning horizon. As it is not possible to represent the optimal solution by a well-defined and practical policy, and the dynamic programming recursion is time-consuming to apply, we propose three heuristic approaches that are easy to understand and to implement in practice: (i) Steady-State Approximation, (ii) Stationary Policy and (iii) Myopic Approach. Through computational analyses, we investigate the effects of the model parameters on the performances of the proposed heuristics.

Managing a Hybrid RDC-DC Inventory System

In this paper, we study a hybrid RDC-DC serial inventory system where the RDC replenishes its stock from an outside supplier, while the DC faces random demand and replenishes its stock from the RDC. However, unlike in the traditional serial system, the DC itself can replenish its inventory from the outside supplier as well. We propose two simple and easy-to-implement policies for the system. The first policy, which we call the three-index policy, combines the characteristics of the echelon-base-stock policy for the serial system (Clark and Scarf 1960) and the dual-index policy for the dual-sourcing system (Veeraraghavan and Scheller 2008). We show that the order-up-to level of the DC from the RDC can be computed by a newsboy fractile. A simulation-based optimization procedure for the policy is provided. We then develop another policy (the ALP policy) based on the three-index policy and the multimodularity of the problem proved by Sapra (2017). The policy applies the linear programming approach to approximately solve the value function of the dynamic programming formulation of the problem. Our numerical results show that both the three-index policy and the ALP policy are comparable to the policy computed via dynamic programming, and that the former performs slightly better. In addition, the outside supply option of the DC can draw considerable cost savings under both policies. We also conduct a numerical study with problem parameters calibrated by the actual data from a consumer goods company in China to glen insights on the management of the system.

An LP based approximate dynamic programming model to address airline overbooking under cancellation, refund and no-show

In this paper we simultaneously address four constraints relevant to airline revenue management problem: flight cancellation, customer no-shows, overbooking, and refunding. We develop a linear program closely related to the dynamic program formulation of the problem, which we later use to approximate the optimal decision rule for rejecting or accepting customers. First, we give a novel proof that the optimal objective function of this linear program is always an upper bound for the dynamic program. Secondly, we construct a decision rule based on this linear program and prove that it is asymptotically optimal under certain circumstances. Finally, using Monte Carlo simulation, we demonstrate that, numerically, the result of the linear programming policy presented in this paper has a short distance to the upper bound of the optimal answer, which makes it a fairly good approximate answer to the intractable dynamic program.

Sequence Alignment on Directed Graphs.

Genomic variations in a reference collection are naturally represented as genome variation graphs. Such graphs encode common subsequences as vertices and the variations are captured using additional vertices and directed edges. The resulting graphs are directed graphs possibly with cycles. Existing algorithms for aligning sequences on such graphs make use of partial order alignment (POA) techniques that work on directed acyclic graphs (DAGs). To achieve this, acyclic extensions of the input graphs are first constructed through expensive loop unrolling steps (DAGification). Furthermore, such graph extensions could have considerable blowup in their size and in the worst case the blow-up factor is proportional to the input sequence length. We provide a novel alignment algorithm V-ALIGN that aligns the input sequence directly on the input graph while avoiding such expensive DAGification steps. V-ALIGN is based on a novel dynamic programming (DP) formulation that allows gapped alignment directly on the input graph. It supports affine and linear gaps. We also propose refinements to V-ALIGN for better performance in practice. With the proposed refinements, the time to fill the DP table has linear dependence on the sizes of the sequence, the graph, and its feedback vertex set. We conducted experiments to compare the proposed algorithm against the existing POA-based techniques. We also performed alignment experiments on the genome variation graphs constructed from the 1000 Genomes data. For aligning short sequences, standard approaches restrict the expensive gapped alignment to small filtered subgraphs having high similarity to the input sequence. In such cases, the performance of V-ALIGN for gapped alignment on the filtered subgraph depends on the subgraph sizes.

A Novel Statistical Cost Model and an Algorithm for Efficient Application Offloading to Clouds

This work presents a novel statistical cost model for applications that can be offloaded to cloud computing environments. The model constructs a tree structure, referred to as the execution dependency tree (EDT), to accurately represent various execution relations, or dependencies (e.g., sequential, parallel and conditional branching) among the application modules, along its different execution paths. Contrary to existing models that assume fixed average offloading costs, each module’s cost is modelled as a random variable described by its Cumulative Distribution Function (CDF) that is statistically estimated through application profiling. Using this model, we generalize the offloading cost optimization functions to those that use more user tailored statistical measures such as cost percentiles. We employ these functions to propose an efficient offloading algorithm based on a dynamic programming formulation. We also show that the proposed model can be used as an efficient tool for application analysis by developers to gain insights on the applications’ statistical performance under varying network conditions and users behaviours. Performance evaluation results show that the achieved mean absolute percentage error between the model-based estimated cost and the measured one for the application execution time can be as small as $5$ percent for applications with sequential and branching module dependencies.

Optimal insurance portfolios risk-adjusted performance through dynamic stochastic programming

The practical adoption of the Solvency II regulatory framework in 2016, together with increasing property and casualty (PC) claims in recent years and an overall reduction of treasury yields across more developed financial markets have profoundly affected traditional risk management approaches by insurance institutions. The adoption of firm-wide risk capital methodologies to monitor the companies’ overall risk exposure has further consolidated the introduction of risk-adjusted performance measures to guide the management medium and long-term strategies. Relying on a dynamic stochastic programming formulation of a 10 year asset-liability management (ALM) problem of a PC company, we analyse in this article the implications on capital allocation and risk-return trade-offs of an optimization problem developed for a global insurance company based on a pair of risk-adjusted return functions. The analysis is relevant for any institutional investor seeking a high risk-adjusted performance as for regulators in their structuring of stress-tests and effective regulatory frameworks. The introduction of the concept of risk capital, or economic capital, in the definition of medium and long term insurance strategies poses a set of modeling and methodological issues tackled in this article. Of particular interest is the study of optimal ALM policies under different assets’ correlation assumptions. From a computational viewpoint it turns out that, depending on the assumed correlation matrix, the stochastic program is linear or of second order conic type. A case study from a real-world company development is presented to highlight the effectiveness of applied stochastic programming in capturing complex risk and return dynamics arising in modern corporate finance and lead to an efficient long-term financial allocation process.

Optimal spare parts management for vessel maintenance scheduling

Condition-based monitoring is used as part of predictive maintenance to collect real-time information on the healthy status of a vessel engine, which allows for a more accurate estimation of the remaining life of an engine or its parts, as well as providing a warning for a potential failure of an engine part. An engine failure results in delays and down-times in the voyage of a vessel, which translates into additional cost and penalties. This paper studies a spare part management problem for maintenance scheduling of a vessel operating on a given route that is defined by a sequence of port visits. When a warning on part failure is received, the problem decides when and to which port each part should be ordered, where the latter is also the location at which the maintenance operation would be performed. The paper describes a mathematical programming model of the problem, as well as a shortest path dynamic programming formulation for a single part which solves the problem in polynomial time complexity. Simulation results are presented in which the models are tested under different scenarios.

Online spatio-temporal matching in stochastic and dynamic domains

Online spatio-temporal matching of servers/services to customers is a problem that arises at a large scale in many domains associated with shared transportation (e.g., taxis, ride sharing, super shuttles, etc.) and delivery services (e.g., food, equipment, clothing, home fuel, etc.). A key characteristic of these problems is that the matching of servers/services to customers in one stage has a direct impact on the matching in the next stage. For instance, it is efficient for taxis to pick up customers closer to the drop off point of the customer from the first stage of matching. Traditionally, greedy/myopic approaches have been adopted to address such large scale online matching problems. While they provide solutions in a scalable manner, due to their myopic nature, the quality of matching obtained can be improved significantly (demonstrated in our experimental results). In this paper, we present a multi-stage stochastic optimization formulation to consider potential future demand scenarios (obtained from past data). We then provide an enhancement to solve large scale problems more effectively and efficiently online. We also provide the worst-case theoretical bounds on the performance of different approaches. Finally, we demonstrate the significant improvement provided by our techniques over myopic approaches and two other multi-stage approaches from literature (Approximate Dynamic Programming and Hybrid Multi-Stage Stochastic optimization formulation) on three real world taxi data sets.

Stochastic dual dynamic integer programming

Multistage stochastic integer programming (MSIP) combines the difficulty of uncertainty, dynamics, and non-convexity, and constitutes a class of extremely challenging problems. A common formulation for these problems is a dynamic programming formulation involving nested cost-to-go functions. In the linear setting, the cost-to-go functions are convex polyhedral, and decomposition algorithms, such as nested Benders’ decomposition and its stochastic variant, stochastic dual dynamic programming (SDDP), which proceed by iteratively approximating these functions by cuts or linear inequalities, have been established as effective approaches. However, it is difficult to directly adapt these algorithms to MSIP due to the nonconvexity of integer programming value functions. In this paper we propose an extension to SDDP—called stochastic dual dynamic integer programming (SDDiP)—for solving MSIP problems with binary state variables. The crucial component of the algorithm is a new reformulation of the subproblems in each stage and a new class of cuts, termed Lagrangian cuts, derived from a Lagrangian relaxation of a specific reformulation of the subproblems in each stage, where local copies of state variables are introduced. We show that the Lagrangian cuts satisfy a tightness condition and provide a rigorous proof of the finite convergence of SDDiP with probability one. We show that, under fairly reasonable assumptions, an MSIP problem with general state variables can be approximated by one with binary state variables to desired precision with only a modest increase in problem size. Thus our proposed SDDiP approach is applicable to very general classes of MSIP problems. Extensive computational experiments on three classes of real-world problems, namely electric generation expansion, financial portfolio management, and network revenue management, show that the proposed methodology is very effective in solving large-scale multistage stochastic integer optimization problems.

Asset-liability management and goal-based investing for retail business

The industry of online personal financial services is expected over the next years to absorb an increasing share of households and individuals' savings and investment decisions with a parallel expansion of tailored decision tools and underlying methodological developments. In this article we present a dynamic stochastic optimisation model formulated to tackle a long-term optimal wealth management problem-based explicitly on the introduction of consumption and investment goals with a terminal inflation-adjusted retirement target. By embedding a goal-based investing philosophy in a dynamic framework we provide a reference modelling approach for increasingly popular households asset-liability management services. In a discrete setting we show that a dynamic stochastic programming formulation will lead to a highly realistic representation and solution of an otherwise hardly manageable optimisation problem and it is consistent with computer-aided decision support tools' operational requirements.

Stochastic Selection Problems with Testing

We study the problem of a decision-maker having to select one of many competing alternatives (e.g., choosing between projects, designs, or suppliers) whose future revenues are a priori unknown and modeled as random variables of known probability distributions. The decision-maker can pay to test each alternative to reveal its specific revenue realization (e.g., by conducting market research), and her goal is to maximize the expected revenue of the selected alternative minus the testing costs. This model captures an interesting trade-off between gaining revenue of a high-yield alternative and spending resources to reduce the uncertainty in selecting it. The combinatorial nature of the problem leads to a dynamic programming (DP) formulation with high-dimensional state space that is computationally intractable. By characterizing the structure of the optimal policy, we derive efficient optimal and near-optimal policies that are simple and easy-to-compute. In fact, these policies are also myopic -- they only consider a limited horizon of one test. Moreover, our policies can be described using intuitive `testing intervals' around the expected revenue of each alternative, and in many cases, the dynamics of an optimal policy can be explained by the interaction between the testing intervals of various alternatives.

Dual sourcing in the age of near-shoring: Trading off stochastic capacity limitations and long lead times

We model a periodic review inventory system with non-stationary stochastic demand, in which a manufacturer is procuring a component from two available supply sources. The faster supply source is modeled as stochastic capacitated with immediate delivery, while the slower supply source is modeled as uncapacitated with a longer fixed lead time. The manufacturer’s objective is to choose how the order should be split between the two supply sources in each period, where the slower supply source is used to compensate for the supply capacity unavailability of the faster supply source. This is different from the conventional dual-sourcing problem, and motivated by the new reality of near-shoring options. We derive the optimal dynamic programming formulation that minimizes the total expected inventory holding and backorder costs over a finite planning horizon and show that the optimal policy is relatively complex. We extend our study by developing an extended myopic two-level base-stock policy and we show numerically that it provides a very close estimate of the optimal costs. Numerical results reveal the benefits of dual-sourcing under near-shoring, where we point out that in most cases the manufacturer should develop a hybrid procurement strategy, taking advantage of both supply sources to minimize its expected total cost.