Dynamic Programming Formulation Research Articles

The single picker routing problem with scattered storage: modeling and evaluation of routing and storage policies

AbstractDespite ongoing automation efforts, most warehouses are still manually operated using a person-to-parts collection strategy. This process of collecting items of customer orders from different storage locations accounts for the majority of the operating costs of the warehouse. Hence, optimizing picker routes is an important instrument to reduce labor costs. We examine the scattered-storage variant of the single picker routing problem in a one-block parallel-aisle warehouse. With scattered storage, an article can be stored at several storage locations within the warehouse, whereas with classic storage, each article has a unique storage location. We use our recently published network-flow model with covering constraints that is based on an extension of the state space of the dynamic-programming formulation by Ratliff and Rosenthal. With modifications in the state graph, this model serves for both exact and all established heuristic routing methods for picker routing. The latter include traversal, return, largest gap, midpoint, and composite. We show that these routing policies can also be implemented through adaptations in the state space. Extensive computational studies highlight a comparison of the different routing and storage policies (in particular class-based storage policies) in the scattered storage context. Analyses demonstrate which combinations of policies are advantageous for the given warehouse layout. For class-based storage policies, we emphasize how the scattering of articles of different classes should be performed: scattering of C-articles is advantageous with reductions of up to 25%. In contrast, when articles are uniformly distributed, A-articles should be scattered.

Cooperative lane-changing for connected autonomous vehicles merging into dedicated lanes in mixed traffic flow

Connected and automated vehicles (CAVs) have enormous potential to enhance traffic safety, efficiency, and emissions reduction. However, in the initial phases of CAV development, mixed traffic comprising CAVs and human-driven vehicles (HDVs) will inevitably coexist in the traffic system. To fully exploit the benefits of CAVs, dedicated lanes with independent rights of way will be established. This paper proposes an optimal control strategy for coordinating the mandatory lane-changing of CAVs from ordinary lanes to dedicated lanes. The strategy develops a centralized two-stage cooperative optimal control model to optimize the lane-changing sequence and trajectories of CAVs. In the first stage, a dynamic programming formulation is designed to determine the lane-changing sequence decisions. The model predictive control (MPC) controller is adopted to dynamically solve the optimal control problem with a fixed terminal state. In the second stage, we dynamically and cooperatively designed the longitudinal trajectories of related CAVs. The lateral trajectories of lane-changing CAVs are planned with a cubic polynomial. The objective function considers driving comfort and state tracking to ensure traffic smoothness. Simulation results show that: (1) the proposed strategy can improve the negative impact of lane-changing behavior under different traffic demand levels. (2) Compared to the benchmark approach, the proposed strategy can significantly enhance traffic efficiency and driving comfort, particularly in medium-traffic demand. The strategy can improve the average speed of CAVs by approximately 12 % and decrease the average acceleration by over 45 %. (3) The average fuel consumption is positively correlated with traffic demands and the difference in arrival speeds between lane-changing and dedicated lane CAVs. (4) The effectiveness of the strategy increases with the length of the lane-changing segment. However, the marginal benefit becomes negligible when the segment exceeds 300 m. Therefore, the findings of this paper can provide theoretical support for the cooperative control of CAVs in dedicated lanes of highways in the future.

Unifying duplication episode clustering and gene-species mapping inference

We present a novel problem, called MetaEC, which aims to infer gene-species assignments in a collection of partially leaf-labeled gene trees labels by minimizing the size of duplication episode clustering (EC). This problem is particularly relevant in metagenomics, where incomplete data often poses a challenge in the accurate reconstruction of gene histories. To solve MetaEC, we propose a polynomial time dynamic programming (DP) formulation that verifies the existence of a set of duplication episodes from a predefined set of episode candidates. In addition, we design a method to infer distributions of gene-species mappings. We then demonstrate how to use DP to design an algorithm that solves MetaEC. Although the algorithm is exponential in the worst case, we introduce a heuristic modification of the algorithm that provides a solution with the knowledge that it is exact. To evaluate our method, we perform two computational experiments on simulated and empirical data containing whole genome duplication events, showing that our algorithm is able to accurately infer the corresponding events.

Erratum to “Competitive Two-Agent Scheduling and Its Applications”

Erratum to “Competitive Two-Agent Scheduling and Its Applications” The paper “Competitive Two-Agent Scheduling and Its Applications” by Leung, Pinedo, and Wan appeared in Operations Research in 2010. Because competitive agent scheduling has become an active area of research, the paper has garnered its share of citations over the years. This particular direction of scheduling research has become so popular that the book Multiagent Scheduling: Models and Algorithms by Agnetis, Billaut, Gawiejnowicz, Pacciarelli, and Soukhal appeared in 2014. However, one of the results in the Leung et al. paper had a minor error in it. The paper considers a two-agent scheduling problem with a single machine, with each one of the two agents having their own set of jobs. The first agent has as the objective the minimization of the total tardiness of her set of jobs, and the second agent has to ensure that the total completion time of her set of jobs is less than or equal to Q. Leung et al. showed, through a dynamic programming formulation, that this problem can be solved in pseudopolynomial time and determined the time complexity of their algorithm. Their proof contains a minor error. The proper algorithm is still pseudopolynomial but of a slightly higher time complexity.

An Auto-Tuned Robust Dispatch Strategy for Virtual Power Plants to Provide Multi-Stage Real-Time Balancing Service

To fully exploit the flexible potential of distributed energy resources (DERs) in providing balancing service to the power system, Virtual Power Plants (VPPs) act as control centers to conduct the optimal real-time dispatch of their managed DERs. This study investigates a VPP’s auto-tuned robust policy based on a multi-stage distributionally robust optimization model (DRO) in response to the uncertainties from both the setpoint of the top-level system operator (SO) and the outputs of renewable DERs. We propose a concise paradigm to reduce the complexity of the original large-scale optimization task. Specifically, we first cast the multi-stage DRO problem into a dynamic programming (DP) formulation and further simplify it to derive a single-stage convex optimization control policy (COCP) at each time stage. Further, an automatic update method based on implicit differentiation is employed to tune the parameters of COCP. Case studies show that this method ensures higher solution quality and faster convergence during training than conventional tuning methods. The proposed COCP outperforms other stochastic optimization techniques in terms of robustness, efficiency, and computational speed.

Performance Guarantees for Network Revenue Management with Flexible Products

Problem definition: We consider network revenue management problems with flexible products. We have a network of resources with limited capacities. To each customer arriving into the system, we offer an assortment of products. The customer chooses a product within the offered assortment or decides to leave without a purchase. The products are flexible in the sense that there are multiple possible combinations of resources that we can use to serve a customer with a purchase for a particular product. We refer to each such combination of resources as a route. The service provider chooses the route to serve a customer with a purchase for a particular product. Such flexible products occur, for example, when customers book at-home cleaning services but leave the timing of service to the company that provides the service. Our goal is to find a policy to decide which assortment of products to offer to each customer to maximize the total expected revenue, making sure that there are always feasible route assignments for the customers with purchased products. Methodology/results: We start by considering the case in which we make the route assignments at the end of the selling horizon. The dynamic programming formulation of the problem is significantly different from its analogue without flexible products as the state variable keeps track of the number of purchases for each product rather than the remaining capacity of each resource. Letting L be the maximum number of resources in a route, we give a policy that obtains at least [Formula: see text] fraction of the optimal total expected revenue. We extend our policy to the case in which we make the route assignments periodically over the selling horizon. Managerial implications: To our knowledge, the policy that we develop is the first with a performance guarantee under flexible products. Thus, our work constructs policies that can be implemented in practice under flexible products, also providing performance guarantees. Funding: The work of H. Topaloglu was partly funded by the National Science Foundation [Grant CMMI-1825406]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/msom.2022.0583 .

Optimal multi-period crop procurement and distribution policy with minimum support prices

Governments of many developing countries offer minimum earnings to farmers through crop Minimum support price (MSP), which guarantees minimum revenue to farmers who cultivate the crop. The crop procured by a government through MSP is later distributed among poor consumers at subsidized prices through public distribution systems (PDS). Thus, governments combine MSP with the PDS scheme to simultaneously ensure farmers’ welfare and tackle poor consumers’ hunger and nutrition problems.However, improper implementation of the schemes causes burgeoning stockpiles in government warehouses, which eventually causes a substantial increase in operational costs and food wastage. In this paper, we characterize the optimal joint MSP and PDS policy over a finite horizon for two objectives: (i) minimize the total cost and (ii) maximize the net value after accounting for farmers’ benefit.Using the data over 15 years, we show that the optimal policy can substantially reduce operational costs (by about 85%) and improve the net value (by about 1.8 trillion INR). We use a linearly constrained quadratic stochastic dynamic program formulation to model the problem. We derive a simple and efficient joint MSP and PDS operational policy that provides the optimal quantity of crop that should be disbursed through PDS in the current period and the optimal MSP that should be announced for the subsequent period.Our policy is easy to compute and implement and can help policymakers efficiently implement MSP and PDS programs jointly. Our model provides a basic framework to solve the joint MSP and PDS problem and can be modified to incorporate other aspects through additional constraints.

Provably Good Solutions to the Knapsack Problem via Neural Networks of Bounded Size

The development of a satisfying and rigorous mathematical understanding of the performance of neural networks is a major challenge in artificial intelligence. Against this background, we study the expressive power of neural networks through the example of the classical NP-hard knapsack problem. Our main contribution is a class of recurrent neural networks (RNNs) with rectified linear units that are iteratively applied to each item of a knapsack instance and thereby compute optimal or provably good solution values. We show that an RNN of depth four and width depending quadratically on the profit of an optimum knapsack solution is sufficient to find optimum knapsack solutions. We also prove the following tradeoff between the size of an RNN and the quality of the computed knapsack solution: for knapsack instances consisting of n items, an RNN of depth five and width w computes a solution of value at least [Formula: see text] times the optimum solution value. Our results build on a classical dynamic programming formulation of the knapsack problem and a careful rounding of profit values that are also at the core of the well-known fully polynomial-time approximation scheme for the knapsack problem. A carefully conducted computational study qualitatively supports our theoretical size bounds. Finally, we point out that our results can be generalized to many other combinatorial optimization problems that admit dynamic programming solution methods, such as various shortest path problems, the longest common subsequence problem, and the traveling salesperson problem. History: Andrea Lodi, Area Editor for Design & Analysis of Algorithms–Discrete. An extended abstract of this article, including Figures 1 – 7 , appeared in the Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, 7685–7693 ( Hertrich and Skutella 2021 ); see https://ojs.aaai.org/index.php/AAAI/article/view/16939 ; copyright © 2021, Association for the Advancement of Artificial Intelligence. Funding: This work was supported by the Deutsche Forschungsgemeinschaft [Grants DFG-GRK 2434 and EXC-2046/1, Project 390685689] and the H2020 European Research Council [ScaleOpt-757481].

Optimal mechanism design of public policies for promoting electromobility: A dynamic programming formulation

Electromobility is considered a promising future of road transportation, driving the current industrial and market shift to the era of Electric Vehicles (EVs). As electromobility’s ecosystem depends on electricity generated by the energy sector, decision-makers face the tedious task of enacting policies that alter the transport and energy sectors simultaneously to promote EVs. This paper proposes an Optimal Mechanism Design framework for planning the shift to electromobility from the urban level. The framework fuses concepts of Mechanism Design, optimization methods and Cost-Benefit Analysis (CBA), incorporating all important elements for a comprehensive and sound policy-making procedure, such as market dynamics and the associated financial, social and technological dimensions for the transport and energy sectors. The policies considered incorporate EV promotion incentives, taxation, charging infrastructure, electricity taxation and infrastructure such as power plants and Renewable Energy Sources (RES) to support the growing energy needs. Market mechanisms comprise direct links between policy actions and sustainability outcomes. The framework identifies and optimizes these mechanisms to maximize the CBA sustainability indicators. The framework is applied to Nicosia, Cyprus’s capital, where accelerated EV adoption appears to maximize the transport sector's socioenvironmental sustainability and the energy sector's financial sustainability. On the other hand, delayed EV adoption maximizes the energy sector's socioenvironmental sustainability and the transport sector's financial sustainability. Taxation policies are used to limit EV penetration, whilst funding incentives are used to accelerate EV adoption. The results provide decision-makers with a new tool for optimally planning EV adoption using a policy mixture that optimizes financial and socioenvironmental performance.

Learning-based importance sampling via stochastic optimal control for stochastic reaction networks

We explore efficient estimation of statistical quantities, particularly rare event probabilities, for stochastic reaction networks. Consequently, we propose an importance sampling (IS) approach to improve the Monte Carlo (MC) estimator efficiency based on an approximate tau-leap scheme. The crucial step in the IS framework is choosing an appropriate change of probability measure to achieve substantial variance reduction. This task is typically challenging and often requires insights into the underlying problem. Therefore, we propose an automated approach to obtain a highly efficient path-dependent measure change based on an original connection in the stochastic reaction network context between finding optimal IS parameters within a class of probability measures and a stochastic optimal control formulation. Optimal IS parameters are obtained by solving a variance minimization problem. First, we derive an associated dynamic programming equation. Analytically solving this backward equation is challenging, hence we propose an approximate dynamic programming formulation to find near-optimal control parameters. To mitigate the curse of dimensionality, we propose a learning-based method to approximate the value function using a neural network, where the parameters are determined via a stochastic optimization algorithm. Our analysis and numerical experiments verify that the proposed learning-based IS approach substantially reduces MC estimator variance, resulting in a lower computational complexity in the rare event regime, compared with standard tau-leap MC estimators.

Robust Scheduling with Temporal Decomposition of Integrated Electrical-Heating System Based on Dynamic Programming Formulation

To realize the potential of the integrated electrical-heating systems (IEHS) for coping with the uncertain renewable energy, a reserve scheduling model based on the dynamic programming transformed multi-stage adaptive robust optimization (DMRO) is proposed in this paper. The DMRO model enforces the non-anticipativity of scheduling process via temporal decomposed framework, and enhances the formulation flexibility for time-coupled equipment, such as energy storage systems (ESS). The whole scheduling model is split into two problems under the decomposition constructure, which consists of a reserve interval generation (RG) problem for making the unit commitment and systems' reserve plan, and a multi-stage economic operation (EO) problem aiming with simulating the worst-case scheduling process. To obtain the global optimal, a nested Benders decomposition algorithm is employed to solve the DMRO model. Case studies on the 6-Bus-5-node IEHS and the 118-Bus-24-node IEHS demonstrate the effectiveness of the proposed model and the solution methodology.

Metamodeling the optimal total revenues of the short-term optimization of a hydropower cascade under uncertainty

This paper deals with the optimization of the short-term production planning of a real cascade of run-of-river hydropower plants. Water inflows and electricity prices are subject to data uncertainty and they are modeled by a finite set of joint scenarios. The optimization problem is written with a two-stage stochastic dynamic mixed-integer linear programming formulation. This problem is solved by replacing the value function of the second stage with a surrogate model. We propose to evaluate the feasibility of fitting the surrogate model by supervised learning during a pre-processing step. The learning data set is constructed by Latin hypercube sampling after discretizing the functional inputs. The surrogate model is chosen among linear models and the dimension of the functional inputs is reduced by principal components analysis. Validation results for one simplified case study are encouraging. The methodology could however be improved to reduce the prediction errors and to be compatible with the time limit of the operational process.

Hessian chain bracketing

Second derivatives of mathematical models for real-world phenomena are fundamental ingredients of a wide range of numerical simulation methods including parameter sensitivity analysis, uncertainty quantification, nonlinear optimization and model calibration. The evaluation of such Hessians often dominates the overall computational effort. The combinatorial {\sc Hessian Accumulation} problem aiming to minimize the number of floating-point operations required for the computation of a Hessian turns out to be NP-complete. We propose a dynamic programming formulation for the solution of {\sc Hessian Accumulation} over a sub-search space. This approach yields improvements by factors of ten and higher over the state of the art based on second-order tangent and adjoint algorithmic differentiation.

Single item periodic review inventory control with sales dependent stochastic return flows

Retailers have to deal with increasing levels of product returns as the shares of e-commerce sales soars. With this increase, it is no longer feasible to dispatch returned products to outlets or landfills, hence retailers must re-evaluate them both to maximize profit and to minimize their environmental impact. Our objective is to study a retailer’s optimal inventory control policy under product returns to maximize expected profit which is the sales revenue minus the procurement, backorder, holding, and salvage costs incurred in a finite horizon. We model a period’s returns to be stochastically dependent on the previous period’s sales quantity. Using dynamic programming formulation, we solve for the optimal periodic review inventory policy and provide structural results on the optimal policy of the final period. Through numerical studies, we show that incorporating detailed sales-dependent returns could increase a retailer’s expected profit by 23%. Ignoring this dependency in determining the optimal inventory policy results with increased order frequency, higher levels of backorders and more leftovers which could eventually end up in a landfill, but above all could lead to a significant overestimation of the resulting profit.

Dynamic inventory decisions for humanitarian aid materials considering budget limitations

In case of disaster, providing relief supplies to the affected people has vital importance. Governmental or non-governmental organizations (NGOs) prepare for disasters by purchasing and stockpiling these aid materials in appropriate quantities. They operate under a limited budget and this budget can either be used before the disaster for stocking decisions under uncertainty or it can be used after the disaster to satisfy the required demand at a higher cost when the uncertainty is resolved. If all the budget is used before the disaster, and if there is no disaster in a long term, there will be a high holding cost. On the other hand, if all the budget is reserved for use after the disaster, meeting the demand will be more costly or the demand may not be met within a certain period of time. Thus, NGOs need to decide how to allocate the budget for pre and post-disaster usage. In this system, the budget of NGOs may also change over time through donations or other incomes. In this point of view, NGOs need to make dynamic stock and budget allocation decisions, under the available budget at hand. In our study, we analyze the dynamic stocking decisions of NGOs using stochastic dynamic programming formulations under budget constraints. We develop infinite horizon stochastic dynamic programming models with and without budget considerations, and compare the results of these models via numerical analysis. Detailed numerical studies and results of the sensitivity analysis show the significance of budget considerations in inventory decisions and the effects of different parameters on the system results.

A stochastic dual dynamic integer programming based approach for remanufacturing planning under uncertainty

ABSTRACT We seek to optimize the production planning of a three-echelon remanufacturing system under uncertain input data. We consider a multi-stage stochastic integer programming approach and use scenario trees to represent the uncertain information structure. We introduce a new dynamic programming formulation that relies on a partial nested decomposition of the scenario tree. We then propose a new approximate stochastic dual dynamic integer programming algorithm based on this partial decomposition. Our numerical results show that the proposed solution approach is able to provide near-optimal solutions for large-size instances with a reasonable computational effort.

Coordinated trajectory planning for lane-changing in the weaving areas of dedicated lanes for connected and automated vehicles

Connected and automated vehicle (CAV) technology is widely recognized as a promising way to improve traffic safety, fuel efficiency and network throughput. However, at the early stage of CAV adoption, there will inevitably be a long period when both CAVs and regular vehicles coexist in the traffic system. To facilitate CAV functionalities in mixed traffic, providing dedicated rights of way for CAVs is necessary. In such scenarios, traffic control requires careful design in the weaving areas that allow CAVs to transfer between dedicated roads and ordinary roads for both safety and efficiency considerations. This paper proposes a dynamic programming approach for the coordinated trajectory planning of CAVs in mandatory lane-changing scenarios, in which CAVs have to switch from a dedicated CAV lane into parallel ordinary lanes with mixed traffic. For a group of CAVs that are poised to enter the ordinary road, the proposed method aims to provide each CAV with a suggested lane-changing gap and a reference trajectory in a coordinated manner. To solve the problem, we establish a dynamic programming formulation to determine the suggested gaps for all CAVs. We formulate a nonconvex quadratic-constrained quadratic programming (QCQP) problem to characterize the trajectory determination problem, and a relaxation-based approximate algorithm is developed to solve the QCQP. Simulation results show that, compared to the benchmark methods, the proposed method offers substantial benefits both in terms of traffic efficiency and in-vehicle passengers experiences.

Hydrogen and the decarbonization of the energy system in europe in 2050: A detailed model-based analysis

The paper aims to address the potential of low-carbon and renewable hydrogen in decarbonizing the European energy system; specifically, reducing emissions by 55% in 2030 compared to 1990, and targeting net-zero emissions by 2050. The methodology relies on a cost-optimization modelling approach using three models complementarily: a detailed European TIMES-type model (MIRET-EU); an aggregated model for the European energy system, allowing endogenous cost reductions based on technology deployment in a dynamic programming formulation for investment strategies (Integrate Europe); and a dedicated model for assessing hydrogen import options for Europe (HyPE). Two policy-relevant scenarios have been developed: Technology Diversification (TD) and Renewable Push (RP). Both lead to climate neutrality in Europe in 2050 but the RP scenario differs by setting new reinforced targets for renewable technologies in Europe. Results show that hydrogen production would increase sharply in the coming decades, exceeding 30 million tons (Mt) by 2030 and more than 100 Mt by 2050 in both scenarios. Polyvalence of hydrogen in decarbonizing the European energy system for certain hard-to-abate energy uses in transport and industry is also observed. European hydrogen production relies on a diverse mix including both renewable and low-carbon technologies. It is complemented by hydrogen imports from neighboring regions, that represent between 10 and 15% of total demand in 2050. Access to existing cross-border pipelines is a critical advantage compared to maritime transport. Notably, there are considerable cost reductions due to technology deployment for solar power and hydrogen production by electrolyzers.

On the optimization of pit stop strategies via dynamic programming

Pit stops are a key element of racing strategy in several motor sports. Typically, these stops involve decisions such as in which laps to stop, and which type of tire, of three possible compounds, to set at each of these stops. There are several factors that increase the complexity of the task: the impact of lap times depending on the tire compound, the wear of the tires, unexpected events on the track such as safety cars and the weather, among others. This work presents a Dynamic Programming formulation that addresses the pit-stop strategy problem in order to optimize the laps in which to stop, and the tire changes that minimize the total race time. We show the relative performance of the optimal strategies for starting with tires of different compounds with different yellow-flag scenarios. Then, we extend the Dynamic Program (DP) to a Stochastic Dynamic Programming (SDP) formulation that incorporates random events such as yellow flags or rainy weather. We are able to visualize and compare these optimal pit-stop strategies obtained with these models in different scenarios. We show that the SDP solution, compared to the DP solution, tends to delay pit stops in order to benefit from a possible yellow flag. Finally, we show that the SDP outperforms the DP, especially in races in which yellow flags are likely to be waved more frequently.

Convexity and Feedback in Approximate Dynamic Programming for Delivery Time Slot Pricing

We consider the revenue management problem of finding profit-maximizing prices for delivery time slots in the context of attended home delivery. This multistage optimal control problem admits a dynamic programming (DP) formulation that is intractable for realistic problem sizes due to the so-called “curse of dimensionality.” We therefore study three approximate DP algorithms both from a numerical and control-theoretical perspective. Our analysis is based on real-world data, from which we generate multiple scenarios to stress-test the robustness of the pricing policies to errors in model parameter estimates. Our theoretical analysis and numerical benchmark tests indicate that one of these algorithms, namely gradient-bounded DP, dominates the others with respect to computation time and profit-generation capabilities of the pricing policies that it generates.