Exact Dynamic Programming Research Articles

Scheduling of e-commerce packaging machines: blocking machines and their impact on the performance–waste tradeoff

AbstractTo streamline their fulfillment processes, many e-commerce retailers today use automated packaging machines for their outbound parcels. An important performance–waste tradeoff is associated with these machines: To reduce packaging waste when handling different sized goods, packaging machines should be able to handle different carton sizes. However, more carton sizes lead to a more involved scheduling process, so that the throughput performance deteriorates (and vice versa). To investigate this tradeoff, this paper develops scheduling procedures for a specific type of packaging machine, called blocking machines. These packaging machines provide multiple back-to-back packaging devices, each continuously processing a dedicated carton size, but blocking each other whenever incoming goods are not properly ordered according to carton sizes on the infeed conveyor. To reduce the resulting throughput loss, we derive various scheduling problems for optimizing the inflow of goods, provide a thorough analysis of the computational complexity, and derive an exact dynamic programming approach that is polynomial in the number of orders to be packed. This allows us to solve even large real-world instances to proven optimality with which we can analyze the performance–waste tradeoff of blocking machines.

Ms.FPOP: a fast exact segmentation algorithm with a multiscale penalty

Given a time series in R n with a piecewise constant mean and independent noises, we propose an exact dynamic programming algorithm for minimizing a least-squares criterion with a multiscale penalty, favoring well-spread changepoints. This penalty was proposed by Verzelen et al. (2023) and achieves optimal rates for changepoint detection and changepoint localization in a non-asymptotic scenario. Our proposed algorithm, Multiscale Functional Pruning Optimal Partitioning (Ms.FPOP), extends functional pruning ideas presented in Rigaill (2015) and Maidstone et al. (2017) to multiscale penalties. For large signals ( n ≥ 10 5 ) with sparse changepoints, Ms.FPOP is shown empirically to be quasi-linear and faster than the Pruned Exact Linear Time (PELT) method of Killick et al. (2012) applied to the multiscale penalty of Verzelen et al. (2023), which exhibits quadratic slowdown in these cases. We propose an efficient implementation of Ms.FPOP coded in C++ interfaced with R that can segment profiles of up to n = 10 6 in a matter of seconds. Our algorithm works for slightly more general multiscale penalties. In particular, it allows a minimum segment length to be imposed. Using simple simulations we then show that where profiles are sufficiently large ( n ≥ 10 4 ), Ms.FPOP using the multiscale penalty of Verzelen et al. (2023) is typically more powerful than optimizing a least-squares criterion with the BIC penalty of Yao (1989), a criterion that was shown by Fearnhead and Rigaill 2020 to perfom well across a wide range of scenarios. Supplementary materials for this article are available online.

Dynamic programming with meta-reinforcement learning: a novel approach for multi-objective optimization

Multi-objective optimization (MOO) endeavors to identify optimal solutions from a finite array of possibilities. In recent years, deep reinforcement learning (RL) has exhibited promise through its well-crafted heuristics in tackling NP-hard combinatorial optimization (CO) problems. Nonetheless, current methodologies grapple with two key challenges: (1) They primarily concentrate on single-objective optimization quandaries, rendering them less adaptable to the more prevalent MOO scenarios encountered in real-world applications. (2) These approaches furnish an approximate solution by imbibing heuristics, lacking a systematic means to enhance or substantiate optimality. Given these challenges, this study introduces an overarching hybrid strategy, dynamic programming with meta-reinforcement learning (DPML), to resolve MOO predicaments. The approach melds meta-learning into an RL framework, addressing multiple subproblems inherent to MOO. Furthermore, the precision of solutions is elevated by endowing exact dynamic programming with the prowess of meta-graph neural networks. Empirical results substantiate the supremacy of our methodology over previous RL and heuristics approaches, bridging the chasm between theoretical underpinnings and real-world applicability within this domain.

ESKEMAP: exact sketch-based read mapping

BackgroundGiven a sequencing read, the broad goal of read mapping is to find the location(s) in the reference genome that have a “similar sequence”. Traditionally, “similar sequence” was defined as having a high alignment score and read mappers were viewed as heuristic solutions to this well-defined problem. For sketch-based mappers, however, there has not been a problem formulation to capture what problem an exact sketch-based mapping algorithm should solve. Moreover, there is no sketch-based method that can find all possible mapping positions for a read above a certain score threshold.ResultsIn this paper, we formulate the problem of read mapping at the level of sequence sketches. We give an exact dynamic programming algorithm that finds all hits above a given similarity threshold. It runs in O(|t|+|p|+ℓ2)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {O} (|t| + |p| + \\ell ^2)$$\\end{document} time and O(ℓlogℓ)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {O} (\\ell \\log \\ell )$$\\end{document} space, where |t| is the number of k\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$k$$\\end{document}-mers inside the sketch of the reference, |p| is the number of k\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$k$$\\end{document}-mers inside the read’s sketch and ℓ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\ell$$\\end{document} is the number of times that k\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$k$$\\end{document}-mers from the pattern sketch occur in the sketch of the text. We evaluate our algorithm’s performance in mapping long reads to the T2T assembly of human chromosome Y, where ampliconic regions make it desirable to find all good mapping positions. For an equivalent level of precision as minimap2, the recall of our algorithm is 0.88, compared to only 0.76 of minimap2.

Rescheduling with New Orders Under Bounded Disruption

Rescheduling problems arise when unpredicted events occur, such as the arrival of new orders. These new jobs should be integrated in a proper way in the existing schedule of the so-called old jobs, with the aim of minimizing an objective function for the joint set of jobs. To avoid a major disruption of the original schedule, each old job is not allowed to deviate from its original completion time by more than a certain threshold. Filling a gap in the existing literature, we consider the minimization of the total weighted completion time. The resulting rescheduling problem is shown to be weakly NP-hard and several properties of the structure of an optimal schedule are derived. These can be used for the construction of an exact dynamic programming algorithm with pseudo-polynomial running time. A fully polynomial time approximation scheme is obtained from the dynamic program by three different scaling and reduction steps. Finally, for the minimization of the number of late jobs a strong NP-hardness result is derived. History: Accepted by Erwin Pesch, Area Editor for Heuristic Search & Approximation Algorithms. Funding: This work was partially supported by the Ministero dell’Istruzione, dell’Università e della Ricerca [Award TESUN-83486178370409 finanziamento dipartimenti di eccellenza CAP. 1694 TIT. 232 ART. 6]. U. Pferschy acknowledges support by the Field of Excellence COLIBRI at the University of Graz.

Exact algorithm and machine learning-based heuristic for the stochastic lot streaming and scheduling problem

This article presents a probabilistic variant of the classic Lot Streaming and Scheduling Problem (LSSP), in which the arrival times of products are stochastic. The LSSP involves a multi-product lot streaming problem and a sublot scheduling problem with a flow shop model and sequence-dependent setup times. Although the deterministic LSSP has been studied in the literature, the problem with stochastic arrival times of products has not been explored. In this article, we first derive some properties of the LSSP solution and propose closed-form expressions to compute the objective function of a given solution under three commonly used stochastic distributions. Based on these expressions, we develop a new exact Dynamic Programming (DP) algorithm and propose an efficient DP-based heuristic algorithm. Additionally, we build a machine learning model to predict whether a DP transition needs to be considered in the heuristic to improve its efficiency. Our computational study of test instances with various arrival time distributions shows that our algorithms can achieve promising results. Furthermore, we find that the machine learning model can simultaneously reduce the computational complexity and improve the algorithm’s accuracy.

Exact Dynamic Programming for Positive Systems with Linear Optimal Cost

Exact Dynamic Programming for Positive Systems with Linear Optimal Cost

Exact Sketch-Based Read Mapping.

Given a sequencing read, the broad goal of read mapping is to find the location(s) in the reference genome that have a "similar sequence". Traditionally, "similar sequence" was defined as having a high alignment score and read mappers were viewed as heuristic solutions to this well-defined problem. For sketch-based mappers, however, there has not been a problem formulation to capture what problem an exact sketch-based mapping algorithm should solve. Moreover, there is no sketch-based method that can find all possible mapping positions for a read above a certain score threshold. In this paper, we formulate the problem of read mapping at the level of sequence sketches. We give an exact dynamic programming algorithm that finds all hits above a given similarity threshold. It runs in time and space, where is the number of -mers inside the sketch of the reference, is the number of -mers inside the read's sketch and is the number of times that -mers from the pattern sketch occur in the sketch of the text. We evaluate our algorithm's performance in mapping long reads to the T2T assembly of human chromosome Y, where ampliconic regions make it desirable to find all good mapping positions. For an equivalent level of precision as minimap2, the recall of our algorithm is 0.88, compared to only 0.76 of minimap2.

Exact approaches for the unconstrained two-dimensional cutting problem with defects

This paper studies the unconstrained two-dimensional cutting problem with defects, which requires cutting a set of rectangular item types from a rectangular sheet with defects. The available number of each type is not limited. Two versions of the problem are studied, based on whether the guillotine cut constraint is required. Due to equipment limitations, the guillotine cutting restrictions are essential in specific industries, requiring cutting from one side to the other with each cut and splitting the sheet into two separate sheets. An integer model is proposed for the non-guillotine cut version, and an exact dynamic programming (DP) approach is proposed for the guillotine cut version. To reduce the number of vertical and horizontal positions, we extend the normal patterns, raster points, and meet-in-the-middle patterns to consider the effect of defects. The experimental results on standard benchmarks show that the proposed model can solve most of the instances in literature in a short time. For the guillotine cut version, our approach can significantly reduce the number of cut positions and improve the efficiency of the DP approach compared to the previous exact approach that considers all possible positions. Our approach solves the large instances with more than 300 item types in a relatively short time.

Pulsed Power Load Coordination in Mission- and Time-critical Cyber-physical Systems

Many mission- and time-critical cyber-physical systems deploy an isolated power system for their power supply. Under extreme conditions, the power system must process critical missions by maximizing the Pulsed Power Load (PPL) utility while maintaining the normal loads in the cyber-physical system. Optimal operation requires careful coordination of PPL deployment and power supply processes. In this work, we formulate the coordination problem for maximizing PPL utility under available resources, capacity, and demand constraints. The coordination problem has two scenarios for different use cases, fixed and general normal loads. We develop an exact pseudo-polynomial time dynamic programming algorithm for each scenario with a proven guarantee to produce an optimal coordination schedule. The performance of the algorithms is also experimentally evaluated, and the results agree with our theoretical analysis, showing the practicality of the solutions.

An exposition of least square Monte Carlo approach for real options valuation

The least square Monte Carlo simulation (LSM) approach is a state-of-the-art approach built upon approximate dynamic programming for the selection of single or multiple exercise options, and it has been extensively used for sequential decision-making and real options valuation. Although it has been broadly discussed and employed in many real-world applications, relatively less attention has been given to some of its implementation details. This paper aims to contribute to an improved understanding of sequential decision-making and real options valuation using the LSM algorithm. In this paper, we illustrate and argue the impact of only including the in-the-money (ITM) paths and the choice of regression functions for a specific example. A simple oil production problem with two common embedded options (shrink versus expands) has been utilized for the analysis.The analyses conducted in this article for the considered decision situation confirm that when at least one of the options (decision alternatives) is relevant to the out-of-the-money (OOTM) paths, it is crucial to consider all the paths to benefit from all the flexibilities. Furthermore, the choice of regression function is shown to have minor effect on the optimal strategies and the expected project value as long as the options are deeply ITM or OOTM. However, when the options are poorly ITM or OOTM, we might reach a different solution than what would have been achieved had we used an exact dynamic programming approach. In addition, excluding OOTM paths from the analysis adversely affects the option valuation part of the LSM for the current application, leading to lower project values. In contrast, the impact of path exclusion on regression performance is less emphasized. This work also verifies the robustness of the LSM approach for the selection of the polynomial order used for tuning the regression function. While some of the findings herein are problem-specific, a similar methodology can be used to evaluate the implementation details and refine problem settings in other real options problems.

A batch reinforcement learning approach to vacant taxi routing

The optimal routing of a single vacant taxi is formulated as a Markov Decision Process (MDP) problem to account for profit maximization over a full working period in a transportation network. A batch offline reinforcement learning (RL) method is proposed to learn action values and the optimal policy from archived trajectory data. The method is model-free, in that no state transition model is needed. It is more efficient than the commonly used online RL methods based on interactions with a simulator, due to batch processing and reuse of transition experiences.The batch RL method is evaluated in a large network of Shanghai, China with GPS trajectories of over 12,000 taxis. The training is conducted with two datasets: one is a synthetic dataset where state transitions are generated in a simulator with a postulated system dynamics model (Yu et al., 2019) whose parameters are derived from observed data; the other contains real-world state transitions extracted from observed taxi trajectories.The batch RL method is more computationally efficient, reducing the training time by dozens of times compared with the online Q-learning method. Its performance in terms of average profit per hour and occupancy rate is assessed in the simulator, against that of a baseline model, the random walk, and an upper bound, generated by the exact Dynamic Programming (DP) method based on the same system model of the simulator. The batch RL based on simulated and observed trajectories both outperform the random walk, and the advantage increases with the training sample size. The batch RL based on simulated trajectories achieves 95% of the performance upper bound with 30-minutes time intervals, suggesting that the model-free method is highly effective. The batch RL based on observed data achieves around 90% of the performance upper bound with 30-minute time intervals, due to the discrepancy between the training and evaluation environments, and its performance in the real world is expected to similarly good since the training and evaluation would be based on the same environment.

Energy efficient speed planning of electric vehicles for car-following scenario using model-based reinforcement learning

Eco-driving is a term used to refer to a strategy for operating vehicles so as to minimize energy consumption. Without any hardware changes, eco-driving is an effective approach to improving vehicle efficiency by optimizing driving behavior, particularly for autonomous vehicles. Several approaches have been proposed for eco-driving, such as dynamic programming, Pontryagin’s minimum principle, and model predictive control; however, it is difficult to control the speed of the vehicle optimally in various driving situations. This study aims to derive an eco-driving strategy for reducing the energy consumption of a vehicle in diverse driving situations, including road slopes and car-following scenarios. A reinforcement learning-based energy efficient speed planning strategy is proposed for autonomous electric vehicles, which learn an optimal control policy through a data-driven learning process. A model-based reinforcement learning algorithm is developed for the eco-driving strategy; based on domain knowledge of the vehicle powertrain, a battery energy consumption model and longitudinal dynamics model of the vehicle are approximated from the driving data and are used for reinforcement learning. The proposed algorithm is tested using a vehicle simulation, and is compared to a global optimal solution obtained using an exact dynamic programming method. The simulation results show that the reinforcement learning algorithm can adjust the speed of the vehicle by considering driving conditions such as the road slope and a safe distance from the leading vehicle while minimizing energy consumption. The reinforcement learning algorithm achieves a near-optimal performance of 93.8% relative to the dynamic programming result.

Transmission scheduling for multi-process multi-sensor remote estimation via approximate dynamic programming

In this paper, we consider a remote estimation problem where multiple dynamical systems are observed by smart sensors, which transmit their local estimates to a remote estimator over channels prone to packet losses. Unlike previous works, we allow multiple sensors to transmit simultaneously even though they can cause interference, thanks to the multi-packet reception capability at the remote estimator. In this setting, the remote estimator can decode multiple sensor transmissions (successful packet arrivals) as long as their signal-to-interference-and-noise ratios (SINR) are above a certain threshold. In this setting, we address the problem of optimal sensor transmission scheduling by minimizing a finite horizon discounted expected estimation error covariance cost across all systems at the remote estimator, subject to an average transmission cost. While this problem can be posed as a stochastic control problem, the optimal solution requires solving a Bellman equation for a dynamic programming (DP) problem, the complexity of which scales exponentially with the number of systems being measured and their state dimensions. In this paper, we resort to a novel Least Squares Temporal Difference (LSTD) Approximate Dynamic Programming (ADP) based approach to approximating the value function. More specifically, an off-policy based LSTD approach, named in short Enhanced-Exploration Greedy LSTD (EG-LSTD), is proposed. We discuss the convergence analysis of the EG-LSTD algorithm and its implementation. A Python based program is developed to implement and analyse the different aspects of the proposed method. Simulation examples are presented to support the results of the proposed approach both for the exact DP and ADP cases.

An exact dynamic programming approach to segmented isotonic regression

This paper proposes a polynomial-time algorithm to construct the monotone stepwise curve that minimizes the sum of squared errors with respect to a given cloud of data points. The fitted curve is also constrained on the maximum number of steps it can be composed of and on the minimum step length. Our algorithm relies on dynamic programming and is built on the basis that said curve-fitting task can be tackled as a shortest-path type of problem. Numerical results on synthetic and realistic data sets reveal that our algorithm is able to provide the globally optimal monotone stepwise curve fit for samples with thousands of data points in less than a few hours. Furthermore, the algorithm gives a certificate on the optimality gap of any incumbent solution it generates. From a practical standpoint, this piece of research is motivated by the roll-out of smart grids and the increasing role played by the small flexible consumption of electricity in the large-scale integration of renewable energy sources into current power systems. Within this context, our algorithm constitutes an useful tool to generate bidding curves for a pool of small flexible consumers to partake in wholesale electricity markets.

A deep reinforcement learning approach to seat inventory control for airline revenue management

Commercial airlines use revenue management systems to maximize their revenue by making real-time decisions on the booking limits of different fare classes offered in each of its scheduled flights. Traditional approaches—such as mathematical programming, dynamic programming, and heuristic rule-based decision models—heavily rely on external mathematical models of demand and passenger arrival, choice, and cancelation, making their performance sensitive to the accuracy of these model estimates. Moreover, many of these approaches scale poorly with increase in problem dimensionality. Additionally, they lack the ability to explore and “directly” learn the true market dynamics from interactions with passengers and adapt to changes in market conditions on their own. To overcome these limitations, this research uses deep reinforcement learning (DRL), a model-free decision-making framework, for finding the optimal policy of the seat inventory control problem. The DRL framework employs a deep neural network to approximate the expected optimal revenues for all possible state-action combinations, allowing it to handle the large state space of the problem. Multiple fare classes with stochastic demand, passenger arrivals, and booking cancelations have been considered in the problem. An air travel market simulator was developed based on the market dynamics and passenger behavior for training and testing the agent. The results demonstrate that the DRL agent is capable of learning the optimal airline revenue management policy through interactions with the market, matching the performance of exact dynamic programming methods. The revenue generated by the agent in different simulated market scenarios was found to be close to the maximum possible flight revenues and surpass that produced by the expected marginal seat revenue-b (EMSRb) method.

Reinforcement learning approach for resource allocation in humanitarian logistics

When a disaster strikes, it is important to allocate limited disaster relief resources to those in need. This paper considers the allocation of resources in humanitarian logistics using three critical performance indicators: efficiency, effectiveness and equity. Three separate costs are considered to represent these metrics, namely, the accessibility-based delivery cost, the starting state-based deprivation cost, and the terminal penalty cost. A mixed-integer nonlinear programming model with multiple objectives and multiple periods is proposed. A Q-learning algorithm, a type of reinforcement learning method, is developed to address the complex optimization problem. The principles of the proposed algorithm, including the learning agent and its actions, the environment and its states, and reward functions, are presented in detail. The parameter settings of the proposed algorithm are also discussed in the experimental section. In addition, the solution quality of the proposed algorithm is compared with that of the exact dynamic programming method and a heuristic algorithm. The experimental results show that the efficiency of the algorithm is better than that of the dynamic programming method and the accuracy of the algorithm is higher than that of the heuristic algorithm. Moreover, the Q-learning algorithm provides close to or even optimal solutions to the resource allocation problem by adjusting the value of the training episode K in practical applications.

Consensus of All Solutions for Intractable Phylogenetic Tree Inference.

Solving median tree problems is a classic approach for inferring species trees from a collection of discordant gene trees. Median tree problems are typically NP-hard and dealt with by local search heuristics. Unfortunately, such heuristics generally lack provable correctness and precision. Algorithmic advances addressing this uncertainty have led to exact dynamic programming formulations suitable to solve a well-studied group of median tree problems for smaller phylogenetic analyses. However, these formulations allow computing only very few optimal species trees out of possibly many such trees, and phylogenetic studies often require the analysis of all optimal solutions through their consensus tree. Here, we describe a significant algorithmic modification of the dynamic programming formulations that compute the cluster counts of all optimal species trees from which various types of consensus trees can be efficiently computed. Through experimental studies, we demonstrate that our parallel implementation of the modified dynamic programming formulation is more efficient than a previous implementation of the original formulation. Finally, we show that the parallel implementation can rapidly identify novel reassorted influenza A viruses potentially facilitating pandemic preparedness efforts.

Dynamic pricing and fleet management for electric autonomous mobility on demand systems

The proliferation of ride sharing systems is a major drive in the advancement of autonomous and electric vehicle technologies. This paper considers the joint routing, battery charging, and pricing problem faced by a profit-maximizing transportation service provider that operates a fleet of autonomous electric vehicles. We first establish the static planning problem by considering time-invariant system parameters and determine the optimal static policy. While the static policy provides stability of customer queues waiting for rides even if consider the system dynamics, we see that it is inefficient to utilize a static policy as it can lead to long wait times for customers and low profits. To accommodate for the stochastic nature of trip demands, renewable energy availability, and electricity prices and to further optimally manage the autonomous fleet given the need to generate integer allocations, a real-time policy is required. The optimal real-time policy that executes actions based on full state information of the system is the solution of a complex dynamic program. However, we argue that it is intractable to exactly solve for the optimal policy using exact dynamic programming methods and therefore apply deep reinforcement learning to develop a near-optimal control policy. The two case studies we conducted in Manhattan and San Francisco demonstrate the efficacy of our real-time policy in terms of network stability and profits, while keeping the queue lengths up to 200 times less than the static policy.

Iterated local search for single machine total weighted tardiness batch scheduling

This paper presents an iterated local search (ILS) algorithm for the single machine total weighted tardiness batch scheduling problem. To our knowledge, this is one of the first attempts to apply ILS to solve a batching scheduling problem. The proposed algorithm contains a local search procedure that explores five neighborhood structures, and we show how to efficiently implement them. Moreover, we compare the performance of our algorithm with dynamic programming-based implementations for the problem, including one from the literature and two other ones inspired in biased random-key genetic algorithms and ILS. We also demonstrate that finding the optimal batching for the problem given a fixed sequence of jobs is $$\mathcal {NP}$$ -hard, and provide an exact pseudo-polynomial time dynamic programming algorithm for solving such problem. Extensive computational experiments were conducted on newly proposed benchmark instances, and the results indicate that our algorithm yields highly competitive results when compared to other strategies. Finally, it was also observed that the methods that rely on dynamic programming tend to be time-consuming, even for small size instances.