Dynamic Programming Solution Research Articles

Representative applications of dynamic-programming in different fields: Analysis on problems of shortest-path, 0/1 knapsack and longest common sub-sequence

This article mainly focuses on exploring the current application status, concrete things implemented by dynamic programming and advantages of dynamic programming in problems (the concept and operation of dynamic programming and the role it plays in programs) in different fields, future development trends and areas for improvement. It also introduces the gaps and problems currently encountered by dynamic programming, then provides corresponding solutions to make it more helpful and efficient in the future. In this regard, by consulting different references and online resources written by previous scientists, it finds out that they all have different points of view on dynamic programming and different solutions to its shortcomings. In the current research, dynamic programming is the most widely used algorithm with few limitations that can achieve the global optimal solution at any time; however, it does not mean that it can achieve the local optimal solution at any time, which also indirectly results in the memory space it requires being larger than other optimization algorithms. After analyzing the literature and researching and testing multiple applications, it is obvious that dynamic programming can make the code run easier and faster than traditional methods, even though it will take up more space.

Cost optimization of electricity in energy storage system by dynamic programming

This paper presents a dynamic programming solution for the cost optimization of an electric storage system. The objective is to minimize the total cost of meeting electricity demand over a specified time interval, considering energy constraints and costs. The proposed algorithm efficiently determines the optimal energy discharge and charge strategies for the storage system, resulting in reduced overall costs. The effectiveness and efficiency of the algorithm are demonstrated through various test cases, highlighting its potential for real-world applications in energy storage systems and electric grid management. It also provides an overview of different types of electrical storage systems, review recent research on optimization techniques for energy storage, and examines recent studies on the optimization of electrical storage systems for specific applications, such as peak load shaving and grid stability. Through this comprehensive analysis, we hope to shed light on the current state of the field and identify areas for further research and improvement.

Optimized Dynamic Vehicle-to-Vehicle Charging for Increased Profit

Many challenges have arisen as a result of the rapid growth of the electric vehicles (EVs) market, due to the lack of charging infrastructure capable of handling such a large number of EVs. To alleviate power grid system overloads and reduce the cost of corresponding infrastructure deployments, a direct vehicle-to-vehicle (V2V) energy exchange strategy has become an emerging research topic. In this paper, we formulate the problem of V2V energy charging on a time–space network and develop a dynamic-programming solution methodology for efficiently finding the solution. The algorithm can pair and route the energy supplier (ES) and the requester (ER) in such a way that maximizes the supplier’s profit. Specifically, the ES is incentivized to rendezvous ERs at any encounter nodes in order to dispense the requested energy amount through platooning. Unlike existing V2V charging solutions, our approach involves charging while vehicles are in motion. We validate the effectiveness of our approach in maximizing the profit of the ES and reducing the incurred overhead on the ER in terms of increased trip time, distance, and energy consumption.

Large-W limit of the knapsack problem.

We formulate the knapsack problem (KP) as a statistical physics system and compute the corresponding partition function as an integral in the complex plane. The introduced formalism allows us to derive three statistical-physics-based algorithms for the KP: one based on the recursive definition of the exact partition function, another based on the large weight limit of that partition function, and a final one based on the zero-temperature limit of the second. Comparing the performances of the algorithms, we find that they do not consistently outperform (in terms of runtime and accuracy) dynamic programming, annealing, or standard greedy algorithms. However, the exact partition function is shown to reproduce the dynamic programming solution to the KP, and the zero-temperature algorithm is shown to produce a greedy solution. Therefore, although dynamic programming and greedy solutions to the KP are conceptually distinct, a statistical physics formalism introduced reveals that the large weight-constraint limit of the former leads to the latter. We conclude by discussing how to extend this formalism in order to obtain more accurate versions of the introduced algorithms and other similar combinatorial optimization problems.

Enacting algorithms: Evolution of the algorythmics storytelling

Visual storytelling, particularly through dance choreographies as showcased in previous AlgoRythmics performances, has been effective in communicating relatively straightforward algorithms in an engaging and memorable way. Nevertheless, when addressing complex algorithmic concepts, an approach with greater expressiveness and flexibility becomes necessary. Consequently, this study introduces stage performances as an innovative solution, using cinematic representation to successfully convey and communicate these intricate concepts and processes. To evaluate the effectiveness of this approach, a short film was designed, produced, and showcased to a second-semester CS2 university course audience studying programming techniques. Following an opening scene that establishes the context, the subsequent three acts vividly depict ad hoc, greedy, and dynamic programming solutions in response to the posed programming challenge. After the screening, a questionnaire was administered, built on four key constructs of the Technology Acceptance Model, as well as other potential facilitating factors. The study reveals 100% positive perceptions of educational benefits, with the vast majority of students expressing agreement regarding the utility, enjoyment, engagement, creativity, filmic quality, and cognitive benefits of short films. Additionally, a remarkable 96% reported the intent to utilize this approach. Our subsequent Structural Equation Modeling analysis discovered that students whose learning styles were in sync with this approach demonstrated a robust correlation between their perception of the method’s value, their enjoyment of the process, and their overall attitude towards this pedagogical method. This study confirms the potential of visual storytelling through short films as an effective tool for delivering programming education. The findings provide valuable insights for computer science educators seeking to engage learners and convey complex information in an attractive and effective way.

Dynamic programming solutions extracted SOC-trajectory online learning generation algorithm based approximate global optimization control strategy for a fuel cell hybrid electric vehicle

The trajectory of the battery state of charge (SOC) optimized by using dynamic programming (DP) is the global optimization solution to enhance the economy performance of the fuel cell hybrid electric vehicles under various driving cycles, however, this method requires prior knowledge of the future driving cycles. To utilize the solutions of DP, a SOC-trajectory online learning generation algorithm based approximate global optimization energy management control strategy is proposed. Initially, the global optimality of DP is used to extract the optimal SOC gradients for diverse driving scenarios. Real-time generation of optimal gradient factors for SOC trajectories is facilitated through the training of a backpropagation neural network with DP solutions. Subsequently, the deterministic rules are designed to plan SOC under actual driving conditions, with a dynamically updated threshold by the trained agents. Finally, based on the above, the optimal calculation of energy allocation is performed by combining sequence quadratic programming. Numerical verification, inclusive of hardware-in-the-loop experiments, show the effectiveness of the proposed strategy. The results demonstrate that the proposed strategy improves fuel economy by 7.39% compared to ECMS. Additionally, it reduces the cost of fuel cell life loss by 32.09% and achieves over 90% optimization of global driving cost.

Provably Good Solutions to the Knapsack Problem via Neural Networks of Bounded Size

The development of a satisfying and rigorous mathematical understanding of the performance of neural networks is a major challenge in artificial intelligence. Against this background, we study the expressive power of neural networks through the example of the classical NP-hard knapsack problem. Our main contribution is a class of recurrent neural networks (RNNs) with rectified linear units that are iteratively applied to each item of a knapsack instance and thereby compute optimal or provably good solution values. We show that an RNN of depth four and width depending quadratically on the profit of an optimum knapsack solution is sufficient to find optimum knapsack solutions. We also prove the following tradeoff between the size of an RNN and the quality of the computed knapsack solution: for knapsack instances consisting of n items, an RNN of depth five and width w computes a solution of value at least [Formula: see text] times the optimum solution value. Our results build on a classical dynamic programming formulation of the knapsack problem and a careful rounding of profit values that are also at the core of the well-known fully polynomial-time approximation scheme for the knapsack problem. A carefully conducted computational study qualitatively supports our theoretical size bounds. Finally, we point out that our results can be generalized to many other combinatorial optimization problems that admit dynamic programming solution methods, such as various shortest path problems, the longest common subsequence problem, and the traveling salesperson problem. History: Andrea Lodi, Area Editor for Design & Analysis of Algorithms–Discrete. An extended abstract of this article, including Figures 1 – 7 , appeared in the Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, 7685–7693 ( Hertrich and Skutella 2021 ); see https://ojs.aaai.org/index.php/AAAI/article/view/16939 ; copyright © 2021, Association for the Advancement of Artificial Intelligence. Funding: This work was supported by the Deutsche Forschungsgemeinschaft [Grants DFG-GRK 2434 and EXC-2046/1, Project 390685689] and the H2020 European Research Council [ScaleOpt-757481].

On scheduling of step-improving jobs to minimize the total weighted completion time

Kim et al. (2022) studied single-machine scheduling of step-improving jobs with a common discount factor to minimize the total weighted completion time. This study proposes a pseudo-polynomial algorithm to solve the problem. We then consider the general problem in which the discount factors are job dependent. It is proven that there is an optimal solution in which the early normal jobs are sequenced in the WSPT order and the late discounted jobs are sequenced in the WSPT order. Based on two WSPT lists, a dynamic programming solution algorithm is proposed. The run time is pseudo-polynomial when the distance of any job is bounded by a constant, where the distance of a job is the absolute difference of the two positions of this job in two WSPT-ordered job lists.

Numerical Strategies for Mixed-Integer Optimization of Power-Split and Gear Selection in Hybrid Electric Vehicles

This paper presents numerical strategies for a computationally efficient energy management system that co-optimizes the power split and gear selection of a hybrid electric vehicle (HEV). We formulate a mixed-integer optimal control problem (MIOCP) that is transcribed using multiple-shooting into a mixed-integer nonlinear program (MINLP) and then solved by nonlinear model predictive control. We present two different numerical strategies, a Selective Relaxation Approach (SRA), which decomposes the MINLP into several subproblems, and a Round-n-Search Approach (RSA), which is an enhancement of the known ‘relax-n-round’ strategy. Subsequently, the resulting algorithmic performance and optimality of the solution of the proposed strategies are analyzed against two benchmark strategies; one using rule-based gear selection, which is typically used in production vehicles, and the other using dynamic programming (DP), which provides a global optimum of a quantized version of the MINLP. The results show that both SRA and RSA enable about 3.6% cost reduction compared to the rule-based strategy, while still being within 1% of the DP solution. Moreover, for the case studied RSA takes about 35% less mean computation time compared to SRA, while both SRA and RSA being about 99 times faster than DP. Furthermore, both SRA and RSA were able to overcome the infeasibilities encountered by a typical rounding strategy under different drive cycles. The results show the computational benefit of the proposed strategies, as well as the energy saving possibility of co-optimization strategies in which actuator dynamics are explicitly included.

Stochastic dynamic programming solution to transmission scheduling: Multi sensor-multi process with wireless noisy channel

We investigate sensor scheduling for remote estimation when multiple smart sensors monitor multiple stochastic dynamical systems. The sensors transmit their measurements to a remote estimator through a noisy wireless communication channel. Such a remote estimator can receive multiple packets simultaneously sent by local sensors. Sensors transmit their measurements if their Signal Interference and Noise Ratio (SINR) is above a threshold. We compute the optimal policy for sensor scheduling by minimizing expected error covariance subject to total signal transmissions from all sensors. We model this problem as Markov Decision Process (MDP) with discounted cost per stage in the finite time horizon framework, then we employ stochastic Dynamic Programming as the optimization method. A novel algorithm based on sampling and machine learning techniques is proposed as the approximation. At each phase of the DP algorithm, samples are collected using a uniform probability distribution. The data is used to feed Neural Network (NN) and Random Forest (RF) models for cost function and policy approximation. The results of the proposed framework are supported by simulation examples comparing RF and NN as Approximate DP (ADP). Note that this idea builds a bridge among the recent advances in the area of data science, Machine Learning, and the ADP.

A Computationally Efficient Algorithm for Perturbed Dynamic Programs (A-PDP)

This letter proposes an algorithm for solving perturbed dynamic programs, where first-order corrections are applied to a pre-computed optimal strategy and its corresponding value function using local quadratic approximation. The technique is developed to handle perturbations of external inputs and parameters affecting system dynamics, objective and constraint functions, allowing the application to a wide variety of perturbed problems. The method is applied to the energy optimization of an electric vehicle thermal management system, formulated as a constrained dynamic program where external conditions and target set-point are perturbed from their nominal values. Simulations result show 80% reduction in computation time from a Dynamic Programming (DP) solution, with only minimal effect on the overall controller performance.

The Dynamic Freight Routing Problem for Less-Than-Truckload Carriers

Less-than-truckload (LTL) carriers transport freight shipments from origins to destinations by consolidating freight using a network of terminals. As daily freight quantities are uncertain, carriers dynamically decide freight routes on the day of operations. We introduce the dynamic freight routing problem (DFRP) and model this problem as a Markov decision process (MDP). To overcome the curses of dimensionality of the MDP model, we introduce an approximate dynamic programming (ADP) solution approach that uses a lookup table to store value function approximations and present and compares a number of aggregation approaches that use features of the postdecision states (PDSs) to aggregate the PDS space and reduce the number of entries in the lookup table. Furthermore, because the decision subproblems are integer programs (IPs), we present a framework for integrating lookup tables into the decision subproblem IPs. This framework consists of (1) a modeling approach for the integration of lookup table value function approximations into subproblem IPs to form extended subproblem IPs; (2) a solution approach, PDS-IP-bounding, which decomposes the extended subproblem IPs into many smaller IPs and uses dynamic bounds to reduce the number of small IPs that have to be solved; and (3) an adaptation of the ϵ-greedy exploration-exploitation algorithm for the IP setting. Our computational experiments show that despite the DFRP having high-dimensional PDS vectors, a two-dimensional aggregation of the space can produce policies that outperform standard myopic policies. Moreover, they demonstrate that the PDS-IP-bounding algorithm provides computational advantages over solving the extended subproblem IPs using a commercial solver. Funding: A. Baubaid acknowledges financial support from the King Fahd University of Petroleum & Minerals. Supplemental Material: The online appendices are available at https://doi.org/10.1287/trsc.2022.1190 .

A Practical End-to-End Inventory Management Model with Deep Learning

We investigate a data-driven multiperiod inventory replenishment problem with uncertain demand and vendor lead time (VLT) with accessibility to a large quantity of historical data. Different from the traditional two-step predict-then-optimize (PTO) solution framework, we propose a one-step end-to-end (E2E) framework that uses deep learning models to output the suggested replenishment amount directly from input features without any intermediate step. The E2E model is trained to capture the behavior of the optimal dynamic programming solution under historical observations without any prior assumptions on the distributions of the demand and the VLT. By conducting a series of thorough numerical experiments using real data from one of the leading e-commerce companies, we demonstrate the advantages of the proposed E2E model over conventional PTO frameworks. We also conduct a field experiment with JD.com, and the results show that our new algorithm reduces holding cost, stockout cost, total inventory cost, and turnover rate substantially compared with JD’s current practice. For the supply chain management industry, our E2E model shortens the decision process and provides an automatic inventory management solution with the possibility to generalize and scale. The concept of E2E, which uses the input information directly for the ultimate goal, can also be useful in practice for other supply chain management circumstances. This paper was accepted by Hamid Nazerzadeh, big data analytics. Funding: This research was supported by the National Key Research and Development Program of China [Grant 2018YFB1700600] and National Natural Science Foundation of China [Grants 71991462 and 91746210]. Supplemental Material: The online data are available at https://doi.org/10.1287/mnsc.2022.4564 .

A comprehensive toolbox for load retrieval in puzzle-based storage systems with simultaneous movements

Puzzle-based storage (PBS) is one of the most space-efficient types of storage systems. In a PBS unit, loads are stored in a grid of cells, where each cell may be empty or contain a load. A load can move only to adjacent empty cells. These cells are termed escorts in the literature, and their number is relatively small. When a load is requested for retrieval, a sequence of load movements is performed in order to bring it to an input/output (I/O) point of the unit. In this paper we minimize a weighted sum of two objectives. The first is the retrieval time of a requested item, and the second objective is the number of moves. While the first objective is associated with the quality of service, the second objective considers energy saving. Note that this operational problem should be solved quickly for every load request. The movement characteristics of loads in the PBS unit are determined by the technology used for its operation. In the most general and intricate case, simultaneous movements of blocks of loads can be performed, while simpler technology may allow only sequential movements or simultaneous movement of single loads. PBS units may also have one or several I/O points. In the latter case, the load may be retrieved via any one of the I/O points, based on the operator’s discretion. In this paper, we suggest a suite of complementary tools to solve load retrieval problems under various technology, including simultaneous block and load movements. First, we present a time-expanded-graph based integer linear-programming (ILP) formulation. This formulation provides optimal solutions for problems with a relatively large number of escorts and obtains lower bounds for other cases. For problems with a small number of escorts, we suggest a dynamic programming (DP) solution approach. A custom-made heuristic based on the DP approach was developed for the rest of the cases. Experiments show that our solution approaches’ ensemble yields optimal or near-optimal solutions to most of our benchmark instances.

Analysis of vascular robot optimization scheme based on dynamic planning and ARIMA model

The study of vascular robots has become a new research hotspot in the interdisciplinary field and is of pivotal importance to the field of minimally invasive therapy. In this paper, we focus on the development of the ordering scheme of vascular robot, establish the optimal ordering model of vascular robot based on the idea of dynamic programming, calculate the minimum operating cost of the ordering scheme through LINGO software programming, and provide a specific weekly ordering scheme for the hospital to order vascular robot. Firstly, the optimal ordering model of the vascular robot is established based on the idea of bottom-up solution of dynamic programming in order to solve the specific to weekly ordering scheme. Second, a mind map format is adopted to sort out the problem, which allows a clearer analysis of the topic conditions. Then, the correctness of the model solution is checked by using segmented programming verification, which enhances the persuasive power of the model. Finally, the principles of machine learning are combined to build.

Power Management of a Plug-in Hybrid Electric Vehicle Using Neural Networks with Comparison to Other Approaches

Many researchers spent much effort on the online power management strategies for plug-in hybrid vehicles (PHEVs) and hybrid electric vehicles (HEVs). Nowadays, artificial neural networks (ANNs), one of the machine learning techniques, have also been applied to this problem due to their good performance in learning non-linear and complicated multi-inputs multi-outputs (MIMO) dynamic systems. In this paper, an ANN is applied to the online power management for a plug-in hybrid electric vehicle (PHEV) by predicting the torque split between an internal combustion engine (ICE) and an electric motor (e-Motor) to optimize the greenhouse gas (GHG) emissions by using dynamic programming (DP) results as training data. Dynamic programming can achieve a global minimum solution while it is computationally intensive and requires prior knowledge of the entire drive cycle. As such, this method cannot be implemented in real-time. The DP-based ANN controller can get the benefit of using an ANN to fit the DP solution so that it can be implemented in real-time for an arbitrary drive cycle. We studied the hyper-parameters’ effects on the ANN model and different structures of ANN models are compared. The minimum training mean square error (MSE) models in each comparison set are selected for comparison with DP and equivalent consumption minimization strategy (ECMS). The total GHG emissions and state of charge (SOC) are the metrics used for the analysis and comparison. All the selected ANNs provide results that are comparable to the optimal DP solution, which indicates that ANNs are almost as good as the DP solution. It is found that the multiple hidden-layer ANN shows more efficiency in the training process than the single hidden-layer ANN. By comparing the results with ECMS, the ANN shows great potential in real-time application with the smallest deviation from the results of DP. In addition, our approach does not require any additional trip information, and its output (torque split) is more directly implementable on real vehicles.

A Multi-Depot Dynamic Vehicle Routing Problem with Stochastic Road Capacity: An MDP Model and Dynamic Policy for Post-Decision State Rollout Algorithm in Reinforcement Learning

In the event of a disaster, the road network is often compromised in terms of its capacity and usability conditions. This is a challenge for humanitarian operations in the context of delivering critical medical supplies. To optimise vehicle routing for such a problem, a Multi-Depot Dynamic Vehicle-Routing Problem with Stochastic Road Capacity (MDDVRPSRC) is formulated as a Markov Decision Processes (MDP) model. An Approximate Dynamic Programming (ADP) solution method is adopted where the Post-Decision State Rollout Algorithm (PDS-RA) is applied as the lookahead approach. To perform the rollout effectively for the problem, the PDS-RA is executed for all vehicles assigned for the problem. Then, at the end, a decision is made by the agent. Five types of constructive base heuristics are proposed for the PDS-RA. First, the Teach Base Insertion Heuristic (TBIH-1) is proposed to study the partial random construction approach for the non-obvious decision. The heuristic is extended by proposing TBIH-2 and TBIH-3 to show how Sequential Insertion Heuristic (SIH) (I1) as well as Clarke and Wright (CW) could be executed, respectively, in a dynamic setting as a modification to the TBIH-1. Additionally, another two heuristics: TBIH-4 and TBIH-5 (TBIH-1 with the addition of Dynamic Lookahead SIH (DLASIH) and Dynamic Lookahead CW (DLACW) respectively) are proposed to improve the on-the-go constructed decision rule (dynamic policy on the go) in the lookahead simulations. The results obtained are compared with the matheuristic approach from previous work based on PDS-RA.

Adaptive equivalent consumption minimization strategy for hybrid electric ship

AbstractIn recent years, with the development of battery technology, hybrid electric ship (HES), as a promising solution to reduce the fuel consumption and emissions, has become a research hotspot. However, frequent use of the battery will accelerate the aging of the battery, and the replacement of scrapped battery will increase the cost of the ship. Therefore, it is necessary to consider delaying battery aging into the energy control strategy of HES. The equivalent consumption minimization strategy (ECMS) is a feasible energy control strategy because it can be implemented in real time. However, under the condition of uncertain initial state of charge (SOC) of the battery, ECMS cannot effectively reduce the fuel consumption unless the equivalent factor (EF) is optimized in real time. In this paper, an adaptive equivalent consumption minimization strategy (A‐ECMS) is proposed, which extracts the global optimal EF trajectory from the dynamic programming (DP) solution and uses the back propagation (BP) neural network to adjust the EF in real time. A trade‐off between the fuel consumption and battery aging is made in the cost function by introducing a weight coefficient. Finally, the effectiveness and the adaptability of the proposed strategy are verified in MATLAB.

Control strategies with multiple closing instants for linear optimal control problems with disturbances*

This paper deals with an optimal control problem for a linear discrete system subject to input and state constraints and unknown bounded disturbances, where the control goal is to minimize a cost function used in linear explicit model predictive control. Since open-loop worst-case solution is conservative or may suffer feasibility problems and dynamic programming solution is computationally demanding, we propose a compromise between those solutions. This compromise is an optimal control strategy that takes into account the state measurements of the system at several future time instants (closing instants). We define control strategies with one and multiple closing instants and propose efficient numerical methods for their construction.

Optimal trade execution with uncertain volume target

In the seminal paper on optimal execution of portfolio transactions, Almgren and Chriss (2001) define the optimal trading strategy to liquidate a fixed volume of a single security under price uncertainty. Yet there exist situations, such as in the power market, in which the volume to be traded can only be estimated and becomes more accurate when approaching a specified delivery time. During the course of execution, a trader should then constantly adapt their trading strategy to meet their fluctuating volume target. In this paper, we develop a model that accounts for volume uncertainty and we show that a risk-averse trader has benefit in delaying their trades. More precisely, we argue that the optimal strategy is a trade-off between early and late trades in order to balance risk associated with both price and volume. By incorporating a risk term related to the volume to trade, the static optimal strategies suggested by our model avoid the explosion in the algorithmic complexity usually associated with dynamic programming solutions, all the while yielding competitive performance.