Dynamic Programming Approach Research Articles

Reinforcement learning versus data-driven dynamic programming: a comparison for finite horizon dynamic pricing markets

Abstract Revenue management (RM) plays a vital role to optimize sales processes in real-life applications under incomplete information. The prediction of consumer demand and the anticipation of price reactions of competitors became key factors in RM to be able to apply classical dynamic programming (DP) methods for expected long-term reward maximization. Modern model-free deep Reinforcement Learning (RL) approaches are able to derive optimized policies without explicit estimations of underlying model dynamics. However, RL algorithms typically require either vast amounts of training data or a suitable synthetic model to be trained on. As existing studies focus on one group of algorithms only, the relation between established DP approaches and new RL techniques is opaque. To address this issue, in this paper, we use a dynamic pricing framework for an airline ticket market to compare state-of-the-art RL algorithms and data-driven versions of classic DP methods regarding (i) performance and (ii) required data to each other. For the DP techniques, we use estimations of market dynamics to be able to compare their performance and data consumption against RL methods. The numerical results of our experiments, which include monopoly as well as duopoly markets, allow to study how the different approaches’ performances relate to each other in exemplary settings. In both setups, we find that with few data (about 10 episodes) fitted DP methods were highly competitive; with medium amounts of data (about 100 episodes) DP methods got outperformed by RL, where PPO provided the best results. Given large amounts of training data (about 1000 episodes), the best RL algorithms, i.e., TD3, DDPG, PPO, and SAC, performed similarly achieving about 90% and more of the optimal solution.

Optimizing Regulatory Floodways: Dynamic Programming Approach

Optimizing Regulatory Floodways: Dynamic Programming Approach

Value-at-risk constrained portfolios in incomplete markets: a dynamic programming approach to Heston’s model

Value-at-risk constrained portfolios in incomplete markets: a dynamic programming approach to Heston’s model

Availability-based maintenance prioritization for data centres: a dynamic programming approach

Digital infrastructures and Data Centres (DCs) have played a significant role in society for several years in conducting digital transformation. Recently after the global COVID-19 crisis, the importance of DCs has greatly increased because of facing new realities. High demand for digital commerce, online meetings, and other online and cloud services in the digitisation process has given the DC industry special attention. Regarding the critical components of DCs, special requirements, such as minimum availability, reliability, quality, and performance levels, should be addressed in DC operations. Therefore, implementing optimised maintenance management is beneficial for these infrastructures to reduce the risk of failure and ensure a minimum level of reliability and availability while minimising maintenance costs. This paper presents a novel, availability-based model for optimising maintenance prioritisation in DCs. The model leverages a Multiple 0-1 Knapsack problem, solved using Dynamic Programming (DP), to determine the optimal set of components for maintenance actions, considering both DC availability requirements and budget limitations. By incorporating reliability and availability analysis, the DP-based approach generates a maintenance prioritisation plan that maximises DC availability within budgetary constraints. This study provides a practical framework for enhancing maintenance decision-making in DCs, ultimately contributing to more efficient and resilient digital infrastructures.

Sufficient approximate optimality condition for the inverse one-phase Stefan problem

Abstract We investigate the identification problem for the one-phase Stefan problem. As the inverse Stefan problem is not well posed, an optimal control problem is considered instead. In the paper we develop a dual dynamic programming approach to derive sufficient approximate optimality conditions for that optimal control problem. As a next step we formulate and prove a verification theorem for approximate solution. The verification Theorem 4.1 is the basis for the development of a numerical algorithm. Having the verification theorem we do not need the convergence of our algorithm.

Optimal investment problem with multiple risky assets and correlation between risk model and financial market for an insurer under the CEV model

This article investigates the optimal investment problem with multiple risky assets and the correlation between risk model and financial market for an insurer. The insurer’s claim process is described by a Brownian motion with drift under the mean-variance premium principle. The insurer is allowed to invest in one risk-free asset and multiple risky assets whose price processes follow the constant elasticity of variance (CEV) model. Moreover, the correlation between the claim process and each risky asset’s price is taken into account. The insurer’s objective is to maximize the exponential utility of the terminal wealth. By applying dynamic programming approach, we propose a new form of the solution to the Hamilton-Jacobi-Bellman (HJB) equation and derive the optimal investment strategy explicitly. This is the first time that an explicit result of the optimal investment strategy is given under the framework of the exponential utility function for the general CEV model based on the correlation between the financial market and the insurance market. In addition, we provide some special cases of our model, i.e., the optimal investment problem with one risky asset and that with two risky assets. We also consider the case that the risk model and the financial market is independent. The results show that ignoring the correlation between the claim process and the risky asset’s price will misestimate the value of risky assets and has a significant effect on the insurer’s investment decision. Finally, numerical simulations are presented to analyze the effects of model parameters on the optimal investment strategy.

Adaptive dynamic programming for trajectory tracking control of a tractor‐trailer wheeled mobile robot

AbstractTractor‐trailer wheeled mobile robots (TTWMRs) possess complex nonlinear dynamics that make their precise trajectory tracking control challenging. This paper explores an adaptive dynamic programming (ADP) approach that utilizes critic neural networks to improve tracking control for continuous‐time TTWMRs. To achieve this, the decoupled kinematic and dynamic loops of the TTWMR are considered, and ADP controllers are proposed aimed at integrated trajectory and velocity tracking. Tis study defines two tracking error systems related to the kinematic and dynamic control loops, which reduces the computational load compared to previous research. The two critic neural networks approximate the optimal cost functions and enable the adaptive tuning of the control policies. Theoretical analysis demonstrates both closed‐loop stability and convergence. Simulation results indicate that the proposed method offers superior tracking performance compared to earlier techniques, exhibiting lower errors and reduced control efforts. This underscores the advantages of using ADP to optimize the control of TTWMRs, even in the presence of partially unknown dynamics.

Dynamic event‐triggered‐based adaptive frequency control of microgrids under cyber‐attacks via adaptive dynamic programming

AbstractThe increasing penetration of renewable energy sources (RES) and the development of the cyber‐physical microgrids (CPMs) make greater demands for frequency control of microgrids. The common approach for frequency control is controlling the micro‐turbine to compensate for frequency deviations, with energy storage systems serving as an auxiliary approach. This article proposes an online adaptive frequency control method to control the governor and energy storage to realize the frequency recovery of microgrid, subject to the external unknown disturbances caused by wind turbine, power load and false data injection (FDI) attacks. First, the non‐zero sum (NZS) games of the considered system are modeled in this work, where the unknown disturbances are also taken into account. For the sake of estimating the unknown disturbances, a disturbance observer (DOB) for the microgrid system is introduced. Then, on the basis of the estimated results of the applied DOB, the disturbance compensation input is derived to offset the interference of the unknown disturbance. Meanwhile, the adaptive dynamic programming (ADP) approach is employed to derive the adaptive optimal control input for the NZS games of microgrid system. Besides, the dynamic event‐triggered (DET) control is introduced, reducing the occupation of computing resources. By utilizing the Lyapunov's method, the stability of the closed‐loop system, the convergence of the estimation weight, the estimation disturbances and the system state are guaranteed. The effectiveness of the proposed method is ultimately verified by the simulation results.

Resilient Control Under Denial-of-service and Uncertainty: An Adaptive Dynamic Programming Approach

Resilient Control Under Denial-of-service and Uncertainty: An Adaptive Dynamic Programming Approach

Floor space optimization of Molds in Boat Manufacturing using Dynamic programming and optimization algorithm

Floor space management in manufacturing of boats requires to be optimized as much as possible and particularly with reference to molds, which are large fixed assets. This article uses dynamic programming to solve the problem of floor space allocation, with particular emphasis on the minimization of unused space and the maximization of mold placement. By specifying the problem as a resource allocation problem, the article naturally considers spatial configurations and operational conditions such as mold size, production timeline, and material transportation. The dynamic programming approach is used to provide a scalable and cost-efficient solution as it is a decomposition of the problem into sub-problems and the global optimum is achieved for layouts. The analysis shows a potential of achieving a high space utilization, the elimination of operational constraints and an improvement of flow rates leading to the formulation of smart manufacturing solutions in the boat industry. This article explains about the analysis of Optimizing Floor Space in Boat Manufacturing, Formulation of the Mold Arrangement Challenge, Key Constraints and Variables in Space Optimization, Dynamic Programming for Mold Layout Optimization, Significance of Efficient Floor Space Utilization and Practical Applications in Boat Manufacturing Facilities were also discussed. In this review article, a total number of 60 papers were choosen from the year 2019 to 2024 accordingly. Moreover, 11 papers were selected from the year 2019, 18 papers were selected from 2020, 13 were selected from the year 2021, 9 papers from the year 2022, 6 papers were selected from 2023 and 3 were from 2024 respectively

Optimal Debt Ratio and Investment-Consumption Strategies with Taxation in the Presence of Jump Risks

This paper derives an optimal debt ratio, consumption rate and investment strategies with taxation for an investor who invest under four background risks: investment, taxation, income and jump risks. The underlying assets considered in this paper are a riskless and risky asset. The risky asset is assumed to follow a jump-diffusion process. We also assume that the income growth rate and tax payment of the investor follow a jump-diffusion process. The aim of the investor is to derive the wealth-after-tax process. The wealth-after-tax process of the investor is taken to be the difference between the wealth-before-tax and the tax payment processes of the investor. The resulting wealth-after-tax process of the investor was solved using dynamic programming approach. As a result, we derive the optimal investment strategies, optimal debt ratio and optimal consumption rate for the investor over time by assuming that the investor chooses a power utility function. The optimal investment strategies were found to involve four components: a speculative portfolio, a tax risks hedging portfolio strategy, an income growth rate risks hedging portfolio strategy and a risk-free fund that holds only the riskless asset. Interestingly, we found that before loan is taken or given, the following must be considered: interest rate on loan to be taken or given, the nominal interest rate, income growth rate, coefficient of the investor willingness to bear the risk of taking debt. We also found that as the income growth rate of the investor increases, the debt profile of the investor decreases. We observe that as the coefficient of risk aversion with respect to debt ratio tends to unity, the amount of debt will be unbearable. It was also observed that the higher an investor willingness to bear the risk of taking debt, the smaller the optimal debt ratio of the investor over time. We further found that when tax rate increases, consumption rate decreases and vice versa. To ascertain the validity of our models, data were collected from six companies in Nigerian Stocks Exchange, and SPSS package was used to analyze the data. Some empirical results were obtained in this paper, using MATLAB software.

An effective and open source interactive 3D medical image segmentation solution

3D medical image segmentation is a key step in numerous clinical applications. Even though many automatic segmentation solutions have been proposed, it is arguably that medical image segmentation is more of a preference than a reference as inter- and intra-variability are widely observed in final segmentation output. Therefore, designing a user oriented and open-source solution for interactive annotation is of great value for the community. In this paper, we present an effective interactive segmentation method that employs an adaptive dynamic programming approach to incorporates users’ interactions efficiently. The method first initializes an segmentation through a feature-based geodesic computation. Then, the segmentation is further refined by using an efficient updating scheme requiring only local computations when new user inputs are available, making it applicable to high resolution images and very complex structures. The proposed method is implemented as a user-oriented software module in 3D Slicer. Our approach demonstrates several strengths and contributions. First, we proposed an efficient and effective 3D interactive algorithm with the adaptive dynamic programming method. Second, this is not just a presented algorithm, but also a software with well-designed GUI for users. Third, its open-source nature allows users to make customized modifications according to their specific requirements.

ReAlign-N: an integrated realignment approach for multiple nucleic acid sequence alignment, combining global and local realignments.

Ensuring accurate multiple sequence alignment (MSA) is essential for comprehensive biological sequence analysis. However, the complexity of evolutionary relationships often results in variations that generic alignment tools may not adequately address. Realignment is crucial to remedy this issue. Currently, there is a lack of realignment methods tailored for nucleic acid sequences, particularly for lengthy sequences. Thus, there's an urgent need for the development of realignment methods better suited to address these challenges. This study presents ReAlign-N, a realignment method explicitly designed for multiple nucleic acid sequence alignment. ReAlign-N integrates both global and local realignment strategies for improved accuracy. In the global realignment phase, ReAlign-N incorporates K-Band and innovative memory-saving technology into the dynamic programming approach, ensuring high efficiency and minimal memory requirements for large-scale realignment tasks. The local realignment stage employs full matching and entropy scoring methods to identify low-quality regions and conducts realignment through MAFFT. Experimental results demonstrate that ReAlign-N consistently outperforms initial alignments on simulated and real datasets. Furthermore, compared to ReformAlign, the only existing multiple nucleic acid sequence realignment tool, ReAlign-N, exhibits shorter runningtimesand occupies less memory space. The source code and test data for ReAlign-N are available on GitHub (https://github.com/malabz/ReAlign-N).

On the solution of Dolichobrachistochrone differential game via dynamic programming approach

On the solution of Dolichobrachistochrone differential game via dynamic programming approach

Climate resilience and profitability thresholds in Chesapeake Bay oyster aquaculture

Shellfish aquaculture producers in coastal systems are facing uncertain future growing conditions as climate change alters weather patterns and raises sea level. We examined expected mid-century (2059–2068) changes in aquaculture profitability from recent conditions by integrating models of climate change, estuarine hydrodynamics and biogeochemistry, oyster growth, oyster mortality, and economics, using the Chesapeake Bay, USA as a case study. We developed an economic stochastic dynamic programming (SDP) approach that generates optimal grower behavior to maximize profits under uncertainty by dynamically choosing planting density, replanting and mitigation use, in response to changing oyster stock status and water quality conditions. Separate models were developed for bottom culture largely serving the cannery market, and container culture largely serving the half-shell market, to reflect different production costs, market prices, and oyster growth and survival. The coupled hydrodynamic-biogeochemical and oyster ecology models projected high spatial variability in oyster growth and mortality with the most favorable growing conditions in the lower north and upper mid bay, where mortality is lowest, and the upper south bay, where growth is highest. Climate change by late mid-century generated modest water quality changes and virtually no mortality rate changes. Nonetheless, our modeling revealed that even if growers made optimal management choices under uncertainty, the majority of modeled sites would see a decline in profitability under climate change, primarily due to potential reductions in food availability. Bottom culture was more resilient to the future climate at most sites, being less sensitive to small changes in growth than container culture. Information on how aquaculture conditions currently vary in space was more important for profitability than future climate forecasts. Our stochastic dynamic programming approach tailored grower behavior to each site and unfolding annual conditions, including highly targeted and cost-effective mitigation adjustments to boost oyster survival or growth.

Computationally efficient assessment of fuel economy of multi-modes and multi-gears hybrid electric vehicles: A hyper rapid dynamic programming approach

The powertrain configuration, sizing, and control are multi-dimensional intertwined. Synergy optimization of these three dimensions can yield the greatest benefits. However, the huge computational load limits its implementation. Especially for multi-modes and multi-gears (MMMG) transmissions. Thus, a more efficient optimization method with acceptable accuracy is urgently required. In this study, a near-global optimal method, called Hyper-Rapid Dynamic Programming (HR-DP), is proposed and discussed. The computation time is significantly reduced by optimization of candidate state and control domains, identification of optimal efficiency operating points, and parallel computation approaches. Subsequently, a thorough comparison of the HR-DP, Rapid-DP and DP methods was performed across various driving cycles. Compared to the DP algorithm, the computational efficiency is boosted by a factor of about 100,000, while the fuel consumption error is limited to 1.5 % in Real-world driving cycle (RWDC). Moreover, the HR-DP, in conjunction with particle swarm optimization (PSO), is employed for the first time to optimize essential sizing for MMMG configuration. The MMMG configuration with optimal sizing is demonstrated to be most energy-efficient, with 7.70%–10.6 % fuel-savings achieved, compared to the Toyota Prius. Therefore, HR-DP is well-suited for the design and optimization of HEV transmission configurations and sizing, significantly accelerating the development progress.

Accelerating chiller sequencing using dynamic programming

Effective management of heating, ventilation, and air conditioning (HVAC) systems is crucial for enhancing building energy efficiency. Chillers, a significant component of HVAC systems, are responsible for more than 50 % of energy usage of the whole cooling plant, underscoring the importance of optimizing chiller sequencing. Although Model Predictive Control (MPC) has demonstrated efficacy in enhancing chiller performance, its widespread adoption is hindered by its high computational complexity. This study introduces a dynamic programming approach coupled with a 2-step optimization method to expedite chiller sequencing optimization while upholding MPC’s real-time control capabilities. The MPC method implemented in this study integrates a load forecasting model with an artificial neural network (ANN) based chiller model. Through simulation tests utilizing historical HVAC operational data from a commercial building located in Shanghai, the proposed chiller sequencing strategy achieves a 9.73 % decrease in energy consumption. Moreover, the dynamic programming approach, especially in conjunction with the 2-step optimization process, significantly reduces the MPC solution time at each control step from 165 min to 3.61 s, making it a viable solution for real-time MPC deployment. Additionally, the research delves into the influence of the chiller model’s complexity, the optimization strategy, and control horizon on computational efficiency. This research provides a feasible solution to implement chiller sequencing in real buildings to pick up the low-hanging fruits in an effective way.

An Approximate Dynamic Programming Approach to Vehicle Platooning Coordination in Networks

An Approximate Dynamic Programming Approach to Vehicle Platooning Coordination in Networks

Optimal Asset Allocation of Pension Funds with a Portfolio Constraint Change

We investigated the asset allocation problem of a pension fund in which the constraints on the risky investment proportion change owing to local regulations or the fund’s policy changes. We assumed that the fund manager has a utility function with respect to the funding ratio of the pension fund. A dynamic programming approach was employed and we obtained a closed-form solution to the problem. To numerically measure the effect of such changes on asset allocation, we define the concept of policy change equivalent return (PCER) as the additional expected return for the value function of the case without such changes at the same level as the value function of the case with them. In the numerical analysis, we analyzed both the effect of regime switching on market parameters and policy change on the PCER and showed the effectiveness of introducing policy changes to asset allocation.

Scheduling of e-commerce packaging machines: blocking machines and their impact on the performance–waste tradeoff

AbstractTo streamline their fulfillment processes, many e-commerce retailers today use automated packaging machines for their outbound parcels. An important performance–waste tradeoff is associated with these machines: To reduce packaging waste when handling different sized goods, packaging machines should be able to handle different carton sizes. However, more carton sizes lead to a more involved scheduling process, so that the throughput performance deteriorates (and vice versa). To investigate this tradeoff, this paper develops scheduling procedures for a specific type of packaging machine, called blocking machines. These packaging machines provide multiple back-to-back packaging devices, each continuously processing a dedicated carton size, but blocking each other whenever incoming goods are not properly ordered according to carton sizes on the infeed conveyor. To reduce the resulting throughput loss, we derive various scheduling problems for optimizing the inflow of goods, provide a thorough analysis of the computational complexity, and derive an exact dynamic programming approach that is polynomial in the number of orders to be packed. This allows us to solve even large real-world instances to proven optimality with which we can analyze the performance–waste tradeoff of blocking machines.