Stochastic Dynamic Programming Research Articles

Hybrid simplicial-randomized approximate stochastic dynamic programming for multireservoir optimization

Hybrid simplicial-randomized approximate stochastic dynamic programming for multireservoir optimization

Optimal control of stochastic differential equations with random impulses and the Hamilton–Jacobi–Bellman equation

AbstractIn this article, we study the optimal control of stochastic differential equations with random impulses. We optimize the performance index and add the influence of random impulses to the performance index with a random compensation function. Using the idea of stochastic analysis and dynamic programming principle, a new Hamilton–Jacobi–Bellman (HJB) equation is obtained, and the existence and uniqueness of its viscosity solution are proved.

Adaptive Double-Spending Attacks on PoW-Based Blockchains

Previous state-of-the-art studies have proposed various analytical models to understand the double-spending attacks (DSA) occurred in Proof-of-Work (PoW) based Blockchains. Although many insights behind the double-spending attacks have been disclosed, we still believe that advanced versions of DSA can be developed to create new threats for the PoW-based blockchains such as the Bitcoin blockchain. In this paper, we present two new types of double-spending attacks in the context of the PoW-based blockchain and discuss the insights behind them. By considering more practical network parameters, such as the number of confirmation blocks, the hashpower of the double-spending attacker, the amount of coins in the target transaction, and the network-status parameter, we first analyze the success probability of the conventional double-spending attack, named Naive DSA. Based on Naive DSA, we create two new adaptive DSA, i.e., the Adaptive DSA and the Reinforcement Adaptive DSA (RA-DSA). In our analytical models, a double-spending attack is converted into a Markov Decision Process. We then exploit the Stochastic Dynamic Programming (SDP) approach to obtain the optimal attack strategies under Adaptive DSA and RA-DSA. Numerical simulation results demonstrate the correlations between each critical network parameter and the expected attacker's reward. Through the proposed analytical models, we aim to alert the PoW-based blockchain ecosystem that the threat of double-spending attacks is still at a dangerous level. For example, our findings show that the attacker can launch a successful attack with a small hashpower proportion much lower than 51% under RA-DSA.

Dynamic Price Dispersion of Seasonal Goods in Bertrand–Edgeworth Competition

The primary distinguishing characteristics of various revenue management systems include fixed capacities, uniform products, and customers' sensitivity to pricing when making purchase decisions. Even with only two or three competitors, the competition remains intense. This study examines the sub-game perfect Nash equilibrium of price competition within an oligopoly market where goods are perishable assets. Each seller possesses a single unit of a non-replenishable good and competes by setting prices to sell their inventory over a finite sales period. Each period, buyers seek to purchase one unit of the good, and the number of buyers entering is random. All sellers' prices are visible to buyers, and there is no cost associated with searching for goods. Through stochastic dynamic programming methods, sellers' optimal responses can be determined by treating the competition as a one-time price-setting game, considering the remaining sales periods and the current demand pattern. Assuming a binary demand model, it is demonstrated that the duopoly model exhibits a unique Nash equilibrium. In addition, it is shown that the oligopoly model does not result in price dispersion based on a discussed metric. Moreover, when a generalized demand model is considered, the duopoly model features a unique Nash equilibrium with mixed strategies, whereas the oligopoly model exhibits a unique symmetric Nash equilibrium with mixed strategies.

Robust time-consistent strategies of DC pension plans with the return of premiums clauses under inflation

. This article investigates the robust time-consistent investment strategy for a DC pension plan with model uncertainty under the mean-variance optimization objective. For the avoidance of reductions in investment returns due to inflation risk and premature death of members, an inflation-indexed bond is introduced into the financial market and a return of premium clause is incorporated into the model. By establishing extended Hamilton-Jacobi-Bellman (HJB) equations, the explicit solutions of both the robust precommitment strategy and the robust time-consistent strategy are presented by virtue of the robust optimal control theory and the stochastic dynamic programming approach. Furthermore, two degradation cases are derived in detail. Finally, a numerical example demonstrates the analysis of the obtained results.

A stochastic programming approach to the antibiotics time machine problem

Antibiotics Time Machine is an important problem to understand antibiotic resistance and how it can be reversed. Mathematically, it can be modeled as follows: Consider a set of genotypes, each of which contain a set of mutated and unmutated genes. Suppose that a set of growth rate measurements of each genotype under a set of antibiotics is given. The transition probabilities of a ‘realization’ of a Markov chain associated with each arc under each antibiotic are computable via a predefined function given the growth rate realizations. The aim is to maximize the expected probability of reaching to the genotype with all unmutated genes given the initial genotype in a predetermined number of transitions, considering the following two sources of uncertainties: (i) the randomness in growth rates, (ii) the randomness in transition probabilities, which are functions of growth rates. We develop stochastic mixed-integer linear programming and dynamic programming approaches to solve static and dynamic versions of the Antibiotics Time Machine Problem under the aforementioned uncertainties. We adapt a Sample Average Approximation approach that exploits the special structure of the problem and provide accurate solutions that perform very well in an out-of-sample analysis.

Economic optimization of Wolbachia-infected Aedes aegypti release to prevent dengue.

Dengue virus, primarily transmitted by the Aedes aegypti mosquito, is a major public health concern affecting ≈3.83 billion people worldwide. Recent releases of Wolbachia-transinfected Ae. aegypti in several cities worldwide have shown that it can reduce dengue transmission. However, these releases are costly, and, to date, no framework has been proposed for determining economically optimal release strategies that account for both costs associated with disease risk and releases. We present a flexible stochastic dynamic programming framework for determining optimal release schedules for Wolbachia-transinfected mosquitoes that balances the cost of dengue infection with the costs of rearing and releasing transinfected mosquitoes. Using an ordinary differential equation model of Wolbachia and dengue in a hypothetical city loosely describing areas at risk of new dengue epidemics, we determined that an all-or-nothing release strategy that quickly brings Wolbachia to fixation is often the optimal solution. Based on this, we examined the optimal facility size, finding that it was inelastic with respect to the mosquito population size, with a 100% increase in population size resulting in a 50-67% increase in optimal facility size. Furthermore, we found that these results are robust to mosquito life-history parameters and are mostly determined by the mosquito population size and the fitness costs associated with Wolbachia. These results reinforce that Wolbachia-transinfected mosquitoes can reduce the cost of dengue epidemics. Furthermore, they emphasize the importance of determining the size of the target population and fitness costs associated with Wolbachia before releases occur. © 2024 The Authors. Pest Management Science published by John Wiley & Sons Ltd on behalf of Society of Chemical Industry.

DYNAMIC PROGRAMMING FOR SOLVE THE INVENTORY MANAGEMENT PROBLEM IN LOGISTICS

Currently, there is a problem of methods insufficient efficiency for finding solutions to the inventory management problem. The research object is the process of solving inventory management problems. The research subject is methods for finding a solution to the inventory management problem based on dynamic programming. The research goal is to increase the efficiency of finding a solution to the inventory management problem through dynamic programming. A method based on deterministic dynamic programming, a method based on stochastic dynamic programming, a method based on Q-learning, and a method based on SARSA were applicated for the inventory management problem. There are advantages of the methods. of Methods modification of deterministic and stochastic dynamic programming, Q-learning, and SARSA due to dynamic parameters makes it possible to increase the learning speed while maintaining the root-mean-square error of the method. The numerical study made it possible to evaluate the methods (for modifying the deterministic and stochastic dynamic programming methods, the number of iterations is close to the number of stages; for both methods of deterministic and stochastic dynamic programming, the root mean square error was 0.02; for modifying the Q-learning and SARSA methods, the number of iterations was 300, for both methods of Q-learning and SARSA, the root mean square error was 0.05). These methods make it possible to expand the scope of dynamic programming, which is confirmed by their adaptation to the inventory management problem and helps to increase the intelligent computer systems efficiency for general and special purposes. The application of these methods for a wide class of artificial intelligence problems are the prospects for further research.

Uncertainty dynamics in energy planning models: An autoregressive and Markov chain modeling approach

This paper deals with the modeling of stochastic processes in long-term multi-stage energy planning problems when limited information is available about the distributions of such processes. Starting from simple estimates of variation intervals for uncertain parameters, such as energy demands and costs, we model the temporal correlation of these parameters through carefully constructed autoregressive (AR) models that respect the intervals defined in each period. We introduce a coefficient for “zigzag” effects in the evolution of uncertain processes, which controls the correlation across periods and also the likelihood of extreme scenarios. To preserve the convexity of the stochastic problem, we discretize the AR models associated with the cost parameters involved in the objective function by Markov chains. The resulting formulation is then solved using a Stochastic Dual Dynamic Programming (SDDP) algorithm available in the literature that handles finite-state Markov chains. Our numerical experiments, carried out on the Swiss energy system, show a very desirable adaptation strategy of investment decisions to uncertainty scenarios, a behavior that is not observed when the temporal correlation is ignored. Moreover, the solutions lead to better out-of-sample cost performances, especially on extreme scenario realizations, than the non-correlated ones which usually result in overcapacities to protect against high, but unlikely, parameter variations over time.

A novel navigation and charging strategy for electric vehicles based on customer classification in power-traffic network

With electric vehicles getting increasingly popular, there has been a lot of interest in encouraging future electric vehicle development in terms of greater charging efficiency, greater benefits to the environment, and greater reliability. One of the most prominent research is on the scheduling of electric vehicle charging and navigation. High-quality navigation and charging scheduling strategy must depend on the state of the coupled network between the transportation and power networks, while the outcomes of scheduling can also have a significant impact on these networks. In this paper, we propose a novel dynamic navigation and charging strategy that fully considers the coupled traffic and power distribution network. Firstly, electric vehicles with similar travel and charging requirements are classified. Secondly, an elastic scheme including fixed route navigation and flexible charging scheduling at charging stations is provided. We also propose an information integration framework for the implementation of electric vehicle routing and charging scheduling, information interaction between sectors including the status of heterogeneous electric vehicles, day-ahead power scheduling, and base load demands. The novel strategy aims at optimal electric vehicle navigation and charging at the system level achieving a lower overall travel cost, charging cost, carbon emission, and load variance of the power distribution network. Classification algorithms, stochastic dynamic programming, and queuing theory are used for mathematical modeling. Numerical results demonstrate the effectiveness of the proposed novel strategy for the navigation and charging scheduling of electric vehicles.

Weighted double Q-learning based eco-driving control for intelligent connected plug-in hybrid electric vehicle platoon with incorporation of driving style recognition

Driving style poses significant impacts on eco-driving performance of vehicles, especially for those with hybrid powertrains. By incorporating driving style recognition, an efficient velocity planning and energy management strategy is developed for a platoon of intelligent connected plug-in hybrid electric vehicles (PHEVs). Firstly, a high-fidelity driving style classification and recognition model is established based on the agglomerative hierarchical clustering algorithm, and then the support vector machine algorithm is employed to recognize the driving style. Next, a multi-objective velocity planning problem is formulated with the consideration of the fuel economy, driving adaptability and comfort, following performance, and driving safety optimization as the optimization objective, and is then solved based on orthogonal collocation direct transcription and the interior-point methods. Finally, an energy management strategy incorporating model predictive control and weighted double Q-learning algorithm is built. The simulation results demonstrated that the velocity planning algorithm incorporating driving styles achieved preferable adaptability and comfort. The proposed strategy can achieve up to most 98.88 % energy economy of the stochastic dynamic programming for the following PHEVs, and reduce the overall fuel consumption by 27.39 % for the platoon, comparing to that without driving style incorporation.

Reinsurance, investment and the rationality with a diffusion model approximating a jump model

This article investigates the optimal excess-loss reinsurance, investment, and the rationality of using a diffusion model to approximate a jump model. We assume that the instantaneous rate of return of the risky asset is modelled by an Ornstein-Uhlenbeck (O-U) process, and the insurance claims are modeled by a compound Poisson (CP) process and a diffusion approximation (DA) model. By using the stochastic dynamic programming method, the closed-form expressions for the optimal reinsurance and investment strategies and the corresponding value functions are derived. We find that the insurer with the CP claim model always has higher reinsurance demand than the insurer with the DA claim model. Moreover, the error of using the DA model to approximate the CP model is very much dependent on the reinsurance price. Numerical analysis shows that the insurer adjusts the investment strategy as the state of the financial market changes.

Long-term operational planning for flexible residential buildings with seasonal storage and capacity-based grid tariffs

When performing operation of the energy system in a building, it is difficult to accurately represent the uncertain long-term objectives in the short-term window, and the correlation between long-term objectives. This work investigates a strategic modelling framework for representing long-term incentives during short-term operational planning for an energy system within flexible buildings. Multi-stage backwards Stochastic Dynamic Programming (SDP) is used to decompose the yearly operational problem into smaller stages, and create long-term cost curves that captures the cost of adjusting long-term price signals, which in this work considers seasonal thermal energy storage and recurring monthly capacity-based grid tariff costs. The proposed method is analyzed for a flexible realistic Norwegian building located in Southern Norway, where the influence both price signals have on long-term operational performance is analyzed. Results show that the framework strategically optimizes the long-term seasonal storage together with the recurring monthly grid tariff, complementing each other by using the seasonal storage to reduce peak costs during winter. With both seasonal storage and monthly demand charge, yearly operational costs are reduced by 4.3% compared to only accounting for the grid tariff. Additionally, the operational performance achieved only a 0.9% higher cost compared to yearly simulation with perfect information.

Dual Stochastic Dual Dynamic Programming for Multi-Stage Economic Dispatch With Renewable Energy and Thermal Energy Storage

Dual Stochastic Dual Dynamic Programming for Multi-Stage Economic Dispatch With Renewable Energy and Thermal Energy Storage

Approximation of multistage stochastic programming problems by smoothed quantization

AbstractWe present an approximation technique for solving multistage stochastic programming problems with an underlying Markov stochastic process. This process is approximated by a discrete skeleton process, which is consequently smoothed down by means of the original unconditional distribution. Approximated in this way, the problem is solvable by means of Markov Stochastic Dual Dynamic Programming. We state an upper bound for the nested distance between the exact process and its approximation and discuss its convergence in the one-dimensional case. We further propose an adjustment of the approximation, which guarantees that the approximate problem is bounded. Finally, we apply our technique to a real-life production-emission trading problem and demonstrate the performance of its approximation given the “true” distribution of the random parameters.

Developing practical measures of the price of pesticide resistance: A flexible computational framework with global sensitivity analysis

AbstractPesticide resistance poses an increasing challenge for agricultural sustainability. Pesticide susceptibility is a depletable biological resource, but resistance management rarely quantifies marginal, forward‐looking economic costs to users of depletion. To facilitate the development of such costs, we use a generic stochastic bioeconomic model of resistance evolution in a crop pest population, stochastic dynamic programming, and global sensitivity analysis to analyze the “marginal user costs” of resistance. The most impactful parameters are population density dependence and pesticide prices. The least impactful is the fitness cost of resistance, which is noteworthy because of prior emphasis on this parameter in the resistance management literature.

Joint optimization of inspection and condition-based maintenance for a deteriorating product under extended warranty

Condition-based maintenance (CBM) has been widely applied in engineering practice, but it has been rarely integrated into extended warranty policies. When the inspection cost is high, it becomes important to schedule the inspection plan when making CBM decisions. This paper jointly optimizes inspection and CBM for a deteriorating product under extended warranty. The deterioration of the product is characterized as a non-stationary gamma process. At each inspection epoch, the decision maker needs to determine whether to repair the product and when to perform the next inspection. The objective is to simultaneously optimize the inspection policy and the maintenance policy that minimize the warranty servicing cost. The joint optimization problem is formulated by stochastic dynamic programming. The optimal policy is proven to be of a control-limit form. A backward induction algorithm integrating the obtained structural properties is proposed to find the optimization results. A numerical example is conducted to illustrate the effectiveness of the proposed approach. A graphical representation of the obtained policy is provided to support practical maintenance implementation. The comparison with a popular CBM policy confirms the superiority of the proposed policy.

Empirical dynamic programming for model‐free ecosystem‐based management

Abstract Quantitative ecosystem‐based management typically relies on hypothetical ecosystem models that are difficult to validate for all but the best‐studied systems. Here, we develop a management scheme that is based on predictive models driven by the observed dynamics. We show that near‐optimal management policies can be constructed from time‐series data by merging empirical dynamic modelling and stochastic dynamic programming. The Empirical Dynamic Programming approach performs well in cases we examined and outperformed a commonly used single‐species alternative. We expect model‐free ecosystem‐based management to be of use wherever ecosystem dynamics are uncertain or observations of the system do not cover all relevant species.

Maximizing the survival probability in a cash flow inventory problem with a joint service level constraint

This paper investigates a multi-period stochastic cash flow inventory problem with the aim of maximizing the long-term survival probability, which may be the objective of some retailers especially in periods of economic distress. Demand in each period is stochastic and can be non-stationary. In order to avoid too many lost sales under this objective, we introduce a joint chance constraint requiring the probability of no stockouts during the planning horizon to be higher than a specified service level. We develop a scenario-based model and a sample average approximation (SAA) model to solve the problem. A statistical upper bound on the survival probability based on SAA is provided and we discuss upper and lower bounds for the problem based on stochastic dynamic programming. We also propose a rolling horizon approach with service rate updating to test the out-of-sample performance of the two stochastic models and solve problems with long planning horizons. We test the two methods in large numerical tests and find that the rolling horizon approach together with the stochastic models can solve realistically sized problems in reasonable time.

A simple heuristic for computing non-stationary inventory policies based on function approximation

We consider a finite-horizon periodic-review inventory system with fixed replenishment costs that faces non-stationary demands. The structure of the optimal control policy for this system has long been known. However, finding optimal policy parameters requires solving a large-scale stochastic dynamic program. To circumvent this, we devise a recursion-free approximation for the cost function of the problem. This translates into an efficient and effective heuristic to compute policy parameters that significantly outperforms earlier heuristics. Our approach is easy-to-understand and easy-to-use as it follows by elementary methods of shortest paths and convex minimization.