Stochastic Dynamic Programming Method Research Articles

Dynamic Price Dispersion of Seasonal Goods in Bertrand–Edgeworth Competition

The primary distinguishing characteristics of various revenue management systems include fixed capacities, uniform products, and customers' sensitivity to pricing when making purchase decisions. Even with only two or three competitors, the competition remains intense. This study examines the sub-game perfect Nash equilibrium of price competition within an oligopoly market where goods are perishable assets. Each seller possesses a single unit of a non-replenishable good and competes by setting prices to sell their inventory over a finite sales period. Each period, buyers seek to purchase one unit of the good, and the number of buyers entering is random. All sellers' prices are visible to buyers, and there is no cost associated with searching for goods. Through stochastic dynamic programming methods, sellers' optimal responses can be determined by treating the competition as a one-time price-setting game, considering the remaining sales periods and the current demand pattern. Assuming a binary demand model, it is demonstrated that the duopoly model exhibits a unique Nash equilibrium. In addition, it is shown that the oligopoly model does not result in price dispersion based on a discussed metric. Moreover, when a generalized demand model is considered, the duopoly model features a unique Nash equilibrium with mixed strategies, whereas the oligopoly model exhibits a unique symmetric Nash equilibrium with mixed strategies.

DYNAMIC PROGRAMMING FOR SOLVE THE INVENTORY MANAGEMENT PROBLEM IN LOGISTICS

Currently, there is a problem of methods insufficient efficiency for finding solutions to the inventory management problem. The research object is the process of solving inventory management problems. The research subject is methods for finding a solution to the inventory management problem based on dynamic programming. The research goal is to increase the efficiency of finding a solution to the inventory management problem through dynamic programming. A method based on deterministic dynamic programming, a method based on stochastic dynamic programming, a method based on Q-learning, and a method based on SARSA were applicated for the inventory management problem. There are advantages of the methods. of Methods modification of deterministic and stochastic dynamic programming, Q-learning, and SARSA due to dynamic parameters makes it possible to increase the learning speed while maintaining the root-mean-square error of the method. The numerical study made it possible to evaluate the methods (for modifying the deterministic and stochastic dynamic programming methods, the number of iterations is close to the number of stages; for both methods of deterministic and stochastic dynamic programming, the root mean square error was 0.02; for modifying the Q-learning and SARSA methods, the number of iterations was 300, for both methods of Q-learning and SARSA, the root mean square error was 0.05). These methods make it possible to expand the scope of dynamic programming, which is confirmed by their adaptation to the inventory management problem and helps to increase the intelligent computer systems efficiency for general and special purposes. The application of these methods for a wide class of artificial intelligence problems are the prospects for further research.

Reinsurance, investment and the rationality with a diffusion model approximating a jump model

This article investigates the optimal excess-loss reinsurance, investment, and the rationality of using a diffusion model to approximate a jump model. We assume that the instantaneous rate of return of the risky asset is modelled by an Ornstein-Uhlenbeck (O-U) process, and the insurance claims are modeled by a compound Poisson (CP) process and a diffusion approximation (DA) model. By using the stochastic dynamic programming method, the closed-form expressions for the optimal reinsurance and investment strategies and the corresponding value functions are derived. We find that the insurer with the CP claim model always has higher reinsurance demand than the insurer with the DA claim model. Moreover, the error of using the DA model to approximate the CP model is very much dependent on the reinsurance price. Numerical analysis shows that the insurer adjusts the investment strategy as the state of the financial market changes.

A Generalized Control Paradigm for Storage Systems: Optimal Energy Management and Stability Certificates

In this brief, we propose a suboptimal control update for lossless storage systems that operates based on the instantaneous value of the load power and generalizes previously suggested solutions for energy management problems. The proposed control update performs under uncertain conditions and does not require statistical information about the load profile. We leverage tools from Pontryagin’s minimum principle and stochastic dynamic programming methods to derive the control strategy and then investigate its stability properties using the Lyapunov analysis. However, the analysis of this control paradigm presents significant challenges, among which the key difficulty arises due to the presence of nonlinear feedback terms. To address this challenge, we first introduce a coordinate transformation, which transforms the original problem into an equivalent stabilization problem, and then verify its stability properties using the Lyapunov function theory and LaSalle’s invariance principle. We show that the proposed control policy works for both continuous-time and stochastic load profiles, and it keeps the stored energy within the capacity bounds by following the optimal path.

Unit Commitment Problem with Energy Storage Under Correlated Renewables Uncertainty

Unit Commitment Problem with Energy Storage Under Correlated Renewables Uncertainty” introduces a novel approach to address the challenges of renewable integration. The study acknowledges the growing variability and correlation in power availability due to renewable generation and proposes a day-ahead unit commitment (UC) problem formulation that incorporates energy storage and considers multistage correlated uncertainty. Using a variant of the stochastic dual dynamic programming (SDDP) method, which can handle temporal correlations effectively, the researchers solve the complex UC problem. Results obtained from the IEEE 118-bus system demonstrate the significant advantages of considering multistage uncertainty and correlations. Applying their approach to the Chilean power system, the researchers present superior UC solutions that adapt generation to changing uncertainty at a lower cost. Additionally, they propose a more efficient deterministic UC solution that outperforms current industry practices. These advancements promise to enhance the integration of renewable energy sources, enabling a more sustainable and environmentally friendly future.

Multi‐stage stochastic dual dynamic programming to low‐carbon economic dispatch for power systems with flexible carbon capture and storage devices

AbstractThe carbon capture and storage (CCS) technique gains much attention due to its role in reducing CO2 emissions. By introducing flexible CCS devices into conventional power plants, the low‐carbon economic operation of the power system can be achieved. However, the uncertainty of renewable energy makes it hard to obtain an optimal operation policy. First, the detailed model of CCS consisting of bypass venting stacks, solvent storage tanks, absorber, and stripper is described. Then, a low‐carbon economic dispatch model of the power system with CCS is proposed to minimize the total operation and renewable energy abandonment penalty costs. Next, the problem is reformulated as multi‐stage stochastic programming, and the stochastic dual dynamic programming (SDDP) method is applied to relieve the computation burden. To control the probability of stopping the algorithm prematurely, a new stopping criterion based on a two‐sided hypothesis test is proposed. Finally, the effectiveness of the proposed model and the new stopping criterion is demonstrated by case studies on a practical power system.

Collaborative Optimization of Company Commuter Bus Stop Selection and Route Programming Considering Stochastic Travel Time and Demand

Company commuter bus service (CCBS) is a convenient commuting mode provided for employees. which has always been popular because of its flexible route planning and low cost. Some companies offer CCBS to their employees for free. However, there are also some companies that need employees to pay the fare for CCBS. For fee-based CCBS, profit is an important consideration. Appropriate stop setting and route programming can attract more commuters and generate greater profits. This paper studies the stop selection and route planning for CCBS considering uncertain demand and travel time. In the data preparation phase, we propose an improved fuzzy c-means clustering to obtain appropriate clusters of commuters’ addresses. In the solution phase, we designed a collaborative framework for stop selection and route programming with the objective to maximize the profit of CCBS. A novel heuristic stochastic dynamic programming (H-SDP) method is then designed for the stop selection sub-problem considering the uncertainties of both traveling time and commuting demand. Meanwhile, we employ a variable neighborhood search algorithm with a novel shaking operation suitable for the routing problem. Finally, we conduct a series of computational experiments to demonstrate the effectiveness and efficiency of the developed algorithms.

Two-timescale coordinated operation of wind-advanced adiabatic compressed air energy storage system: A bilevel stochastic dynamic programming method

Renewable energy is a promising solution to address the energy crisis and environmental issues, but it comes with challenges due to its inherent volatility and limited dispatchability. Advanced adiabatic compressed air energy storage (AA-CAES) is a favorable partner for centralized renewable integration, due to its numerous benefits, such as large capacity, long lifetime, fast response capability, and zero carbon emissions. For the economic management of a wind-AACAES system, a battery-like AA-CAES dispatch model is proposed, where the charging/discharging efficiencies and capacities are dependent on the operating power and state-of-charge (SoC) to capture the part-load characteristics. A hybrid control strategy is proposed to enhance the flexibility and efficiency of the compression side under off-design conditions caused by simultaneous changes in back pressure and load. To provide dispatch and control strategies of AA-CAES, a multi-timescale optimization problem is established. The slow timescale determines the baseline output scheduling on an hourly basis to maximize total revenue, while the fast timescale updates the real-time adjustment every five minutes to minimize the penalty of tracking error. A bi-level stochastic dynamic programming (SDP) framework is formulated to incorporate the wind uncertainties and multi timescale coordination, where terminal SoCs and value functions are essential to integrate decisions across slow and fast timescales. A non-iterative parametric programming-based method is proposed to solve the slow-timescale SDP, and a simulation-based rollout algorithm is applied to extract the fast-timescale actions, which yields evident improvement over any heuristic base policy that is sequentially consistent. The numerical results demonstrate that the proposed method has a performance gap of approximately 7.1%, which outperforms the rule-based method by 32.6%. The installation of AA-CAES improves the total revenue by 11.9%. Additionally, the hybrid control strategy enhances the charging flexibility and efficiency by 30.7% and 4.3%, respectively, resulting in a 3.9% reduction in tracking deviation penalty.

Deep Reinforcement Learning Based Unit Commitment Scheduling under Load and Wind Power Uncertainty

The intermittent nature of renewable energy sources and fluctuating electricity demand induce significant uncertainty that needs to be tackled with computationally efficient solution techniques to provide reliable and cost-effective generation schedules of power systems. In this work, we present a deep reinforcement learning (DRL) based approach for the day-ahead scheduling of generation resources under demand and wind power uncertainties. The proposed approach yields a causal policy relying only on historical uncertainty realizations and forecast data that is trained with an actor-critic-based reinforcement learning algorithm. Through safe exploration, the DRL-based approach guarantees a feasible commitment schedule without any operational constraint violations. We conduct computational experiments on the IEEE 39-bus and 118-bus test cases to demonstrate the effectiveness of the proposed solution strategy and improvement over existing approaches, including a deterministic approach with point forecasts and the stochastic dual dynamic integer programming method. The results show that the proposed approach enjoys superior performance in terms of computational efficiency and incurred operational costs, with significant reduction in penalty costs caused by insufficient net load supply than the deterministic approach.

Foreign exchange trading and management with the stochastic dual dynamic programming method

We present a novel tool for generating speculative and hedging foreign exchange (FX) trading policies. Our solution provides a schedule that determines trades in each rebalancing period based on future currency prices, net foreign account positions, and incoming (outgoing) flows from business operations. To obtain such policies, we construct a multistage stochastic programming (MSP) model and solve it using the stochastic dual dynamic programming (SDDP) numerical method, which specializes in solving high-dimensional MSP models. We construct our methodology within an open-source SDDP package, avoiding implementing the method from scratch. To measure the performance of our policies, we model FX prices as a mean-reverting stochastic process with random events that incorporate stochastic trends. We calibrate this price model on seven currency pairs, demonstrating that our trading policies not only outperform the benchmarks for each currency, but may also be close to ex-post optimal solutions. We also show how the tool can be used to generate more or less conservative strategies by adjusting the risk tolerance, and how it can be used in a variety of contexts and time scales, ranging from intraday speculative trading to monthly hedging for business operations. Finally, we examine the impact of increasing trade policy uncertainty (TPU) levels on our findings. Our findings show that the volatility of currencies from emerging economies rises in comparison to currencies from developed markets. We discover that an increase in the TPU level has no effect on the average profit obtained by our method. However, the risk exposure of the policies increases (decreases) for the group of currencies from emerging (developed) markets.

Two-Timescale Dynamic Energy and Reserve Dispatch With Wind Power and Energy Storage

The integration of volatile renewable resources and energy storage entails making dispatch decisions for conventional coal-fired units and fast-response devices in different timescales. This paper studies intraday dynamic energy-reserve dispatch following a two-timescale setting. The coarse timescale determines the hourly reference output and reserve allocation, which offers a sufficient backup for the fine-timescale operation; the fine timescale determines the adjustment of gas-fired units and energy storage system every 15 minutes in response to the actual wind power. A stochastic dynamic programming method is proposed to make decisions at the coarse timescale while guaranteeing the robust feasibility of the fast process via vertex scenarios of uncertainty set and bounding the state-of-charge intervals for energy storage; the fast-response actions at the fine timescale are updated using a truncated rolling-horizon optimization, which incorporates the cost-to-go functions calculated at the coarse timescale to prevent the fast decisions from being myopic. Case studies on the modified IEEE 5-bus system and 118-bus system validate the proposed framework.

Optimal High-Frequency Trading in a Pro-Rata Microstructure with Predictive Information

We propose a framework to study optimal trading policies in a one-tick pro-rata limit order book, as typically arises in short-term interest rate futures contracts. The high-frequency trader has the choice to trade via market orders or limit orders, which are represented respectively by impulse controls and regular controls. We model and discuss the consequences of the two main features of this particular microstructure: first, the limit orders sent by the high frequency trader are only partially executed, and therefore she has no control on the executed quantity. For this purpose, cumulative executed volumes are modelled by compound Poisson processes. Second, the high frequency trader faces the overtrading risk, which is the risk of brutal variations in her inventory. The consequences of this risk are investigated in the context of optimal liquidation. The optimal trading problem is studied by stochastic control and dynamic programming methods, which lead to a characterization of the value function in terms of an integro quasi-variational inequality. We then provide the associated numerical resolution procedure, and convergence of this computational scheme is proved. Next, we examine several situations where we can on one hand simplify the numerical procedure by reducing the number of state variables, and on the other hand focus on specific cases of practical interest. We examine both a market making problem and a best execution problem in the case where the mid-price process is a martingale. We also detail a high frequency trading strategy in the case where a (predictive) directional information on the mid-price is available. Each of the resulting strategies are illustrated by numerical tests.

DNN-SSDP for hydropower system operation using small state sets

Sampling stochastic dynamic programming (SSDP) method is one of the most popular methods in solving reservoir operation models, especially in long term operation models. However, the dimension problems seriously confine the model scale and solving efficiency. To increase the solution efficiency of SSDP for multiple reservoir operations, a deep neural network SSDP (DNN-SSDP) method is proposed, in which DNN, SSDP, and simulation are combined. The solution uses an initial state set, obtained using a deterministic model, in the first round SSDP run. In the SSDP, the future values and the decisions are computed using a series of deep neural networks replacing the interpolation algorithms to make it possible to adopt a small-scale and irregularly distributed state set. Then the SSDP run is repeated after the state set is extended according to the simulation results using the obtained operation rule. The process is repeated several times until the performance change between two rounds of SSDP run is small enough. The scales of the state sets used are much smaller than the commonly used vertically intersecting discretization. The proposed method is applied to derive the long term operation rule of the hydropower system on the Wu River in Southwest China for energy maximization. Results show that the computing time using the proposed method is only 18% of the common method without considerable accuracy loss.

Asset-liability management with state-dependent utility in the regime-switching market

This paper considers a state-dependent optimal asset-liability management problem in continuous-time settings. The investor maximizes the expected state-dependent utility of the terminal asset-liability ratio in a regime-switching market to better describe insurance companies’ needs for asset-liability matching and regulation with background risk. We apply the stochastic dynamic programming method to get closed-form results. We find that the optimal strategy in the state-dependent utility case is the same, and the optimal value function differs from the state-independent utility case. Finally, we study how the parameters impact the optimal investment strategy and corresponding value functions through numerical examples.

Profit sharing mechanisms in multi-owned cascaded hydrosystems

We consider the optimal management of hydropower generated by a cascade of three interconnected reservoirs owned by different agents. In this setting, water availability at the downhill reservoirs depends on decisions taken by the agents upstream. This creates an opportunity for the hydroplant at the top to withhold water and take advantage of situations with higher selling prices, which makes the overall decisions of the agents deviate from what can be considered best for the cascade as whole. In order to mitigate the market power of the hydroplant uphill, we propose a mechanism to enforce some collaborative behavior among the agents. This is achieved by agents transferring upstream fractions of their profit in exchange for water released from the top. The corresponding mathematical models are trilevel deterministic and trilevel stochastic linear programming problems, with uncertainty in prices and streamflows (exogenous inflows). For the stochastic variants, analyzed both in two- and multi-stage formulations, we propose new solution methods, extending to the trilevel nested setting the well-known L-Shaped and Stochastic Dual Dynamic Programming methods. A successful implementation of the later depends on certain floating cuts, that represent symbolically Benders-like linearizations. Convergence properties are discussed for some of the procedures. Numerical experiments confirm the interest of the approach, because with the proposed mechanism the top owner as well as the downhill hydroplants earn more money than when acting in an individualistic manner.

Разработка алгоритмов сопровождения процессов реализации программы цифровизации на железнодорожном транспорте

Objective: To develop a methodology for minimizing costs, associated with additional investment attraction in the course of monitoring of accomplishment process and updating of digitalization program (Digitalization Programs DP). To consider the need for a strict quantitative justification and coordination of digitalization activities at various hierarchical levels of the projects. To show the necessity to solve tasks on calculating parameters of situational response to potential violations of DP accomplishment progress, to work out algorithms to determine rational relation between technical-technological procedures and control procedures. Methods: For to achieve the stated goals, stochastic dynamic programming method is applied; developed algorithms are based on the hypothesis on the possibility of distinguishing a law of DP stage duration distribution. Results: The system of algorithms, allowing to gradually select rational managing impacts while synchronization of usage of being additionally involved investments, combined with control procedures, is developed and tested. The methodology requires actualization due to inevitable changes in databases, containing information on links between randomized entities of next being predicted corrective measures, their cost and the assessment of being liquidated delay. To increase the calculation result reliability in parallel to realization of the developed algorithms, it is necessary to embody marketing research and efficiency model evaluation of various DP reboot options. Practical significance: An approach is proposed to provide for guaranteed implementation of transport system DPs within the given timeframes with minimal further investment. The developed algorithms make it possible to define not “deterministic” scheme of program reconfiguration but to carry out digitalization process situational management consistently. The developed algorithms can be recommended for practical use in the course of administrative-scientific accompaniment of the implementation of DP component project realization.

Real options with overextrapolation

This study incorporates the overextrapolation belief into the classic real options model. Using the stochastic dynamic programming method, we obtain the semiclosed-form solutions for the optimal investment and valuation of real options and the welfare loss owing to overextrapolation. The theoretical results show that overextrapolation has significant effects on the investment and pricing, which depend on the belief that overextrapolation induces the agent to underinvest in and undervalue the option with low belief concerning the anticipated return. Conversely, overextrapolation leads to overinvestment and overvaluation for agents with high belief. Moreover, our model predicts that welfare loss owing to distorted investment due to overextrapolation is inessential.

Fast Stochastic Dual Dynamic Programming for Economic Dispatch in Distribution Systems

The multistage economic dispatch solution is of paramount importance for achieving the optimal unit commitment and the optimal decentralized decision-making in smart grid operations. Through the approximations of the optimal expected cost-to-go functions, the stochastic dual dynamic programming (SDDP) has been shown effective to obtain the optimum for the multistage stochastic programming (SP) problem, while without finite termination guarantee. Thus, we propose a fast stochastic dual dynamic programming (FSDDP) method to accelerate the convergence rate of the SDDP, including updating the candidate points based on the case with the maximum difference between the upper and lower bounds of the optimal value functions, adding a backward-forward-backward inner scheme in the backward pass, and initializing the bounds with finite constant values. We can verify that the FSDDP method enjoys a nice property with a finite convergence guarantee for the multistage SP problems. Numerical tests on 4-, 33-, 34-, 69-, 118- and 123-bus distribution systems have demonstrated that FSDDP can approach the optimal economic performance at significantly improved computational efficiency.

Multi-Stage Distributionally Robust Stochastic Dual Dynamic Programming to Multi-Period Economic Dispatch With Virtual Energy Storage

A virtual energy storage (VES) model is proposed in this paper to accommodate renewable energy under a special market regulation. Such VESs can provide or consume electricity to the main power grid under the premise that the daily net electricity energy is balanced. Furthermore, a multi-stage distributionally robust optimization (MSDRO) model is set up in this paper to address the temporal uncertainties in the day-ahead economic dispatch model. Compared with the traditional two-stage distributionally robust optimization, the proposed multi-stage approach provides more flexibilities so that the decision variables can be adjusted at each time period, leading to a complex nested formulation. To efficiently solve the MSDRO model, a stochastic dual dynamic programming method is employed to decompose the original large-scale optimization model into several sub-problems in the stages, as two steps: forward pass and backward pass. In the forward pass, the expected cost-to-go function is approximated by piecewise-linear functions and then several samples are used to generate a lower bound; the backward pass will generate Benders’ cuts at each stage from the solution of the forward pass. The forward and backward passes are performed iteratively until the convergence is reached. Numerical results on an IEEE 118-bus system and a practical power system in China verify the proposed method.

Robust asset-liability management under CRRA utility criterion with regime switching: a continuous-time model

This article describes a robust continuous-time asset-liability management problem under Markov regime-switching. First, we employ the “homothetic robustness” to preserve the performance of robustness for the ALM model, which runs well in precisely modified state variables and performs reasonably if some forms of model misspecification exist. Second, we consider the asset-to-liability ratio instead of the surplus, which ensures that we use relative values instead of absolute values to modify the wealth process. Besides, we use the stochastic dynamic programming method to get some closed-form results and analyze the impacts of parameters on the investment strategy and value function, respectively, by numerical examples.