Stochastic Dynamic Optimization Research Articles

Drive force allocation control based on pump displacement optimization for hydraulic hybrid motor-assisted system

To achieve rational coordination of the front and rear axle torque of commercial vehicles equipped with hydraulic hybrid motor-assisted systems (HHMAS), and further enhance the vehicle’s fuel efficiency and traction capability. A hydraulic pump displacement control strategy based on energy-efficiency-orientation (EEO) is proposed for driving force distribution. Firstly, the efficiency is analyzed from the perspective of system energy flow. Then, considering the time-varying efficiency characteristics of the hydraulic variable pump and hydraulic motor, the stochastic dynamic optimization problem is modeled as a markov decision process, and a value function estimation method is introduced to approximate the optimal energy efficiency of the entire HHMAS. Finally, the results indicate that the EEO strategy can enhance the fuel economy of vehicles, with a fuel-saving rate of 6.3% compared to the wheel-speed-following-control (WSFC) strategy. In addition, the proposed strategy exhibits superior traction capacity and climbing ability compared to the traditional WSFC strategy.

Regression based stochastic energy management strategy for a grid connected microgrid using Artificial Electric Field Algorithm

Recent decade has witnessed steep technological advancement in the renewable energy sector due to growing concern of climate change and emission cut target. The lack of availability of fossil fuels and the high price of the same have made the use of Renewable Energy Sources (RES) as a promising approach for generating clean and sustainable energy at local level. This persuades the implementation of Microgrid (MG) supported with RES, to cater various types of load demand. However, planning and optimizing a MG operation have become challenging due to the intermittent nature of RES and uncertain loads. To address this issue, this paper introduces a Stochastic Energy Management Strategy (SEMS) for a grid connected MG comprising Photovoltaic, Wind Turbine, Fuel Cell, Micro Turbine, Battery Energy Storage and electrical as well as heat energy demand. The volatilities of RES power output and energy demand are modeled via Gaussian Process Regression (GPR) and Monte Carlo Simulation (MCS) respectively. The problem is framed as non-linear (NL) dynamic stochastic optimization problem and solved using ‘Artificial Electric Field Algorithm (AEFA)’. Here, minimization of expected value of operational and emission cost are taken as objective functions; while various practicality aspects are modeled as constraints in the problem. Further, the efficacy of the proposed approach is validated around the first central moment of the load. Various case studies are performed to estimate the day ahead optimum operating schedule of the Distributed Energy Resources (DER) and to assess the impact of load uncertainties on DER operation and total MG operation cost. Further, the efficacy of the GPR is verified through comparison with MCS based result.

Performance analysis of machine learning based prediction models in assessing optimal operation of microgrid under uncertainty

Of late, the exponential rise in the global population is driving higher energy demand. However, the rapid depletion of conventional fossil fuels and growing environmental concerns have prompted the evolution of alternative energy sources. To this end, Microgrid (MG) with Renewable Energy Sources (RES) has emerged as popular means of small-scale localized power grid. However, planning of MG operation poses challenges due to the inherent variability and stochasticity in RES power output and energy demand. On account of this, the present study introduces a Stochastic Energy Management Strategy (SEMS) for a grid-connected MG incorporating Micro-Turbine, Fuel-Cell, RES, Battery Energy Storage, and electrical and heat energy demand. The stochasticity of RES is forecasted through a hybrid prediction model (sARIMA-GRU) and the uncertain demand is estimated via 'Monte Carlo Simulation.' The proposed problem is formulated as a dynamic non-linear stochastic optimization problem. It seeks to minimize the expected value of MG operational cost satisfying the practical constraints. Addressing this, a newly developed ‘Artificial Electric Field Algorithm (AEFA)' is utilized. Several case studies are performed to assess MG operation under varied operating conditions. Moreover, the present study analyses the impact of uncertainty on energy contribution from DER, grid dependency, and MG operation cost. Comparative analysis reveals that sARIMA-GRU outperforms other contemporary prediction models. It is noteworthy that the superior prediction accuracy of sARIMA-GRU leads to lower MG operation costs. Moreover, statistical analysis and convergence confirm the proficiency of applied AEFA over state-of-the-art Grey Wolf Optimization and Firefly Algorithm in solving the proposed problem.

Valuing Combinations of Flexible Planning, Design, and Operations in Water Supply Infrastructure

AbstractUncertainty arising from climate change poses a central challenge to the long‐term performance of many engineered water systems. Water supply infrastructure projects can leverage different types of flexibility, in planning, design, or operations, to adapt infrastructure systems in response to climate change over time. Both flexible planning and design enable future capacity expansion if‐and‐when needed, with flexible design proactively incorporating physical design changes that enable retrofits. All three forms of flexibility have not previously been analyzed together to explicitly assess their relative value in mitigating cost and water supply reliability risk. In this paper, we propose a new framework to evaluate combinations of flexible planning, design, and operations. We develop a nested stochastic dynamic optimization approach that jointly optimizes dam development and operating policies under dynamic climate uncertainty. We demonstrate this approach on a reservoir project near Mombasa, Kenya. Our results find that flexible operations have the greatest potential to reduce costs. Flexible design and flexible planning can amplify the value of flexible operations under higher discounting scenarios and when initial infrastructure capacities are undersized. This approach provides insight on the climate change and techno‐economic conditions under which flexible planning, design, and operations can be best leveraged individually or in combination to reduce climate change uncertainty risks in water supply infrastructure projects.

Rational expectations: an approach of anticipated linear-quadratic social optima

Motivated by rational expectations theory, this paper studies a class of stochastic linear-quadratic dynamic optimisation problems involving a large number of weakly-coupled heterogeneous agents. Different from well-studied mean field games, these agents formalise a team with cooperation to minimise some social cost functional. Moreover, the state here evolves by some anticipated backward stochastic differential equation in which the terminal instead initial condition is specified and the anticipated terms are involved. Applying a so-called anticipated person-by-person optimality principle, we construct an auxiliary control problem for each agent based on decentralised information. The decentralised social strategy is derived by a class of new consistency condition systems, which are mean-field-type anticipated forward–backward stochastic differential delay equations (AFBSDDEs). The well-posedness of such consistency condition system is obtained via discounting method. The corresponding asymptotic social optimality is also verified as the population size goes to infinity.

Eurosystem stability: A stochastic dynamic optimization approach to public debt

In this paper we focus on the dynamics of tax revenue and debt that should be necessary for the European countries to comply with the rules on which the Stability and Growth Pact (SGP) is founded, when public budgets are expounded to unexpected expenses.We specify a stochastic control model with infinite horizon formalizing the optimal rules implied by the SGP and compute tax revenue and debt satisfying the model. We find that, even though in long run equilibrium, they are non-stationary processes. From a technical point of view, we make use of a matrix discounted Riccati equation and show that, even in the presence of a unitary root, a real solution exists.The implementation of the optimal model with real data initial conditions suggests for European countries a more intense use of debt rather than tax revenue to finance the public budget in the absence of financial turmoil. Furthermore, there emerge also some small margins for this policy during the Covid-19 period.

Risk-averse stochastic dynamic power dispatch based on deep reinforcement learning with risk-oriented Graph-Gan sampling

The increasing penetration of renewable energy sources (RES) brings volatile stochasticity, which significantly challenge the optimal dispatch of power systems. This paper aims at developing a cost-effective and robust policy for stochastic dynamic optimization of power systems, which improves the economy as well as avoiding the risk of high costs in some critical scenarios with small probability. However, it is hard for existing risk-neutral methods to incorporate risk measure since most samples are normal. For this regard, a novel risk-averse policy learning approach based on deep reinforcement learning with risk-oriented sampling is proposed. Firstly, a generative adversarial network (GAN) with graph convolutional neural network (GCN) is proposed to learn from historical data and achieve risk-oriented sampling. Specifically, system state is modelled as graph data and GCN is employed to capture the underlying correlation of the uncertainty corresponding to the system topology. Risk knowledge is the embedded to encourage more critical scenarios are sampled while aligning with historical data distributions. Secondly, a modified deep reinforcement learning (DRL) with risk-measure under soft actor critic framework is proposed to learn the optimal dispatch policy from sampling data. Compared with the traditional deep reinforcement learning which is risk-neutral, the proposed method is more robust and adaptable to uncertainties. Comparative simulations verify the effectiveness of the proposed method.

ICU capacity expansion under uncertainty in the early stages of a pandemic

We propose a general modular approach to support decision‐makers' response in the early stages of a pandemic with resource expansion, motivated by the shortage of Covid‐19‐related intensive care units (ICU) capacity in 2020 in Italy. Our approach uses (1) a stochastic extension of an epidemic model for scenarios of projected infections, (2) a capacity load model to translate infections into scenarios of demand for the resources of interest, and (3) an optimization model to allocate this demand to the projected levels of resources based on different values of investment. We demonstrate this approach with the onset of the first and second Covid‐19 waves in three Italian regions, using the data available at that time. For epidemic modeling, we used a parsimonious stochastic susceptible‐infected‐removed model with a robust estimation procedure based on bootstrap resampling, suitable for a noisy and data‐limited environment. For capacity loading, we used a Cox queuing model to translate the projected infections into demand for ICU, using stochastic intensity to capture the variability of the patient arrival process. Finally, we used stochastic dynamic optimization to select the best policy (when and how much to expand) to minimize the expected number of patients denied ICU for any level of investment in capacity expansion and obtain an efficient frontier. The frontier allows a trade‐off between investment in additional resources and the number of patients denied intensive care. Moreover, in the panic‐driven early days of a pandemic, decision‐makers can also obtain the time until which they can postpone action, potentially reducing investment costs without increasing the expected number of denied patients.

Flattening Energy-Consumption Curves by Monthly Constrained Direct Load Control Contracts

California Utility Firm Implements Innovative Model, Reducing Costs by 4% A California utility firm has successfully implemented a pioneering model to balance electricity demand and supply while minimizing costs. By utilizing direct load control contracts (DLCCs), the firm can reduce energy consumption during peak hours. Researchers developed an integer stochastic dynamic optimization problem that considers monthly and annual constraints, allowing for effective execution of DLCCs. Incorporating a “reduce-to-threshold” policy to flatten energy-consumption curves during high demand, the model was verified using real data from the California Independent System Operator. When implemented, the utility firm achieved an impressive cost reduction of approximately 4%. Sensitivity analysis was conducted to enhance customer experience and improve DLCC contract features. The success of this innovative model highlights the potential of DLCCs and advanced optimization techniques in the energy sector, offering a blueprint for other utility companies seeking to optimize grid stability and reduce costs.

Potential Welfare Gains from Optimal Macro Hedging for Oil Exporters

This paper computes the welfare gains from optimal hedging with futures contracts for an oil-exporting country. Unlike previous studies, this paper derives the welfare gains under a more realistic futures hedging model. This is accomplished by considering basis risk and by relaxing the full-hedging assumption. Furthermore, this is the first paper to derive the welfare gains under optimal hedging strategies. We also incorporate the empirical relationship between spot and futures prices within our models, rather than the theoretical relationship which most studies employ. The models were developed under a dynamic stochastic optimization framework and the optimal consumption and value functions were found using the method of Endogenous Gridpoints. The results showed that the choice of the optimal hedging strategy employed led to a slight improvement in the country’s welfare gains relative to full hedging. We also found that the strategies with the highest welfare gains were the most effective at volatility reduction. Finally, this paper provides compelling evidence for the use of optimal macro futures hedging as an effective risk management tool for oil-exporting developing countries.

A Nonparametric Algorithm for Optimal Stopping Based on Robust Optimization

Optimal stopping is a fundamental class of stochastic dynamic optimization problems with numerous applications in finance and operations management. In “A nonparametric algorithm for optimal stopping based on robust optimization,” the author introduces an approach based on robust optimization for computing near-optimal solutions for computationally demanding stochastic optimal stopping problems with known probability distributions. Through this new use of robust optimization, the paper develops new algorithms for solving stochastic optimal stopping problems that are practical and can outperform state-of-the-art simulation-based algorithms in the context of pricing high-dimensional financial options.

A note on the existence of optimal stationary policies for average Markov decision processes with countable states

In many practical stochastic dynamic optimization problems with countable states, the optimal policy possesses certain structural properties. For example, the (s,S) policy in inventory control, the well-known cμ-rule and the recently discovered c/μ-rule (Xia et al. (2022)) in scheduling of queues. A presumption of such results is that an optimal stationary policy exists. There are many research works regarding to the existence of optimal stationary policies of Markov decision processes with countable state spaces (see, e.g., Bertsekas (2012); Hernández-Lerma and Lasserre (1996); Puterman (1994); Sennott (1999)). However, these conditions are usually not easy to verify in such optimization problems. In this paper, we study the optimization of long-run average of continuous-time Markov decision processes with countable state spaces. We provide an intuitive approach to prove the existence of an optimal stationary policy. The approach is simply based on compactness of the policy space, with a special designed metric, and the continuity of the long-run average in the space. Our method is capable to handle cost functions unbounded from both above and below, which makes a complementary contribution to the literature work where the cost function is unbounded from only one side. Examples are provided to illustrate the application of our main results.

Optimal stochastic inventory-distribution strategy for damaged multi-echelon humanitarian logistics network

The quick distribution of relief goods is vital in alleviating human suffering in affected areas. A relief strategy that delivers (pushes) goods to the affected population depending on the predicted demand is critical, especially during the early post-disaster period when accurate demand information is lacking. Several inventory-distribution strategies based on multi-echelon humanitarian logistics networks have been previously investigated to facilitate a quick response to demand information. However, natural disasters have raised doubts about the performance of these networks. This paper presents the disaster conditions under which conventional multi-echelon networks are conducive to push-mode strategies that deliver relief goods depending on the predicted demand. Specifically, an approximate solution to the optimal inventory-distribution strategy is analytically derived from a dynamic stochastic optimization problem using deterministic approximation and decomposition. Several numerical validations ensure the optimality and practical applicability of the strategy. The approximate strategy identifies disaster conditions that do not contribute to push-mode strategies in conventional multi-echelon networks and proposes an alternative network structure.

Estimating the Operating Reserve Demand Curve for Efficient Adoption of Renewable Sources in Korea

As the proportions of variable renewable sources (VRSs) such as solar and wind energy increase rapidly in the power system, their uncertainties inevitably undermine power supply reliability and increase the amount of operating reserve resources required to manage the system. However, because operating reserves have the characteristics of a public good and their value is related to the social cost of blackouts, it is difficult to determine their market price efficiently, which leads to inefficiencies in procuring operating reserves. This study estimates the operating reserve demand curve (ORDC) of the Korean power system to provide an effective basis for measuring the proper value and quantity of operating reserves needed to meet the reliability standard. A stochastic dynamic optimization model is applied to incorporate the probabilistic characteristics of VRS and the inter-hour constraint, which is necessary for analyzing load-following reserves. An econometric model and the Monte Carlo simulation method are used to generate the forecast profiles of solar and wind generation. The results indicate that the proper amount of hourly operating reserves needed in 2034 is approximately 4.4 times higher than that in 2020 at the current reserve offer price. The ORDC of 2020 has a price-inelastic shape, whereas the ORDC of 2034 has a price-elastic shape because the reserve requirement varies considerably with its offer price level in the high-VRS penetration case. This variability is due to alternatives, such as VRS curtailment or load shedding, which can replace the reserve requirement. This study also showed that VRS curtailment is an effective balancing resource as an alternative to reserves.

Optimal dynamic pricing and production policy for a stochastic inventory system with perishable products and inventory-level-dependent demand

In this paper, we consider a joint dynamic pricing and production policy for a stochastic inventory system with perishable products. The demand is stochastic and dependent on the price and the level of the on-hand inventory. Combined dynamic pricing and production control, a stochastic dynamic optimization model that maximizes the total discounted profit is built. Applying the stochastic optimal control theory, we formulate the problem of finding the optimal joint dynamic pricing and production schedule as the problem of solving a Hamilton-Jacobi-Bellman (HJB) equation. By solving the HJB equation, analytical solutions for optimal pricing and production rate are obtained. In addition, a numerical example is given to illustrate the validness of the theoretical results and sensitivity analysis on major system parameters is carried out.

Prioritising the eradication of invasive species from island archipelagos with high reinvasion risk

Abstract Eradicating invasive species from islands is a proven method for safeguarding threatened and endangered species from extinction. Island eradications can deliver lasting benefits, but require large up‐front expenditure of limited conservation resources. The choice of islands must therefore be prioritised. Numerous tools have been developed to prioritise island eradications, but none fully account for the risk of those eradicated species later returning to the island: reinvasion. In this paper, we develop a prioritisation method for island eradications that accounts for the complexity of the reinvasion process. By merging spatially explicit metapopulation modelling with stochastic dynamic optimisation techniques, we construct a decision‐support tool that optimises conservation outcomes in the presence of reinvasion risk. We applied this tool to two different case studies—rat (Rattus rattus) invasions in the Seaforth archipelago in New Zealand, and cane toad (Rhinella marina) invasions in the Dampier archipelago in Australia—to illustrate how state‐dependent optimal policies can maximise expected conservation gains. In both case studies, incorporating reinvasion risk dramatically altered the optimal order of island eradications, and improved the potential conservation benefits. The increase in benefits was larger in Dampier than Seaforth (42% improvement versus 6%), as a consequence of both the characteristics of the invasive species, and the arrangement of the islands. Synthesis and applications. Our results illustrate the potential consequences of ignoring reinvasion risk. We recommend that reinvasion risk be explicitly included in any island eradication prioritisation involving an archipelago, particularly when some islands are close to the mainland.

Stochastic Dynamic Optimization for Forest Rotation with Uncertain Stumpage Prices

Abstract This study investigates the effect of uncertain stumpage prices on the optimal forest rotation decision and related profitability under various silvicultural scenarios. The study applies loblolly pine (Pinus taeda L.) growth and yield models with distinct silvicultural scenarios in the Piedmont and Upper Coastal Plain region of the southeastern United States. It then applies chance-constrained dynamic optimization to derive optimal rotation age and related profitability on per acre basis. The growth and yield models show that using a combination of herbicide and fertilizer silvicultural treatment leads to a higher timber yield and profit at a lower optimal rotation age than other treatment scenarios. Furthermore, the stochastic optimization model shows that low risk tolerance results in low returns. Within a particular risk tolerance level, silvicultural treatment options with higher standard deviations almost always produce higher returns. The findings of this article contribute to the literature on optimizing forest management when considering uncertain stumpage prices over time, given different risk tolerance levels.

Designing Watersheds for Integrated Development (DWID): A stochastic dynamic optimization approach for understanding expected land use changes to meet potential water quality regulations

This study investigates the tradeoff between agricultural economic losses and potential water quality regulations at the watershed level over time. Specifically, we applied a stochastic dynamic mathematical programming approach to evaluate the impact of land use changes on agricultural economic losses under uncertain water quality regulations and crop yields. We selected the Little River Experimental Watershed in South Georgia as a case study. Seven potential water pollution reduction trajectories for total phosphorus (TP) were developed. The proposed model applied a chance-constraint technique to allow some degree of uncertainty in meeting the water quality standards. The result shows that agricultural profits decrease when uncertainties are considered at the watershed level. The findings further show that uncertainties of meeting the water quality standard expedited the land use transition from croplands to forestlands and magnified the share of forestlands, highlighting the crucial role of forests in managing water pollution in the selected watershed. The results also indicate that under a moderate pollution reduction scenario, an economic loss of $232/ha/yr is expected to achieve a 16 % reduction in TP concentration. In a more ambitious pollution mitigation scenario, agricultural economic losses are expected to be $778/ha/yr and may result in a 45 % decrease in TP concentration. The modeling approach developed in Designing Watersheds for Integrated Development (DWID) could easily feed into those policy deliberations which operate at the interface of economics and water-related policies for ensuring sustainable management of watersheds worldwide.

Response adjustable performance of a visco-elastomer sandwich plate with harmonic parameters and distributed supported masses under random loading

Vibration control of composite structures with distributed masses under random loadings is a significant issue. Adjustability of dynamic characteristics including response spectrum peaks and valleys is important for structural vibration control. The vibration control design in space contains structure and conformation designs which combination results in periodic composite structures. In the present paper, spatial periodicity control design is proposed. Stochastic response adjustable performance of a visco-elastomer sandwich plate with harmonic distribution of geometrical and physical parameters and distributed supported masses under random base motion loading is studied. Both facial layer thickness and core layer modulus of the sandwich plate are considered as harmonic distribution in length and width directions as well as periodically distributed masses. Partial differential equations of coupling motions of the sandwich plate system are derived and converted into ordinary differential equations for multi-mode coupling vibration. Generalized stiffness, damping, and mass coefficients are functions of the harmonic distribution parameters. An analysis solution with frequency response function and response spectral density expressions of the sandwich plate system is obtained. Numerical results are given to show the response adjustable performance through the harmonic geometrical and physical parameters and distributed masses. The results have a potential application to stochastic vibration control or dynamic optimization design of smart composite structure systems.

Dynamic Optimization of on‐Grid Integrated Energy System Considering Peak‐Shaving Demand Via Learning Methods

An integrated energy system (IES) is combined with electricity, gas and heating subnetworks to satisfy various loads, which results in a complicated energy coupling. Considering the power interaction between IES and power grid, the real-time dynamic optimal dispatch policies for the on-grid IES are proposed and compared by learning methods to improve the operation economic and peak-shaving performances in this study. Owing to the uncertainties of the load demands and real-time peak-shaving demand and renewable energy, together with the sequential operation of the controllable sources, the energy dispatch optimization problem of the IES is described as a stochastic dynamic optimization process. A source-load collaborative operation mode is proposed as the coupling of thermal energy and electrical power in the IES and compared with different energy dispatching mode. And reinforcement learning methods are adopted to achieve the dynamic optimal policy. Since the multiple systems with energy coupling would result in modeling difficulty and dimension curse during the optimization, a deep reinforcement learning method (DDQN) is used to solve this problem. A numerical analysis is performed by comprehensively comparing the different learning algorithms and operation modes, and the simulation results show that the derived dispatch policy achieved by DDQN method with source-load collaboration can effectively improve the performance of the IES in the stochastic environment. © 2022 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.