Stochastic Dynamic Programming Method Research Articles

Stochastic Optimization of Grid to Vehicle Frequency Regulation Capacity Bids

This paper investigates the application of stochastic dynamic programming to the optimization of charging and frequency regulation capacity bids of an electric vehicle (EV) in a smart electric grid environment. We formulate a Markov decision problem to minimize an EV's expected cost over a fixed charging horizon. We account for both Markov random prices and a Markov random regulation signal. We also propose an enhancement to the classical discrete stochastic dynamic programming method. This enhancement allows optimization over a continuous space of decision variables via linear programming at each state. Simple stochastic process models are built from real data and used to simulate the implementation of the proposed method. The proposed method is shown to outperform deterministic model predictive control in terms of average EV charging cost.

Divergence Behaviour of the Successive Geometric Mean Method of Pairwise Comparison Matrix Generation for a Multiple Stage, Multiple Objective Optimization Problem

This paper focuses on the divergence behaviour of the successive geometric mean (SGM) method used to generate pairwise comparison matrices while solving a multiple stage, multiple objective (MSMO) optimization problem. The SGM method can be used in the matrix generation phase of our three-phase methodology to obtain pairwise comparison matrix at each stage of an MSMO optimization problem, which can be subsequently used to obtain the weight vector at the corresponding stage. The weight vectors across the stages can be used to convert an MSMO problem into a multiple stage, single objective (MSSO) problem, which can be solved using dynamic programming-based approaches. To obtain a practical set of non-dominated solutions (also referred to as Pareto optimal solutions) to the MSMO optimization problem, it is important to use a solution approach that has the potential to allow for a better exploration of the Pareto optimal solution space. To accomplish a more exhaustive exploration of the Pareto optimal solution space, the weight vectors that are used to scalarize the MSMO optimization problem into its corresponding MSSO optimization problem should vary across the stages. Distinct weight vectors across the stages are tied directly with distinct pairwise comparison matrices across the stages. A pairwise comparison matrix generation method is said to diverge if it can generate distinct pairwise comparison matrices across the stages of an MSMO optimization problem. In this paper, we demonstrate the SGM method's divergence behaviour when the three-phase methodology is used in conjunction with an augmented high-dimensional, continuous-state stochastic dynamic programming method to solve a large-scale MSMO optimization problem. Copyright © 2013 John Wiley & Sons, Ltd.

Worst-Case-Expectation Approach to Optimization Under Uncertainty

In this paper we discuss multistage programming with the data process subject to uncertainty. We consider a situation where the data process can be naturally separated into two components: one can be modeled as a random process, with a specified probability distribution, and the other one can be treated from a robust (worst-case) point of view. We formulate this in a time consistent way and derive the corresponding dynamic programming equations. To solve the obtained multistage problem, we develop a variant of the stochastic dual dynamic programming method. We give a general description of the algorithm and present computational studies related to planning of the Brazilian interconnected power system.

OPTIMAL HIGH‐FREQUENCY TRADING IN A PRO RATA MICROSTRUCTURE WITH PREDICTIVE INFORMATION

We propose a framework to study optimal trading policies in a one‐tick pro rata limit order book, as typically arises in short‐term interest rate futures contracts. The high‐frequency trader chooses to post either market orders or limit orders, which are represented, respectively, by impulse controls and regular controls. We discuss the consequences of the two main features of this microstructure: first, the limit orders are only partially executed, and therefore she has no control on the executed quantity. Second, the high‐frequency trader faces the overtrading risk, which is the risk of large variations in her inventory. The consequences of this risk are investigated in the context of optimal liquidation. The optimal trading problem is studied by stochastic control and dynamic programming methods, and we provide the associated numerical resolution procedure and prove its convergence. We propose dimension reduction techniques in several cases of practical interest. We also detail a high‐frequency trading strategy in the case where a (predictive) directional information on the price is available. Each of the resulting strategies is illustrated by numerical tests.

Optimal dynamic asset allocation strategy for ELA scheme of DC pension plan during the distribution phase

In this paper, we study the optimal dynamic asset allocation strategy for the ELA scheme of DC pension plan during the distribution phase. In an ELA scheme of DC pension plan, the assets are invested in equities and bonds, and are distributed to the plan participants by an actuarial method. The survived participant can also obtain a survival credit from the mortality risk-sharing implicit in the pension plan. The goal of the scheme is to maintain the stable purchasing power of the plan participants, i.e., to minimize the square deviations of the distribution and a predetermined level by choosing the optimal dynamic asset allocation proportions. We formalize the problem into a continuous-time stochastic optimal control problem and establish the optimal dynamic asset allocation strategy by stochastic dynamic programming method. We obtain the optimal dynamic asset allocation proportions invested in the equities and bonds, and give an economical explanation of the key factors influencing the strategy.

Nonlinear Dynamic Characteristics and Optimal Control of Giant Magnetostrictive Laminated Plate Subjected to In-plane Stochastic Excitation

In this paper, nonlinear dynamic characteristics and optimal control of giant magnetostrictive laminated plate (GMLP) subjected to in-plane stochastic excitation were studied. Von del Pol nonlinear item was introduced to interpret the hysteresis phenomenon of the strain–magnetic field intensity curve of giant magnetostrictive material, and the nonlinear dynamic model of GMLP subjected to in-plane stochastic excitation was developed. Local and global stochastic stabilities were analyzed according to largest Lyapunov exponent theory and singular boundary theory. The functions of steady-state probability density and joint probability density were obtained, and the condition of stochastic Hopf bifurcation was analyzed. The reliability function was solved from backward Kolmogorov equation, and the probability density of the first-passage time was obtained. Finally, the optimal control strategy was proposed in stochastic dynamic programming method. Numerical simulation shows that the stability of the solution varies with parameter, and stochastic Hopf bifurcation appears in the process; the reliability of the system was improved by optimal control, and the first-passage time was delayed. The result is helpful to engineering applications of GMLP.

Inexact stochastic dynamic programming method and application to water resources management in Shandong China under uncertainty

In this study, an interval parameter multistage joint-probability programming (IMJP) approach has been developed to deal with water resources allocation under uncertainty. The IMJP can be used not only to deal with uncertainties in terms of joint-probability and intervals, but also to examine the risk of violating joint probabilistic constraints in the context of multistage. The proposed model can handle the economic expenditure caused by regional water shortage and flood control. The model can also reflect the related dynamic changes in the multi-stage cases and the system safety under uncertainty. The developed method is applied to a case study of water resources allocation in Shandong, China, under multistage, multi-reservoir and multi-industry. The violating reservoir constraints are addressed in terms of joint-probability. Different risk levels of constraint lead to different planning. The obtained results can help water resources managers to identify desired system designs under various economic, environment and system reliability scenarios.

Risk neutral and risk averse Stochastic Dual Dynamic Programming method

In this paper we discuss risk neutral and risk averse approaches to multistage (linear) stochastic programming problems based on the Stochastic Dual Dynamic Programming (SDDP) method. We give a general description of the algorithm and present computational studies related to planning of the Brazilian interconnected power system.

Energy contracts management by stochastic programming techniques

We consider the problem of optimal management of energy contracts, with bounds on the local (time step) amounts and global (whole period) amounts to be traded, integer constraint on the decision variables and uncertainty on prices only. After building a finite state Markov chain by using vectorial quantization tree method, we rely on the stochastic dual dynamic programming (SDDP) method to solve the continuous relaxation of this stochastic optimization problem. An heuristic for computing sub optimal solutions to the integer optimization problem, based on the Bellman values of the continuous relaxation, is provided. Combining the previous techniques, we are able to deal with high-dimensional state variables problems. Numerical tests applied to realistic energy markets problems have been performed.

Simultaneous control of production, repair/replacement and preventive maintenance of deteriorating manufacturing systems

Abstract This paper presents a method to find the optimal production, repair/replacement and preventive maintenance policies for a degraded manufacturing system. The system is subject to random machine failures and repairs. The status of the system is deemed to degrade with repair activities. When a failure occurs, the machine is either repaired or replaced, and a replacement action renews the machine, while a repair action brings it to a degraded operational state, with the next repair time increasing as the number of repairs increases as well. A preventive maintenance action is considered in order to improve the reliability of the machine, thereby reducing the amount of disruptions caused by machine failures. The decision variables are the production rate, the preventive maintenance rate and the repair/replacement switching policy upon machine failure. The objective of the study is to find the decision variables that minimize the overall cost, including repair, replacement, preventive maintenance, inventory holding and backlog costs over an infinite planning horizon. The proposed model is based on a semi-Markov decision process, and the stochastic dynamic programming method is used to obtain the optimality conditions. A numerical example is given to illustrate the proposed model, and a sensitivity analysis is considered in order to confirm the structure of the control policy and to illustrate the usefulness of the proposed approach.

Suitability of short or long conservation contracts under ecological and socio-economic uncertainty

Acquisition of land rights has become a primary tool used to protect terrestrial biodiversity. Fixed length contracts are often used when trying to secure conservation benefits on private land in agri-environment schemes and payment for environmental services schemes, but the duration of the conservation contracts used in different programmes varies. To date, very little research has been undertaken to determine the situations in which contracts of differing lengths are optimal or when conservation agencies or groups should use a portfolio of different contract lengths rather than relying on a single type. Using stochastic dynamic programming and related heuristic methods, we investigate how the choice between short or long conservation contracts is affected by uncertainty regarding the future availability of sites and their ecological condition. We also examine the benefits offered by using a portfolio of different contract lengths. Conservation agencies must pay private landowners a premium to secure longer agreements and because of this, shorter contracts are advantageous if sites are likely to remain available for conservation in the future. Long contracts are preferred when future site availability becomes more unlikely. In contrast to uncertainty over site availability, uncertainty over future ecological conditions has little effect on contract selection and only markedly influences the choice between short and long contracts when there is heterogeneity across sites in expected conservation outcomes and future availability of sites is also uncertain. Finally, when future site availability is unlikely, the use of a portfolio of short and long contracts would offer greater conservation gains than using either type in isolation, even though this option is not yet one that is commonly found in conservation practice.

A fuzzy-Markov-chain-based analysis method for reservoir operation

In this study, a fuzzy-Markov-chain-based stochastic dynamic programming (FM-SDP) method is developed for tackling uncertainties expressed as fuzzy sets and distributions with fuzzy probability (DFPs) in reservoir operation. The concept of DFPs used in Markov chain is presented as an extended form for expressing uncertainties including both stochastic and fuzzy characteristics. A fuzzy dominance index analysis approach is proposed for solving multiple fuzzy sets and DPFs in the proposed FM-SDP model. Solutions under a set of α-cut levels and fuzzy dominance indices can be generated by solving a series of deterministic submodels. The developed method is applied to a case study of a reservoir operation system. Solutions from FM-SDP provide a range of desired water-release policies under various system conditions for reservoir operation decision makers, reflecting dynamic and dual uncertain features of water availability simultaneously. The results indicate that the FM-SDP method could be applicable to practical problems for decision makers to obtain insight regarding the tradeoffs between economic and system reliability criteria. Willingness to obtain a lower benefit may guarantee meeting system-constraint demands; conversely, a desire to acquire a higher benefit could run into a higher risk of violating system constraints.

AN OPTION VALUE PROBLEM FROM SEINFELD

This is a paper about nothing.(JEL C61, D91, G11)

Analysis of stochastic dual dynamic programming method

In this paper we discuss statistical properties and convergence of the Stochastic Dual Dynamic Programming (SDDP) method applied to multistage linear stochastic programming problems. We assume that the underline data process is stagewise independent and consider the framework where at first a random sample from the original (true) distribution is generated and consequently the SDDP algorithm is applied to the constructed Sample Average Approximation (SAA) problem. Then we proceed to analysis of the SDDP solutions of the SAA problem and their relations to solutions of the “true” problem. Finally we discuss an extension of the SDDP method to a risk averse formulation of multistage stochastic programs. We argue that the computational complexity of the corresponding SDDP algorithm is almost the same as in the risk neutral case.

An Effective Approach for the Scheduling of Steel-making and Continuous Casting System with Stochastic Processing Requirements

AbstractThe steel-making and continuous casting system (SCCS) is the bottleneck in the iron and steel production, SCCS often involve various uncertainties such as the emergency customer orders, inaccurate estimate of ingredient components, the unpredictable machine breakdown or the inaccurate estimate of processing time. How to consider such uncertainties to build a better schedule in a limited time by a computationally efficient manner is becoming critical for the production of iron and steel. The stochastic dynamic programming method is adopt to solve the obtained subproblems which are relaxed by Lagrangian relaxation multipliers, a good dual solution is selected by using “ordinal optimization,” and the actual schedule is dynamically constructed based on the dual solution and the realization of random processing requirements. The method has been tested by using practical data from the Shanghai Bashan steel plant in China and could get near optimal solutions in a limited time; the stochastic processing requirements are effectively handled for the production.

Stochastic optimisation of Hydro-Quebec hydropower installations : a statistical comparison between SDP and SSDP methods

Cet article etudie le probleme de trouver une politique optimale d'exploitation des centrales hydroelectriques d'Hydro-Quebec sur les rivieres Manicouagan et aux Outardes. La methode de resolution est basee sur l'algorithme de programmation dynamique stochastique par echantillonnage (SSDP — sampling stochastic dynamic programming). Nous utilisons une nouvelle variable de l'etat hydrologique pour saisir les debits entrants; cette variable est donnee par une combinaison lineaire de l'equivalent en eau de la neige et de l'eau dans le sol. Dans une exploitation en temps reel, cette variable est calculee par un modele hydrologique et incorporee dans la politique d'exploitation afin de calculer la quantite d'eau liberee par chaque centrale. L'algorithme est compare a la programmation dynamique stochastique (SDP — stochastic dynamic programming) lag-1 deja utilise a Hydro-Quebec par le truchement d'une analyse statistique d'un ensemble de 40 scenarios historiques de debits entrants obtenus par modelisation hydrologique utilisant une temperature et une precipitation artificielles produites par un generateur meteorologique stochastique. Les resultats de l'analyse montrent que la politique d'exploitation SSDP est superieure d'un point de vue statistique a la politique d'exploitation du modele SDP lag-1. Contrairement au SDP, la politique d'exploitation SSDP ne sous-estime pas le volume de ruissellement printanier. Ainsi, il y a reduction des hm 3 d'eau deverses tout en augmentant la moyenne annuelle produite.

Beyond stochastic dynamic programming: a heuristic sampling method for optimizing conservation decisions in very large state spaces

Summary1. When managing endangered species the consequences of making a poor decision can be extinction. To make a good decision, we must account for the stochastic dynamic of the population over time. To this end stochastic dynamic programming (SDP) has become the most widely used tool to calculate the optimal policy to manage a population over time and under uncertainty.2. However, as a result of its prohibitive computational complexity, SDP has been limited to solving small dimension problems, which results in SDP models that are either oversimplified or approximated using greedy heuristics that only consider the immediate rewards of an action.3. We present a heuristic sampling (HS) method that approximates the optimal policy for any starting state. The method is attractive for problems with large state spaces as the running time is independent of the size of the problem state space and improves with time.4. We demonstrate that the HS method out‐performs a commonly used greedy heuristic and can quickly solve a problem with 33 million states. This is roughly 3 orders of magnitude larger than the largest problems that can currently be solved with SDP methods.5. We found that HS out‐performs greedy heuristics and can give near‐optimal policies in shorter timeframes than SDP. HS can solve problems with state spaces that are too large to optimize with SDP. Where the state space size precludes SDP, we argue that HS is the best technique.

Conservation decision-making in large state spaces

When looking for the best course of management decisions to efficiently conserve metapopulation systems, a classic approach in the ecology literature is to model the optimisation problem as a Markov decision process and find an optimal control policy using exact stochastic dynamic programming techniques. Stochastic dynamic programming is an iterative procedure that seeks to optimise a value function at each timestep by evaluating the benefits of each of the actions in each state of the system defined in the Markov decision process. Although stochastic dynamic programming methods provide an optimal solution to conservation management questions in a stochastic world, their applicability in metapopulation problems has always been limited by the so-called curse of dimensionality. The curse of dimensionality is the problem that adding new state variables inevitably results in much larger (often exponential) increases in the size of the state space, which can make solving superficially small problems impossible. The high computational requirements of stochastic dynamic programming methods mean that only simple metapopulation management problems can be analysed. In this paper we overcome the complexity burden of exact stochastic dynamic programming methods and present the benefits of an on-line sparse sampling algorithm proposed by Kearns, Mansour and Ng (2002). The algorithm is particularly attractive for problems with large state spaces as the running time is independent of the size of the state space of the problem. This appealing improvement is achieved at a cost: the solutions found are no longer guaranteed to be optimal. We apply the algorithm of Kearns et al. (2002) to a hypothetical fish metapopulation problem where the management objective is to maximise the number of occupied patches over the management time horizon. Our model has multiple management options to combat the threats of water abstraction and waterhole sedimentation. We compare the performance of the optimal solution to the results of the on-line sparse sampling algorithm for a simple 3-waterhole case. We find that three look-ahead steps minimises the error between the optimal solution and the approximation algorithm. This paper introduces a new algorithm to conservation management that provides a way to avoid the effects of the curse of dimensionality. The work has the potential to allow us to approximate solutions to much more complex metapopulation management problems in the future.

Modeling and Control of a Power-Split Hybrid Vehicle

Toyota hybrid system (THS) is used in the current best selling hybrid vehicle on the market-the Toyota Prius. This hybrid system contains a power-split planetary gear system which combines the benefits of series and parallel hybrid vehicles. In this paper, we developed a dynamic model of the THS powertrain and then apply it for model-based control development. Two control algorithms are introduced: one based on the stochastic dynamic programming method, and the other based on the equivalent consumption minimization strategy. Both approaches determine the engine power based on the overall vehicle efficiency and apply the electrical machines to optimize the engine operation. The performance of these two algorithms is assessed by comparing against the dynamic programming results, which are non-causal but provide theoretical benchmarks for other implementable control algorithms.

Genome-wide nucleotide-level mammalian ancestor reconstruction.

Recently attention has been turned to the problem of reconstructing complete ancestral sequences from large multiple alignments. Successful generation of these genome-wide reconstructions will facilitate a greater knowledge of the events that have driven evolution. We present a new evolutionary alignment modeler, called "Ortheus," for inferring the evolutionary history of a multiple alignment, in terms of both substitutions and, importantly, insertions and deletions. Based on a multiple sequence probabilistic transducer model of the type proposed by Holmes, Ortheus uses efficient stochastic graph-based dynamic programming methods. Unlike other methods, Ortheus does not rely on a single fixed alignment from which to work. Ortheus is also more scaleable than previous methods while being fast, stable, and open source. Large-scale simulations show that Ortheus performs close to optimally on a deep mammalian phylogeny. Simulations also indicate that significant proportions of errors due to insertions and deletions can be avoided by not assuming a fixed alignment. We additionally use a challenging hold-out cross-validation procedure to test the method; using the reconstructions to predict extant sequence bases, we demonstrate significant improvements over using closest extant neighbor sequences. Accompanying this paper, a new, public, and genome-wide set of Ortheus ancestor alignments provide an intriguing new resource for evolutionary studies in mammals. As a first piece of analysis, we attempt to recover "fossilized" ancestral pseudogenes. We confidently find 31 cases in which the ancestral sequence had a more complete sequence than any of the extant sequences.