Stochastic Dynamic Optimization Problem Research Articles

Linear-quadratic approximation, external habit and targeting rules

Abstract We examine the linear-quadratic approximation of nonlinear dynamic stochastic optimization problems. A discrete-time version of Magill [1977a. A local analysis of N-sector capital accumulation under uncertainty. Journal of Economic Theory 15(2), 211–219] is generalized to models with forward-looking variables paying special attention to second-order conditions. This is the ‘large distortions’ case in the literature. We apply the approach to monetary policy in a DSGE model with external habit in consumption. We then develop a condition for ‘target-implementability’, a concept related to ‘targeting rules’. Finally, we extend the approach to a comparison between cooperative and non-cooperative equilibria in a two-country model and show that the ‘small distortions’ approximation is inappropriate for this exercise.

Linear-Quadratic Approximation, External Habit and Targeting Rules

We examine the linear-quadratic (LQ) approximation of non-linear stochastic dynamic optimization problems in macroeconomics, in particular for monetary policy. We make four main contributions: first, we draw attention to a general Hamiltonian framework for LQ approximation due to Magill (1977). We show that the procedure for the ‘large distortions’ case of Benigno and Woodford (2003, 2005) is equivalent to the Hamiltonian approach, but the latter is far easier to implement. Second, we apply the Hamiltonian approach to a Dynamic Stochastic General Equilibrium model with external habit in consumption. Third, we introduce the concept of target-implementability which fits in with the general notion of targeting rules proposed by Svensson (2003, 2005). We derive sufficient conditions for the LQ approximation to have this property in the vicinity of a zero-inflation steady state. Finally, we extend the Hamiltonian approach to a non-cooperative equilibrium in a two-country model. JEL Classification: E52, E37, E58

Elevator Group Supervisory Control System Using Genetic Network Programming with Macro Nodes and Reinforcement Learning

Elevator Group Supervisory Control System (EGSCS) is a very large scale stochastic dynamic optimization problem. Due to its vast state space, significant uncertainty and numerous resource constraints such as finite car capacities and registered hall/car calls, it is hard to manage EGSCS using conventional control methods. Recently, many solutions for EGSCS using Artificial Intelligence (AI) technologies have been reported. Genetic Network Programming (GNP), which is proposed as a new evolutionary computation method several years ago, is also proved to be efficient when applied to EGSCS problem. In this paper, we propose an extended algorithm for EGSCS by introducing Reinforcement Learning (RL) into GNP framework, and an improvement of the EGSCS' performances is expected since the efficiency of GNP with RL has been clarified in some other studies like tile-world problem. Simulation tests using traffic flows in a typical office building have been made, and the results show an actual improvement of the EGSCS' performances comparing to the algorithms using original GNP and conventional control methods. Furthermore, as a further study, an importance weight optimization algorithm is employed based on GNP with RL and its efficiency is also verified with the better performances.

A generalization of the endogenous grid method

This paper extends Carroll's [2005. The method of endogenous gridpoints for solving dynamic stochastic optimization problems. National Bureau of Economic Research Technical Working Paper 309]. endogenous grid method to perform value function iteration in models with more than one control variable. We propose to mix the endogenous grid method with standard value function iteration to achieve higher efficiency. We illustrate the method using the stochastic neoclassical growth model with and without labor–leisure choice. We report important gains in efficiency that make value function iteration an attractive computational method in terms of both computing time and accuracy.

The method of endogenous gridpoints for solving dynamic stochastic optimization problems

This paper introduces a solution method for numerical dynamic stochastic optimization problems that avoids rootfinding operations. The idea is applicable to many microeconomic and macroeconomic problems, including life cycle, buffer-stock, and stochastic growth problems. Software is provided.

Measuring Risk for Income Streams

A measure of risk is introduced for a sequence of random incomes adapted to some filtration. This measure is formulated as the optimal net present value of a stream of adaptively planned commitments for consumption. The new measure is calculated by solving a stochastic dynamic linear optimization problem which, for finite filtrations, reduces to a deterministic linear programming problem. We analyze properties of the new measure by exploiting the convexity and duality structure of the stochastic dynamic linear problem. The measure depends on the full distribution of the income process (not only on its marginal distributions) as well as on the filtration, which is interpreted as the available information about the future. The features of the new approach are illustrated by a numerical example.

Stochastic intertemporal duality: An application to investment under uncertainty

We present a duality concept for a class of stochastic dynamic optimization problems that especially includes models of investment under uncertainty. This concept allows us to derive explicit characterizations of optimal investment behavior from an adjustment cost model with expectations described by linear or nonlinear stochastic processes. It covers settings that are impenetrable with a direct approach, like expectations with a multimodal density or certain forms of non-convex technologies.

Optimal consumption and investment problems under GARCH with transaction costs

General multiperiod optimal consumption and investment problems with proportional transaction costs are investigated in this paper, a GARCH-type process is used to model the risky asset’s return series so that its time-varying moments and conditional heteroskedasticity can be properly described. We model this kind of consumption and investment problems as dynamic stochastic optimization problems, which can easily cope with different utility functions and any number of time periods. The procedure to efficiently solve the resulting nonlinear stochastic optimization problem is discussed in detail and a parallelizable decomposition algorithm is devised. Numerical results show the suitability and promise of our methodology.

Epi-Convergent Discretizations of Multistage Stochastic Programs

In many dynamic stochastic optimization problems in practice, the uncertain factors are best modeled as random variables with an infinite support. This results in infinite-dimensional optimization problems that can rarely be solved directly. Therefore, the random variables (stochastic processes) are often approximated by finitely supported ones (scenario trees), which result in finite-dimensional optimization problems that are more likely to be solvable by available optimization tools. This paper presents conditions under which such finite-dimensional optimization problems can be shown to epi-converge to the original infinite-dimensional problem. Epi-convergence implies the convergence of optimal values and solutions as the discretizations are made finer. Our convergence result applies to a general class of convex problems where neither linearity nor complete recourse are assumed.

The Timing of Uncertainty and the Intensity of Policy

This article analyzes the trade-off between `caution' and `intensity' in the use of the control variable in a one-state one-control dynamic stochastic quadratic linear optimization problem with discount factor. It studies the effects that changes in uncertainty of the control parameter have on the optimal first period response of the control variable, showing that the trade-off between `caution' and `intensity' depends on the timing of the uncertainty. Given an increase in current uncertainty and an equal increase in future uncertainty, caution will always prevail over intensity. Moreover, the prevalence of caution will be enlarged as the increase in future uncertainty moves farther away into the future, while this prevalence will be reduced as the increase in future uncertainty expands into the future. Finally, for the infinite horizon case, caution is the optimal policy response.

Integrated risk management of hydro power scheduling and contract management

The paper describes the implementation of a new integrated tool for risk management in hydropower systems. Earlier practice in Scandinavia has been to separate operations scheduling and contract management. In the present approach operation scheduling and hedging by future contacts are integrated in one model. The risk level is controlled by setting revenue targets. Revenues below target are penalized; this implicitly defines a revenue utility function to reduce risk. The possibility of dynamically changing the future contract portfolio is now represented. The resulting large stochastic dynamic optimization problem is solved using a combination of stochastic dynamic programming and stochastic dual dynamic programming. Simulations for a test case show that the profit in the lower range is considerably improved with the new tool. The approach can be useful for hydropower companies that face price risks in addition to the inflow uncertainty, as is the case in a deregulated system.

INDIVIDUAL LEARNING ABOUT CONSUMPTION

The standard approach to modeling consumption/saving problems is to assume that the decisionmaker is solving a dynamic stochastic optimization problem. However, under realistic descriptions of utility and uncertainty, the optimal consumption/saving decision is so difficult that only recently have economists managed to find solutions, using numerical methods that require previously infeasible amounts of computation. Yet, empirical evidence suggests that household behavior conforms fairly well with the prescriptions of the optimal solution, raising the question of how average households can solve problems that economists, until recently, could not. This paper examines whether consumers might be able to find a reasonably good rule-of-thumb approximation to optimal behavior by trial-and-error methods, as Milton Friedman proposed long ago. We find that such individual learning methods can reliably identify reasonably good rules of thumb only if the consumer is able to spend absurdly large amounts of time searching for a good rule.

Oligopoly equilibria in nonrenewable resource markets

Most Nash–Cournot oligopoly models of nonrenewable resources apply open-loop equilibrium concepts and are based on physical resource depletion. This paper studies feedback equilibria and economic depletion. Assuming affine-quadratic functional forms, the existence, uniqueness, and explicit solutions for the equilibria are derived for duopoly and n-player oligopoly with multiple resource stocks. For the cases of nonquadratic criteria, we develop a numeric solution scheme for the Nash feedback equilibrium. This scheme is an application of a discrete time, discrete state controlled Markov chain approximation method originally developed for solving deterministic and stochastic dynamic optimization problems. In our Nash–Cournot equilibrium, the degree of concentration in supply declines over time whereas the previous models with physical depletion and open-loop equilibrium concepts predict that a Nash–Cournot resource market will develop in the direction of monopoly supply.

Optimal Asset Allocation towards the End of the Life Cycle: To Annuitize or Not to Annuitize?

INTRODUCTION ...It is a well known fact that annuity contracts, other than in the form of group insurance through pension systems, are extremely rare. Why this should be so is a subject of considerable current interest. It is still ill-understood. Adverse selection, causing an unfavorable payout, and the fact that some utility may be derived from bequest are, presumably, an important part of the answer.... Franco Modigliani, December 9, 1985 Nobel Prize acceptance speech in Stockholm, Sweden.(1) Most individuals must decide how much, if any, of their wealth should be annuitized near the time they retire. For many individuals a large portion of wealth is forcefully annuitized; for example, pensions and government social security. In other cases they have discretion in the matter. In its most general form, purchasing a life annuity involves paying a non-refundable lump sum to an insurance company in exchange for a guaranteed constant life-long consumption stream that cannot be outlived. The natural alternative to annuitization is investment amongst the various asset classes, such as equity, fixed income and real estate, together with a fixed periodic withdrawal equivalent to the consumption stream generated by the annuity. This do-it-yourself strategy incurs the financial risk of under-funding retirement in the event of long-run inferior investment returns in conjunction with unexpected human longevity. As Modigliani (1986), Friedman and Warshawsky (1990), Miter (1994), Poterba and Wise (1996) and many others have pointed out, very few people consciously choose to annuitize their marketable (liquid or discretionary) wealth, as evidenced by the comprehensive Health and Retirement Survey (HRS), conducted in the United States.(2) Only 1.57 percent of the HRS respondents reported annuity income. Likewise, only 8.0 percent of HRS respondents with a defined contribution pension plan selected an annuity payout. This phenomenon is especially puzzling within the paradigm of the Ando and Modigliani (1963) Life Cycle Hypothesis (LCH), under which individuals seek to their lifetime consumption by annuitizing wealth. What better way is there to smooth and consumption for the rest of one's natural life? The most common answer is to simply abandon the strict form of the life-cycle hypothesis and declare that individuals have strong bequest motives, as Bernheim (1991), Hurt (1989) and many others have suggested. Another approach, which we prefer, is to argue that even when individuals have negligible bequest motives, annuities are simply too expensive, as was hypothesized by Warshawsky (1988) and Friedman and Warshawsky (1990). This means that the implied rates of return from life annuities are much lower as a result of transaction costs, or loads, than those available from other investment assets, considering the life-long consumption guarantee which they provide. The contribution of this paper is to give prescriptive advice in the face of these costs. We suggest that most individuals should defer annuitization, via the do-it-yourself scheme, until is no longer possible to beat the mortality adjusted rate of return from the life annuity. We call this approach the do-it-yourself-and-then-switch strategy. Our methodology deviates from the traditional financial economic utility maximizing approach to asset allocation by focusing on the probability of consumption shortfall, as the measure of risk. The classic utility-based approach would involve solving the following dynamic stochastic optimization problem: [Mathematical Expression Omitted], (1) where U([C.sub.t]) denotes the instantaneous utility of consumption; [Rho] is the personal discount rate for consumption; B([W.sub.T]) denotes the utility of bequest; [Gamma] is the discount rate for bequest; T is the stochastic time of death; [a.sub.t] is the amount of annuities; [C.sub.t] is the rate of consumption; [W. …

Elevator Group Control Using Multiple Reinforcement Learning Agents

Recent algorithmic and theoretical advances in reinforcement learning (RL) have attracted widespread interest. RL algorithms have appeared that approximate dynamic programming on an incremental basis. They can be trained on the basis of real or simulated experiences, focusing their computation on areas of state space that are actually visited during control, making them computationally tractable on very large problems. If each member of a team of agents employs one of these algorithms, a new collective learning algorithm emerges for the team as a whole. In this paper we demonstrate that such collective RL algorithms can be powerful heuristic methods for addressing large-scale control problems. Elevator group control serves as our testbed. It is a difficult domain posing a combination of challenges not seen in most multi-agent learning research to date. We use a team of RL agents, each of which is responsible for controlling one elevator car. The team receives a global reward signal which appears noisy to each agent due to the effects of the actions of the other agents, the random nature of the arrivals and the incomplete observation of the state. In spite of these complications, we show results that in simulation surpass the best of the heuristic elevator control algorithms of which we are aware. These results demonstrate the power of multi-agent RL on a very large scale stochastic dynamic optimization problem of practical utility.

Chow's method of optimal control

The note investigates the numerical properties of the method proposed by Chow (1993) to solve stochastic dynamic optimization problems. The results are compared to the standard method of linear quadratic approximation. The properties of these approximation schemes under heteroskedastic noise are examined.

Optimal power system expansion planning under uncertain CO2 emissions control policies

AbstractMany efforts towards a settlement of the CO2 problem have already been made through internationally organized meetings. It is not unlikely that some targets will be set for the reduction of CO2 emissions. If a stringent reduction target should be imposed on CO2 emitted from a national energy system, this can possibly have as severe an impact upon electric power sectors as the oil price crises experienced in the seventies. It is, however, quite uncertain whether or not political muscles will be used specifically for tackling the CO2 problem in the near future.In such a context, this paper presents a new method for optimal power system planning under uncertain CO2 emissions control policies. The derived optimal expansion plan gives us the minimum expected value of the sum of the total system cost and the total amount of carbon taxes levied on net CO2 emissions from fossil fuel‐fired power plants. The stochastic dynamic optimization problem discussed here is formulated as a linear programming problem decomposable into several small subproblems. The proposed method can take into account a specific future scenario of the magnitude of control policy uncertainties, which can be presumably resolved in the first half of the next century due to the increase in scientific wisdom.In order to evaluate the usefulness of the method proposed here, the authors also present a simple case study with the Japanese national power system up to the year 2050.

A recursive forward simulation method for solving nonlinear rational expectations models

This paper describes a recursive forward simulation method to solve dynamic stochastic optimization problems. The proposed simulation method uses a combination of nonlinear Euler equations and linearized decision rules to obtain numerical solution paths for the endogenous variables of the model. Diagnostic tests from 1000 repetitions for each of the ten parameter configurations of a one-sector stochastic growth model indicate that forward simulation and standard simulation are about equally successful in generating a martingale Euler equation shock when the shock variance is small, but that forward simulation outperforms standard simulation when the shock variance is large.

A theory of rolling horizon decision making

In this paper, we develop a theoretical framework for the common business practice of rolling horizon decision making. The main idea of our approach is that the usefulness of rolling horizon methods is, to a great extent, implied by the fact that forecasting the future is a costly activity. We, therefore, consider a general, discrete-time, stochastic dynamic optimization problem in which the decision maker has the possibility to obtain information on the uncertain future at given cost. For this non-standard optimization problem with optimal stopping decisions, we develop a dynamic programming formulation. We treat both finite and infinite horizon cases. We also provide a careful interpretation of the dynamic programming equations and illustrate our results by a simple numerical example. Various generalizations are shown to be captured by straightforward modifications of our model.

Rule-based production planning in flexible manufacturing systems

Production planning in flexible manufacturing may require the solution of a large-scale discrete-event dynamic stochastic optimization problem, due to the complexity of the system to be optimized, and to the occurrence of discrete events (new orders and hard failures). The production planning problem is here approached for a multistage multipart-type manufacturing shop, where each work cell can share its processing time among the different types of parts. The solution of this problem is obtained by an open-loop-feedback control strategy, updated each time a new event occurs. At each event time, two coupled problems are solved: 1) a product-order scheduling problem, conditioned on estimated values of the production capacities of all component work cells; and 2) a production-capacity planning problem, conditioned on predefined sequences of the product orders to be processed. In particular, the article aims at defining a production planning procedure that integrates both analytical tools, derived from mathematical programming, and knowledge-based rules, coming from experience. The objective is to formulate a hybrid (knowledge-based/analytical) planning architecture, and to analyze its use for multicell multipart-type manufacturing systems.