Infinite Horizon Stochastic Control Problem Research Articles

Infinite horizon backward stochastic Volterra integral equations and discounted control problems

Infinite horizon backward stochastic Volterra integral equations (BSVIEs for short) are investigated. We prove the existence and uniqueness of the adapted M-solution in a weightedL2-space. Furthermore, we extend some important known results for finite horizon BSVIEs to the infinite horizon setting. We provide a variation of constant formula for a class of infinite horizon linear BSVIEs and prove a duality principle between a linear (forward) stochastic Volterra integral equation (SVIE for short) and an infinite horizon linear BSVIE in a weightedL2-space. As an application, we investigate infinite horizon stochastic control problems for SVIEs with discounted cost functional. We establish both necessary and sufficient conditions for optimality by means of Pontryagin’s maximum principle, where the adjoint equation is described as an infinite horizon BSVIE. These results are applied to discounted control problems for fractional stochastic differential equations and stochastic integro-differential equations.

Optimization of Constrained Stochastic Linear-Quadratic Control on an Infinite Horizon: A Direct-Comparison Based Approach

In this paper we study the optimization of the discrete-time stochastic linear-quadratic (LQ) control problem with conic control constraints on an infinite horizon, considering multiplicative noises. Stochastic control systems can be formulated as Markov Decision Problems (MDPs) with continuous state spaces and therefore we can apply the direct-comparison based optimization approach to solve the problem. We first derive the performance difference formula for the LQ problem by utilizing the state separation property of the system structure. Based on this, we successfully derive the optimality conditions and the stationary optimal feedback control. By introducing the optimization, we establish a general framework for infinite horizon stochastic control problems. The direct-comparison based approach is applicable to both linear and nonlinear systems. Our work provides a new perspective in LQ control problems; based on this approach, learning based algorithms can be developed without identifying all of the system parameters.

Ergodic control of infinite-dimensional stochastic differential equations with degenerate noise

This paper is devoted to the study of the asymptotic behaviour of the value functions of both finite and infinite horizon stochastic control problems and to the investigation of their relationship with suitable stochastic ergodic control problems. Our methodology is based only on probabilistic techniques, as, for instance, the so-called randomisation of the control method, thus avoiding completely analytical tools from the theory of viscosity solutions. We are then able to treat with the case where the state process takes values in a general (possibly infinite-dimensional) real separable Hilbert space, and the diffusion coefficient is allowed to be degenerate.

Modeling and Analysis of the Role of Energy Storage for Renewable Integration: Power Balancing

The high variability of renewable energy is a major obstacle toward its increased penetration. Energy storage can help reduce the power imbalance due to the mismatch between the available renewable power and the load. How much can storage reduce this power imbalance? How much storage is needed to achieve this reduction? This paper presents a simple analytic model that leads to some answers to these questions. Considering the multitimescale grid operation, we formulate the power imbalance problem for each timescale as an infinite horizon stochastic control problem and show that a greedy policy minimizes the average magnitude of the residual power imbalance. Observing from the wind power data that in shorter timescales the power imbalance can be modeled as an iid zero-mean Laplace distributed process, we obtain closed form expressions for the minimum cost and the stationary distribution of the stored power. We show that most of the reduction in the power imbalance can be achieved with relatively small storage capacity. In longer timescales, the correlation in the power imbalance cannot be ignored. As such, we relax the iid assumption to a weakly dependent stationary process and quantify the limit on the minimum cost for arbitrarily large storage capacity.

Stationary Hamilton--Jacobi Equations in Hilbert Spaces and Applications to a Stochastic Optimal Control Problem

We study an infinite horizon stochastic control problem associated with a class of stochastic reaction-diffusion systems with coefficients having polynomial growth. The hamiltonian is assumed to be only locally Lipschitz continuous so that the quadratic case can be covered. We prove that the value function V corresponding to the control problem is given by the solution of the stationary Hamilton--Jacobi equation associated with the state system. To this purpose we write the Hamilton--Jacobi equation in integral form, and, by using the smoothing properties of the transitionsemigroup relative to the state system and the theory of m-dissipative operators, we show that it admits a unique solution. Moreover, the value function V is obtained as the limit of minima for some approximating control problems which admit unique optimal controls and states.

On the Properties of Linear Decision Rules and Their Derivation by an Iterative Procedure

The paper develops a simple iterative procedure for deriving linear decision rules which provide the optimal control policy for a stochastic dynamic linear system. The procedure works for a quadratic objective function with any time horizon up to and including infinity, either with or without time discounting. The role of target variables is conisidered and there is a discussion of the results which ensue if these targets are incompatible, that is, if they do not satisfy the underlying structural model. The paper concludes with some consideration of the convergence and other properties of the controlled system. THIS PAPER DEVELOPS a simple iterative method for deriving linear decision rules which provide the control policy for a stochastic dynamic linear system which is optimal for a quadratic criterion. The basic theory in economics was developed by Holt, Simon, Theil, Phillips, and others2 in the fifties and has recently been extended by Aoki [1], Chow [2 and 3], and Turnovsky [10]. The method described here is similar to that used by Chow [3] where the dynamic structure of the model is used to develop a suitable iterative procedure. This procedure is computationally simple, of low dimensionality, and may be applied to a system with any number of lags, irrespective of whether it is stable or unstable. For economic applications, the underlying system would typically be an econometric model in reduced form which has either been specially estimated as a completely linear model or has been suitably linearized. In Section 2 of the paper we derive a general procedure for solving an infinite horizon quadratic programming problem, proving both its convergence and optimality properties. In Sections 3 and 4 we discuss how this procedure may be adapted to solve finite and infinite horizon stochastic control problems and demonstrate some properties of the optimal path. Since the method produces an analytically explicit solution we are enabled to develop some further convergence properties of the infinite horizon, optimal path in Section 5. The specific control problem to be discussed in this paper is one of the following