Horizon Stochastic Optimal Control Problem Research Articles

Nonconvex homogenization for one-dimensional controlled random walks in random potential

We consider a finite horizon stochastic optimal control problem for nearest-neighbor random walk $\{X_{i}\}$ on the set of integers. The cost function is the expectation of the exponential of the path sum of a random stationary and ergodic bounded potential plus $\theta X_{n}$. The random walk policies are measurable with respect to the random potential, and are adapted, with their drifts uniformly bounded in magnitude by a parameter $\delta\in[0,1]$. Under natural conditions on the potential, we prove that the normalized logarithm of the optimal cost function converges. The proof is constructive in the sense that we identify asymptotically optimal policies given the value of the parameter $\delta$, as well as the law of the potential. It relies on correctors from large deviation theory as opposed to arguments based on subadditivity which do not seem to work except when $\delta=0$. The Bellman equation associated to this control problem is a second-order Hamilton–Jacobi (HJ) partial difference equation with a separable random Hamiltonian which is nonconvex in $\theta$ unless $\delta=0$. We prove that this equation homogenizes under linear initial data to a first-order HJ equation with a deterministic effective Hamiltonian. When $\delta=0$, the effective Hamiltonian is the tilted free energy of random walk in random potential and it is convex in $\theta$. In contrast, when $\delta=1$, the effective Hamiltonian is piecewise linear and nonconvex in $\theta$. Finally, when $\delta\in(0,1)$, the effective Hamiltonian is expressed completely in terms of the tilted free energy for the $\delta=0$ case and its convexity/nonconvexity in $\theta$ is characterized by a simple inequality involving $\delta$ and the magnitude of the potential, thereby marking two qualitatively distinct control regimes.

Infinite Horizon Stochastic Optimal Control Problems with Running Maximum Cost

An infinite horizon stochastic optimal control problem with running maximum cost is considered. The value function is characterized as the viscosity solution of a second-order HJB equation with mixed boundary condition. A general numerical scheme is proposed and convergence is established under the assumptions of consistency, monotonicity and stability of the scheme. A convergent semi-Lagrangian scheme is presented in detail.

Solution to Discrete-Time Linear FBSDEs with Application to Stochastic Control Problem

In this paper, we consider a class of infinite horizon forward and backward stochastic difference equations (FBSDEs) that are fully coupled. The adapted solution is given by establishing an explicit relation between the forward and backward components in terms of a generalized algebraic Riccati equation. The equivalence between the exponential stabilizability of the stochastic control system and the infinite horizon FBSDEs is derived. As an application, the FBSDEs are used to characterize the maximum principle of the infinite horizon stochastic optimal control problem.

Neural Network-Based Solutions for Stochastic Optimal Control Using Path Integrals.

In this paper, an offline approximate dynamic programming approach using neural networks is proposed for solving a class of finite horizon stochastic optimal control problems. There are two approaches available in the literature, one based on stochastic maximum principle (SMP) formalism and the other based on solving the stochastic Hamilton-Jacobi-Bellman (HJB) equation. However, in the presence of noise, the SMP formalism becomes complex and results in having to solve a couple of backward stochastic differential equations. Hence, current solution methodologies typically ignore the noise effect. On the other hand, the inclusion of noise in the HJB framework is very straightforward. Furthermore, the stochastic HJB equation of a control-affine nonlinear stochastic system with a quadratic control cost function and an arbitrary state cost function can be formulated as a path integral (PI) problem. However, due to curse of dimensionality, it might not be possible to utilize the PI formulation for obtaining comprehensive solutions over the entire operating domain. A neural network structure called the adaptive critic design paradigm is used to effectively handle this difficulty. In this paper, a novel adaptive critic approach using the PI formulation is proposed for solving stochastic optimal control problems. The potential of the algorithm is demonstrated through simulation results from a couple of benchmark problems.

Benefits of Storage Control for Wind Power Producers in Power Markets

We consider a wind power producer (WPP) participating in a dynamically evolving two settlement power market. We study the utility of energy storage for a WPP in maximizing its expected profit. With random wind and price processes, the optimal forward contract and storage charging/discharging decisions are formulated as solutions of an infinite horizon stochastic optimal control problem. For the asymptotically small storage capacity regime, we precisely characterize the maximum profit increase brought by utilizing energy storage. We prove that, in this regime, an optimal policy uses storage to compensate for power delivery shortfall/surplus in real time, without changing the forward contracts from the optimal ones in the absence of energy storage. This policy also serves as an approximately optimal policy for the case of relatively small storage capacity. We also design a policy based on model predictive control (MPC) that is approximately optimal for general storage capacities. We numerically evaluate the developed policies for wind and price processes with representative statistics from real world data. It is observed that, as expected, the simple small storage approximation policy performs closely to the optimum when storage is relatively small, while the more complex stochastic MPC policy performs better for larger storage capacities.

Nonquadratic stochastic model predictive control: A tractable approach

This paper deals with the finite horizon stochastic optimal control problem with the expectation of the p-norm as the objective function and jointly Gaussian, although not necessarily independent, additive disturbance process. We develop an approximation strategy that solves the problem in a certain class of nonlinear feedback policies while ensuring satisfaction of hard input constraints. A bound on suboptimality of the proposed strategy in this class of nonlinear feedback policies is given for the special case of p=1. We also develop a recursively feasible receding horizon policy with respect to state chance constraints and/or hard control input constraints in the presence of bounded disturbances. The performance of the proposed policies is examined in two numerical examples.

Analytic solutions for infinite horizon stochastic optimal control problems via finite horizon approximation: A practical guide

We show how infinite horizon stochastic optimal control problems can be solved via studying their finite horizon approximations. This often leads to analytical solutions for the infinite horizon problem by studying phase diagrams, even in cases where the complexity of the finite horizon case does not permit analytic solutions. Our approach can be applied to many problems in dynamic economics.

Analytic Solutions for Infinite Horizon Stochastic Optimal Control Problems via Finite Horizon Approximation: A Practical Guide

We show how infinite horizon stochastic optimal control problems can be solved via studying their finite horizon approximations. This often leads to analytical solutions for the infinite horizon problem, even when the complexity of the finite horizon approximation is to large, as in order to allow analytical solutions in the finite horizon case.

Extended Ritz method for reservoir management over an infinite horizon

The management problem of water reservoirs can be formulated as a stochastic optimal control (SOC) problem, where the objective function is an aggregated cost that accounts for the interests acting in the water system (e.g. hydropower production, irrigation supply, etc.) and the design variable is the reservoirs release policy. Solving the SOC problem through stochastic dynamic programming is often impossible, since the numerical resolution of the Bellman equation is computationally prohibitive even for small reservoir networks. An approximate solution can be searched for by assuming a priori the family of function to which the control laws belong and replacing the SOC problem with a nonlinear programming one. Recently, a method based on this approach has been proposed in the literature, coupled with the use of nonlinear approximating networks to approximate the optimal control laws. This optimization method, called Extended RItz Method (ERIM), is suited for finite horizon SOC problems. However, management problems for environmental systems are spontaneously formulated over an infinite horizon, since the life time of the system is infinite. This paper thus presents an extension of the ERIM to the infinite horizon case. The algorithm that implements such method is tested on a numerical example where a 10-reservoirs network is optimized for hydropower production and irrigation supply.

Infinite Horizon Stochastic Optimal Control Problems with Degenerate Noise and Elliptic Equations in Hilbert Spaces

Semilinear elliptic partial differential equations are solved in a mild sense in an infinite-dimensional Hilbert space. These results are applied to a stochastic optimal control problem with infinite horizon. Applications to controlled stochastic heat and wave equations are given.

Distributed dynamic programming

We consider distributed algorithms for solving dynamic programming problems whereby several processors participate simultaneously in the computation while maintaining coordination by information exchange via communication links. A model of asynchronous distributed computation is developed which requires very weak assumptions on the ordering of computations, the timing of information exchange, the amount of local information needed at each computation node, and the initial conditions for the algorithm. The class of problems considered is very broad and includes shortest path problems, and finite and infinite horizon stochastic optimal control problems. When specialized to a shortest path problem the algorithm reduces to the algorithm originally implemented for routing of messages in the ARPANET.