Discrete-time Stochastic Optimal Control Problem Research Articles

First- and Second-Order Maximum Principles for Discrete-Time Stochastic Optimal Control With Recursive Utilities

This paper deals with the discrete-time stochastic optimal control problems with recursive utilities under weakened convexity assumption. A new stochastic maximum principle is established. Moreover, by constructing two new adjoint equations and two new variational equations for the backward stochastic difference equation (BS <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\Delta$</tex-math></inline-formula> E), we obtain the second-order necessary optimality condition of quasi-singular control. Finally, as an illustration, a discrete-time mean-variance portfolio selection mixed with a recursive utility functional optimization problem is solved.

A maximum principle for discrete-time stochastic optimal control problem with delay

A maximum principle for discrete-time stochastic optimal control problem with delay

Modified dynamic programming algorithms for GLOSA systems with stochastic signal switching times

A discrete-time stochastic optimal control problem was recently proposed to address the GLOSA (Green Light Optimal Speed Advisory) problem in cases where the next signal switching time is decided in real time and is therefore uncertain in advance. The corresponding numerical solution via SDP (Stochastic Dynamic Programming) calls for substantial computation time, which excludes problem solution in the vehicle’s on-board computer in real time. In this context, the present paper concentrates on the challenge of developing numerical algorithms to solve efficiently the stochastic GLOSA problem. As a first attempt to overcome the computation time bottleneck, a modified version of Dynamic Programming, known as Discrete Differential Dynamic Programming (DDDP) was recently employed for the numerical solution of the stochastic optimal control problem and was demonstrated to achieve results equivalent to those obtained with the ordinary SDP algorithm, albeit with significantly reduced computation times. After an outline of the DDDP approach, the present work considers a second modified version of Dynamic Programming, known as Differential Dynamic Programming (DDP). For the stochastic GLOSA problem, it is demonstrated that DDP achieves quasi-instantaneous (extremely fast) solutions in terms of CPU times, which allows for the proposed approach to be readily executable online, in an MPC (Model Predictive Control) framework, in the vehicle’s on-board computer. The novel numerical approach is tested and demonstrated by use of realistic examples and is compared to the SDP and DDDP solutions. It should be noted that DDP does not require discretization of variables, hence the obtained solutions may be slightly superior to the standard SDP solutions.

State estimation and optimal control of an inverted pendulum on a cart system with stochastic approximation approach

&lt;abstract&gt; &lt;p&gt;In this paper, optimal control of an inverted pendulum on a cart system is studied. Since the nonlinear structure of the system is complex, and in the presence of random disturbances, optimization and control of the motion of the system become more challenging. For handling this system, a discrete-time stochastic optimal control problem for the system is described, where the external force is considered as the control input. By defining a loss function, namely, the mean squared errors to be minimized, the stochastic approximation (SA) approach is applied to estimate the state dynamics. In addition, the Hamiltonian function is defined, and the first-order necessary conditions are derived. The gradient of the cost function is determined so that the SA approach is employed to update the control sequences. For illustration, considering the values of the related parameters in the system, the discrete-time stochastic optimal control problem is solved iteratively by using the SA algorithm. The simulation results show that the state estimation and the optimal control law design are well performed with the SA algorithm, and the motion of the inverted pendulum cart is addressed satisfactorily. In conclusion, the efficiency of the SA approach for solving the inverted pendulum on a cart system is verified.&lt;/p&gt; &lt;/abstract&gt;

Optimal Trading Algorithms under Regime Switching

In this article, the author is concerned with the problem of efficiently trading a large position in the marketplace when the stock price dynamic follows a regime-switching process. If the execution of a large order is not done properly, this will certainly lead to large losses. Given that the execution of a large position may take several trading days, it is therefore reasonable to assume that the market microstructure may change during the execution of the order. To address this possibility, the author assumes that the stock price follows a regime-switching model. This article is particularly interested in trading algorithms that track market benchmarks such as the volume-weighted average price (VWAP) and the minimum execution shortfall. The author proposes trading algorithms that break the execution order into small pieces and execute them over a predetermined period of time so as to minimize the overall execution shortfall or exceed the overall market VWAP. The underlying problem is formulated as a discrete-time stochastic optimal control problem with resource constraints. The value function and optimal trading strategies are derived in closed form. Numerical simulations with market data are reported to illustrate the pertinence of the approach.

Maximum principle for discrete-time stochastic optimal control problem and stochastic game

&lt;p style='text-indent:20px;'&gt;This paper is first concerned with one kind of discrete-time stochastic optimal control problem with convex control domains, for which necessary condition in the form of Pontryagin's maximum principle and sufficient condition of optimality are derived. The results are then extended to two kinds of discrete-time stochastic games. Two illustrative examples are studied, for which the explicit optimal strategies are given. This paper establishes a rigorous version of discrete-time stochastic maximum principle in a clear and concise way and paves a road for further related topics.&lt;/p&gt;

An efficient algorithm for stochastic optimal control problems by means of a least-squares Monte-Carlo method

In this work, we provide discrete optimality conditions of the optimal control problems of stochastic differential equations. Euler and Runge–Kutta methods are used for discretization. A Lagrange multiplier method for a discrete-time stochastic optimal control problem is formulated. The discrete adjoint process is obtained in terms of conditional expectations and for both methods. To estimate these nested conditional expectations at each time step via simulation, we use a very powerful new approach, least-squares Monte-Carlo method, developed by Longstaff–Schwartz. This is the first time to solve a stochastic optimal control problem by calculating the nested conditional expectations numerically with the help of a least-squares Monte-Carlo method. Some examples are studied to test and demonstrate the efficiency of the Lagrange multiplier combined with the least-squares Monte-Carlo method.

Necessary First-Order and Second-Order Optimality Conditions in Discrete-Time Stochastic Systems

In this paper, first-order and second-order necessary conditions for optimality for discrete-time stochastic optimal control problems governed by discrete-time Ito equations are established. A new discrete-time backward stochastic equation and discrete-time backward stochastic matrix equation are introduced. Based on the discrete-time backward stochastic Ito equation, a discrete-time stochastic maximum principle for the stochastic discrete optimal control problems is obtained. Moreover, using the discrete-time backward stochastic matrix equation the second-order necessary conditions for optimality are obtained.

A Maximum Principle for Optimal Control of Discrete-Time Stochastic Systems With Multiplicative Noise

The maximum principle (MP) for the discrete-time stochastic optimal control problems is proved. It is shown that the adjoint equations of the MP are a pair of backward stochastic difference equations.

Exact optimal solution for a class of dual control problems

This paper considers a discrete-time stochastic optimal control problem for which only measurement equation is partially observed with unknown constant parameters taking value in a finite set of stochastic systems. Because of the fact that the cost-to-go function at each stage contains variance and the non-separability of the variance is so complicated that the dynamic programming cannot be successfully applied, the optimal solution has not been found. In this paper, a new approach to the optimal solution is proposed by embedding the original non-separable problem into a separable auxiliary problem. The theoretical condition on which the optimal solution of the original problem can be attained from a set of solutions of the auxiliary problem is established. In addition, the optimality of the interchanging algorithm is proved and the analytical solution of the optimal control is also obtained. The performance of this controller is illustrated with a simple example.

Optimal algorithms for trading large positions

In this paper, we are concerned with the problem of efficiently trading a large position on the market place. If the execution of a large order is not dealt with appropriately this will certainly break the price equilibrium and result in large losses. Thus, we consider a trading strategy that breaks the order into small pieces and execute them over a predetermined period of time so as to minimize the overall execution shortfall while matching or exceeding major execution benchmarks such as the volume-weighted average price (VWAP). The underlying problem is formulated as a discrete-time stochastic optimal control problem with resource constraints. The value function and optimal trading strategies are derived in closed form. Numerical simulations with market data are reported to illustrate the pertinence of these results.

Solution of the discrete-time stochastic optimal control problem in the 2-domain

The design of discrete-time optimal multivariate systems is considered in the z-domain. The constant plant can be non-square, unstable and/or non-minimum phase and feedback system dynamics can be modelled. The stationary coloured noise processes are assumed to be represented by discrete rational spectral densities. The system can contain transport delay elements and the effects of plant saturation can be limited by the choice of performance criterion. The system inputs are assumed to contain both stochastic and deterministic components. The two-stage design procedure is original and it enables the stochastic and deterministic control functions to be separated, A performance criterion is first defined which is insensitive to the deterministic signals and this defines the closed-loop optimal controller. The resulting closed-loop system acts as an optimum regulator to minimize the effects of stochastic disturbances. A second tracking error performance criterion is then specified which determines the optimal ...

Convergence of discretization procedures in dynamic programming

The computational solution of discrete-time stochastic optimal control problems by dynamic programming requires, in most cases, discretization of the state and control spaces whenever these spaces are infinite. In this short paper we consider a discretization procedure often employed in practice. Under certain compactness and Lipschitz continuity assumptions we show that the solution of the discretized algorithm converges to the solution of the continuous algorithm, as the discretization grids become finer and finer. Furthermore, any control law obtained from the discretized algorithm results in a value of the cost functional which converges to the optimal value of the problem.

Performance bounds for discrete-time stochastic optimal control problems†

The purpose of this paper is to obtain some upper and lower bounds for the optimal performance of discrete-time stochastic optimal control problems with incomplete state observation and/or with unknown constant parameters. These bounds are useful for evaluation of various suboptimal policies. For a discrete-state case, Astrom (l965) has shown two theorems which give these bounds. Our methods are extensions of his in several ways. Utilizing the result of our methods, an evaluation of the state information obtained from the observation signal is also developed.