Discrete-time Stochastic Control Research Articles

The Optimal Recourse Problem in Discrete Time: $L^1 $-Multipliers for Inequality Constraints

An optimal recourse problem is an optimization problem with both stochastic and dynamic aspects, involving the interplay of observations and responses. In discrete time (with a finite horizon), there are finitely many stages, at each of which a decision is selected on the basis of prior observations of random events and subject to costs and constraints affected by these observations as well as past decisions. The goal is to minimize expected cost, taking into account the known distribution of future random events. This paper is concerned with the derivation of necessary and sufficient conditions for optimality in the case of convex costs and constraints. It is shown that if the recourse problem is strictly feasible and satisfies a new condition called essentially complete recourse, optimal solutions can be characterized by a “pointwise” Kuhn–Tucker property involving $L^1 $-multipliers. Applications to multistage stochastic programs with special structures are developed in the last two sections of the paper. In particular, the relation between the general model and discrete-time stochastic control models is brought out by applying the basic results to a linear stochastic problem with state constraints.

Separation of Estimation and Control for Decentralized Stochastic Control Systems

A sufficient condition is given for a separation of estimation and control in decentralized discrete-time stochastic control systems. It is shown that the one-step delay sharing system is a special case satisfying this condition. A counterexample is given, however, to the well-known conjecture that the separation will hold for the more-than-one-step delay sharing systems.

Convergence of discretization procedures in dynamic programming

The computational solution of discrete-time stochastic optimal control problems by dynamic programming requires, in most cases, discretization of the state and control spaces whenever these spaces are infinite. In this short paper we consider a discretization procedure often employed in practice. Under certain compactness and Lipschitz continuity assumptions we show that the solution of the discretized algorithm converges to the solution of the continuous algorithm, as the discretization grids become finer and finer. Furthermore, any control law obtained from the discretized algorithm results in a value of the cost functional which converges to the optimal value of the problem.

A standard form for sequential stochastic control

Causal discrete-time stochastic control problems with nonclassical information patterns are not necessarily sequential in the sense that the order of the various control actions is fixed in advance. When this sequential condition does hold, then they can be reduced to a (theoretical) standard form which enables the establishment of a maximum principle, a reachable set concept and dynamic programming, all in a straightforward and transparent way.

Separation of estimation and control for discrete time systems

An attempt is made to coordinate the numerous results relating to separation of estimation and control in discrete time stochastic control theory. The results vary widely depending upon the assumptions about linearity, criteria, information pattern, constraints, and noise distributions. Some of the less well-known underlying concepts are discussed with the help of a fairly general model.

Performance bounds for discrete-time stochastic optimal control problems†

The purpose of this paper is to obtain some upper and lower bounds for the optimal performance of discrete-time stochastic optimal control problems with incomplete state observation and/or with unknown constant parameters. These bounds are useful for evaluation of various suboptimal policies. For a discrete-state case, Astrom (l965) has shown two theorems which give these bounds. Our methods are extensions of his in several ways. Utilizing the result of our methods, an evaluation of the state information obtained from the observation signal is also developed.

A Stochastic Multiperiod Multimode Transportation Model

This paper develops a dynamic programming model for selecting an optimal combination of transportation modes over a midtiperiod planning horizon. The formulation explicitly incorporates uncertainty regarding future requirements or demands for a number of commodity classes. In addition to determining the optimal modes to employ, the model assigns individual commodity classes to various modes, determines which supply points serve which destinations, and reroutes carriers from destinations to alternative sources where they will be most effective. The model is formulated as an optimal discrete time stochastic control problem where cost is quadratic and dynamic equations linear in the state and control variables. This model may be solved in closed form by an efficient dynamic programming algorithm that permits the treatment of relatively large scale systems. Also developed is an alternative, generally suboptimal method of solution, based upon solving a sequence of convex programming problems over time. This technique may be employed for a more general class of problems. In both methods the use of “shadow prices” that arise is discussed.

Contraction Mappings in the Theory Underlying Dynamic Programming

Contraction Mappings in the Theory Underlying Dynamic Programming

Discrete-time interrupted stochastic control processes

This paper is concerned with the determination of optimal policies for applying inputs to discrete-time processes which are interrupted in the sense that at each time t, t = 1, 2, …, there is a probability p t (called the probability of interruption) that the state vector of the system cannot be observed. The policy is to be optimal with respect to a specified loss functional. It is shown that the optimal policy for controlling such processes can be expressed in terms of functional equations obtained by the usual procedures of dynamic programming. An interrupted process involving a linear system and quadratic loss functional is examined in detail and the optimal policy as well as the expected cost of the process under the optimal policy is expressed in analytic form. The optimal policy is found to be independent of the probability of interruption. Asymptotic properties of the optimal policy and the expected cost of the process under the optimal policy are also obtained.