Deterministic Optimal Control Problem Research Articles

On the Relation between Discounted and Average Optimal Value Functions

We investigate the relation between discounted and average deterministic optimal control problems for nonlinear control systems. In particular we are interested in the corresponding optimal value functions. Using the concepts of Viability, Chain Controllability, and Controllability, a global convergence result for vanishing discount rate is obtained. Basic ingredients for the analysis are an Abelian type theorem, controllability properties of the system, and the Morse decomposition of the corresponding control flow.

On asymptotic optimization of a class of nonlinear stochastic hybrid systems

We consider the problem of control for continuous time stochastic hybrid systems in finite time horizon. The systems considered are nonlinear: the state evolution is a nonlinear function of both the control and the state. The control parameters change at discrete times according to an underlying controlled Markov chain which has finite state and action spaces. The objective is to design a controller which would minimize an expected nonlinear cost of the state trajectory. We show using an averaging procedure, that the above minimization problem can be approximated by the solution of some deterministic optimal control problem. This paper generalizes our previous results obtained for systems whose state evolution is linear in the control.

Rates of Convergence for Approximation Schemes in Optimal Control

We present a simple method for obtaining rate of convergence estimates for approximations in optimal control problems. Although the method is applicable to a wide range of approximation problems, it requires in all cases some type of smoothness of the quantity being approximated. We illustrate the method by presenting a number of examples, including finite difference schemes for stochastic and deterministic optimal control problems. A general principle can be abstracted, and indeed the method may be applied to a variety of approximation problems, such as the numerical approximation of nonlinear PDEs not a priori related to control theory.

Asymptotic buffer overflow probabilities in multiclass multiplexers: an optimal control approach

We consider a multiclass multiplexer with support for multiple service classes and dedicated buffers for each service class. Under specific scheduling policies for sharing bandwidth among these classes, we seek the asymptotic (as the buffer size goes to infinity) tail of the buffer overflow probability for each dedicated buffer. We assume dependent arrival and service processes as is usually the case in models of bursty traffic. In the standard large deviations methodology, we provide a lower and a matching (up to first degree in the exponent) upper bound on the buffer overflow probabilities. We introduce a novel optimal control approach to address these problems. In particular, we relate the lower bound derivation to a deterministic optimal control problem, which we explicitly solve. Optimal state trajectories of the control problem correspond to typical congestion scenarios. We explicitly and in detail characterize the most likely modes of overflow. We specialize our results to the generalized processor sharing policy (GPS) and the generalized longest queue first policy (GLQF). The performance of strict priority policies is obtained as a corollary. We compare the GPS and GLQF policies and conclude that GLQF achieves smaller overflow probabilities than GPS for all arrival and service processes for which our analysis holds. Our results have important implications for traffic management of high-speed networks and can be used as a basis for an admission control mechanism which guarantees a different loss probability for each class.

Asymptotic Optimization of a Nonlinear Hybrid System Governed by a Markov Decision Process

We consider in this paper a continuous time stochastic hybrid control system with finite time horizon. The objective is to minimize a nonlinear function of the state trajectory. The state evolves according to a nonlinear dynamics. The parameters of the dynamics of the system may change at discrete times $l\epsilon$, $l=0,1,...$, according to a controlled Markov chain which has finite state and action spaces. Under the assumption that $\epsilon$ is a small parameter, we justify an averaging procedure allowing us to establish that our problem can be approximated by the solution of some deterministic optimal control problem.

Measurement scheduling for recursive team estimation

We consider a decentralized LQG measurement scheduling problem in which every measurement is costly, no communication between observers is permitted, and the observers' estimation errors are coupled quadratically. This setup, motivated by considerations from organization theory, models measurement scheduling problems in which cost, bandwidth, or security constraints necessitate that estimates be decentralized, although their errors are coupled. We show that, unlike the centralized case, in the decentralized case the problem of optimizing the time integral of the measurement cost and the quadratic estimation error is fundamentally stochastic, and we characterize the E-optimal open-loop schedules as chattering solutions of a deterministic Lagrange optimal control problem. Using a numerical example, we describe also how this deterministic optimal control problem can be solved by non- linear programming.

Filtering and stochastic control: a historical perspective

We attempt to give a historical account of the main ideas leading to the development of nonlinear filtering and stochastic control as we know it today. We present a development of linear filtering theory, beginning with Wiener-Kolmogoroff filtering and ending with Kalman filtering. The linear-quadratic-Gaussian problem of stochastic control is considered and states that for this problem the optimal stochastic control can be constructed by solving separately a state estimation problem and a deterministic optimal control problem. Many of the ideas presented here generalize to the nonlinear situation. A reasonably detailed discussion of nonlinear filtering, again from the innovations viewpoint, is given. Finally, we deal with optimal stochastic control. The general method of discussing these problems is dynamic programming.

The linear programming approach to deterministic optimal control problems

Given a deterministic optimal control problem (OCP) with value function, say $J^*$, we introduce a linear program $(P)$ and its dual $(P^*)$ whose values satisfy $\sup(P^*) \leq\inf(P)\leq J^*(t,x)$. Then we give conditions under which (i) there is no dua

Optimal feedback production planning in a stochastic N-machine flowshop

We consider a production planning problem in an N-machine flowshop subject to breakdown and repair of machines and to non-negativity constraints on work-in-process. The machine capacities and demand processes are assumed to be finite-state Markov chains. The problem is to choose the rates of production on the N machines over time to minimize the expected discounted cost of production and inventory/backlog over an infinite horizon. It is formulated as a stochastic dynamic programming problem. We show that the value function of the problem is locally Lipschitz and is a solution to a dynamic programming equation together with a certain boundary condition. We provide an interpretation of the boundary condition. We also prove a verification theorem and derive the optimal feedback control policy in terms of the directional derivatives of the value function. Finally, we obtain a deterministic optimal control problem that is equivalent to the stochastic production planning problem under consideration.

Two-degree-of-freedom linear quadratic Gaussian predictive control

A two-degree-of-freedom LQGPC optimal control law is derived, which has properties in common with both the LQG and GPC control laws. The stability and robustness properties are the same as for an LQG optimal controller, but the cost of future predictive error and control action is dealt with in the same manner as for the GPC control law. An advantage of the approach is that true predictive action is possible, so the control at time t will optimally use the future reference-trajectory knowledge. The feedback controller to be implemented is time-invariant, but the future predicted control action is obtained from the polynomial equivalent of a time-varying solution, which is analogous to the solution of the finite-time deterministic LQ optimal control problem. The results can be presented in a form similar to that employed in GPC algorithms, suggesting an immediate way of achieving input and output constraints.

A GRID REFINEMENT METHOD FOR DETERMINISTIC CONTROL AND DIFFERENTIAL GAMES

We develop here a simple method for the computation of value functions of deterministic optimal control or differential games problems which allows to refine locally a grid and reduce memory space. Given an approximation of optimal trajectories, one can solve the associated Hamilton-Jacobi equation in a tubular neighborhood with state-constraints boundary conditions. We study here the validity of such an approach and we illustrate it on various examples.

Optimum design of measurement channels and control policies for linear-quadratic stochastic systems

In the design of optimal controllers for linear-quadratic stochastic systems, a standard assumption is that the measurement channels are fixed and linear, and the measurement noise is Gaussian. In this paper we relax the first part of this restriction and raise the issue of the derivation of optimum measurement structures as a part of the overall design. Toward this end, we take the measuement process as one given by a Wiener integral, and modify the cost function so that it now places some soft constraints on the measurement strategy. Using some results from information theory, we show that the scalar version (for both finite and infinite horizons) of this joint design problem admits an optimum, dictating linear designs for both the controller and the measurement strategy. For the vector version, however, it is possible for a nonlinear design to improve over the best linear one. In both cases, best linear designs involve the solutions of nonlinear (deterministic) optimal control problems.

The Bellman equation for constrained deterministic optimal control problems

The Bellman equation for constrained deterministic optimal control problems

Optimal maintenance of stochastically deteriorating systems

A machine maintenance problem were the deterioration of the machine is subject to additive random noise is considered. The objective is then to maximize the discounted net return of the machine. Furthermore, it is also required that the machine maintains a sufficiently good quality state with certain degrees of confidence in part or whole of the machine life span. This problem can be formulated as a constrained stochastic optimal control problem. It is then shown that the stochastic optimal control problem can be converted into an equivalent deterministic optimal control problem and subsequently solved by the technique of control parametrization. Numerical examples are presented to illustrate some of the interesting features of the model.

Two new techniques for optimal control

Two techniques for deterministic discrete-time optimal control problems with inequality and equality constraints on state and control variables are presented. The first one tackles a problem by solving a sequence of two-stage subproblems successively. Convergence properties are discussed under general conditions. The second method, a version of the first one, is constructed for parallel computation. It solves multiple subproblems simultaneously and can therefore calculate much faster. The Kuhn-Tucker conditions for discrete-time optimal-control are discussed. >

Hamilton-Jacobi equations with state constraints

In the present paper we consider Hamilton-Jacobi equations of the form H ( x , u , ∇ u ) = 0 , x ∈ Ω H(x,u,\nabla u) = 0,\;x \in \Omega , where Ω \Omega is a bounded open subset of R n , H {R^n},H is a given continuous real-valued function of ( x , s , p ) ∈ Ω × R × R n (x,s,p) \in \Omega \times R \times {R^n} and ∇ u \nabla u is the gradient of the unknown function u u . We are interested in particular solutions of the above equation which are required to be supersolutions, in a suitable weak sense, of the same equation up to the boundary of Ω \Omega . This requirement plays the role of a boundary condition. The main motivation for this kind of solution comes from deterministic optimal control and differential games problems with constraints on the state of the system, as well from related questions in constrained geodesics.

An optimal control problem for a system with small noise

A deterministic optimal control problem involving a system of the form dx/ dt = f( t, x, a P ), t > 0, x( t) ∈ R 4, where a P is the control function, is posed. Since f(·,·,·) is not smooth enough, the problem is transformed to a stochastic optimal control problem involving the system dx = f( t, x, a P ) dt + ϵdw, t > 0, where W is an R 4-valued standard Wiener process and ϵ, 0 < ϵ ⪡ 1, is a given number.

Species preservation in an optimal harvest model with random prices

In this paper, we consider an optimal harvest model in which the objective is to maximize the expected return. The unit price of biomass is assumed constant until a random time when the price increases by a given amount. Furthermore, due to obvious environmental protection requirements, it is assumed that the fishery population is bounded from below for all time so as to reduce the danger of species extinction. Clearly, this problem is an optimal control problem in which a random parameter is involved. However, due to its special structure, it is shown that the problem is convertible into a deterministic optimal control problem and hence is solvable by an existing optimal control software package, MISER. The practical implication of several computed results obtained by this approach is discussed. They are also compared with other related results in the literature.

Solution of a class of stochastic linear-convex control problems using deterministic equivalents

In this paper, we identify a new class of stochastic linearconvex optimal control problems, whose solution can be obtained by solving appropriate equivalent deterministic optimal control problems. The term “linear-convex” is meant to imply that the dynamics is linear and the cost function is convex in the state variables, linear in the control variables, and separable. Moreover, some of the coefficients in the dynamics are allowed to be random and the expectations of the control variables are allowed to be constrained. For any stochastic linear-convex problem, the equivalent deterministic problem is obtained. Furthermore, it is shown that the optimal feedback policy of the stochastic problem is affine in its current state, where the affine transformation depends explicitly on the optimal solution of the equivalent deterministic problem in a simple way. The result is illustrated by its application to a simple stochastic inventory control problem.

Estimation and control of systems with unknown covariance and multiplicative noise

The problem of estimation and control for systems with multiplicative noise and unknown second-order statistics is considered. Conditions are found for the existence of a solution based on game theoretic ideas. The conditions for the existence of a saddle point for the time-invariant filtering problem are necessary and sufficient, whereas for all other cases only necessary. The central idea of the solution is to convert the stochastic problem to a deterministic optimal control problem whose minimax point is sought with respect to the control, filter, and unknown statistics parameters. The results that are derived show that the problem of estimation for systems with unknown covariances depends on the costate matrix, which in turn is a function of the performance measure. Thus, the filter loses one of its best known properties, that of independence of the performance functional. This property holds not only for the classical Kalman filter but also for multiplicative systems.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>