Existence Of Optimal Control Policy Research Articles

Discrete-time hybrid control with risk-sensitive discounted costs

AbstractThis paper studies discrete-time hybrid control problems where the controller’s payoff function follows a risk-sensitive discounted model. The associated discount factor can depend on state and action variables, which can even take values of zero or one. We prove the existence of optimal control policies under two different hypotheses. The following technique is the dynamic programming method, where we characterize the minimum cost as the solution of the so-called dynamic programming equations and find the above optimal policies through these equations. We illustrate our theory with practical examples involving inventory-manufacturing systems, and pollution management.

Some advances on constrained Markov decision processes in Borel spaces with random state-dependent discount factors

ABSTRACT This paper addresses a class of discrete-time Markov decision processes in Borel spaces with a finite number of cost constraints. The constrained control model considers costs of discounted type with state-dependent discount factors which are subject to external disturbances. Our objective is to prove the existence of optimal control policies and characterize them according to certain optimality criteria. Specifically, by rewriting appropriately our original constrained problem as a new one on a space of occupation measures, we apply the direct method to show solvability. Next, the problem is defined as a convex program, and we prove that the existence of a saddle point of the associated Lagrangian operator is equivalent to the existence of an optimal control policy for the constrained problem. Finally, we turn our attention to multi-objective optimization problems, where the existence of Pareto optimal policies can be obtained from the existence of saddle-points of the aforementioned Lagrangian or equivalently from the existence of optimal control policies of constrained problems.

Optimal Control with Partially Observed Regime Switching: Discounted and Average Payoffs

We consider an optimal control problem with the discounted and average payoff. The reward rate (or cost rate) can be unbounded from above and below, and a Markovian switching stochastic differential equation gives the state variable dynamic. Markovian switching is represented by a hidden continuous-time Markov chain that can only be observed in Gaussian white noise. Our general aim is to give conditions for the existence of optimal Markov stationary controls. This fact generalizes the conditions that ensure the existence of optimal control policies for optimal control problems completely observed. We use standard dynamic programming techniques and the method of hidden Markov model filtering to achieve our goals. As applications of our results, we study the discounted linear quadratic regulator (LQR) problem, the ergodic LQR problem for the modeled quarter-car suspension, the average LQR problem for the modeled quarter-car suspension with damp, and an explicit application for an optimal pollution control.

Discrete-time control with non-constant discount factor

This paper deals with discrete-time Markov decision processes (MDPs) with Borel state and action spaces, and total expected discounted cost optimality criterion. We assume that the discount factor is not constant: it may depend on the state and action; moreover, it can even take the extreme values zero or one. We propose sufficient conditions on the data of the model ensuring the existence of optimal control policies and allowing the characterization of the optimal value function as a solution to the dynamic programming equation. As a particular case of these MDPs with varying discount factor, we study MDPs with stopping, as well as the corresponding optimal stopping times and contact set. We show applications to switching MDPs models and, in particular, we study a pollution accumulation problem.

Discrete-Time Hybrid Control in Borel Spaces

A discrete-time hybrid control model with Borel state and action spaces is introduced. In this type of models, the dynamic of the system is composed by two sub-dynamics affecting the evolution of the state; one is of a standard-type that runs almost every time and another is of a special-type that is active under special circumstances. The controller is able to use two different type of actions, each of them is applied to each of the two sub-dynamics, and the activations of these sub-dynamics are possible according to an activation rule that can be handled by the controller. The aim for the controller is to find a control policy, containing a mix of actions (of either standard- or special-type), with the purpose of minimizing an infinite-horizon discounted cost criterion whose discount factor is dependent on the state-action history and may be equal to one at some stages. Two different sets of conditions are proposed to guarantee (i) the finiteness of the cost criterion, (ii) the characterization of the optimal value function and (iii) the existence of optimal control policies; to do so, we employ the dynamic programming approach. A useful characterization that signalizes the accurate times between changes of sub-dynamics in terms of the so-named contact set is also provided. Finally, we introduce two examples that illustrate our results and also show that control models such as discrete-time impulse control models and discrete-time switching control models become special cases of our present hybrid model.

Optimal control for an age-dependent competitive species model in a polluted environment

In this paper, a new age-dependent toxicant population model in an environment with small toxicant capacity is formulated, and the optimal control problem for this kind hybrid system is investigated. The existence and uniqueness of nonnegative solution of this model is obtained by using a fixed point theorem. The optimality conditions for the control problem are given by means of tangent-normal cone techniques in nonlinear functional analysis. The existence of optimal control policy is carefully verified by using of Ekeland’s variational principle.

Optimal control for age-dependent population hybrid system in a polluted environment

In this paper, we study an optimal control problem for an age-dependent population hybrid system in a polluted environment. Namely, we establish a new toxicant-population model with age-dependence in a small content of the environment. By fixed point theorem, we obtain the existence and uniqueness of nonnegative solution of the toxicant-population model with age-dependence. We employ tangent-normal cone techniques in nonlinear functional analysis to deduct the optimality conditions for control problem. The existence of optimal control policy is carefully verified by means of Ekeland’s variational principle.

Congestion control of TCP flows in Internet routers by means of index policy

In this paper we address the problem of fast and fair transmission of flows in a router, which is a fundamental issue in networks like the Internet. We model the interaction between a source using the Transmission Control Protocol (TCP) and a bottleneck router with the objective of designing optimal packet admission controls in the router queue. We focus on the relaxed version of the problem obtained by relaxing the fixed buffer capacity constraint that must be satisfied at all time epoch. The relaxation allows us to reduce the multi-flow problem into a family of single-flow problems, for which we can analyze both theoretically and numerically the existence of optimal control policies of special structure. In particular, we show that for a variety of parameters, TCP flows can be optimally controlled in routers by so-called index policies, but not always by threshold policies. We have also implemented the index policy in Network Simulator-3 and tested in a simple topology their applicability in real networks. The simulation results show that the index policy achieves a wide range of desirable properties with respect to fairness between different TCP versions, across users with different round-trip-time and minimum buffer required to achieve full utility of the queue.

Optimal control for an age-dependent predator-prey system in a polluted environment

In this paper, we investigate an optimal control problem for an age-dependent predator-prey hybrid system in a polluted environment. Namely, we establish a new toxicant-population model with age-dependence in an environment with small toxicant capacity. By using a fixed point theorem, we obtain the existence and uniqueness of nonnegative solution of the toxicant-population model with age-dependence. We employ tangent-normal cone techniques in nonlinear functional analysis to deduct the optimality conditions for the control problem. The existence of optimal control policy is carefully verified by means of Ekeland’s variational principle.

Optimal congestion control of TCP flows for internet routers

In this work we address the problem of fast and fair transmission of ows in a router, which is a fundamental issue in networks like the Internet. We model the interaction between a TCP source and a bottleneck queue with the objective of designing optimal packet admission controls in the bottleneck queue. We focus on the relaxed version of the problem obtained by relaxing the fixed buffer capacity constraint that must be satisfied at all time epoch. The relaxation allows us to reduce the multi-ow problem into a family of single-ow problems, for which we can analyze both theoretically and numerically the existence of optimal control policies of special structure. In particular, we show that for a variety of parameters, TCP ows can be optimally controlled in routers by so-called index policies. We have implemented index policies in Network Simulator-3 (NS-3) and compared its performance with DropTail and RED buffers. The simulation results show that the index policy has several desirable properties with respect to fairness and efficiency.

Average cost Markov Decision Processes: Optimality conditions

We are concerned in this paper with discrete-time Markov Decision Processes (MDPs) with Borel state and action spaces X and A, respectively, and the long run expected average cost criterion. When X is a denumerable set, man y necessary and/or sufficient conditions for the existence of optimal control policies are known. However, when X is a Borel space (i.e., a Borel subset of a complete separable metric space), most of the available results impose on the MDP very restrictive topological conditions (e.g., compactness) and/or strong recurrence assumptions (such as Doeblin's condition); see, e.g., [4,9, 12] and their references. Another related work is [7] where we have studied MDPs from the viewpoint of the recurrence (or ergodicity) properties of the state process. ln the present paper, however, we are concerned with the existence of average optimal policies by looking at (static) optimization problems (see condition CS in Section 4) related-in sorne cases equivalent-to the existence of a bounded solution to the so-called Optimality Equation (see C4 in Section 4). These optimization problems are dual in the sense that, under appropriate conditions, the existence of an optimal solution to one of the problems implies existence of an optimal

Optimal Control of Generating Policies in a Power System Governed by a Second Order Hyperbolic Partial Differential Equation

In this paper we present necessary and sufficient conditions for determining the optimum generating policies for each of a system of power stations feeding into a primary transmission line. The system is modeled as a linear second order partial differential equation of hyperbolic type with appropriate initial and boundary conditions. Further, existence of optimal control policies for the system is proved. A computational algorithm for determining the optimal policies is also presented.