Infinite-horizon Discrete-time Optimal Control Research Articles

Affine Monotonic and Risk-Sensitive Models in Dynamic Programming

In this paper, we consider a broad class of infinite horizon discrete-time optimal control models that involve a nonnegative cost function and an affine mapping in their dynamic programming equation. They include as special cases several classical models, such as stochastic undiscounted nonnegative cost problems, stochastic multiplicative cost problems, and risk-sensitive problems with exponential cost. We focus on the case where the state space is finite and the control space has some compactness properties, and we emphasize shortest path-type models. We assume that the affine mapping has a semicontractive character, whereby for some policies it is a contraction, whereas for others it is not. In one line of analysis, we impose assumptions guaranteeing that the noncontractive policies cannot be optimal. Under these assumptions, we prove strong results that resemble those for discounted Markovian decision problems, such as the uniqueness of solution of Bellman's equation, and the validity of forms of value and policy iteration. In the absence of these assumptions, the results are weaker and unusual in character: the optimal cost function need not be a solution of Bellman's equation, and may not be found by value or policy iteration. Instead the optimal cost function over just the contractive policies is the largest solution of Bellman's equation, and can be computed by a variety of algorithms.

Agreeable solutions of discrete time optimal control problems

In this paper we study solutions of an infinite horizon discrete-time optimal control problem with a compact metric space of states which arises in economic dynamics. We consider locally maximal solutions whose prototypes were introduced and used for models of solid-state physics and in the theory of thermodynamical equilibrium for materials, good solutions and agreeable solutions which were introduced for models of economic dynamics. We show that if an objective function possesses an asymptotic turnpike property, then an admissible program is agreeable if and only if it is locally maximal and good.

Existence of solutions for a class of nonconcave infinite horizon optimal control problems

In this article, we establish the existence of optimal solutions for a large class of nonconvex infinite horizon discrete-time optimal control problems. This class contains optimal control problems arising in economic dynamics which describe a model with nonconcave utility functions representing the preferences of the planner.

GOOD SOLUTIONS FOR A CLASS OF INFINITE HORIZON DISCRETE-TIME OPTIMAL CONTROL PROBLEMS

In this paper we establish the existence of good solutions for a large class of infinite horizon discrete-time optimal control problems. This class contains optimal control problems arising in economic dynamics which describe a model proposed by Robinson, Solow and Srinivasan with nonconcave utility functions representing the preferences of the planner.

Uniqueness and Stability of Optimal Policies of Finite State Markov Decision Processes

In this paper we consider infinite horizon discrete-time optimal control of Markov decision processes (MDPs) with finite state spaces and compact action sets. We restrict attention to unicost MDPs, which form a class that contains all the weakly communicating MDPs. The unicost MDPs are characterized as those MDPs for which there exists a solution to the single optimality equation. We address the problem of uniqueness and stability of minimizing Markov actions. Our main result asserts that when we endow the set of unicost MDPs with a certain natural metric, under which it is complete, then the class of MDPs with essentially unique and stable minimizing Markov actions contains the intersection of countably many open dense sets (hence is itself dense). Thus, the property of having essentially unique and stable minimizing Markov actions is generic for unicost MDPs.