Continuous-time Infinite Horizon Research Articles

On continuous-time infinite horizon optimal control—Dissipativity, stability, and transversality

This paper analyzes the interplay between dissipativity and stability properties in continuous-time infinite-horizon Optimal Control Problems (OCPs). We establish several relations between these properties, which culminate in a set of equivalence conditions. Moreover, we investigate convergence and stability of the infinite-horizon optimal adjoint trajectories. The workhorse for our investigations is a notion of strict dissipativity in OCPs, which has been coined in the context of economic model predictive control. With respect to the link between stability and dissipativity, the present paper can be seen as an extension of the seminal work on least squares optimal control by Jan C. Willems (1971). Furthermore, we show that strict dissipativity provides a conclusive answer to the question of adjoint transversality conditions in infinite-horizon optimal control which has been raised by Hubert Halkin (1974). Put differently, we establish conditions under which the adjoints converge to their optimal steady-state value. We draw upon several examples to illustrate our findings. Moreover, we discuss the relation of our findings to results available in the literature.

Optimal pricing and deteriorating inventory control when inventory stimulates stochastic demand with reference price effect

We study the joint dynamic pricing and deteriorating inventory management problem in the presence of reference price and stock display effects. In a random potential market setting with convex replenishment/ordering and holding/shortage cost functions, the retailer replenishes/orders and sells a single deteriorating product over a continuous-time infinite horizon to maximize its profit. The demand rate depends negatively on the sales price and positively on the consumers’ reference price and displayed stock quantity. The inventories deteriorate physically at a constant fraction of the on-hand stocks. A stochastic optimal control model is established to characterize the optimal policy in a linear feedback form of the state variables when the observed inventory level is either positive or negative. We also investigate the asymptotic behavior of the system and provide sufficient conditions for the stability and monotone convergence of the expected long-run behavior. Finally, we perform numerical examples to illustrate the theoretical results and sensitivity analysis to derive insights into deteriorating inventory management under reference price and stock display effects. The findings suggest that current reference price level has a positive effect while inventory level has a negative on the optimal replenishment rate and price. We characterize the sample paths of optimal strategies and find that the initial consumer reference price has an important impact on the firm’s optimal operations management. On the expected long-run behavior, it is beneficial for the firm to reduce demand uncertainty and deteriorating rate. We also observe that a large factor of displayed stock effect brings a positive effect on the total expected profit. Additionally, the firm needs to reduce (increase) the price and increase (reduce) the replenishment rate with high reference price effect intensity (memory factor).

Solution for the Continuous-Time Infinite-Horizon Linear Quadratic Regulator Subject to Scalar State Constraints

This letter provides a solution for the continuous-time linear quadratic regulator (LQR) subject to a scalar state constraint. Using a dichotomy transformation, novel properties for the finite-horizon LQR are derived; the unknown boundary conditions are explicitly expressed as a function of the horizon length, the initial state, and the final state or cost of the final state. Practical relevance of these novel properties are demonstrated with an algorithm to compute the continuous-time LQR subject to a scalar state constraint. The proposed algorithm uses the analytical conditions for optimality, without a priori discretization, to find only those sampling time instances that mark the start and end of a constrained interval. Each subinterval consists of a finite-horizon LQR, hence, a solution can be efficiently computed and the computational complexity does not grow with the horizon length. In fact, an infinite horizon can be handled. The algorithm is demonstrated with a simulation example.

Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem

In this paper we discuss an online algorithm based on policy iteration for learning the continuous-time (CT) optimal control solution with infinite horizon cost for nonlinear systems with known dynamics. That is, the algorithm learns online in real-time the solution to the optimal control design HJ equation. This method finds in real-time suitable approximations of both the optimal cost and the optimal control policy, while also guaranteeing closed-loop stability. We present an online adaptive algorithm implemented as an actor/critic structure which involves simultaneous continuous-time adaptation of both actor and critic neural networks. We call this ‘synchronous’ policy iteration. A persistence of excitation condition is shown to guarantee convergence of the critic to the actual optimal value function. Novel tuning algorithms are given for both critic and actor networks, with extra nonstandard terms in the actor tuning law being required to guarantee closed-loop dynamical stability. The convergence to the optimal controller is proven, and the stability of the system is also guaranteed. Simulation examples show the effectiveness of the new algorithm.

Continuous Time Linear Quadratic Regulator With Control Constraints via Convex Duality

A continuous time infinite horizon linear quadratic regulator with input constraints is studied. Optimality conditions, both in the open loop and feedback form, and continuity and differentiability properties of the optimal value function and of the optimal feedback are shown. Arguments rely on basic ideas of convex conjugacy, and in particular, use a dual optimal control problem

Optimal time aggregation of infinite horizon control problems

This paper proposes a novel method that enhances numerical approximation of infinite horizon optimal control problems. For direct numerical optimization, a continuous-time infinite horizon model needs to be first recast as a discrete-time, finite-horizon control problem. The very transformation itself may significantly degrade the quality of the optimization results, if due care is not taken to preserve the salient features in the original model. Mercenier and Michel (1994. Econometrica 62, 635–656, 2001. Journal of Economic Dynamics and Control 25, 1179–1191), for instance, propose time aggregation methods that minimize approximation errors at the steady-state. Using their scheme, we show that overall optimization performance can be further improved if the discretization of the transient phase is optimal as well. Three sample problems are numerically solved to demonstrate the potential benefits.

Optimal growth with decreasing marginal impatience

This paper analyses the long run dynamics of a continuous time infinite horizon optimal growth model with identical households where the households’ rate of time preference is endogenously determined. However, unlike the existing literature, we assume that households have decreasing marginal impatience, i.e., the instantaneous discount rate is negatively related to the current consumption. With this assumption we analyse the stability and uniqueness of the long run equilibrium. We show that contrary to the general belief, a negative relationship between the instantaneous discount rate and the household's current consumption does not necessarily result in instability of the dynamic system. However, even a stable system may be characterized by multiple equilibria so that the initial condition becomes important in determining the long run behaviour of the economy. We derive a set of sufficient conditions for stability and uniqueness in this context.

Existence of optimal controls for continuous time infinite horizon problems

An existence theorem is given for general continuous time infinite horizon optimal control formulations. Specifically, we establish conditions on the plant model, the cost integrand and the set of initial states to be considered under which the infinite horizon optimal control problem has a solution. The conditions on the objective function are of a sufficiently general nature to allow flexibility in the choice of cost function and therefore the hypotheses easily fit standard process systems applications.

Infinite-horizon continuous-time growth models

In this paper we examine a continuous-time infinite-horizon economic growth model with non-convexities. First we show that an optimal program exists and the value of the problem can be obtained as the limit of the values of the finite-horizon approximations. If we are only interested in the existence part, then we show that the hypotheses can be weakened substantially to incorporate economically significant cases. Finally we obtain a transversality condition for optimality that does not involve dual variables (prices).

Sensitivity analysis for a continuous time infinite horizon growth model

In this paper, we consider an infinite horizon, continuous time growth model with discounted future utilities. We show that the set of optimal programs is nonempty, compact, and convex in the space C( R +, R N ) and depends in an upper semicontinuous way on the initial capital stock, while the value of the problem is a continuous function of the initial capital stock.

Increasing Returns and the Optimality of Open Capital Markets in a Small Growing Economy

This paper examines the welfare consequences of liberalizing capital markets in the presence of increasing returns driven by externalities in knowledge and capital accumulation. Using a revealed preference argument in a continuous-time infinite horizon model, this paper shows that the opening of capital markets can result in welfare deterioration if foreign interest rates are high and capital outflows are likely. Does opening capital markets improve welfare? A simple reinterpretation of the static theory of gains-from-trade in an intertemporal framework leads us to answer in the affirmative. Opening up capital markets should result in welfare gains by exploiting the possibility of international borrowing and lending at world interest rates different from autarky rates. Despite the theoretical prediction, most countries have been imposing various restrictions on international capital transactions. Besides difficulties in managing exchange regimes arising from massive short-term capital movements with free capital mobility, growth oriented economies are also concerned about possible adverse effects of capital flight on domestic capital accumulation. When the outside world offers higher returns to capital than domestic markets, domestic savings may go abroad to finance foreign capital accumulation instead of domestic accumulation. However, if no distortion exists in the market mechanism, the cost of decreased domestic capital accumulation and slower growth must be more than offset by the benefit of increased capital income from abroad. The defense of capital controls on the grounds of loss in domestic production when capital outflows are likely after liberalization can be justified only when there is a substantial amount of external economies associated with containing knowledge and capital within the border. External economies from scale of operation, knowledge creation, and physical or human capital accumulation have received renewed attention, and have become one of the building blocks in endogenous growth models. (See, for example, Romer 1986 and 1987, Lucas 1988, and Grossman and Helpman 1990.) In these models, an individual production unit has the standard neoclassical technology but spillovers of external exonomies from the aggregate stock of knowledge and capital make the economy as a whole exhibit scale economies and prevent the marginal product of capital from falling below the subjective discount rate, sustaining unbounded

The existence of optimal consumption policies in optimal economic growth models with nonconvex technologies

An existence theorem for a class of continuous time infinite horizon optimal growth models is developed. The underlying technology set is not assumed to be convex, instead the “slices” of the technology set corresponding to a fixed capital stock vector are assumed convex and compact in the consumption and net investment variables. This allows consideration of the case of increasing returns to scale. Existence of an optimal capital stock and consumption policy is proved directly without consideration of the underlying Hamiltonian dynamical system that arises from applying Pontryagin's maximum principle.

Existence and Characterization of Optimal Growth Paths Including Models with Non-Convexities in Utilities and Technologies

This paper is concerned with the existence and characterization of optimal growth paths in continuous time infinite horizon problems, and brings to bear on these issues mathematical techniques presented by the author in Chichilnisky (1977). These techniques, non-linear functional analysis in Hilbert (weighted L2 and Sobolev) spaces, permit generalization, clarification and simplification of existing results. In particular, the following points are of importance: (i) As Hilbert spaces are self-dual, continuous linear functionals on the commodity space are elements of the space, and thus have a totally natural interpretation as prices. We therefore obtain that commodity and price paths are in the same spaces, a useful property to prove existence in general equilibrium models. A failure to meet this condition has led to earlier work on characterization being troubled by the problem that supporting linear functionals are not always interpretable as prices, so that one cannot always give a price characterization of optimal and efficient paths. In the present framework this problem does not arise. (ii) A natural way to pose the problem of finding an optimal path is as one of maximizing a continuous function on a compact set. A major advantage of the techniques we use here is demonstrated by the fact that we are actually able to follow this approach. (iii) Because of (ii), it is possible to establish the existence of an optimal path without the use of any convexity assumptions on either the technology or the preferences. This is a major relaxation of earlier conditions, and enables existence results to be applied to models with increasing returns in production, non-convex preferences and endogenous population. For instance, in Lane (1977) both the feasible net production set and the preference ordering are in general non-convex due to the endogenous response of population growth to income. In Ryder and Heal (1973) and Wan (1970) non-convexities appear as well as in environmental and biological problems discussed for example in Clark (1976). In this latter case the logistic growth functions of biological populations introduces non-convexities in a natural way. (iv) It can be shown that the prices that arise within this framework provide a well-defined present value for every feasible program. This makes it possible to provide a full characterization of optimal and efficient paths in terms of profit maximization without any reference to transversality conditions. This characterization is thus analogous to that