Continuous-time Optimal Control Research Articles

Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem

In this paper we discuss an online algorithm based on policy iteration for learning the continuous-time (CT) optimal control solution with infinite horizon cost for nonlinear systems with known dynamics. That is, the algorithm learns online in real-time the solution to the optimal control design HJ equation. This method finds in real-time suitable approximations of both the optimal cost and the optimal control policy, while also guaranteeing closed-loop stability. We present an online adaptive algorithm implemented as an actor/critic structure which involves simultaneous continuous-time adaptation of both actor and critic neural networks. We call this ‘synchronous’ policy iteration. A persistence of excitation condition is shown to guarantee convergence of the critic to the actual optimal value function. Novel tuning algorithms are given for both critic and actor networks, with extra nonstandard terms in the actor tuning law being required to guarantee closed-loop dynamical stability. The convergence to the optimal controller is proven, and the stability of the system is also guaranteed. Simulation examples show the effectiveness of the new algorithm.

A pseudospectral method for optimal control of open quantum systems

In this paper, we present a unified computational method based on pseudospectral approximations for the design of optimal pulse sequences in open quantum systems. The proposed method transforms the problem of optimal pulse design, which is formulated as a continuous-time optimal control problem, to a finite-dimensional constrained nonlinear programming problem. This resulting optimization problem can then be solved using existing numerical optimization suites. We apply the Legendre pseudospectral method to a series of optimal control problems on open quantum systems that arise in nuclear magnetic resonance spectroscopy in liquids. These problems have been well studied in previous literature and analytical optimal controls have been found. We find an excellent agreement between the maximum transfer efficiency produced by our computational method and the analytical expressions. Moreover, our method permits us to extend the analysis and address practical concerns, including smoothing discontinuous controls as well as deriving minimum-energy and time-optimal controls. The method is not restricted to the systems studied in this article and is applicable to optimal manipulation of both closed and open quantum systems.

An External Active-Set Strategy for Solving Optimal Control Problems

We present a new, <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">external</i> , active constraints set strategy for solving nonlinear programming problems with a large number of inequality constraints that arise in the process of discretizing continuous-time optimal control problems with state-space constraints. This strategy constructs a sequence of inequality constrained nonlinear programming problems, containing a progressively larger subset of the constraints in the original problem, and submits them to a nonlinear programming solver for a fixed number of iterations. We prove that this scheme computes a solution of the original problem and show by means of numerical experiments that this strategy results in reductions in computing time ranging from a factor of 6 to a factor of over 100.

Optimal Control for Nonconvex Rate-Independent Evolution Processes

Energetic solutions to rate-independent systems allow for an effective modeling of many physical systems displaying hysteretic effects, e.g., phase transformations in shape-memory alloys, elastoplasticity, and ferroelectricity. For some engineering applications, optimal control of such systems is desirable. We establish existence results for these continuous-time optimal control problems by a combination of the direct method with $\Gamma$-convergence arguments. Applicability to the common situation of a controlled external loading is demonstrated and a concrete partial differential inclusion as well as an academic model of the foreign exchange market including trading costs are investigated.

Risk in Revenue Management and Dynamic Pricing

We present a new model for optimal dynamic pricing of perishable services or products that incorporates a simple risk measure permitting control of the probability that total revenues fall below a minimum acceptable level. The formulation assumes that sales must occur within a finite time period, that there is a finite—possibly large—set of available prices, and that demand follows a price-dependent, nonhomogeneous Poisson process. This model is particularly appropriate for applications in which attainment of a revenue target is an important consideration for managers; for example, in event management, in seasonal clearance of high-value items, or for business subunits operating under performance targets. We formulate the model as a continuous-time optimal control problem, obtain optimality conditions, explore structural properties of the solution, and report numerical results on problems of realistic size.

An Accelerator for Packages Solving Discrete-Time Optimal Control Problems

We present an accelerator scheme for use with existing packages that solve nonlinear programming problems with a large number of inequality constraints that arise in the process of discretizing continuous-time optimal control problems with state-space constraints. This scheme is based on the concept of outer approximations used in semi-infinite programming and acts as an external, active constraints set strategy.Our scheme constructs a finite sequence of inequality constrained nonlinear programming problems, containing a progressively larger subset of the constraints in the original problem, and submits these problems to a nonlinear programming solver for a fixed number of iterations. We prove that this scheme computes a solution of the original problem and show, by means of numerical experiments, that it results in reductions in computing time ranging from a factor of 6 to a factor of over 400.

Control Parametrization Enhancing Technique and Simulation on the Design of a Flexible Rotating Beam

In this paper, a computational approach is adopted to solve the optimal control and optimal parameter selection problems of a rotating flexible beam fully covered with active constrained layer damping (ACLD) treatment. The beam rotates in a vertical plane under gravity with variable angular velocity and carries an end mass. The problem is posed as a continuous-time optimal control problem and solved via MISER3. Using the control parametrization enhancing technique (CPET), the results show that substantial improvements are obtained with ACLD as compared to the passive constrained layer damping (PCLD) treatment. In addition, accurate switching times of the system are determined. A parametric analysis is also included through simulation results.

A nonlinear continuous time optimal control model of dynamic pricing and inventory control with no backorders

AbstractIn this paper, we present a continuous time optimal control model for studying a dynamic pricing and inventory control problem for a make‐to‐stock manufacturing system. We consider a multiproduct capacitated, dynamic setting. We introduce a demand‐based model where the demand is a linear function of the price, the inventory cost is linear, the production cost is an increasing strictly convex function of the production rate, and all coefficients are time‐dependent. A key part of the model is that no backorders are allowed. We introduce and study an algorithm that computes the optimal production and pricing policy as a function of the time on a finite time horizon, and discuss some insights. Our results illustrate the role of capacity and the effects of the dynamic nature of demand in the model. © 2007 Wiley Periodicals, Inc. Naval Research Logistics, 2007

Hybrid approach to the rational management of machine replacement

The paper analyzes a nonlinear integer-valued optimization problem that describes the optimal machine replacement under improving technology. The problem is extended into the classes of real-valued discrete-time optimization problems and continuous-time optimal control. The results include some qualitative properties of optimal management policies and the construction of an efficient numeric algorithm.

Direct Trajectory Optimization and Costate Estimation via an Orthogonal Collocation Method

A pseudospectral method, called the Gauss pseudospectral method, for solving nonlinear optimal control problems is presented. In the method presented here, orthogonal collocation of the dynamics is performed at the Legendre-Gauss points. This form of orthogonal collocation leads a nonlinear programming problem (NLP) whose Karush-Kuhn-Tucker (KKT) multipliers can be mapped to the costates of the continuous-time optimal control problem. In particular the Legendre-Gauss collocation leads to a costate mapping at the boundary points. The method is demonstrated on an example problem where it is shown that highly accurate costates are obtained. The results presented in this paper show that the Gauss pseudospectral method is a viable apprach for direct trajectory optimization and costate estimation.

A Bolza problem in hydrothermal optimization

This paper studies the optimization of large-scale hydrothermal power systems. For the general problem with n hydro-plants, we present an algorithm using a particular strategy related to the Gauss–Southwell method of nonlinear optimization. The algorithm offers a constructive method for producing sequences of problems with one hydro-plant. For this simple problem we use Pontryagin’s minimum principle to prove a condition for the extremals of the functional. We set out our problem in terms of optimal control in continuous time, with the Bolza-type functional. Finally, we present one example employing a program developed with the “Mathematica” package and analyze the convergence of the algorithm.

A Pseudospectral Method for the Optimal Control of Constrained Feedback Linearizable Systems

We consider the optimal control of feedback linearizable dynamical systems subject to mixed state and control constraints. In general, a linearizing feedback control does not minimize the cost function. Such problems arise frequently in astronautical applications where stringent performance requirements demand optimality over feedback linearizing controls. In this paper, we consider a pseudospectral (PS) method to compute optimal controls. We prove that a sequence of solutions to the PS-discretized constrained problem converges to the optimal solution of the continuous-time optimal control problem under mild and numerically verifiable conditions. The spectral coefficients of the state trajectories provide a practical method to verify the convergence of the computed solution. The proposed ideas are illustrated by several numerical examples.

Synthesis of optimal controllers for piecewise affine systems with sampled-data switching

This paper discusses the optimal control problem of the continuous-time piecewise affine (PWA) systems with sampled-data switching, where the switching action is executed based upon a condition on the state at each sampling time. First, an algebraic characterization for the problem to be feasible is derived. Next, an optimal continuous-time controller is derived for a general class of PWA systems with sampled-data switching, for which the optimal control problem is feasible but whose subsystems in some modes may be uncontrollable in the usual sense. Finally, as an application of the proposed approach, the high-speed and energy-saving control problem of the CPU processing is formulated, and the validity of the proposed methods is shown by numerical simulations.

The Effect of Market Structure on Pest Resistance Buildup

In this paper we present a continuous time optimal control analysis of resistance buildup in agricultural chemicals when the supplier is a monopolist. We solve a general model and examine the solutions for Cobb-Douglas demand, including a simulation of changes in the interest rate and pest growth rate. The central finding is that with a monopolistic supplier, resistance buildup is lower than socially optimal. This directly opposes the standard result in the resistance literature. This perspective thus allows us to develop a policy prescription that depends on the structure of the industry supplying the pesticide. It suggests that there may be underuse of chemicals and a slow buildup of resistance in the early stages of the product life when it is controlled by a monopolist. However, when the patent expires and the supply of the chemical becomes more competitive, then there may be tendency to overuse the chemicals.

On equivalence of pencils from discrete-time and continuous-time control

We study two matrix pencils that arise, respectively, in discrete-time and continuous-time optimal and robust control. We introduce a one-to-one transformation between these two pencils. We show that for the pencils under the transformation, their regularity is preserved and their eigenvalues and deflating subspaces are equivalently related. The eigen-structures of the pencils under consideration have strong connections with the associated control problems. Our result may be applied to connect the discrete-time and continuous-time control problems and eventually lead to a unified treatment of these two types of control problems.

ON COMPUTING THE WORST-CASE NORM OF CONVOLUTION SYSTEMS: A COMPARISON OF CONTINUOUS-TIME AND DISCRETE-TIME APPROACHES

This paper compares two approaches to compute the worst-case norm of finite-dimensional convolution systems. All admissible inputs are defined to have bounded magnitude and limited rate of change. Due to physical and mathematical reasons, the inputs are also specified to start from zero. The first approach is based on continuous-time optimal control formulation. Necessary conditions obtained via the Pontryagin's maximum principle provide a systematic means to characterize and construct the worst-case input. The second approach is based on discretization of the norm-computation problem which results in a large-scale finite-dimensional linear programming. We also investigate computational errors including truncation errors and discretization errors. Although the second approach seems to be simpler, the first approach is deemed to yield better accuracy.

Dynamic Congestion Pricing in Disequilibrium

A theory of dynamic congestion pricing for the day-to-day time scale is presented which takes the form of a continuous time optimal control problem. The formulation accomodates elastic nonseparable travel demands and nonseparable travel costs. Necessary conditions for optimal congestion prices are analyzed to uncover bang-bang, singular and synthesized optimal control decison rules for setting network tolls in a dynamic environment. These decision rules are shown to be sufficient for optimality under plausible regularity conditions.

Deterministic Flow Routing and Oligopolistic Competition in Dynamic Data Networks: Modeling and Numerical Solution Using Multigrid Optimization Techniques

In this paper we study the dynamic telecommunications flow routing problem from two points of view: (1) that of a single firm attempting to operate its network efficiently, and (2) oligopolistic competition among several firms competing for network capacity and market share. The mathematical perspective is that of continuous time optimal control and infinite dimensional variational inequalities.

Combined H-Infinity Model-Matching Control and Dual-rate Digital Redesign for Missile Acceleration Autopilots

This paper proposes a missile acceleration autopilot obtained via H∞ model-matching control and dual-rate digital redesign. The first stage of the procedure consists in solving an H∞ model-matching problem in continuous-time. The second stage of the proposed approach is the conversion of the optimal continuous-time control system to a fast-inner-loop rate, slow-outer-loop rate digital control system using a dual-rate digital redesign approach which takes into account the closed-loop dynamics of the missile autopilot in the conversion. Reducing the order of the autopilot transfer functions and scheduling the controllers over the entire flight envelope constitute the last stages of the proposed approach. The proposed dual-rate missile acceleration autopilot is very useful in practice since it provides satisfactory closed-loop performances over an extended range of sampling rates, as compared with other digital autopilots. A nonlinear missile airframe dynamics model is used to demonstrate the application of ...

Optimal control of sampled-data piecewise affine systems

This paper discusses the optimal continuous-time control problem of a class of piecewise affine (PWA) systems, where the switching action of the discrete state is determined at each sampling time according to a condition on the continuous state. Such a system is called here the sampled-data PWA (SD-PWA) system. First, important remarks on the control design of this system via the continuous-(or discrete-)time PWA model are pointed out, which motivate us to use the SD-PWA model. Next, based on the good properties of the proposed model, an optimal continuous-time controller of the SD-PWA systems is proposed.