Dynamic Optimal Control Problem Research Articles

An aspect of the turnpike property. Long time horizon behavior

The turnpike phenomenon concerns the structure of the optimal control and the optimal state of dynamic optimal control problems for long time horizons. The focus is regularly placed on the study of the interior of the time interval. Classical turnpike results state how the solution of the dynamic optimal control problems approaches the solution of the corresponding static optimal control problem in the interior of the time interval.
 In this paper we look at a new aspect of the turnpike phenomenon. We show that for problems without explicit terminal condition, for large time horizons in the last part of the time interval the optimal state approaches a certain limit trajectory that is independent of the terminal time exponentially fast. For large time horizons also the optimal state in the initial part of the time interval approaches exponentially fast a limit state.

Dynamic optimal control of flow front position in injection molding process: A control parameterization-based method

High dynamic and precise control of process variables within permissible limits is the key to high product quality in the injection molding industry. Among these variables, the injection flow rate of molten polymer is particularly significant. In this paper, an effective optimal open-loop and state feedback controller design method is designed to realize the optimal tracking control of the melt polymer flow front position flow of melt polymer fluid during injection molding. To this end, the dynamic model of the filling process is firstly established, taking into account the nonlinearity of hydraulic control system and the effect of molten polymer flow on the dynamics of the injection ram, and then the dynamic optimal tracking control problem of flow front position of molten polymer in injection molding machine is proposed. Then, an open-loop controller and a multi-level feedback control law are well designed, wherein the control signal and the control feedback kernels are parameterized by control parameterization. The controller design problems are transformed into sequential optimal parameter decision problems. The explicit expressions of the gradient information of the objective function and constraints on the parameters of the decision variables are solved by using the state sensitivity equation analysis method, and the gradient information is combined with the nonlinear programming algorithm to solve the optimization problems efficiently. Finally, the viability and effectiveness of the proposed design method are verified by numerical simulations.

Dynamic event-triggered control for discrete-time nonlinear Markov jump systems using policy iteration-based adaptive dynamic programming

This paper investigates a dynamic event-triggered optimal control problem of discrete-time (DT) nonlinear Markov jump systems (MJSs) via exploring policy iteration (PI) adaptive dynamic programming (ADP) algorithms. The performance index function (PIF) defined in each subsystem is updated by utilizing an online PI algorithm, and the corresponding control policy is derived via solving the optimal PIF. Then, we adopt neural network (NN) techniques, including an actor network and a critic network, to estimate the iterative PIF and control policy. Moreover, the designed dynamic event-triggered mechanism (DETM) is employed to avoid wasting additional resources when the estimated iterative control policy is updated. Finally, based on the Lyapunov difference method, it is proved that the system stability and the convergence of all signals can be guaranteed under the developed control scheme. A simulation example for DT nonlinear MJSs with two system modes is presented to demonstrate the feasibility of the control design scheme.

Fractional dynamic analysis and optimal control problem for an SEIQR model on complex networks.

A fractional order susceptible-exposed-infected-quarantined-recovered model is established on the complex networks. We calculate a specific expression for the basic reproduction number R0, prove the existence and uniqueness with respect to the solution, and prove the Ulam-Hyers stability of the model. Using the Latin hypercube sampling-partial rank correlation coefficient method, the influence of parameters on the R0 is analyzed. Based on the results of the analysis, the optimal control of the model is investigated as the control variables with vaccination rate and quarantine rate applying Pontryagin's minimum principle. The effects of α, degree of nodes, and network size on the model dynamics are simulated separately by the prediction correction method.

The application of model predictive control on stock portfolio optimization with prediction based on Geometric Brownian Motion-Kalman Filter

<p style='text-indent:20px;'>A stock portfolio is a collection of assets owned by investors, such as companies or individuals. The determination of the optimal stock portfolio is an important issue for the investors. Management of investors' capital in a portfolio can be regarded as a dynamic optimal control problem. At the same time, the investors should also consider about the prediction of stock prices in the future time. Therefore, in this research, we propose Geometric Brownian Motion-Kalman Filter (GBM-KF) method to predict the future stock prices. Subsequently, the stock returns will be calculated based on the forecasting results of stock prices. Furthermore, Model Predictive Control (MPC) will be used to solve the portfolio optimization problem. It is noticeable that the management strategy of stock portfolio in this research considers the constraints on assets in the portfolio and the cost of transactions. Finally, a practical application of the solution is implemented on 3 company's stocks. The simulation results show that the performance of the proposed controller satisfies the state's and the control's constraints. In addition, the amount of capital owned by the investor as the output of system shows a significant increase.</p>

Binary optimal control by trust-region steepest descent

We present a trust-region steepest descent method for dynamic optimal control problems with binary-valued integrable control functions. Our method interprets the control function as an indicator function of a measurable set and makes set-valued adjustments derived from the sublevel sets of a topological gradient function. By combining this type of update with a trust-region framework, we are able to show by theoretical argument that our method achieves asymptotic stationarity despite possible discretization errors and truncation errors during step determination. To demonstrate the practical applicability of our method, we solve two optimal control problems constrained by ordinary and partial differential equations, respectively, and one topological optimization problem.

Full Information H2 Control of Borel-Measurable Markov Jump Systems with Multiplicative Noises

This paper addresses an H2 optimal control problem for a class of discrete-time stochastic systems with Markov jump parameter and multiplicative noises. The involved Markov jump parameter is a uniform ergodic Markov chain taking values in a Borel-measurable set. In the presence of exogenous white noise disturbance, Gramian characterization is derived for the H2 norm, which quantifies the stationary variance of output response for the considered systems. Moreover, under the condition that full information of the system state is accessible to measurement, an H2 dynamic optimal control problem is shown to be solved by a zero-order stabilizing feedback controller, which can be represented in terms of the stabilizing solution to a set of coupled stochastic algebraic Riccati equations. Finally, an iterative algorithm is provided to get the approximate solution of the obtained Riccati equations, and a numerical example illustrates the effectiveness of the proposed algorithm.

Symbolic Regression Methods for Reinforcement Learning

Reinforcement learning algorithms can solve dynamic decision-making and optimal control problems. With continuous-valued state and input variables, reinforcement learning algorithms must rely on function approximators to represent the value function and policy mappings. Commonly used numerical approximators, such as neural networks or basis function expansions, have two main drawbacks: they are black-box models offering little insight into the mappings learned, and they require extensive trial and error tuning of their hyper-parameters. In this paper, we propose a new approach to constructing smooth value functions in the form of analytic expressions by using symbolic regression. We introduce three off-line methods for finding value functions based on a state-transition model: symbolic value iteration, symbolic policy iteration, and a direct solution of the Bellman equation. The methods are illustrated on four nonlinear control problems: velocity control under friction, one-link and two-link pendulum swing-up, and magnetic manipulation. The results show that the value functions yield well-performing policies and are compact, mathematically tractable, and easy to plug into other algorithms. This makes them potentially suitable for further analysis of the closed-loop system. A comparison with an alternative approach using neural networks shows that our method outperforms the neural network-based one.

Engineering-oriented dynamic optimal control of a greenhouse environment using an improved genetic algorithm with engineering constraint rules

To effectively and practically solve the dynamic economic optimal control problem in greenhouse environments, an improved genetic algorithm with engineering constraint rules (R-GA) is proposed. Based on a dynamic greenhouse-crop model and control vector parameterization (CVP) method to discretize the control variables, the economic optimal control problem is transformed into a nonlinear programming (NLP) problem with finite-dimension parameters, and then R-GA is used to effectively solve the NLP problem. Three components were investigated, such as a smooth penalty function to deal with state variable path constraints, engineering constraint rules to improve the optimization performance and the algorithm feasibility, and the number of collocation points (Nc) to meet the actual control laws. The simulation results demonstrate that the proposed approach greatly improves the effectiveness and feasibility to solve the dynamic optimal control problem in greenhouse environments.

Generalized Gradients in Dynamic Optimization, Optimal Control, and Machine Learning Problems*

Problems of nonsmooth nonconvex dynamic optimization, optimal control (in discrete time), including feedback control, and machine learning are considered from a common point of view. An analogy between controlling discrete dynamical systems and multilayer neural network learning problems with nonsmooth objective functionals and connections is traced. Methods for computing generalized gradients for such systems based on the Hamilton–Pontryagin functions are developed. Gradient (stochastic) algorithms for optimal control and learning are extended to nonconvex nonsmooth dynamic systems.

Dynamic model of real investment with lags

The dynamic optimal control problem of corporation’s real investment in case of a limited budget is analyzed. Initial optimization problem is formulated as a system of linear integral equations of Volterra type. The offered model takes into account the accumulated effect of real investment and previous revenue on sales. It is supposed that the previous revenue indirectly affects the current one and reflects the influence of other factors. The accumulated impact of investment is distributed over a fix time interval. The total profit maximization problem under restrictions is stated. The existence theorem for the solutions of the total profit maximization problem is proved. The model is tested with the real data. In this case the Pontryagin’s maximum principle and numerical methods of solving the integral and differential equations are used. The results of experiment are in accordance with the understanding of optimal investment strategy. Considering it the authors state that the offered model makes a solution which is possible to increase the total profit over the planning period.

On the Turnpike Phenomenon for Optimal Boundary Control Problems with Hyperbolic Systems

We study problems of optimal boundary control with systems governed by linear hyperbolic partial differential equations. The objective function is quadratic and given by an integral over the finite time interval $(0,\, T)$ that depends on the boundary traces of the solution. If the time horizon $T$ is sufficiently large, the solution of the dynamic optimal boundary control problem can be approximated by the solution of a steady state optimization problem. We show that for $T\rightarrow\infty$ the approximation error converges to zero in the sense of the norm in $L^2(0,\,1)$ with the rate $1/T$, if the time interval $(0,\, T)$ is transformed to the fixed interval $(0,\, 1)$. Moreover, we show that also for optimal boundary control problems with integer constraints for the controls the turnpike phenomenon occurs. In this case the steady state optimization problem also has the integer constraints. If $T$ is sufficiently large, the integer part of each solution of the dynamic optimal boundary control problem with integer constraints is equal to the integer part of a solution of the static problem. A numerical verification is given for a control problem in gas pipeline operations.

Optimal control of bacterial growth for the maximization of metabolite production.

Microorganisms have evolved complex strategies for controlling the distribution of available resources over cellular functions. Biotechnology aims at interfering with these strategies, so as to optimize the production of metabolites and other compounds of interest, by (re)engineering the underlying regulatory networks of the cell. The resulting reallocation of resources can be described by simple so-called self-replicator models and the maximization of the synthesis of a product of interest formulated as a dynamic optimal control problem. Motivated by recent experimental work, we are specifically interested in the maximization of metabolite production in cases where growth can be switched off through an external control signal. We study various optimal control problems for the corresponding self-replicator models by means of a combination of analytical and computational techniques. We show that the optimal solutions for biomass maximization and product maximization are very similar in the case of unlimited nutrient supply, but diverge when nutrients are limited. Moreover, external growth control overrides natural feedback growth control and leads to an optimal scheme consisting of a first phase of growth maximization followed by a second phase of product maximization. This two-phase scheme agrees with strategies that have been proposed in metabolic engineering. More generally, our work shows the potential of optimal control theory for better understanding and improving biotechnological production processes.

A controllability perspective of dynamic soaring

Dynamic soaring is an exquisite flying technique to acquire energy from the atmospheric wind shear. In this study, a geometric nonlinear controllability analysis of an unmanned aerial vehicle (UAV) under dynamic soaring conditions is performed. To achieve such an objective, the state-of-the-art mathematical tools of nonlinear controllability are summarized and presented to an aeronautical engineering audience. The dynamic soaring optimal control problem is then formulated and solved numerically. The controllability of the UAV along the optimal soaring trajectory is analyzed. More importantly, the geometric nonlinear controllability characteristics of generic flight dynamics are analyzed in the presence and absence of wind shear to provide a controllability explanation for the role of wind shear in the physics of dynamic soaring flight. It is found that the wind shear is instrumental in ensuring controllability as it allows the UAV attitude controls (pitch and roll) to play the role of thrust in controlling the flight path angle. The presented analysis represents a controllability-based mathematical proof for the energetics of flight physics.

Dynamic Optimal Control of a One-Dimensional Magnetohydrodynamic System With Bilinear Actuation

The manipulation of the magnetic field has been a proven effective method to change the flow velocity in magnetohydrodynamics (MHD) flow systems. In this paper, we consider a novel bilinear magnetic control problem arising in a 1-D MHD flow system modeled by a set of coupled partial differential equations (PDEs). We formulate the control of the magnetic field as a finite-time PDE-constrained dynamic optimal control problem and our aim is to realize the desired stationary state of the flow velocity at a specific terminal time. A model order reduction technique, based on proper orthogonal decomposition and Galerkin projection procedure, is first adopted to approximate the original complex optimization problem governed by PDEs into a semi-discrete approximation problem governed by a low-dimensional reduced-order model, and therefore can efficiently reduce the computational burden of the dynamic system. Then, the piecewise-linear control parameterization method is used to obtain an approximate optimal parameter selection problem that can be solved using nonlinear optimization techniques such as sequential quadratic programming. The exact formulas for the gradients of the defined cost functional with respect to the decision parameters are analytically derived via state sensitivity method. Numerical simulation results verify the effectiveness of our proposed computational method. The methodology proposed in this paper is a potential implementation of a real-time control strategy in a number of MHD flow systems.

Stochastic impulse control with regime-switching dynamics

Optimal product management problems with multiple product generations in continuous time lead to the consideration of dynamic optimal control problems that feature both intervention costs and partially controlled regime shifts. We therefore investigate and solve such stochastic impulse control problems with regime-switching in a general setting. We analyze the associated coupled systems of quasi-variational inequalities in suitable Sobolev spaces, and we establish a direct approach to construct both the value function and optimal strategies. Our results in particular yield a numerical method for the computation of the optimal value function and the associated strategies.

A stochastic control problem and related free boundaries in finance

In this paper, we investigate an optimal stopping problem (mixed with stochastic controls) for a manager whose utility is nonsmooth and nonconcave over a finite time horizon. The paper aims to develop a new methodology, which is significantly different from those of mixed dynamic optimal control and stopping problems in the existing literature, so as to figure out the manager's best strategies. The problem is first reformulated into a free boundary problem with a fully nonlinear operator. Then, by means of a dual transformation, it is further converted into a free boundary problem with a linear operator, which can be consequently tackled by the classical method. Finally, using the inverse transformation, we obtain the properties of the optimal trading strategy and the optimal stopping time for the original problem.

Optimal combined long-term facility design and short-term operational strategy for CHP capacity investments

This work presents a detailed case study for the optimization of the expansion of a district energy system evaluating the investment decision timing, type of capacity expansion, and fine-scale operational modes. The study develops an optimization framework to find the investment schedule over 30 years with options of investing in traditional heating sources (boilers) or a next-generation combined heat and power (CHP) plant that provides heat and electricity. In district energy systems, the selected capacity and type of system is dependent on demand-side requirements, energy prices, and environmental costs. This work formulates capacity planning over a time horizon as a dynamic optimal control problem considering both operational modes and capital investment decisions. The initial plant is modified by the dynamic optimization throughout the 30 years to maximize profitability. The combined optimal controller and capital investment planner solves a large scale mixed integer nonlinear programming problem to provide the timing and size of the capacity investment (30 year outlook) and also guidance on the mode of operation (1 h time intervals). The optimizer meets optimal economic, environmental, and regulatory constraints with the suggested design and operational guidance with daily cyclical load following of heat and electricity demand.

Bi-objective optimization of dynamic systems by continuation methods

Abstract As a proof of concept the properties of path-following methods are studied for multi-objective optimization problems involving dynamic systems (also called multi-objective dynamic optimization or multi-objective optimal control problems), which have never been presented before. Two case studies with two objectives are considered to cover convex, as well as non-convex trade-off curves or Pareto sets. In order for the method to be applicable, the infinite dimensional dynamic problems have to be discretized and scalarization parameters have to be introduced, which leads to large-scale parametric nonlinear optimization problems. For both the chemical tubular reactor and the fed-batch bioreactor case study it is found that a path-following continuation approach is able to compute the Pareto fronts accurately and efficiently. A branch switching technique is required whenever a constraint switches from active to inactive or vice versa. When dealing with non-convex problems, a technique for detecting inflection points is required. Simple switching techniques are suggested and have been tested successfully.

A comparative economic study of two configurations of hydro-wind power plants

This paper presents a comparison between two options for operating a wind park in combination with a pumped-storage hydro-plant. First, we analyze the behavior of a wind farm that goes to the electricity market having previously forecasted the wind speed, while accepting the deviation penalties these forecasts may incur. Second, we study the possibility of the wind farm not going to the market individually, but as part of a hydro-wind power plant. Considerations about the optimal size of the wind farm and the hydro-pumped storage plant have previously been analyzed in the literature. However, most of these papers do not analyze dynamic considerations. The dimensioning of the system is not studied in our paper; its main feature will instead be the consideration of the dynamic optimal control problem. The use of Pontryagin's Maximum Principle allows us to obtain a very efficient optimization algorithm. The hydro-plant is modelled in great detail, using a variable-head model for the pumped hydro-plant. Our study provides a useful tool for electricity companies. The algorithm is able to solve the problem of the day-ahead market, a problem that is not solved in other studies.