Continuous-time Optimal Control Problem Research Articles

Innovative Optimal Control Path Planning for PMCU in Cruise Ship Construction: A Kinematic and Obstacle Avoidance Model

In cruise ship manufacturing, it is common practice for prefabricated modular cabin units (PMCU) to be assembled in the workshop and transported to a designated installation location on board the ship. This paper proposed a Radau pseudospectral method based on a probabilistic roadmap (PRM) for PMCU path planning in complex deck environments. Firstly, a kinematic model is constructed, and the continuous-time optimal control path planning problem is established. Among them, an area-based obstacle avoidance constraint is applied to the approach. Then, the continuous-time optimal control problem is discretized into a nonlinear programming (NLP) problem using the orthogonal collocation method. For the initial value-sensitive NLP problem solving, a strategy using PRM combined with a heuristic rule is used to obtain a high-quality initial guess. Finally, experiments under two operating scenarios of PMCU pushing demonstrate that the method is capable of generating a better path while ensuring that the whole process is collision-free. The results show that the algorithm proposed in this paper can effectively solve the PMCU pushing path planning problem in the complex deck environment, and the path considering kinematic characteristics can be used for cabin installation, which has strong operability and applicability.

Policy-Iteration-Based Finite-Horizon Approximate Dynamic Programming for Continuous-Time Nonlinear Optimal Control.

The Hamilton-Jacobi-Bellman (HJB) equation serves as the necessary and sufficient condition for the optimal solution to the continuous-time (CT) optimal control problem (OCP). Compared with the infinite-horizon HJB equation, the solving of the finite-horizon (FH) HJB equation has been a long-standing challenge, because the partial time derivative of the value function is involved as an additional unknown term. To address this problem, this study first-time bridges the link between the partial time derivative and the terminal-time utility function, and thus it facilitates the use of the policy iteration (PI) technique to solve the CT FH OCPs. Based on this key finding, the FH approximate dynamic programming (ADP) algorithm is proposed leveraging an actor-critic framework. It is shown that the algorithm exhibits important properties in terms of convergence and optimality. Rather importantly, with the use of multilayer neural networks (NNs) in the actor-critic architecture, the algorithm is suitable for CT FH OCPs toward more general nonlinear and complex systems. Finally, the effectiveness of the proposed algorithm is demonstrated by conducting a series of simulations on both a linear quadratic regulator (LQR) problem and a nonlinear vehicle tracking problem.

Analysis of Theoretical and Numerical Properties of Sequential Convex Programming for Continuous-Time Optimal Control

Sequential Convex Programming (SCP) has recently gained significant popularity as an effective method for solving optimal control problems and has been successfully applied in several different domains. However, the theoretical analysis of SCP has received comparatively limited attention, and it is often restricted to discrete-time formulations. In this paper, we present a unifying theoretical analysis of a fairly general class of SCP procedures for continuous-time optimal control problems. In addition to the derivation of convergence guarantees in a continuous-time setting, our analysis reveals two new numerical and practical insights. First, we show how one can more easily account for manifold-type constraints, which are a defining feature of optimal control of mechanical systems. Second, we show how our theoretical analysis can be leveraged to accelerate SCP-based optimal control methods by infusing techniques from indirect optimal control.

Structure-exploiting Newton-type method for optimal control of switched systems

ABSTRACT This study proposes an efficient Newton-type method for the optimal control of switched systems under a given mode sequence. A mesh-refinement-based approach is utilised to discretise continuous-time optimal control problems (OCPs) and formulate a nonlinear program (NLP), which guarantees the local convergence of a Newton-type method. A dedicated structure-exploiting algorithm (Riccati recursion) is proposed to perform a Newton-type method for the NLP efficiently because its sparsity structure is different from a standard OCP. The proposed method computes each Newton step with linear time-complexity for the total number of discretization grids as the standard Riccati recursion algorithm. Additionally, the computation is always successful if the solution is sufficiently close to a local minimum. Conversely, general quadratic programming (QP) solvers cannot accomplish this because the Hessian matrix is inherently indefinite. Moreover, a modification on the reduced Hessian matrix is proposed using the nature of the Riccati recursion algorithm as the dynamic programming for a QP subproblem to enhance the convergence. A numerical comparison is conducted with off-the-shelf NLP solvers, which demonstrates that the proposed method is up to two orders of magnitude faster. Whole-body optimal control of quadrupedal gaits is also demonstrated and shows that the proposed method can achieve the whole-body model predictive control (MPC) of robotic systems with rigid contacts.

Adaptive discounted control for piecewise deterministic Markov processes

The main goal of this paper is to study the adaptive infinite-horizon discounted continuous-time optimal control problem of piecewise deterministic Markov processes (PDMPs) with the control acting continuously on the jump intensity λ and on the transition measure Q of the process. It is assumed that jump parameters (λ and Q), as well the continuous and boundary costs (Cg and Ci respectively), depend on an unknown parameter β⁎. It is shown that the principle of estimation and control holds, that is, the strategy consisting of choosing, at each stage n, an action according to an optimal stationary policy, where the true but unknown parameter β⁎ is replaced by its estimated value βˆn, is asymptotically discount optimal, provided that the sequence of estimators {βˆn} of β⁎ is strongly consistent, that is, βˆn converge to β⁎ almost surely. In the framework of PDMPs, the so-called discrepancy function depends on the derivative along the flow of the value function as well as on some boundary conditions, which brings new challenges in the analysis of this problem.

Inaccuracy Matters: Accounting for Solution Accuracy in Event-Triggered Nonlinear Model Predictive Control

We consider the effect of using approximate system predictions in event-triggered control schemes. These approximations often result from using numerical transcription methods for solving continuous-time optimal control problems. Mesh refinement can guarantee upper bounds on the error in the differential equations that model the system dynamics. We employ the accuracy guarantees of a mesh refinement scheme to show that the proposed event-triggering scheme, which compares the measured system with approximate state predictions, can be used with a guaranteed strictly positive inter-update time. Furthermore, if knowledge of the employed transcription scheme or the approximation errors are available, then better online estimates of inter-update times can be obtained. We also detail a method of tightening constraints on the approximate system trajectory to guarantee constraint satisfaction of the continuous-time system. This is the first work to incorporate prediction accuracy in triggering metrics to guarantee reliable lower bounds for inter-update times and perform solution-dependent constraint tightening.

Maximum Entropy Optimal Control of Continuous-Time Dynamical Systems

Maximum entropy reinforcement learning methods have been successfully applied to a range of challenging sequential decision-making and control tasks. However, most of the existing techniques are designed for discrete-time systems although there has been a growing interest to handle physical processes evolving in continuous time. As a first step toward their extension to continuous-time systems, this article aims to study the theory of maximum entropy optimal control in continuous time. Applying the dynamic programming principle, we derive a novel class of Hamilton–Jacobi–Bellman (HJB) equations and prove that the optimal value function of the maximum entropy control problem corresponds to the unique viscosity solution of the HJB equation. We further show that the optimal control is uniquely characterized as Gaussian in the case of control-affine systems and that, for linear-quadratic problems, the HJB equation is reduced to a Riccati equation, which can be used to obtain an explicit expression of the optimal control. The results of our numerical experiments demonstrate the performance of our maximum entropy method in continuous-time optimal control and reinforcement learning problems.

Influence of Discretization in Dynamically Embedded Model Predictive Control

The paper analyzes the closed-loop stability of Dynamically Embedded Model Predictive Control for input-constrained continuous-time nonlinear systems. Given a stabilizing continuous-time optimal control problem, the proposed method performs a discrete approximation to obtain a finite number of optimization variables. The resulting optimization problem is then embedded into a dynamic feedback law that evolves in parallel to the system. Using Input-to-State Stability, it is shown that the dynamic interconnection between the ideal continuous-time model predictive controller and the dynamically embedded solver is asymptotically stable for a sufficiently small discretization step and sufficiently fast solver dynamics. Numerical results, however, highlight a counter-intuitive behavior: as the discretization step decreases, the stability of the closed-loop system tends to deteriorate. This suggests that, although the discretization should be sufficiently accurate to correctly capture the behavior of the system, oversampling the system dynamics may be just as harmful as undersampling.

Performance-based data-driven optimal tracking control of shape memory alloy actuated manipulator through reinforcement learning

This article focuses on the continuous-time optimal tracking control problem of a shape memory alloy (SMA) actuated manipulator subject to prescribed error constraints and completely unknown nonlinear dynamics. Firstly, prespecified error constraints imposed by the prescribed performance function (PPF) are transformed into an equivalent unconstrained task by adopting a transformation function, where the newly designed PPF is irrelated to the initial condition. An unconstrained augmented system and a long-term discounted performance considering tracking errors and input cost are constructed for the convenience of designing the optimal controller. Then, a model-based optimal control algorithm is derived to solve the Hamilton–Jacobi–Bellman equation (HJBE). Next, data-driven reinforcement learning (RL) is employed to approximate the solution of the HJBE iteratively to obviate requiring an SMA manipulator model. Moreover, critic neural networks (NNs) and actor NNs are introduced to the RL-based optimal controller, and the least-squares method is used to find the parameters for the NNs-based RL algorithm. Rigorous theoretical analyses demonstrate that the proposed controller can stable the SMA manipulator system, and the tracking errors are always constrained within the prescribed region. Finally, experiments are conducted on the established SMA actuated manipulator platform, and the results illustrate that the proposed controller is feasible and effective.

Dynamic optimization analyses and algorithm design for the parallel heat exchange system in spacecraft

Dynamic optimization of the fluid loop is critical of the active thermal control system (ATCS) for future spacecraft. In this paper, the transient heat current model of the parallel heat exchanger system is constructed by the thermoelectric analogy method to analyze the optimal control problem of dynamic flow allocation. The continuous free time optimal flow control problem is transformed into a fixed-time nonlinear programming problem which can be solved by the sequential quadratic programming (SQP) algorithm through the control variable parameterization (CVP) and time-scaling methods. The simulation showed that the optimized flow allocation significantly reduced the process time required for system heat dissipation. Compared with the non-gradient particle swarm optimization (PSO) algorithm, the SQP method constructed in this paper decreased the optimization index by about 4%∼6% and the calculation time by about one order of magnitude. To diminish the sensitivity of the SQP algorithm to the initial value, this paper simplifies the model according to the system characteristics. Through the Pontryagin extremum principle, it is found and proved that, under the simplified model, the optimal flow rates are a set of time-independent variables, which significantly reduces the difficulty of the optimization problem, according to which a hybrid algorithm is designed that uses the optimal result of the simplified model as the initial value predictor. Compared with the average of using the random initial value, using the estimated initial value decreased the optimization index by more than 2% and the computation time by about 60%, under the same SQP algorithm.

Applications of Markov chain approximation methods to optimal control problems in economics

In this paper we explore some benefits of using the finite-state Markov chain approximation (MCA) method of Kushner and Dupuis (2001) to solve continuous-time optimal control problems in economics. We first show that the implicit finite-difference scheme of Achdou et al. (2022) amounts to a limiting form of the MCA method for a certain choice of approximating chains and policy function iteration for the resulting system of equations. We then illustrate that relative to the implicit finite-difference approach, using variations of modified policy function iteration to solve income fluctuation problems both with and without discrete choices can lead to an increase in the speed of convergence of more than an order of magnitude. Finally, we provide several consistent chain constructions for stationary portfolio problems with correlated state variables, and illustrate the flexibility of the MCA approach by using it to construct and compare two simple solution methods for a general equilibrium model with financial frictions.

Data-driven optimal tracking control for SMA actuated systems with prescribed performance via reinforcement learning

This article addresses the data-driven performance-prescribed continuous-time optimal tracking control problem for the shape memory alloy (SMA) actuated systems with completely unknown model knowledge. Firstly, the error constraint problem is transformed into the unconstrained error tracking problem by the prescribed performance control (PPC) method. Then, the optimal tracking control problem (OTCP) is pre-processed by establishing an unconstrained augmented system. Furthermore, the Hamilton–Jacobi–Bellman equation (HJBE) of the OTCP is solved iteratively by utilizing reinforcement learning (RL) without the SMA actuator model information requirement. The value function and execution strategy of the RL are approximated by two neural networks, acting as actor and critic, respectively, and the actor–critic based RL is implemented using the least-squares method. In addition, the Lyapunov method ensures the stability of the closed-loop system actuated by SMA, as well as the user-specified error constraints concerning the error convergence rate, overshoot, and tracking accuracy. Finally, the experimental results and comparisons illustrate the validity of the proposed method.

Deep Learning for Efficient and Optimal Motion Planning for AUVs with Disturbances.

We use the recent advances in Deep Learning to solve an underwater motion planning problem by making use of optimal control tools—namely, we propose using the Deep Galerkin Method (DGM) to approximate the Hamilton–Jacobi–Bellman PDE that can be used to solve continuous time and state optimal control problems. In order to make our approach more realistic, we consider that there are disturbances in the underwater medium that affect the trajectory of the autonomous vehicle. After adapting DGM by making use of a surrogate approach, our results show that our method is able to efficiently solve the proposed problem, providing large improvements over a baseline control in terms of costs, especially in the case in which the disturbances effects are more significant.

Potential difference games and applications

This paper concerns noncooperative difference games with infinite horizon discounted payoffs. More precisely, it is a sequel to a previous paper [A. Fonseca-Morales and O. Hern'andez-Lerma, Potential differential games, Dynamic Games Appl., 8(2) (2018), pp. 254–279.] where the notion of continuous-time potential games was introduced. That is, a noncooperative differential game to which we can associate a continuous-time optimal control problem (OCP) whose solutions are Nash equilibria for the original game. Thus, finding or analysing the properties of Nash equilibria for the game reduces to that of the optimal solution of an OCP. Here, we study difference games, that is, the discrete-time case. First, we give several mild conditions for which a difference game is a potential difference game (PDG). Then, we illustrate our results with several examples and applications.

Pontryagin maximum principle for state constrained optimal sampled-data control problems on time scales

In this paper we consider optimal sampled-data control problems on time scales with inequality state constraints. A Pontryagin maximum principle is established, extending to the state constrained case existing results in the time scale literature. The proof is based on the Ekeland variational principle and on the concept of implicit spike variations adapted to the time scale setting. The main result is then applied to continuous-time min-max optimal sampled-data control problems, and a maximal velocity minimization problem for the harmonic oscillator with sampled-data control is numerically solved for illustration.

The hub location’s method for solving optimal control problems

In this paper, the p-hub median’s algorithm is modified to find an approximate solution of bounded continuous-time nonlinear optimal control problems (NOCP). For this purpose, after presenting the similarities between uncapacitated single allocation p-Hub median problem (USApHMP) which is a special case of the well-studied hub location problem and a discretized form of NOCP, an improvement on sequential general variable neighborhood search algorithm (SGVNS) is proposed. This algorithm that denoted as new-sequential general variable neighborhood search (N-SGVNS) is based on three local searches that use efficient neighborhood interchange. In addition, comparing the NOCP with the USApHMP has led us to introduce a suitable method to find an approximate solution for NOCP. Our results of the implementation of the N-SGVNS on the real-world data set for the USApHMP show the high accuracy of the obtained solutions. Because of the proposed algorithm is able to solve the USApHMP successfully, we hoped that this method would also be successful for NOCP. Solving of benchmark NOCP and then comparing the results of the N-SGVNS with recently developed algorithms suggest that the presented strategy may provide an outstanding framework for designing cost-effective control functions.

Optimal Multivehicle Motion Planning Using Bernstein Approximants

This article presents a computational framework to efficiently generate feasible and safe trajectories for multiple autonomous vehicle operations. We formulate the optimal motion planning problem as a continuous-time optimal control problem, and approximate its solutions in a discretized setting using Bernstein polynomials. The latter possess convenient properties that allow to efficiently compute and enforce constraints along the vehicles' trajectories, such as maximum speed and angular rates, minimum distance between trajectories and between the vehicles and known obstacles, etc. Thus, the proposed method is particularly suitable for generating trajectories in real-time for safe operations in complex environments and multiple vehicle missions. We show, using a rigorous mathematical framework, that the solution to the discretized optimal motion planning problem converges to that of the continuous-time one. The advantages of the proposed method are investigated through numerical examples.

An identifier-actor-optimizer policy learning architecture for optimal control of continuous-time nonlinear systems

An intelligent solution method is proposed to achieve real-time optimal control for continuous-time nonlinear systems using a novel identifier-actor-optimizer (IAO) policy learning architecture. In this IAO-based policy learning approach, a dynamical identifier is developed to approximate the unknown part of system dynamics using deep neural networks (DNNs). Then, an indirect-method-based optimizer is proposed to generate high-quality optimal actions for system control considering both the constraints and performance index. Furthermore, a DNN-based actor is developed to approximate the obtained optimal actions and return good initial guesses to the optimizer. In this way, the traditional optimal control methods and state-of-the-art DNN techniques are combined in the IAO-based optimal policy learning method. Compared to the reinforcement learning algorithms with actor-critic architectures that suffer hard reward design and low computational efficiency, the IAO-based optimal policy learning algorithm enjoys fewer user-defined parameters, higher learning speeds, and steadier convergence properties in solving complex continuous-time optimal control problems (OCPs). Simulation results of three space flight control missions are given to substantiate the effectiveness of this IAO-based policy learning strategy and to illustrate the performance of the developed DNN-based optimal control method for continuous-time OCPs.

Approximating optimal finite horizon feedback by model predictive control

We consider a finite-horizon continuous-time optimal control problem with nonlinear dynamics, an integral cost, control constraints and a time-varying parameter which represents perturbations or uncertainty. After discretizing the problem we employ a Model Predictive Control (MPC) approach by first solving the problem over the entire remaining time horizon and then applying the first element of the optimal discrete-time control sequence, as a constant in time function, to the continuous-time system over the sampling interval. Then the state at the end of the sampling interval is measured (estimated) with certain error, and the process is repeated at each step over the remaining horizon. As a result, we obtain a piecewise constant function of time representing MPC-generated control signal. Hence MPC turns out to be an approximation to the optimal feedback control for the continuous-time system. In our main result we derive an estimate of the difference between the MPC-generated state and control trajectories and the optimal feedback generated state and control trajectories, both obtained for the same value of the perturbation parameter, in terms of the step-size of the discretization and the measurement error. Numerical results illustrating our estimate are reported.

Spatio-Temporal Value of Energy Storage in Transmission Networks

This article proposes a continuous-time modeling approach to calculate the spatio-temporal value of energy storage (ES) devices in power transmission networks. More specifically, this article formulates the dynamic operation of ES devices in power transmission networks as a continuous-time optimal control problem, where the Lagrange multiplier trajectories of nodal power balance constraints are defined as continuous-time locational marginal prices (LMPs). In addition, the adjoint functions associated with ES state equations are calculated in closed-form, which define the locational value of stored energy (LVSE) in ES devices. The notion of LVSE is, then, leveraged to define incremental charging utility and discharging cost rates of ES devices, which pass on the spatio-temporal values of ES devices to the continuous-time LMPs. The numerical results of implementing the proposed continuous-time model on a test 3-bus power system and the IEEE-Reliability Test System highlight its effectiveness in calculating the value of ES energy shifting and congestion relief services in transmission networks.