Finite-horizon Optimal Control Research Articles

A lie group PMP approach for optimal stabilization and tracking control of autonomous underwater vehicles

AbstractIn this research, we explore a finite horizon optimal stabilization and tracking control scheme for the dynamical model of a 6‐DOF Autonomous Underwater Vehicle (AUV). Dynamical equations of the AUV are represented in a Lie group () framework, encompassing both translational and rotational motions. Utilizing a left Lie group action on , we define error function for velocities via a right transport map to effectively address optimal trajectory tracking. The optimal control objective is formulated as a trade‐off problem, aiming to minimize both errors and control effort simultaneously. Left action on yields the left trivialized Hamiltonian function from which the concomitant state and costate dynamical equations are derived using Pontryagin's Minimum Principle (PMP). Consequently, the resulting two‐point boundary value problem is solved to obtain optimal trajectories. We demonstrate the optimality of the resulting solution obtained from the derived control law. For ensuring boundedness in the presence of small disturbances, this study incorporates the effects of internal parametric uncertainties associated with added mass and inertia components, along with the influence of external disturbances induced by ocean currents. Through simulation validations, we confirm the alignment of our results with the theoretical developments, demonstrating that the proposed control law effectively mitigates both parametric uncertainties and ocean current disturbances.

Q-learning based tracking control with novel finite-horizon performance index

A data-driven method is designed to realize the model-free finite-horizon optimal tracking control (FHOTC) of unknown linear discrete-time systems based on Q-learning in this paper. First, a novel finite-horizon performance index (FHPI) that only depends on the next-step tracking error is introduced. Then, an augmented system is formulated, which incorporates with the system model and the trajectory model. Based on the novel FHPI, a derivation of the augmented time-varying Riccati equation (ATVRE) is provided. We present a data-driven FHOTC method that uses Q-learning to optimize the defined time-varying Q-function. This allows us to estimate the solutions of the ATVRE without the system dynamics. Finally, the validity and features of the proposed Q-learning-based FHOTC method are demonstrated by means of conducting comparative simulation studies.

Finite-Horizon Optimal Control for Linear and Nonlinear Systems Relying on Constant Optimal Costate

Finite-Horizon Optimal Control for Linear and Nonlinear Systems Relying on Constant Optimal Costate

Optimal control for reachability of Markov jump switching Boolean control networks subject to output trackability

This article aims at designing the finite horizon optimal control sequence (FHOCS) for ensuring the reachability of Markov jump switching Boolean control networks (MJSBCNs) subject to output trackability. The key step in solving such a problem is to construct a Luenberger-like observer (LLO). Our approach includes a few steps as follows. We firstly propose the formulation of the optimal control problem (OCP) of maximising the probability of finite-time reachability subject to output trackability. Then, by converting the dynamics of MJSBCNs into an equivalent expectation system, an LLO is established for MJSBCNs. In this setup, a Luenberger-like observation matrix is constructed, which is an effect tool to solve the finite-time reachability problem. Finally, an LLO method is developed to find the FHOCS by maximising the T -step Luenberger-like observation matrix. In a biological example, we show the effectiveness of our developed methods.

Application of Hamilton–Jacobi–Bellman Equation/Pontryagin’s Principle for Constrained Optimal Control

This article applies novel results for infinite- and finite-horizon optimal control problems with nonlinear dynamics and constraints. We use the Valentine transformation to convert a constrained optimal control problem into an unconstrained one and show uniqueness of the value function to the corresponding Hamilton–Jacobi–Bellman (HJB) equation. From there, we show how to approximate the solution of the initial (in)finite-horizon problem with a family of solutions that is Γ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varGamma $$\\end{document}-convergent. Optimal solutions are efficiently obtained via a solver based on Pontryagin’s Principle (PP). The proposed methodology is demonstrated on the path planning problem using the full nonlinear dynamics of an unmanned aerial vehicle (UAV) and autonomous underwater vehicle (AUV) involving state constraints in 3D environments with obstacles.

Dynamic Event-Driven Finite-Horizon Optimal Consensus Control for Constrained Multiagent Systems.

This article investigates the event-driven finite-horizon optimal consensus control problem for multiagent systems with symmetric or asymmetric input constraints. Initially, in order to overcome the difficulty that the Hamilton-Jacobi-Bellman equation is time-varying in finite-horizon optimal control, a single critic neural network (NN) with time-varying activation function is applied to obtain the approximate optimal control. Meanwhile, for minimizing the terminal error to satisfy the terminal constraint of the value function, an augmented error vector containing the Bellman residual and the terminal error is constructed to update the weight of the NN. Furthermore, an improved learning law is proposed, which relaxes the tricky persistence excitation condition and eliminates the requirement of initial stability control. Moreover, a specific algorithm is designed to update the historical dataset, which can effectively accelerate the convergence rate of network weight. In addition, to improve the utilization rate of the communication resource, an effective dynamic event-triggering mechanism (DETM) composed of dynamic threshold parameters (DTPs) and auxiliary dynamic variables (ADVs) is designed, which is more flexible compared with the ADV-based DETM or DTP-based DETM. Finally, to support the effectiveness of the proposed method and the superiority of the designed DETM, a simulation example is provided.

UKF-Based Optimal Tracking Control for Uncertain Dynamic Systems With Asymmetric Input Constraints.

To enhance system robustness in the face of uncertainty and achieve adaptive optimization of control strategies, a novel algorithm based on the unscented Kalman filter (UKF) is developed. This algorithm addresses the finite-horizon optimal tracking control problem (FHOTCP) for nonlinear discrete-time (DT) systems with uncertainty and asymmetric input constraints. An augmented system is constructed with asymmetric control constraints being considered. The augmented problem is addressed with a DT Hamilton-Jacobi-Bellman equation (DTHJBE). By analyzing convergence with regard to the cost function and control law, the UKF-based iterative adaptive dynamic programming (ADP) algorithm is proposed. This algorithm approximates the solution of the DTHJBE, ensuring that the cost function converges to its optimal value within a bounded range. To execute the UKF-based iterative ADP algorithm, the actor-estimator-critic framework is built, in which the estimator refers to system state estimation through the application of UKF. Ultimately, simulation examples are presented to show the performance of the proposed method.

Value functional and optimal feedback control in linear-quadratic optimal control problem for fractional-order system

In this paper, a finite-horizon optimal control problem involving a dynamical system described by a linear Caputo fractional differential equation and a quadratic cost functional is considered. An explicit formula for the value functional is given, which includes a solution of a certain Fredholm integral equation. A step-by-step feedback control procedure for constructing $ \varepsilon $-optimal controls with any accuracy $ \varepsilon > 0 $ is proposed. The basis for obtaining these results is the study of a solution of the associated Hamilton–Jacobi–Bellman equation with so-called fractional coinvariant derivatives.

Set-point tracking MPC with avoidance features

This work proposes a finite-horizon optimal control strategy to solve the tracking problem while providing avoidance features to the closed-loop system. Inspired by the set-point tracking model predictive control (MPC) framework, the central idea of including artificial variables into the optimal control problem is considered. This approach allows us to add avoidance features into the set-point tracking MPC strategy without losing the properties of an enlarged domain of attraction and feasibility insurances in the face of any changing reference. Besides, the artificial variables are considered together with an avoidance cost functional to establish the basis of the strategy, maintaining the recursive feasibility property in the presence of a previously unknown number of regions to be avoided. It is shown that the closed-loop system is recursively feasible and input-to-state-stable under the mild assumption that the avoidance cost is uniformly bounded over time. Finally, two numerical examples illustrate the controller behavior.

Deep reinforcement learning in finite-horizon to explore the most probable transition pathway

In many scientific and engineering problems, noise and nonlinearity are unavoidable, which could induce interesting mathematical problem such as transition phenomena. This paper focuses on efficiently discovering the most probable transition pathway for stochastic dynamical systems employing reinforcement learning. With the Onsager–Machlup action functional theory to quantify rare events in stochastic dynamical systems, finding the most probable pathway is equivalent to solving a variational problem on the action functional. When the action function cannot be explicitly expressed by paths near the reference orbit, the variational problem needs to be converted into an optimal control problem. First, by integrating terminal prediction into the reinforcement learning framework, we develop a Terminal Prediction Deep Deterministic Policy Gradient (TP-DDPG) algorithm to deal with the finite-horizon optimal control issue in a forward way. Next, we present the convergence analysis of our algorithm for the value function in terms of the neural network’s approximation error and estimation error. Finally, we conduct various experiments in different dimensions for the transition problems in applications to illustrate the effectiveness of our algorithm.

Multitrajectory Model Predictive Control for Safe UAV Navigation in an Unknown Environment

The problem of navigating an unmanned aerial vehicle (UAV) in an unknown environment is addressed with a novel model predictive control (MPC) formulation, named multitrajectory MPC (mt-MPC). The objective is to safely drive the vehicle to the desired target location by relying only on the partial description of the surroundings provided by an exteroceptive sensor. This information results in time-varying constraints during the navigation among obstacles. The proposed mt-MPC generates a sequence of position set points that are fed to control loops at lower hierarchical levels. To do so, the mt-MPC predicts two different state trajectories, a safe one and an exploiting one, in the same finite horizon optimal control problem (FHOCP). This formulation, particularly suitable for problems with uncertain time-varying constraints, allows one to partially decouple constraint satisfaction (safety) from cost function minimization (exploitation). Uncertainty due to modeling errors and sensors noise is taken into account as well, in a set membership (SM) framework. Theoretical guarantees of persistent obstacle avoidance are derived under suitable assumptions, and the approach is demonstrated experimentally out-of-the-laboratory on a prototype built with off-the-shelf components.

Data-Driven Finite-Horizon Tracking Control With Event-Triggered Mechanism for the Continuous-Time Nonlinear Systems.

In this article, the neural network (NN)-based adaptive dynamic programming (ADP) event-triggered control method is presented to obtain the near-optimal control policy for the model-free finite-horizon optimal tracking control problem with constrained control input. First, using available input-output data, a data-driven model is established by a recurrent NN (RNN) to reconstruct the unknown system. Then, an augmented system with event-triggered mechanism is obtained by a tracking error system and a command generator. We present a novel event-triggering condition without Zeno behavior. On this basis, the relationship between event-triggered Hamilton-Jacobi-Isaacs (HJI) equation and time-triggered HJI equation is given in Theorem 3. Since the solution of the HJI equation is time-dependent for the augmented system, the time-dependent activation functions of NNs are considered. Moreover, an extra error is incorporated to satisfy the terminal constraints of cost function. This adaptive control pattern finds, in real time, approximations of the optimal value while also ensuring the uniform ultimate boundedness of the closed-loop system. Finally, the effectiveness of the proposed near-optimal control pattern is verified by two simulation examples.

Three-Dimensional Guidance Approach with Obstacle Avoidance against Maneuvering Targets

A guidance approach with obstacle avoidance is proposed for maneuvering target interception. Firstly, by decomposing the constrained optimal guidance problem into two associated subproblems, the overall scheme of guidance is presented. Secondly, to improve the guidance accuracy against the maneuvering target, an MPC controller with disturbance estimation is designed, which transforms the global optimization problem into a finite-horizon optimal control problem. An auxiliary controller is also developed to ensure the system’s stability. Then, to achieve obstacle avoidance without affecting the guidance accuracy, a unified function is built to assess the threats from obstacles of different shapes, based on which the optimal guidance-obstacle avoidance model is established. Finally, to solve the above optimization problems, a hybrid solver is developed based on the adaptive moment estimation (Adam) algorithm, whose output will serve as the real-time guidance command. The numerical simulation results show that the guidance approach presented in this paper can achieve precise guidance and obstacle avoidance in various scenarios, and the Monte Carlo experiment further demonstrates the robustness and computation efficiency of the approach.

Distributed Model Predictive Control with Particle Swarm Optimizer for Collision-Free Trajectory Tracking of MWMR Formation

The distributed model predictive control (DMPC) strategy with particle swarm optimization (PSO) is applied to solve the collision-free trajectory tracking problem for the mecanum-wheeled mobile robot (MWMR) formation. Under the leader–follower framework, the predictive model is established considering the kinematics and dynamics of the MWMR with the uncertainties and external disturbances. Based on the information from itself and its neighbors, each MWMR is assigned its own finite-horizon optimal control problem, of which the objective/cost function consists of formation maintenance, trajectory tracking, and collision avoidance terms, and the control inputs of each MWMR are computed synchronously in a distributed manner. PSO serves as the fast and effective optimizer to find feasible solutions to these finite-horizon optimal control problems. Further, the feedback emendation is implemented using a double closed-loop compensator to efficiently inhibit the influence of unknown dynamics in real time. The stability of the proposed distributed formation control approach is strictly analyzed. Numerical simulations confirmed the robustness and effectiveness of the control approach in obstacle environments.

Personalized Constrained MPC for glucose regulation

The technological advances reached in the last years overcome the relevant hardware limitations of the past artificial pancreas releasing portable devices with high computational capabilities. In view of this, the use of a Constrained MPC (CMPC) that needs the solution of a finite horizon optimal control problem can be considered. The real advantage of the explicit inclusion of the state constraints in the optimization problem is mainly related to the quality of the model used for the predictions. In a recent work, a CMPC based on an average model has been tested on the 100 in silico patients of the new UVA/Padova simulator, showing satisfactory results in terms of average glucose, time spent in tight range and time above 180 mg/dl. A percentage of time below 70 mg/dl grater than 3% is present for more than 25% of the patients. In this work, a new personalized CMPC is proposed to mitigate the hypoglycemia phenomena for these patients, using personalized impulse-response models for glucose prediction.

A direct integral pseudospectral method for solving a class of infinite-horizon optimal control problems using Gegenbauer polynomials and certain parametric maps

<abstract><p>We present a novel direct integral pseudospectral (PS) method (a direct IPS method) for solving a class of continuous-time infinite-horizon optimal control problems (IHOCs). The method transforms the IHOCs into finite-horizon optimal control problems in their integral forms by means of certain parametric mappings, which are then approximated by finite-dimensional nonlinear programming problems (NLPs) through rational collocations based on Gegenbauer polynomials and Gegenbauer-Gauss-Radau (GGR) points. The paper also analyzes the interplay between the parametric maps, barycentric rational collocations based on Gegenbauer polynomials and GGR points and the convergence properties of the collocated solutions for IHOCs. Some novel formulas for the construction of the rational interpolation weights and the GGR-based integration and differentiation matrices in barycentric-trigonometric forms are derived. A rigorous study on the error and convergence of the proposed method is presented. A stability analysis based on the Lebesgue constant for GGR-based rational interpolation is investigated. Two easy-to-implement pseudocodes of computational algorithms for computing the barycentric-trigonometric rational weights are described. Three illustrative test examples are presented to support the theoretical results. We show that the proposed collocation method leveraged with a fast and accurate NLP solver converges exponentially to near-optimal approximations for a coarse collocation mesh grid size. The paper also shows that typical direct spectral/PS and IPS methods based on classical Jacobi polynomials and certain parametric maps usually diverge as the number of collocation points grow large if the computations are carried out using floating-point arithmetic and the discretizations use a single mesh grid, regardless of whether they are of Gauss/Gauss-Radau type or equally spaced.</p></abstract>

Experimental Validation on Aerial Vehicles of Real-Time Motion Planning with Continuous-Time Q-Learning

In this paper, we propose an algorithm and implementation for real-time optimized kinodynamic motion planning for aerial vehicles with unknown dynamics in crowded environments. A random-sampling space-filling tree is used for both planning and rapidly replanning a path through the environment. Then, continuous-time Q-learning is used to approximately solve the resulting finite-horizon optimal control problem online to optimally track the planned path. To facilitate the Q-learning, we propose an actor-critic structure with integral reinforcement learning to approximate the Hamilton-Jacobi-Bellman equation. The critic approximates the Q-function while the actor approximates the control policy. We demonstrate our approach on custom drone hardware in which all planning, learning, and control computations are conducted onboard in real-time.

A dynamic programming approach for controlled fractional SIS models

We investigate a susceptible-infected-susceptible (SIS) epidemic model based on the Caputo–Fabrizio operator. After performing an asymptotic analysis of the system, we study a related finite horizon optimal control problem with state constraints. We prove that the corresponding value function satisfies in the viscosity sense a dynamic programming equation. We then turn to the asymptotic behavior of the value function, proving its convergence to the solution of a stationary problem, as the planning horizon tends to infinity. Finally, we present some numerical simulations providing a qualitative description of the optimal dynamics and the value functions involved.

An efficient NMPC-based multi-task control toolkit for remote handling applications

This paper presents an efficient NMPC-based multi-task control toolkit for complex robotic systems used in remote handling applications. Instead of resolving multi-task problem by solving instantaneous optimization problems like some existing controllers, in our toolkit a NMPC based control method is proposed which computes the control commands by solving a finite time horizon optimal control problem. The predictive feature of the controller enhances the intelligence and safety of robotic systems. However, it causes the computation more time-consuming. To improve the computational efficiency, we develop a two-step primal–dual interior-point solver ensuring the NMPC problem to be solved in real-time by exploiting the special structure of KKT matrix. Furthermore, our toolkit supports symbolic computation and C-code generation, which greatly reduces the effort in programming and deploying the controller on different computing platforms. The effectiveness and efficiency of our control method are demonstrated by numerical and dynamic simulations on different types of robotic systems conducting diverse RH applications.

Inverse linear-quadratic discrete-time finite-horizon optimal control for indistinguishable homogeneous agents: A convex optimization approach

The inverse linear-quadratic optimal control problem is a system identification problem whose aim is to recover the quadratic cost function and hence the closed-loop system matrices based on observations of optimal trajectories. In this paper, the discrete-time, finite-horizon case is considered, where the agents are also assumed to be homogeneous and indistinguishable. The latter means that the agents all have the same dynamics and objective functions and the observations are in terms of “snap shots” of all agents at different time instants, but what is not known is “which agent moved where” for consecutive observations. This absence of linked optimal trajectories makes the problem challenging. We first show that this problem is globally identifiable. Then, for the case of noiseless observations, we show that the true cost matrix, and hence the closed-loop system matrices, can be recovered as the unique global optimal solution to a convex optimization problem. Next, for the case of noisy observations, we formulate an estimator as the unique global optimal solution to a modified convex optimization problem. Moreover, the statistical consistency of this estimator is shown. Finally, the performance of the proposed method is demonstrated by a number of numerical examples.