Finite-horizon Optimal Control Problem Research Articles

Application of Hamilton–Jacobi–Bellman Equation/Pontryagin’s Principle for Constrained Optimal Control

This article applies novel results for infinite- and finite-horizon optimal control problems with nonlinear dynamics and constraints. We use the Valentine transformation to convert a constrained optimal control problem into an unconstrained one and show uniqueness of the value function to the corresponding Hamilton–Jacobi–Bellman (HJB) equation. From there, we show how to approximate the solution of the initial (in)finite-horizon problem with a family of solutions that is Γ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varGamma $$\\end{document}-convergent. Optimal solutions are efficiently obtained via a solver based on Pontryagin’s Principle (PP). The proposed methodology is demonstrated on the path planning problem using the full nonlinear dynamics of an unmanned aerial vehicle (UAV) and autonomous underwater vehicle (AUV) involving state constraints in 3D environments with obstacles.

Dynamic Event-Driven Finite-Horizon Optimal Consensus Control for Constrained Multiagent Systems.

This article investigates the event-driven finite-horizon optimal consensus control problem for multiagent systems with symmetric or asymmetric input constraints. Initially, in order to overcome the difficulty that the Hamilton-Jacobi-Bellman equation is time-varying in finite-horizon optimal control, a single critic neural network (NN) with time-varying activation function is applied to obtain the approximate optimal control. Meanwhile, for minimizing the terminal error to satisfy the terminal constraint of the value function, an augmented error vector containing the Bellman residual and the terminal error is constructed to update the weight of the NN. Furthermore, an improved learning law is proposed, which relaxes the tricky persistence excitation condition and eliminates the requirement of initial stability control. Moreover, a specific algorithm is designed to update the historical dataset, which can effectively accelerate the convergence rate of network weight. In addition, to improve the utilization rate of the communication resource, an effective dynamic event-triggering mechanism (DETM) composed of dynamic threshold parameters (DTPs) and auxiliary dynamic variables (ADVs) is designed, which is more flexible compared with the ADV-based DETM or DTP-based DETM. Finally, to support the effectiveness of the proposed method and the superiority of the designed DETM, a simulation example is provided.

Value functional and optimal feedback control in linear-quadratic optimal control problem for fractional-order system

In this paper, a finite-horizon optimal control problem involving a dynamical system described by a linear Caputo fractional differential equation and a quadratic cost functional is considered. An explicit formula for the value functional is given, which includes a solution of a certain Fredholm integral equation. A step-by-step feedback control procedure for constructing $ \varepsilon $-optimal controls with any accuracy $ \varepsilon > 0 $ is proposed. The basis for obtaining these results is the study of a solution of the associated Hamilton–Jacobi–Bellman equation with so-called fractional coinvariant derivatives.

Multitrajectory Model Predictive Control for Safe UAV Navigation in an Unknown Environment

The problem of navigating an unmanned aerial vehicle (UAV) in an unknown environment is addressed with a novel model predictive control (MPC) formulation, named multitrajectory MPC (mt-MPC). The objective is to safely drive the vehicle to the desired target location by relying only on the partial description of the surroundings provided by an exteroceptive sensor. This information results in time-varying constraints during the navigation among obstacles. The proposed mt-MPC generates a sequence of position set points that are fed to control loops at lower hierarchical levels. To do so, the mt-MPC predicts two different state trajectories, a safe one and an exploiting one, in the same finite horizon optimal control problem (FHOCP). This formulation, particularly suitable for problems with uncertain time-varying constraints, allows one to partially decouple constraint satisfaction (safety) from cost function minimization (exploitation). Uncertainty due to modeling errors and sensors noise is taken into account as well, in a set membership (SM) framework. Theoretical guarantees of persistent obstacle avoidance are derived under suitable assumptions, and the approach is demonstrated experimentally out-of-the-laboratory on a prototype built with off-the-shelf components.

Policy-Iteration-Based Finite-Horizon Approximate Dynamic Programming for Continuous-Time Nonlinear Optimal Control.

The Hamilton-Jacobi-Bellman (HJB) equation serves as the necessary and sufficient condition for the optimal solution to the continuous-time (CT) optimal control problem (OCP). Compared with the infinite-horizon HJB equation, the solving of the finite-horizon (FH) HJB equation has been a long-standing challenge, because the partial time derivative of the value function is involved as an additional unknown term. To address this problem, this study first-time bridges the link between the partial time derivative and the terminal-time utility function, and thus it facilitates the use of the policy iteration (PI) technique to solve the CT FH OCPs. Based on this key finding, the FH approximate dynamic programming (ADP) algorithm is proposed leveraging an actor-critic framework. It is shown that the algorithm exhibits important properties in terms of convergence and optimality. Rather importantly, with the use of multilayer neural networks (NNs) in the actor-critic architecture, the algorithm is suitable for CT FH OCPs toward more general nonlinear and complex systems. Finally, the effectiveness of the proposed algorithm is demonstrated by conducting a series of simulations on both a linear quadratic regulator (LQR) problem and a nonlinear vehicle tracking problem.

Data-Driven Finite-Horizon Tracking Control With Event-Triggered Mechanism for the Continuous-Time Nonlinear Systems.

In this article, the neural network (NN)-based adaptive dynamic programming (ADP) event-triggered control method is presented to obtain the near-optimal control policy for the model-free finite-horizon optimal tracking control problem with constrained control input. First, using available input-output data, a data-driven model is established by a recurrent NN (RNN) to reconstruct the unknown system. Then, an augmented system with event-triggered mechanism is obtained by a tracking error system and a command generator. We present a novel event-triggering condition without Zeno behavior. On this basis, the relationship between event-triggered Hamilton-Jacobi-Isaacs (HJI) equation and time-triggered HJI equation is given in Theorem 3. Since the solution of the HJI equation is time-dependent for the augmented system, the time-dependent activation functions of NNs are considered. Moreover, an extra error is incorporated to satisfy the terminal constraints of cost function. This adaptive control pattern finds, in real time, approximations of the optimal value while also ensuring the uniform ultimate boundedness of the closed-loop system. Finally, the effectiveness of the proposed near-optimal control pattern is verified by two simulation examples.

Three-Dimensional Guidance Approach with Obstacle Avoidance against Maneuvering Targets

A guidance approach with obstacle avoidance is proposed for maneuvering target interception. Firstly, by decomposing the constrained optimal guidance problem into two associated subproblems, the overall scheme of guidance is presented. Secondly, to improve the guidance accuracy against the maneuvering target, an MPC controller with disturbance estimation is designed, which transforms the global optimization problem into a finite-horizon optimal control problem. An auxiliary controller is also developed to ensure the system’s stability. Then, to achieve obstacle avoidance without affecting the guidance accuracy, a unified function is built to assess the threats from obstacles of different shapes, based on which the optimal guidance-obstacle avoidance model is established. Finally, to solve the above optimization problems, a hybrid solver is developed based on the adaptive moment estimation (Adam) algorithm, whose output will serve as the real-time guidance command. The numerical simulation results show that the guidance approach presented in this paper can achieve precise guidance and obstacle avoidance in various scenarios, and the Monte Carlo experiment further demonstrates the robustness and computation efficiency of the approach.

Distributed Model Predictive Control with Particle Swarm Optimizer for Collision-Free Trajectory Tracking of MWMR Formation

The distributed model predictive control (DMPC) strategy with particle swarm optimization (PSO) is applied to solve the collision-free trajectory tracking problem for the mecanum-wheeled mobile robot (MWMR) formation. Under the leader–follower framework, the predictive model is established considering the kinematics and dynamics of the MWMR with the uncertainties and external disturbances. Based on the information from itself and its neighbors, each MWMR is assigned its own finite-horizon optimal control problem, of which the objective/cost function consists of formation maintenance, trajectory tracking, and collision avoidance terms, and the control inputs of each MWMR are computed synchronously in a distributed manner. PSO serves as the fast and effective optimizer to find feasible solutions to these finite-horizon optimal control problems. Further, the feedback emendation is implemented using a double closed-loop compensator to efficiently inhibit the influence of unknown dynamics in real time. The stability of the proposed distributed formation control approach is strictly analyzed. Numerical simulations confirmed the robustness and effectiveness of the control approach in obstacle environments.

Personalized Constrained MPC for glucose regulation

The technological advances reached in the last years overcome the relevant hardware limitations of the past artificial pancreas releasing portable devices with high computational capabilities. In view of this, the use of a Constrained MPC (CMPC) that needs the solution of a finite horizon optimal control problem can be considered. The real advantage of the explicit inclusion of the state constraints in the optimization problem is mainly related to the quality of the model used for the predictions. In a recent work, a CMPC based on an average model has been tested on the 100 in silico patients of the new UVA/Padova simulator, showing satisfactory results in terms of average glucose, time spent in tight range and time above 180 mg/dl. A percentage of time below 70 mg/dl grater than 3% is present for more than 25% of the patients. In this work, a new personalized CMPC is proposed to mitigate the hypoglycemia phenomena for these patients, using personalized impulse-response models for glucose prediction.

A direct integral pseudospectral method for solving a class of infinite-horizon optimal control problems using Gegenbauer polynomials and certain parametric maps

<abstract><p>We present a novel direct integral pseudospectral (PS) method (a direct IPS method) for solving a class of continuous-time infinite-horizon optimal control problems (IHOCs). The method transforms the IHOCs into finite-horizon optimal control problems in their integral forms by means of certain parametric mappings, which are then approximated by finite-dimensional nonlinear programming problems (NLPs) through rational collocations based on Gegenbauer polynomials and Gegenbauer-Gauss-Radau (GGR) points. The paper also analyzes the interplay between the parametric maps, barycentric rational collocations based on Gegenbauer polynomials and GGR points and the convergence properties of the collocated solutions for IHOCs. Some novel formulas for the construction of the rational interpolation weights and the GGR-based integration and differentiation matrices in barycentric-trigonometric forms are derived. A rigorous study on the error and convergence of the proposed method is presented. A stability analysis based on the Lebesgue constant for GGR-based rational interpolation is investigated. Two easy-to-implement pseudocodes of computational algorithms for computing the barycentric-trigonometric rational weights are described. Three illustrative test examples are presented to support the theoretical results. We show that the proposed collocation method leveraged with a fast and accurate NLP solver converges exponentially to near-optimal approximations for a coarse collocation mesh grid size. The paper also shows that typical direct spectral/PS and IPS methods based on classical Jacobi polynomials and certain parametric maps usually diverge as the number of collocation points grow large if the computations are carried out using floating-point arithmetic and the discretizations use a single mesh grid, regardless of whether they are of Gauss/Gauss-Radau type or equally spaced.</p></abstract>

Experimental Validation on Aerial Vehicles of Real-Time Motion Planning with Continuous-Time Q-Learning

In this paper, we propose an algorithm and implementation for real-time optimized kinodynamic motion planning for aerial vehicles with unknown dynamics in crowded environments. A random-sampling space-filling tree is used for both planning and rapidly replanning a path through the environment. Then, continuous-time Q-learning is used to approximately solve the resulting finite-horizon optimal control problem online to optimally track the planned path. To facilitate the Q-learning, we propose an actor-critic structure with integral reinforcement learning to approximate the Hamilton-Jacobi-Bellman equation. The critic approximates the Q-function while the actor approximates the control policy. We demonstrate our approach on custom drone hardware in which all planning, learning, and control computations are conducted onboard in real-time.

A dynamic programming approach for controlled fractional SIS models

We investigate a susceptible-infected-susceptible (SIS) epidemic model based on the Caputo–Fabrizio operator. After performing an asymptotic analysis of the system, we study a related finite horizon optimal control problem with state constraints. We prove that the corresponding value function satisfies in the viscosity sense a dynamic programming equation. We then turn to the asymptotic behavior of the value function, proving its convergence to the solution of a stationary problem, as the planning horizon tends to infinity. Finally, we present some numerical simulations providing a qualitative description of the optimal dynamics and the value functions involved.

An efficient NMPC-based multi-task control toolkit for remote handling applications

This paper presents an efficient NMPC-based multi-task control toolkit for complex robotic systems used in remote handling applications. Instead of resolving multi-task problem by solving instantaneous optimization problems like some existing controllers, in our toolkit a NMPC based control method is proposed which computes the control commands by solving a finite time horizon optimal control problem. The predictive feature of the controller enhances the intelligence and safety of robotic systems. However, it causes the computation more time-consuming. To improve the computational efficiency, we develop a two-step primal–dual interior-point solver ensuring the NMPC problem to be solved in real-time by exploiting the special structure of KKT matrix. Furthermore, our toolkit supports symbolic computation and C-code generation, which greatly reduces the effort in programming and deploying the controller on different computing platforms. The effectiveness and efficiency of our control method are demonstrated by numerical and dynamic simulations on different types of robotic systems conducting diverse RH applications.

Guidance and Control of Spin-Stabilized Projectiles Based on Super Twisting Algorithm

In this paper, a hybrid integrated guidance and control design exploiting the time-scale separation principle for a canard-guided dual-spin projectile (DSP) is proposed. The DSP consists of two parts: i) the aft part, which has a high spin rate for providing gyroscopic and aerodynamic stability, and ii) the forward part, which contains a course correction fuze that houses two pairs of movable canards to ensure precise interception. The guidance and control algorithms operate in a two-loop structure where the outer guidance loop generates required pitch and yaw rate commands, whereas the inner loop tracks these commands by generating canard deflections. The rolling motion of the forward part is controlled using a coaxial motor. The multibody dynamic model of DSP has seven degrees of freedom and is highly coupled and nonlinear. For the guidance, a finite horizon optimal control problem is formulated as an output regulation problem with output vector taken as pitch and yaw components of zero effort miss vector. A coupled pitch–yaw autopilot for pitch and yaw rate command tracking and a roll autopilot for roll stabilization of the forward part are designed based on super twisting algorithm. The efficacy of the control design is shown through numerical simulations.

When does stabilizability imply the existence of infinite horizon optimal control in nonlinear systems?

The paper addresses an existence problem for infinite horizon optimal control when the system under control is exponentially stabilizable or stable. Classes of nonlinear control systems for which infinite horizon optimal controls exist are identified in terms of stability, stabilizability, detectability and growth conditions. The result then applies to estimate the existence region of stable manifolds in the associated Hamiltonian systems. Applications of the results also include the analysis for turnpike property in nonlinear finite horizon optimal control problems by a geometric approach.

Recurrent Model Predictive Control: Learning an Explicit Recurrent Controller for Nonlinear Systems

This paper proposes an offline control algorithm, called Recurrent Model Predictive Control (RMPC), to solve large-scale nonlinear finite-horizon optimal control problems. It can be regarded as an explicit solver of traditional Model Predictive Control (MPC) algorithms, which can adaptively select appropriate model prediction horizon according to current computing resources, so as to improve the policy performance. Our algorithm employs a recurrent function to approximate the optimal policy, which maps the system states and reference values directly to the control inputs. The output of the learned policy network after N recurrent cycles corresponds to the nearly optimal solution of N-step MPC. A policy optimization objective is designed by decomposing the MPC cost function according to the Bellman's principle of optimality. The optimal recurrent policy can be obtained by directly minimizing the designed objective function, which is applicable for general nonlinear and non input-affine systems. Both simulation-based and real-robot path-tracking tasks are utilized to demonstrate the effectiveness of the proposed method.

On representation formulas for optimal control: A Lagrangian perspective

In this paper, we study representation formulas for finite-horizon optimal control problems with or without state constraints, unifying two different viewpoints: the Lagrangian and dynamic programming (DP) frameworks. In a recent work [1], the generalized Lax formula is obtained via DP for optimal control problems with state constraints and nonlinear systems. We revisit the formula from the Lagrangian perspective to provide a unified framework for understanding and implementing the nontrivial representation of the value function. Our simple derivation makes direct use of the Lagrangian formula from the theory of Hamilton-Jacobi (HJ) equations. We also discuss a rigorous way to construct an optimal control using a $\delta$-net, as well as a numerical scheme for controller synthesis via convex optimization.

Robust Walking Based on MPC With Viability Guarantees

Model predictive control (MPC) has shown great success for controlling complex systems, such as legged robots. However, when closing the loop, the performance and feasibility of the finite horizon optimal control problem (OCP) solved at each control cycle are not guaranteed anymore. This is due to model discrepancies, the effect of low-level controllers, uncertainties, and sensor noise. To address these issues, we propose a modified version of a standard MPC approach used in legged locomotion with viability (weak forward invariance) guarantees. In this approach, instead of adding a (conservative) terminal constraint to the problem, we propose to use the measured state projected to the viability kernel in the OCP solved at each control cycle. Moreover, we use past experimental data to find the best cost weights, which measure a combination of performance, constraint satisfaction robustness, or stability (invariance). These interpretable costs measure the tradeoff between robustness and performance. For this purpose, we use Bayesian optimization to systematically design experiments that help efficiently collect data to learn a cost function leading to robust performance. Our simulation results with different realistic disturbances (i.e., external pushes, unmodeled actuator dynamics, and computational delay) show the effectiveness of our approach to create robust controllers for humanoid robots.

Self‐triggered predictive control of nonlinear systems using approximation model

AbstractThis article studies discrete‐time model predictive control of continuous‐time nonlinear systems with measurement noises and exogenous disturbances. We consider the co‐design of discrete‐time finite horizon optimal control problem (FHOCP) and the associated self‐triggering schemes that schedule the time instants for sampling the state and computing the FHOCP. The state‐dependent nature of the self‐triggered scheduling can not only dynamically adjust the inter‐sampling period according to the system status, but dynamically discretize the model used in the FHOCP as well, in order to reduce the complexity of the FHOCP if designed appropriately. It is shown that the system can be stabilized with uniform ultimate boundedness, as long as the scheduling scheme matches the approximation model such that the one‐step approximation error between the predicted state and the actual state is below a threshold related to the cost function. These results can be applied to most existing model approximation methods with either fixed or time‐varying sampling rates.

A Gradient Flow Equation for Optimal Control Problems With End-point Cost

In this paper, we consider a control system of the form dot x = F(x)u, linear in the control variable u. Given a fixed starting point, we study a finite-horizon optimal control problem, where we want to minimize a weighted sum of an end-point cost and the squared 2-norm of the control. This functional induces a gradient flow on the Hilbert space of admissible controls, and we prove a convergence result by means of the Lojasiewicz-Simon inequality. Finally, we show that, if we let the weight of the end-point cost tend to infinity, the resulting family of functionals is Γ-convergent, and it turns out that the limiting problem consists in joining the starting point and a minimizer of the end-point cost with a horizontal length-minimizer path.