Approximate Optimal Control Research Articles

Approximate Optimal Controller for Weakly Nonlinear Evolutionary Equation of Parabolic Type

Approximate Optimal Controller for Weakly Nonlinear Evolutionary Equation of Parabolic Type

Learning‐basedT‐sHDP() for optimal control of a class of nonlinear discrete‐time systems

AbstractThis article investigates the optimal control problem via reinforcement learning for a class of nonlinear discrete‐time systems. The nonlinear system under consideration is assumed to be partially unknown. A new learning‐based algorithm,T‐step heuristic dynamic programming with eligibility traces (T‐sHDP()), is proposed to tackle the optimal control problem for such partially unknown system. First, the concerned optimal control problem is turned into its equivalence problem, that is, solving a Bellman equation. Then, theT‐sHDP() is utilized to get an approximate solution of Bellman equation, and a rigorous convergence analysis is also conducted as well. Instead of the commonly used single step update approach, theT‐sHDP() stores finite step past returns by introducing a parameter, and then utilizes these knowledge to update the value function (VF) of multiple moments synchronously, so as to achieve higher convergence speed. For implementation ofT‐sHDP(), a neural network‐based actor‐critic architecture is applied to approximate VF and optimal control scheme. Finally, the feasibility of the algorithm is demonstrated by two illustrative simulation examples.

Applications of Approximate Optimal Control to Nonlinear Systems of Tracked Vehicle Suspensions

Technique of approximate optimal vibration control and simulation for vehicle active suspension systems are developed. Considered the nonlinear damping of springs, mechanical model and a nonlinear dynamic system for a class of tracked vehicle suspension vibration control are established and the corresponding system of state space form is described. To prolong the working life of suspension system and improve ride comfort, based on the active suspension vibration control devices and using optimal control approach, an approximate optimal vibration controller is designed, and an algorithm is presented for the vibration controller. Numerical simulation results illustrate the effectiveness of the proposed technique.

Critic-observer-based decentralized force/position approximate optimal control for modular and reconfigurable manipulators with uncertain environmental constraints

A critic-observer decentralized force/position approximate optimal control method is presented to address the joint trajectory and contacted force tracking problem of modular and reconfigurable manipulators (MRMs) with uncertain environmental constraints. The dynamic model of the MRM systems is formulated as an integration of joint subsystems via extensive state observer (ESO) associated with the effect of interconnected dynamic coupling (IDC). A radial basis function neural network (RBF-NN) is developed to deal with the IDC effects among the independent joint subsystems. Based on adaptive dynamic programming (ADP) approach and policy iteration (PI) algorithm, the Hamilton–Jacobi–Bellman (HJB) equation is approximately solved by establishing critic NN structure and then the approximated optimal control policy can be derived. The closed-loop manipulator system is proved to be asymptotic stable by using the Lyapunov theory. Finally, simulation results are provided to demonstrate the effectiveness and advantages of the proposed control method.

Adaptive dynamic programing design for the neural control of hypersonic flight vehicles

An adaptive dynamic programming controller based on backstepping method is designed for the optimal tracking control of hypersonic flight vehicles. The control input is divided into two parts namely stable control and optimal control. First, the back-stepping method is exploited via neural networks (NNs) to estimate unknown functions. Then, the computational load is reduced by the minimal-learning-parameter (MLP) scheme. To avoid the problem of “explosion of terms”, a first-order filter is adopted. Next, the optimal controller is designed based on the adaptive dynamic programming. In order to solve the Hamiltonian equation, NNs estimators are introduced to approximate performance indicators, achieving the approximate optimal control of hypersonic flight vehicles. Finally, the effectiveness and advantages of the control method are verified by simulation results.

The Adaptive Dynamic Programming Toolbox.

The paper develops the adaptive dynamic programming toolbox (ADPT), which is a MATLAB-based software package and computationally solves optimal control problems for continuous-time control-affine systems. The ADPT produces approximate optimal feedback controls by employing the adaptive dynamic programming technique and solving the Hamilton–Jacobi–Bellman equation approximately. A novel implementation method is derived to optimize the memory consumption by the ADPT throughout its execution. The ADPT supports two working modes: model-based mode and model-free mode. In the former mode, the ADPT computes optimal feedback controls provided the system dynamics. In the latter mode, optimal feedback controls are generated from the measurements of system trajectories, without the requirement of knowledge of the system model. Multiple setting options are provided in the ADPT, such that various customized circumstances can be accommodated. Compared to other popular software toolboxes for optimal control, the ADPT features computational precision and time efficiency, which is illustrated with its applications to a highly non-linear satellite attitude control problem.

Reinforcement Learning-Based Approximate Optimal Control for Attitude Reorientation Under State Constraints

This article addresses the attitude reorientation problems of rigid bodies under multiple state constraints. A novel reinforcement learning (RL)-based approximate optimal control method is proposed to make the tradeoff between control cost and performance. The novelty lies in that it guarantees constraint handling abilities on attitude forbidden zones and angular velocity limits. To achieve this, barrier functions are employed to encode the constraint information into the cost function. Then, an RL-based learning strategy is developed to approximate the optimal cost function and control policy. A simplified critic-only neural network (NN) is employed to replace the conventional actor–critic structure once adequate data are collected online. This design guarantees the uniform boundedness of reorientation errors and NN weight estimation errors subject to the satisfaction of a finite excitation condition, which is a relaxation compared with the persistent excitation condition that is typically required for this class of problems. More importantly, all underlying state constraints are strictly obeyed during the online learning process. The effectiveness and advantages of the proposed controller are verified by both numerical simulations and experimental tests based on a comprehensive hardware-in-loop testbed.

Complexity Analysis of stochastic gradient methods for PDE-constrained optimal Control Problems with uncertain parameters

We consider the numerical approximation of an optimal control problem for an elliptic Partial Differential Equation (PDE) with random coefficients. Specifically, the control function is a deterministic, distributed forcing term that minimizes the expected squaredL2misfit between the state (i.e.solution to the PDE) and a target function, subject to a regularization for well posedness. For the numerical treatment of this risk-averse Optimal Control Problem (OCP) we consider a Finite Element discretization of the underlying PDE, a Monte Carlo sampling method, and gradient-type iterations to obtain the approximate optimal control. We provide full error and complexity analyses of the proposed numerical schemes. In particular we investigate the complexity of a conjugate gradient method applied to the fully discretized OCP (so called Sample Average Approximation), in which the Finite Element discretization and Monte Carlo sample are chosen in advance and kept fixed over the iterations. This is compared with aStochastic Gradientmethod on a fixed or varying Finite Element discretization, in which the expectation in the computation of the steepest descent direction is approximated by Monte Carlo estimators, independent across iterations, with small sample sizes. We show in particular that the second strategy results in an improved computational complexity. The theoretical error estimates and complexity results are confirmed by numerical experiments.

Integrated Capacity Planning and Production Control of an Assembly Manufacturing System

For an assembly manufacturing system, its capacity planning problem and production control problem are usually considered separately. However, production capacity cannot exert its maximum productivity without employing optimal production control; and optimal production control may not satisfy the demand without sufficient production capacity. Therefore, it is important to jointly optimize capacity planning and production control. First, the mathematical formulation of integrated production control and capacity planning problem for an assembly manufacturing system with multiple upstream production lines and one downstream assembly line is proposed in this paper. Second, we develop a method to obtain the marginal probability distribution of the inventory level in the assembly system, which can be employed to calculate the average production cost under an approximate optimal production control policy. Since optimal control policy is subject to production capacity (i.e., machine number), we employ the integer programming method to optimize the machine number (i.e., production capacity), which consequently influences the control policy. Numerical experiments are conducted to verify the correctness of the method of calculating steady-state probability distribution, and the effectiveness of the algorithm of integrated optimization of production control and production capacity.

Robust Neurooptimal Control for a Robot via Adaptive Dynamic Programming.

We aim at the optimization of the tracking control of a robot to improve the robustness, under the effect of unknown nonlinear perturbations. First, an auxiliary system is introduced, and optimal control of the auxiliary system can be seen as an approximate optimal control of the robot. Then, neural networks (NNs) are employed to approximate the solution of the Hamilton-Jacobi-Isaacs equation under the frame of adaptive dynamic programming. Next, based on the standard gradient attenuation algorithm and adaptive critic design, NNs are trained depending on the designed updating law with relaxing the requirement of initial stabilizing control. In light of the Lyapunov stability theory, all the error signals can be proved to be uniformly ultimately bounded. A series of simulation studies are carried out to show the effectiveness of the proposed control.

A subdivision strategy for adjoining cell mapping on the global optimal control in multi‐input‐multi‐output systems

AbstractIn this paper, a subdivision strategy for the adjoining cell mapping method to solve the global optimal control problems for multi‐input‐multi‐output systems is proposed. The subdivision strategy constructs the discrete optimal control table first by a rough discretization of the state space, which can identify uncontrollable regions. Then the optimal feedback control from the selected initial state is instituted. The rough cells where the feedback control trajectory crosses are subdivided into smaller cells and the finer discrete optimal control table is obtained after the second search. The further discretization of the target cell and control inputs can further enhance the terminate control precision. The calculation can also take the advantage of GPU parallel technology that will greatly enhance the computation efficiency. Two examples illustrate the feasibility and validity of the subdivision strategy to solve optimal control problems for multi‐input‐multi‐output systems. The proposed method offers an efficient alternative numerical way to obtain the approximate optimal control solution with global optimum for nonlinear multi‐input‐multi‐output systems.

Sliding-Mode Surface-Based Approximate Optimal Control for Uncertain Nonlinear Systems With Asymptotically Stable Critic Structure.

This article develops a novel sliding-mode surface (SMS)-based approximate optimal control scheme for a large class of nonlinear systems affected by unknown mismatched perturbations. The observer-based perturbation estimation procedure is employed to establish the online updated value function. The solution to the Hamilton-Jacobi-Bellman equation is approximated by an SMS-based critic neural network whose weights error dynamics is designed to be asymptotically stable by nested update laws. The sliding-mode control strategy is combined with the approximate optimal control design procedure to obtain a faster control action. The stability is proved based on the Lyapunov's direct method. The simulation results show the effectiveness of the developed control scheme.

Fixed-time disturbance observer-based nearly optimal control for reusable launch vehicle with input constraints

In this paper, a fixed-time disturbance observer-based nearly optimal control (FTDO-NOC) scheme is proposed for reusable launch vehicle (RLV) subject to model uncertainties, input constraints, and unknown mismatched/matched disturbances. The dynamics of RLV attitude motion are divided into outer-loop subsystem and inner-loop subsystem. For the outer-loop subsystem, to address the problems of unknown mismatched disturbances and model uncertainties, a novel adaptive-gain multivariable generalized super-twisting (AMGST) controller is proposed. Two modified gain-adaptation laws are derived for tuning the control gains of AMGST controller, which attenuates chattering efficiently. For the inner-loop subsystem, considering the effect of unknown matched disturbances, a fixed-time disturbance observer (FTDO) is utilized to estimate the matched disturbances and the time derivative of virtual control input. Incorporated with the designed FTDO, a nearly optimal controller (NOC), which is based on the critic–actor neural networks (NNs), is utilized to generate the approximate optimal control moments satisfying the input constraints. The tracking errors of inner-loop subsystem and the weight estimation errors of the critic–actor NNs are proved to be uniformly ultimately bounded (UUB) via Lyapunov technique. Finally, we provide simulation results to validate the effectiveness and superiority of the proposed control scheme.

Adaptive dynamic programming-based feature tracking control of visual servoing manipulators with unknown dynamics

This paper investigates a feature tracking control method for visual servoing (VS) manipulators adaptive dynamic programming (ADP)-based the unknown dynamics. The major superiority of ADP-based optimal control lies in that the visual tracking problem is converted to the feature tracking error control with optimal cost function. Moreover, an adaptive neural network observer is developed to approximate the entire uncertainties, which are utilized to construct an improved cost function. By establishing a critic neural network, the Hamilton–Jacobi–Bellman (HJB) equation is solved, and the approximate optimal error control policy is derived. The closed-loop VS manipulator system is verified to be ultimately uniformly bounded with the developed ADP-based feature tracking control strategy according to the Lyapunov theory. Finally, simulation results under various situations demonstrate that the proposed method achieves higher tracking accuracy than other methods, as well as satisfies energy optimal requirements.

Parametric approximate optimal control of uncertain differential game with application to counter terror

Abstract The linear quadratic differential game plays an important role in many fields. It is well known that the saddle point of linear quadratic differential game is given in a feedback form with a solution of Riccati differential equation. However, the control-related Riccati differential equation cannot be solved analytically in many cases. Then the optimal controls may be difficult to be implemented in practice. In order to simplify the forms of optimal controls, in this paper, we investigate a parametric approximate optimal control problem of linear quadratic differential game under uncertain environment. First, we introduce an uncertain linear quadratic differential game model and its analytic optimal controls. Then, an uncertain linear quadratic differential game model with constrained parametric control domain is formulated. Moreover, a parametric approximate optimization method is presented for solving the optimal control parameters. Finally, a counter terror problem is analyzed to show the efficiency of our presented method.

Droop-Free Optimal Feedback Control of Distributed Generators in Islanded DC Microgrids

This article introduces a droop-free, approximate optimal feedback control strategy to optimally control distributed generators (DGs) in islanded dc microgrids (DCMGs). Each DG is modeled as a control affine dynamical system, and constrained input of each DG is designed to minimize the infinite horizon quadratic state cost and nonquadratic control effort. The approximate dynamic programming (ADP) method is employed to solve the infinite horizon optimal control problem by successive approximation of the value function via a linear in the parameter (LIP) neural network (NN). The NN weights are updated online by a reinforcement learning (RL)-based tuning algorithm, and the convergence of the unknown weights to a neighborhood of the optimal weights is guaranteed without the persistence of excitation (PE). Simulation and experimental results are presented to demonstrate the effectiveness and applicability of the proposed concept.

Fundamental solutions for semi-linear neutral retarded integro-differential systems and applications to control problems

In this work, we establish the theory of fundamental solution, including the existence and some essential properties, for neutral linear integro-differential equations with finite delay. In this way, we are able to represent the mild solutions of semi-linear neutral retarded integro-differential equations by fundamental solutions through the Laplace transformation method. Then, as applications, we investigate the problems of approximate controllability, optimal control and time optimal control for this kind of integro-differential evolution systems based on the theory of fundamental solution, resolvent operators and fractional powers, and sufficient conditions for these control problems are obtained, respectively. Finally, an example is presented to illustrate the achieved results.

Compactification in optimal control of McKean‐Vlasov stochastic differential equations

AbstractWe study existence and approximation of optimal controls for systems governed by McKean‐Vlasov stochastic differential equations. It is well known in simple examples that in the absence of convexity conditions, the strict control problem has no optimal solution. The compactification of the set of such strict admissible controls leads to measure valued controls called relaxed controls. The space of relaxed controls enjoys nice topological properties. We prove that under pathwise uniqueness of solutions of the state equation, the relaxed state process is continuous with respect to the control variable. This means that the relaxed and strict control problems have the same value function. Moreover, we show, under merely continuity of the coefficients, that an optimal control exists in the space of relaxed controls. Under additional convexity hypothesis, we show that the optimal relaxed control is a strict control. These two results extend known results to general nonlinear MVSDEs, under minimal assumptions on the coefficients.

Adaptive dynamic programming enhanced admittance control for robots with environment interaction and actuator saturation

This paper focuses on the optimal tracking control problem for robot systems with environment interaction and actuator saturation. A control scheme combined with admittance adaptation and adaptive dynamic programming (ADP) is developed. The unknown environment is modelled as a linear system and admittance controller is derived to achieve compliant behaviour of the robot. In the ADP framework, the cost function is defined with non-quadratic form and the critic network is designed with radial basis function neural network which introduces to obtain an approximate optimal control of the Hamilton–Jacobi–Bellman equation, which guarantees the optimal trajectory tracking. The system stability is analysed by Lyapunov theorem and simulations demonstrate the effectiveness of the proposed strategy.

Event-triggered design for discrete-time nonlinear systems with control constraints

In order to solve the constrained-input problem and reduce the computing resources, a novel event-triggered optimal control method is proposed for a class of discrete-time nonlinear systems. In the proposed method, the event-triggered control policy is applied to the globalized dual heuristic dynamic programming (GDHP) algorithm. Compared with the traditional adaptive dynamic programming (ADP) control, the event-triggered GDHP control can reduce the computation while ensuring the system performance. In this paper, a non-quadratic function is given to code the control constraints and the trigger condition with the stability analysis is provided. Neural networks (NN) are constructed in the GDHP structure, where the model network is designed to identify the unknown nonlinear system, the critic network is used to learn the cost function and its partial derivative, and the action network is designed to obtain the approximate optimal control law. Three simulation examples are presented to demonstrate the performance of the proposed event-triggered design for constrained discrete-time nonlinear systems.