Hamilton Jacobi Isaacs Equation Research Articles

Bounded and inverse optimal formation stabilization of second-order agents

This paper formulates and solves a new problem of both bounded and inverse optimal formation stabilization control for a group of second-order dynamic mobile agents with collision avoidance and limited sensing range. The control design is based on new Lyapunov functions, new non-zero convergence and dominating lemmas, new pairwise collision avoidance functions, and forwarding and inverse optimal control design methods. The proposed formation stabilization control design guarantees no collision between any agents, “almost global” asymptotic stability of desired equilibrium points and instability of undesired equilibrium points, an infinite gain margin, bounded controls by a pre-specified constant, and minimization of a cost function that penalizes both stabilization errors and the control inputs without having to solve a Hamilton–Jacobi–Bellman or Hamilton–Jacobi–Isaacs equation.

Value Iteration-Based H∞ Controller Design for Continuous-Time Nonlinear Systems Subject to Input Constraints

In this paper, a novel integral reinforcement learning method is proposed based on value iteration (VI) to design the $H_{\infty }$ controller for continuous-time nonlinear systems subject to input constraints. To confront the control constraints, a nonquadratic function is introduced to reconstruct the ${L_{2}}$ -gain condition for the $H_{\infty }$ control problem. Then, the VI method is proposed to solve the corresponding Hamilton–Jacobi–Isaacs equation initialized with an arbitrary positive semi-definite value function. Compared with most existing works developed based on policy iteration, the initial admissible control policy is no longer required which results in a more free initial condition. The iterative process of the proposed VI method is analyzed and the convergence to the saddle point solution is proved in a general way. For the implementation of the proposed method, only one neural network is introduced to approximate the iterative value function, which results in a simpler architecture with less computational load compared with utilizing three neural networks. To verify the effectiveness of the VI-based method, two nonlinear cases are presented, respectively.

Synchronous Fault-Tolerant Near-Optimal Control for Discrete-Time Nonlinear PE Game.

In this article, the synchronous fault-tolerant near-optimal control strategy design problem is studied for a class of discrete-time nonlinear pursuit-evasion (PE) games. In the studied PE game, the input saturation phenomenon and possible actuator fault are simultaneously taken into consideration. To accelerate the estimation speed, a novel nonlinear fault estimator is designed by introducing a nonlinear function. Then, for the purpose of obtaining the synchronous control strategy for the discrete-time PE games, an approximate Hamilton-Jacobi-Isaacs (HJI) equation is established, which is seldom utilized for the discrete-time approximate dynamic programming in most existing results. It should be noticed that the synchronous control strategy designed based on the approximate HJI equation can be convergent very fast because of its quasi-Newton's iteration form. Furthermore, the sufficient condition is established to guarantee that the studied system is uniformly ultimately bounded. Finally, a numerical simulation of the hypersonic vehicle system is carried out to validate the proposed methodology.

Event-driven [formula omitted] control with critic learning for nonlinear systems

In this paper, we study an event-driven H∞ control problem of continuous-time nonlinear systems. Initially, with the introduction of a discounted cost function, we convert the nonlinear H∞ control problem into an event-driven nonlinear two-player zero-sum game. Then, we develop an event-driven Hamilton–Jacobi–Isaacs equation (HJIE) related to the two-player zero-sum game. After that, we propose a novel event-triggering condition guaranteeing Zeno behavior not to happen. The triggering threshold in the newly proposed event-triggering condition can be kept positive without requiring to properly choose the prescribed level of disturbance attenuation. To solve the event-driven HJIE, we employ an adaptive critic architecture which contains a unique critic neural network (NN). The weight parameters used in the critic NN are tuned via the gradient descent method. After that, we carry out stability analysis of the hybrid closed-loop system based on Lyapunov’s direct approach. Finally, we provide two nonlinear plants, including the pendulum system, to validate the proposed event-driven H∞ control scheme.

Event-Driven H∞-Constrained Control Using Adaptive Critic Learning.

This article considers an event-driven H∞ control problem of continuous-time nonlinear systems with asymmetric input constraints. Initially, the H∞ -constrained control problem is converted into a two-person zero-sum game with the discounted nonquadratic cost function. Then, we present the event-driven Hamilton-Jacobi-Isaacs equation (HJIE) associated with the two-person zero-sum game. Meanwhile, we develop a novel event-triggering condition making Zeno behavior excluded. The present event-triggering condition differs from the existing literature in that it can make the triggering threshold non-negative without the requirement of properly selecting the prescribed level of disturbance attenuation. After that, under the framework of adaptive critic learning, we use a single critic network to solve the event-driven HJIE and tune its weight parameters by using historical and instantaneous state data simultaneously. Based on the Lyapunov approach, we demonstrate that the uniform ultimate boundedness of all the signals in the closed-loop system is guaranteed. Finally, simulations of a nonlinear plant are presented to validate the developed event-driven H∞ control strategy.

Inverse Optimal Stabilization of Dynamical Systems by Wiener Processes

Abstract This paper formulates and solves new problems of inverse optimal stabilization and inverse optimal stabilization with gain assignment for nonlinear systems by Wiener processes. First, a theorem is developed to design inverse optimal stabilizers (i.e., covariance matrix multiplied by variance of Wiener processes), where it does not require to solve a Hamilton–Jacobi–Belman equation. Second, another theorem is developed to design inverse optimal stabilizers with gain assignment for nonlinear systems perturbed by both nonvanishing deterministic and stochastic (Wiener processes) disturbances without having to solve a Hamilton–Jacobi–Isaacs equation.

Zero-sum risk-sensitive stochastic differential games with reflecting diffusions in the orthant

We study zero-sum games with risk-sensitive cost criterion on the infinite horizon where the state is a controlled reflecting diffusion in the nonnegative orthant. We consider two cost evaluation criteria: discounted cost and ergodic cost. Under certain assumptions, we establish the existence of saddle point equilibria. We obtain our results by studying the corresponding Hamilton–Jacobi–Isaacs equations. For the ergodic cost criterion, exploiting the stochastic representation of the principal eigenfunction, we have completely characterized saddle point equilibrium in the space of stationary Markov strategies.

Optimal distributed learning for disturbance rejection in networked non‐linear games under unknown dynamics

In this study, an online distributed optimal adaptive algorithm is introduced for continuous-time non-linear differential graphical games under unknown systems subject to external disturbances. The proposed algorithm learns online the approximate solution to the coupled Hamilton–Jacobi–Isaacs equations. Each of the players in the game uses an actor-critic network structure and an intelligent identifier to find the unknown parameters of the systems. The authors use recorded past observations concurrently with current data to speed up convergence by exploring the state space. The closed-loop stability and convergence of the policies to Nash equilibrium are ensured by using Lyapunov stability theory. Finally, a simulation example shows the efficiency of the proposed algorithm.

A Differential Game with Exit Costs

We study a differential game where two players separately control their own dynamics, pay a running cost, and moreover pay an exit cost (quitting the game) when they leave a fixed domain. In particular, each player has its own domain and the exit cost consists of three different exit costs, depending whether either the first player only leaves its domain, or the second player only leaves its domain, or they both simultaneously leave their own domain. We prove that, under suitable hypotheses, the lower and upper values are continuous and are, respectively, the unique viscosity solution of a suitable Dirichlet problem for a Hamilton–Jacobi–Isaacs equation. The continuity of the values relies on the existence of suitable non-anticipating strategies respecting the domain constraint. This problem is also treated in this work.

A New Hamilton–Jacobi Differential Game Framework for Nonlinear Estimation and Output Feedback Control

In this paper, we develop a new framework for designing state estimators/filters and output measurement feedback controllers for affine nonlinear systems in state space. The problems are formulated as zero-sum differential games, and sufficient conditions for their solvability are given in terms of Hamilton–Jacobi–Isaacs equations (HJIEs). These HJIEs are new, in the sense that they are both state-dependent and measurement output dependent. This allows for the filter and observer gains to be optimized over all possible nonlinear gains. Examples and simulation results are also presented to support the theory.

Solutions for Multiagent Pursuit-Evasion Games on Communication Graphs: Finite-Time Capture and Asymptotic Behaviors

In this paper, the multiagent pursuit-evasion (MPE) games are solved in order to obtain optimal strategic policies for all players. In these games, multiple pursuers attempt to intercept multiple evaders who try to avoid capture. A graph-theoretic approach is employed to study the interactions of the agents with limited sensing capabilities, such that distributed control policies are obtained for every agent. Furthermore, the minimization of performance indices associated with the goals of the agents is guaranteed. Nash equilibrium among the players is obtained by means of optimal policies that use the solutions of the Hamilton–Jacobi–Isaacs (HJI) equations of the game. Minmax strategies are also proposed to guarantee a security-level performance when the solutions of the HJI equations for Nash equilibrium do not exist. Scenarios for finite-time capture and for asymptotic rendezvous are analyzed, and emergent behaviors are obtained by means of modifications of the proposed general-case performance indices. The containment control results are shown to be special cases of the solutions of the MPE games.

Tracking control using optimal discrete-time [formula omitted] for mechanical systems: Applied to Robotics

In this work, the H∞ control for mechanical systems and its application in Robotics is discussed. The controller is designed in discrete time and it is synthesized for mechanical systems that are modeled by means of the Euler–Lagrange formulation. Making use of the discrete Hamilton–Jacobi–Isaacs equation the control law is derived. The discrete control law is then applied to a continuous-time 6-DoF bipedal robot model in order to track the walking pattern references for each link. The system along with the control law is simulated, with the system subjected to an external disturbance that emulates the action of a group of unknown bounded forces over the links of the bipedal robot. Furthermore, an algorithm to diminish the effect of the disturbance is proposed such that the full knowledge of the plant is not needed but only the linear part of the mass and inertia matrix; this algorithm is combined with the H∞ controller and applied to a robotic arm. Finally, this work is compared to a similar approach that uses H∞ technique in continuous time.

Inverse optimal control of stochastic systems driven by Lévy processes

This paper formulates and solves new problems of inverse optimal stabilization and inverse optimal gain assignment for stochastic nonlinear systems driven by Lévy processes. First, a theorem is developed to design inverse optimal stabilizers based on inverse pre-optimal stabilization controllers for stochastic systems with known noise characteristics, where it does not require to solve a Hamilton–Jacobi–Bellman equation. Second, another theorem is developed to design inverse optimal gain assignment controllers for stochastic systems with unknown noise characteristics, where there is no need to solve a Hamilton–Jacobi–Isaacs equation. Third, the results are applied to design inverse optimal stabilizers for Euler–Lagrange systems.

Decentralized robust zero-sum neuro-optimal control for modular robot manipulators in contact with uncertain environments: theory and experimental verification

This paper presents a decentralized robust zero-sum optimal control approach for modular robot manipulators (MRMs) in contact with uncertain environments based on the adaptive dynamic programming (ADP) algorithm. The dynamic model of MRMs is formulated via joint torque feedback technique that is deployed for each joint module to design the model compensation controller. An uncertainty decomposition-based robust control is developed to compensate the model uncertainties, and then, the robust optimal control problem of the MRM system is transformed into a two-player zero-sum optimal control one. According to the ADP algorithm, the Hamilton–Jacobi–Isaacs equation can be solved by establishing action and critic neural networks, thus making the derivation of the approximate optimal control policy feasible. Based on the Lyapunov theory, the closed-loop robotic system is proved to be asymptotic stable under the developed decentralized control method. Finally, experiments are conducted to verify the effectiveness and advantages of the proposed method.

Adaptive Dynamics Programming for H∞ Control of Continuous-Time Unknown Nonlinear Systems via Generalized Fuzzy Hyperbolic Models

In this paper, a novel adaptive dynamic programming (ADP) algorithm is developed for the infinite-horizon ( $H_{\infty}$ ) optimal control problems with unknown continuous-time (CT) nonlinear systems subject to external disturbances. To facilitate the implementation of the algorithm, generalized fuzzy hyperbolic models (GFHMs) are utilized to establish an identifier–critic architecture, where the identifier is designed to reconstruct the unknown system dynamics, and the GFHM-based critic network is employed to approximate the value functions. The CT $H_{\infty}$ optimal control issue is converted into a two-player zero-sum game and the corresponding Hamilton–Jacobi–Isaacs equation is derived. The learning procedure of the critic design is adaptively implemented with the help of the reconstructed model, thus the requirement of the complete system dynamics is relaxed. Furthermore, by the means of Lyapunov direct method, the uniform ultimate boundedness stability analysis of the closed-loop control system is explicitly provided. Finally, to compare the control performances and disturbance attenuation properties of the proposed method and the existing ADP algorithms, two numerical examples are given.

Stable value iteration for two-player zero-sum game of discrete-time nonlinear systems based on adaptive dynamic programming

In this paper, a stable value iteration (SVI) algorithm is developed to solve discrete-time two-player zero-sum game (TP-ZSG) for nonlinear systems based on adaptive dynamic programming (ADP). In the SVI algorithm, both optimality and stability of nonlinear systems are considered with proofs given. First, an iterative ADP algorithm is presented to obtain the approximate optimal solutions by solving Hamilton–Jacobi–Isaacs (HJI) equation. Second, a range of the discount factor is shown, which guarantees HJI equation serving as a Lyapunov equation. Moreover, we prove that if the iteration number reaches a given number, then the iterative control inputs make the closed-loop system asymptotic stable. Third, in order to improve the practicability of the developed stability condition, a simple criteria is established based on Lyapunov stability theory. Neural networks (NNs) are used to approximate the system states, the value function, the control and disturbance inputs. Finally, simulation results are given to illustrate the performance of the developed optimal control method.

Critic-only adaptive dynamic programming algorithms’ applications to the secure control of cyber–physical systems

Industrial cyber–physical systems generally suffer from the malicious attacks and unmatched perturbation, and thus the security issue is always the core research topic in the related fields. This paper proposes a novel intelligent secure control scheme, which integrates optimal control theory, zero-sum game theory, reinforcement learning and neural networks. First, the secure control problem of the compromised system is converted into the zero-sum game issue of the nominal auxiliary system, and then both policy-iteration-based and value-iteration-based adaptive dynamic programming methods are introduced to solve the Hamilton–Jacobi–Isaacs equations. The proposed secure control scheme can mitigate the effects of actuator attacks and unmatched perturbation, and stabilize the compromised cyber–physical systems by tuning the system performance parameters, which is proved through the Lyapunov stability theory. Finally, the proposed approach is applied to the Quanser helicopter to verify the effectiveness.

Event-trigger-based robust control for nonlinear constrained-input systems using reinforcement learning method

In this paper, an online integral reinforcement learning strategy is proposed to deal with robust constrained control problems using event-triggered mechanism for nonlinear Continuous-Time (C-T) systems with external disturbances. The novel design of constrained control law is addressed together with the adaptive event-triggered condition by guaranteeing the optimal performance and system stability. An adaptive online actor-critic Neural Network (NN) reinforcement learning scheme is developed to approximate the optimal solution of the complicated Hamilton–Jacobi–Isaacs equation. Meanwhile, the convergence of NN weight errors and the event-triggered closed-loop system stability are demonstrated to be uniform ultimate bounded by Lyapunov analysis under the proposed triggering condition. Moreover, event-triggered H∞ tracking control with input constrains and limited network communication is also presented by establishing an augmented system. Finally, simulation results are provided to show the algorithm validity.

Approximate-optimal control algorithm for constrained zero-sum differential games through event-triggering mechanism

This paper investigates the optimization problem of two-player zero-sum differential game with control constraints in the framework of event triggering. Relying on reinforcement learning, an adaptive dynamic programming algorithm is developed to approximate the optimal solution of zero-sum game, i.e., the saddle-point equilibrium. A single-network structure is adopted, wherein a critic neural network (NN) evaluates the action. First, the constrained Hamilton–Jacobi–Isaacs equation is mathematically derived in the presence of control constraints; the event-triggering mechanism is then incorporated to reduce calculations and actions. Then, based on the gradient-descent technique, a novel weight updating law is designed for the critic NN, which ensures the solution can converge to the optimal value online. Moreover, the stability of closed-loop system is guaranteed and the unfavorable Zeno behavior is excluded by calculating the theoretical minimum triggering interval. Finally, two numerical examples are provided to verify the reliability and effectiveness of proposed algorithm.

Adaptive dynamic surface-based differential games for a class of pure-feedback nonlinear systems with output constraints

ABSTRACTThis paper investigates the zero-sum differential game problem for a class of uncertain nonlinear pure-feedback systems with output constraints and unknown external disturbances. A barrier Lyapunov function is introduced to tackle the output constraints. By constructing an affine variable at each dynamic surface control design step rather than utilising the mean-value theorem, the tracking control problem for pure-feedback systems can be transformed into an equivalent zero-sum differential game problem for affine systems. Then, the solution of associated Hamilton–Jacobi–Isaacs equation can be obtained online by using the adaptive dynamic programming technique. Finally, the whole control scheme that is composed of a feedforward dynamic surface controller and a feedback differential game control strategy guarantees the stability of the closed-loop system, and the tracking error is remained in a bounded compact set. The simulation results demonstrate the effectiveness of the proposed control scheme.