Differential Graphical Games Research Articles

Dynamic event-driven optimal consensus control for state-constrained multiagent zero-sum differential graphical games

In this paper, a dynamic event-driven optimal control scheme is proposed for the zero-sum differential graphical games in nonlinear multiagent systems with full-state constraints. Initially, to address the dual demands of optimality and state constraints, a set of system transformation functions are introduced to satisfy the state constraints of the agents. Then, by applying the principle of differential game theory, the distributed optimal control problem affected by external disturbances is formulated as a zero-sum differential graphical game, and the performance index function related to neighbor informations and disturbances is designed for each follower. Afterwards, to enhance the utilization of communication resource, a novel dynamic event-triggered mechanism characterized by a dynamic threshold parameter and an auxiliary dynamic variable is developed, which not only exhibits greater flexibility but also diminishes the frequency of triggers. Furthermore, the approximate optimal control strategies are obtained by employing an event-driven adaptive dynamic programming algorithm. Ultimately, a simulation example is presented to verify the applicability of the proposed control approach.

Bipartite containment control of multi-agent systems subject to adversarial inputs based on zero-sum game

In this paper, we investigate bipartite containment control problem of multi-agent systems (MASs) with signed directed graph under adversarial inputs. Firstly, we define the bipartite containment error and establish the equivalence between the bipartite containment error converging to zero and the achievement of bipartite containment control. Subsequently, we prove that the bounded L2-gain bipartite containment problem under adversarial inputs can be reformulated as a multi-player zero-sum differential graphical game problem and can be solved via the solution to the coupled Hamilton-Jacobi-Isaacs (HJI) equation. To address this, we propose a policy iteration (PI) algorithm and prove its convergence under different updating cases. The proposed algorithm is implemented by neural networks (NNs) and a numerical simulation example is provided to show its effectiveness.

Distributed optimal control of nonlinear multi‐agent systems based on integral reinforcement learning

AbstractIn this article, a distributed optimal control approach is proposed for a class of affine nonlinear multi‐agent systems (MASs) with unknown nonlinear dynamics. The game theory is used to formulate the distributed optimal control problem into a differential graphical game problem with synchronized updates of all nodes. A data‐based integral reinforcement learning (IRL) algorithm is used to learn the solution of the coupled Hamilton–Jacobi (HJ) equation without prior knowledge of the drift dynamics, and the actor‐critic neural networks (A‐C NNs) are used to approximate the control law and the cost function, respectively. To update the parameters synchronously, the gradient descent algorithm is used to design the weight update laws of the A‐C NNs. Combining the IRL and the A‐C NNs, a distributed consensus optimal control method is designed. By using the Lyapunov stability theory, the developed optimal control method can show that all signals in the considered system are uniformly ultimately bounded (UUB), and the systems can achieve Nash equilibrium when all agents update their controllers simultaneously. Finally, simulation results are given to illustrate the effectiveness of the developed optimal control approach.

Robust distributed Nash equilibrium solution for multi‐agent differential graphical games

AbstractThis paper studies the differential graphical games for linear multi‐agent systems with modelling uncertainties. A robust optimal control policy that seeks the distributed Nash equilibrium solution and guarantees the leader‐following consensus is designed. The weighting matrices rely on modelling uncertainties, leading to the Nash equilibrium solution, and the solution can be obtained by solving a decoupled algebraic Riccati equation. Simulation studies are finally reported to illustrate the effectiveness of proposed policy.

Differential graphical games of multiagent systems with nonzero leader's control input and external disturbances

AbstractIn this article, we investigate the problem of differential graphical games for multiagent systems (MASs) with external disturbances. In particular, the system considered in this article allows the leader to have an unknown bounded control input, and the communication topology between agents is allowed to be a weakly restricted structure with only containing a directed spanning tree. We propose a novel dynamic sliding mode control strategy to dispose the presence of leader's nonzero control input and external disturbances. Furthermore, a nominal controller based on the worst‐case control strategies of neighbors is designed to provide a distributed solution for differential graphics games of MASs. Finally, our algorithms are applied to solve multiple unmanned surface vessels games. Simulation studies are provided to test the effectiveness of the proposed algorithms.

Differential graphical game‐based multi‐agent tracking control using integral reinforcement learning

AbstractThis paper studies the cooperative tracking control problem of interacted multi‐agent systems (MASs) under undirected communication. Based on differential graphical game theory, the MAS tracking control problem is formulated as an infinite horizon cooperative differential graphical game‐theoretic tracking control framework, where a multi‐objective optimization problem is designed and then cast into a Pareto‐equivalent single‐objective optimization problem using a scalarization method. Necessary and sufficient conditions for the existence of the Pareto‐optimal strategy to the game theoretic tracking control are established, where it has been proven that the solution to the integral Bellman optimality equation leads to Pareto‐optimal strategy. Then, an off‐policy integral reinforcement learning scheme to find optimal control strategy using a pure data‐driven manner is developed, which consumes less computation efforts than the traditional learning scheme. Simulated results are conducted to validate the effectiveness of the proposed game and IRL‐based tracking control method.

Coordinated transportation of tethered multi-rotor UAVs based on differential graphical games

This paper investigates the stability control of the tethered multi-rotor unmanned aerial vehicles (T-MRUAVs) system in cargo delivery, where precise payload trajectory control is deemed unnecessary, with a focus on enhancing controller simplicity and robustness. To achieve this objective, it is imperative to employ an optimal formation tracking control, ensuring not only the symmetrical positioning of UAVs concerning the payload but also the achievement of distributed optimal solutions. Therefore, a formation tracking control strategy based on differential graphical games is designed to ensure optimized performance metrics and minimized suspended load swing. The designed control framework includes an outer loop and an inner loop control, which respectively achieve formation tracking and attitude stability. In addition to maintaining equilibrium states, each UAV is required to track the desired trajectory. Consequently, the formation tracking control consists of a gravity compensation term at the equilibrium point and a formation tracking control term based on differential graphical games. Then, a PID control strategy is displayed for the inner loop control, and the altitude control adopts feedback linearization. Finally, numerical simulations have validated the effectiveness of the designed algorithm.

Distributed Consensus Protocol for Multi-Agent Differential Graphical Games

Distributed Consensus Protocol for Multi-Agent Differential Graphical Games

Distributed Optimal Formation Control of Multiple Unmanned Surface Vehicles with Stackelberg Differential Graphical Game

Distributed Optimal Formation Control of Multiple Unmanned Surface Vehicles with Stackelberg Differential Graphical Game

A novel Cooperative–Competitive Differential Graphical Game with distributed global Nash solution

A novel Cooperative–Competitive Differential Graphical Game with distributed global Nash solution

Distributed Minmax Strategy for Consensus Tracking in Differential Graphical Games: A Model-Free Approach

Distributed Minmax Strategy for Consensus Tracking in Differential Graphical Games: A Model-Free Approach

Beyond Nash Solutions for Differential Graphical Games

Differential graphical games model the interaction between networked dynamical agents that do not possess global state information available. In general, this lack of global information prevents the agents from playing their optimal Nash strategies. In this paper, two different solution concepts for graphical games are proposed as practical alternatives when Nash equilibrium is not attainable among the players. The proposed solutions consider the case on which the players make optimistic assumptions about their neighbors, and are characterized by providing distributed, linear control policies. The robustness properties of each solution, in terms of the gain and phase margins, are also determined.

Distributed Adaptive Nash Equilibrium Solution for Differential Graphical Games.

This article investigates differential graphical games for linear multiagent systems with a leader on fixed communication graphs. The objective is to make each agent synchronize to the leader and, meanwhile, optimize a performance index, which depends on the control policies of its own and its neighbors. To this end, a distributed adaptive Nash equilibrium solution is proposed for the differential graphical games. This solution, in contrast to the existing ones, is not only Nash but also fully distributed in the sense that each agent only uses local information of its own and its immediate neighbors without using any global information of the communication graph. Moreover, the asymptotic stability and global Nash equilibrium properties are analyzed for the proposed distributed adaptive Nash equilibrium solution. As an illustrative example, the differential graphical game solution is applied to the microgrid secondary control problem to achieve fully distributed voltage synchronization with optimized performance.

Fault-Tolerant Coverage Control of Multiple Satellites: A Differential Graphical Game Approach

This paper considers the regional coverage problem of multiple satellites with full consideration of the coupled position-attitude characteristics, the optimality, and the cooperation among satellites. In the fault-free case, a configuration condition of satellite formation flying (SFF) is established under which the desired coverage configuration is designed; In the presence of completely failed satellites, a configuration adjustment scheme is provided such that the coverage tasks of failed satellites are handed over to other healthy ones; Based on the proposed configurations, a distributed near-optimal coverage controller that achieves the optimal coverage control goal is designed for each satellite by utilizing the differential graphical game theory and the approximate dynamic programming (ADP) method. Simulation results show the effectiveness of the proposed methods.

Differential Graphical Games for Constrained Autonomous Vehicles Based on Viability Theory.

This article proposes an optimal-distributed control protocol for multivehicle systems with an unknown switching communication graph. The optimal-distributed control problem is formulated to differential graphical games, and the Pareto optimum to multiplayer games is sought based on the viability theory and reinforcement learning techniques. The viability theory characterizes the controllability of a wide range of constrained nonlinear systems; and the viability kernel and the capture basin are the pillars of the viability theory. The capture basin is the set of all initial states, in which there exist control strategies that enable the states to reach the target in finite time while remaining inside a set before reaching the target. In this regard, the feasible learning region is characterized by the reinforcement learner. In addition, the approximation of the capture basin provides the learner with prior knowledge. Unlike the existing works that employ the viability theory to solve control problems with only one agent and differential games with only two players, the viability theory, in this article, is utilized to solve multiagent control problems and multiplayer differential games. The distributed control law is composed of two parts: 1) the approximation of the capture basin and 2) reinforcement learning, which are computed offline and online, respectively. The convergence properties of the parameters' estimation errors in reinforcement learning are proved, and the convergence of the control policy to the Pareto optimum of the differential graphical game is discussed. The guaranteed approximation results of the capture basin are provided and the simulation results of the differential graphical game are provided for multivehicle systems with the proposed distributed control policy.

Optimal Leader-Following Consensus Control of Multi-Agent Systems: A Neural Network Based Graphical Game Approach

In this article, the optimal leader-following consensus control problem of multi-agent systems is solved using a novel neural network-based (NN-based) integrated heuristic dynamic programming (IHDP) algorithm under a distributed learning framework. The optimal controller is designed to find a Nash equilibrium of the multi-agent differential graphical game. According to the principle of optimality, this problem can be transformed into solving a set of coupled Hamilton-Jacobi-Bellman (HJB) equations, which are difficult to solve directly because of the nonlinearity and strong coupling of the equations. The proposed algorithm combines policy iteration (PI) and value iteration (VI), and the Nash equilibrium of performance index among agents is achieved distributedly, i.e., all agents update their control inputs only use local information. The convergence of the algorithm is rigorously proved. Neural network (NN) approximation implementation is given for the unknown structures of local value functions of agents. The effectiveness of the algorithm is verified by numerical simulation.

Distributed Optimal Fault Estimation and Fault-Tolerant Control for Interconnected Systems: A Stackelberg Differential Graphical Game Approach

This article addresses the distributed optimal fault estimation (FE) and fault-tolerant control (FTC) issues under the bidirectional interaction between a possibly faulty interconnected system and its associated diagnostic interconnected observer. The proposed methodology is based on a framework, named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Stackelberg differential graphical game</i> , in which the interconnected observer and its corresponding interconnected system are formulated as two graphical games, and they, respectively, play the role of the follower and leader to make <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sequential</i> decisions of the FE and the FTC. Such a framework allows us to fully reveal the property of the bidirectional interaction, based on which a distributed optimal FE and FTC scheme is developed. Moreover, a new concept named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">interactive Stackelberg equilibrium</i> is proposed, where, to reach this equilibrium, an auxiliary controller is suitably designed for each subobserver. In this way, this auxiliary controller can cooperate with other auxiliary controllers of subobservers, and noncooperate with controllers of subsystems, which helps to achieve the distributed optimal FE and FTC goals.

Model-Free distributed optimal consensus control of nonlinear multi-agent systems: A graphical game approach

In this paper, the optimal consensus control problem of nonlinear multi-agent systems(MASs) with completely unknown dynamics is considered. The problem is formulated in a differential graphical game approach which can be solved by Hamilton-Jacobi (HJ) equations. The main difficulty in solving the HJ equations lies in the nonlinear coupling between equations. Based on the Adaptive Dynamic Programming (ADP) technique, an VI-PI mixed HDP algorithm is proposed to solve the HJ equations distributedly. With the PI step, a suitable iterative initial value can be obtained according to the initial policies. Then, VI steps are run to get the optimal solution with exponential convergence rate. Neural networks (NNs) are applied to approximate the value functions, which makes the data-driven end-to-end learning possible. A numerical simulation is conducted to show the effectiveness of the proposed algorithm.

Differential Graphical Game With Distributed Global Nash Solution

This article studies multiagent differential graphical games in linear dynamical networks. We point out that in existing graphical game formulations, being global Nash and being distributed are two contradicting properties. In particular, we prove that the best response strategy, most widely used in solving graphical games, leads to Nash but does not provide distributed solutions. On the other hand, the minmax strategy, recently developed in solving graphical games, provides distributed solutions but prevents the agents from reaching a global Nash equilibrium. In this article, we propose a novel differential graphical game formulation, which promises the existence of solutions that 1) lead to global Nash equilibrium and 2) are distributed in the sense that each agent only uses the state information of its own and its direct neighbors. Stability and Nash equilibrium properties of the proposed graphical game are proven, respectively. Simulation studies are conducted to illustrate the theoretical results.

Stability and robustness analysis of minmax solutions for differential graphical games

Recent studies have shown that, in general, Nash equilibrium cannot be achieved by the players of a differential graphical game by using distributed control policies. Alternative solution concepts that do not necessarily lead to Nash equilibrium can be proposed to allow the players in the game determine distributed optimal strategies. This paper analyzes the performance properties of the solution concept regarded as minmax strategies. The minmax formulation is shown to provide distributed control policies for linear systems under mild assumptions. The stability and robustness characteristics of the proposed solution are studied in terms of gain and phase margins, and related to the robustness properties of the single-agent LQR controller. The results of our analysis are finally tested by means of a simulation example.