Multi-agent Differential Games Research Articles

An adaptable fuzzy reinforcement learning method for non-stationary environments

How do we know when a reinforcement learning policy needs to adapt? In non-stationary environments, agents must adapt and learn in environments that change dynamically. We propose a finite-horizon model-free solution using a hierarchical learning structure with fuzzy systems. The higher-level learning policy advises the lower-level policy when to start and stop learning based on the temporal differences calculated within the lower-level. Major differences in the temporal difference of each action produced by an agent may indicate environment change. This structure is tested with multi-agent differential games in both the cooperative and competitive aspect. Results show that this method is quick to notice and adapt the policy within relatively few learning episodes.

Accelerating Fuzzy Actor–Critic Learning via Suboptimal Knowledge for a Multi-Agent Tracking Problem

Multi-agent differential games usually include tracking policies and escaping policies. To obtain the proper policies in unknown environments, agents can learn through reinforcement learning. This typically requires a large amount of interaction with the environment, which is time-consuming and inefficient. However, if one can obtain an estimated model based on some prior knowledge, the control policy can be obtained based on suboptimal knowledge. Although there exists an error between the estimated model and the environment, the suboptimal guided policy will avoid unnecessary exploration; thus, the learning process can be significantly accelerated. Facing the problem of tracking policy optimization for multiple pursuers, this study proposed a new form of fuzzy actor–critic learning algorithm based on suboptimal knowledge (SK-FACL). In the SK-FACL, the information about the environment that can be obtained is abstracted as an estimated model, and the suboptimal guided policy is calculated based on the Apollonius circle. The guided policy is combined with the fuzzy actor–critic learning algorithm, improving the learning efficiency. Considering the ground game of two pursuers and one evader, the experimental results verified the advantages of the SK-FACL in reducing tracking error, adapting model error and adapting to sudden changes made by the evader compared with pure knowledge control and the pure fuzzy actor–critic learning algorithm.

Shadow Monopolies: Sparse Structures of Strategic Cooperation

In this work, we will show a result derived from a class of economically inspired multi-agent differential games. The result shows that in the class of games we present, only a sparse cooperation structure is needed by the agents to achieve an outcome that is optimal in the sense of the utilitarian social welfare function. We show that neither full cooperation amongst players nor full information from a centralized controller is ever necessary to do so. We precisely characterize Shadow Monopolies, the nonzero-cooperation structures that equilibrate the optimal outcome, and we show that only the sparsest kind of Shadow Monopoly is ever needed.

Robust multi-agent differential games with application to cooperative guidance

Robust multi-agent differential games and its application to cooperative guidance are investigated in this paper. The multi-agent system is modeled by general linear dynamics with norm-bound parameter uncertainties. The players in the game are divided into two teams. One team consists of a leader, and the other team consists of a fixed number of followers agents that aim to achieve leader-following consensus. The two teams constitute the adversaries of a two-player zero-sum differential game. Based on quadratic stabilization techniques, a set of saddle point strategies of the game are designed to stabilize the closed-loop multi-agent system. By modifying the optimal solution of the multi-agent differential game of the nominal model, the sufficient conditions are presented such that the stabilization of the uncertain system is guaranteed. It is proved that the modified solution achieves quasi-optimality. It is also shown that L2-bounded consensus error can be achieved, if the control of the leader deviates from its quasi-optimal strategy. The model of cooperative guidance problem is established, based on which the robust multi-agent differential games approach is applied to design the cooperative guidance laws. Numerical examples are given to verify the effectiveness of the theoretical results and the performance of the proposed cooperative guidance law.

Robust multi-agent differential games for general linear systems with model uncertainties

The problem of infinite-horizon multi-agent differential games is investigated, where the process can be modeled by a set of uncertain linear dynamics. The players are divided into two teams, one of which consists of a fixed number of follower agents while the other has one leader agent. The two teams constitute the adversaries. The multi-agent differential games can be transformed into a two-player game. The dynamics of the agents are subjected to norm-bound model uncertainties. Based on quadratic stabilization techniques, a set of saddle point strategies of the game is designed to stabilize the closed-loop multi-agent system, where the weighting matrices of the cost function are properly selected. For any given cost function, by modifying the solution of the linear quadratic differential game of the nominal model, the sufficient conditions are presented such that the stabilization of the system is guaranteed and the uncertainties are compensated. It is proved that the modified solution achieves optimality. A numerical example is given to verify the effectiveness of the theoretical results.

Optimal distributed synchronization control for continuous-time heterogeneous multi-agent differential graphical games

In this paper, a new optimal distributed synchronization control scheme for the consensus problem of heterogeneous multi-agent differential graphical games is developed by iterative adaptive dynamic programming (ADP). The main idea is to use iterative ADP technique to obtain the iterative control law which makes all the agents track a given dynamics and simultaneously makes the iterative value function reach the Nash equilibrium. In the developed heterogeneous multi-agent differential graphical games, the agent of each node is different from one another. The dynamics and performance index function for each node depend only on local neighborhood information. A cooperative policy iteration algorithm is presented to achieve the optimal distributed synchronization control law for the agent of each node, where the coupled Hamilton–Jacobi equations for optimal synchronization control of heterogeneous multi-agent differential games can be avoided. Convergence analysis is developed to show that the iterative value functions of heterogeneous multi-agent differential graphical games can converge to the Nash equilibrium. Two simulation examples are given to show the effectiveness of the developed optimal control scheme.

Leader-Based Optimal Coordination Control for the Consensus Problem of Multiagent Differential Games via Fuzzy Adaptive Dynamic Programming

In this paper, a new on-line scheme is presented to design the optimal coordination control for the consensus problem of multi-agent differential games by fuzzy adaptive dynamic programming (FADP), which brings together game theory, generalized fuzzy hyperbolic model (GFHM) and adaptive dynamic programming. In general, the optimal coordination control for multi-agent differential games is the solution of the coupled Hamilton-Jacobi (HJ) equations. Here, for the first time, GFHMs are used to approximate the solution (value functions) of the coupled HJ equations, based on policy iteration (PI) algorithm. Namely, for each agent, GFHM is used to capture the mapping between the local consensus error and local value function. Since our scheme uses the single-network rchitecture for each agent (which eliminates the action network model compared with dual-network architecture), it is a more reasonable architecture for multi-agent systems. Furthermore, the approximation solution is utilized to obtain the optimal coordination controls. Finally, we give the stability analysis for our scheme, and prove the weight estimation error and the local consensus error are uniformly ultimately bounded. Further, the control node trajectory is proven to be cooperative uniformly ultimately bounded.